Introduction
I have been running an uninterruptible power supply (UPS) for a few reasons.
- I want my router to always have power and be able to shutdown gracefully my servers during power outages
- I run a pihole on a raspberry pi that I want to always be running mostly to block tracking, phishing, malware, and ransom-ware sites
- I don’t want my servers to shutdown while I am transferring data
- I need surge protection from a voltage spike to avoid damaging any equipment
That said, I also want to be able to monitor the UPS’s battery, energy consumption and be able to compare with past data.
The Setup
My current setup is quite simple:
- Main Server : raspberry pi 3b+ running pihole
- Modem/Router providing Wifi and Lan
- Secondary Server : Running multiple VMs on Proxmox VE (AMD Ryzen 3600X, 16GB of RAM for now, few terabytes of storage)
- Maybe a third server in the future?
- UPS : APC Back-UPS XS 950U (Costs around 100€ on Amazon FR, Amazon DE)
The UPS has 4 outputs so I still have the ability to add an other server and most importantly is that it have a USB data port for management.
So the goal is to connect my main server to the UPS via the USB data port and share the data readings with the rest of the servers. For APC UPSs apcupsd package can be and used to extract the data and also configure the servers to shutdown when the battery reaches a certain level of charge.
The UPS has one USB data port so only one device can be connected directly to the UPS. the good news is that the apcupsd comes with a NIS server so instances of upcupsd on other machines can get connected using a polling mechanism via ethernet.
Installing apcupsd
On the primary server We will have to verify that the USB connection is functional.
$ lsusb
Bus 001 Device 004: ID 051d:0002 American Power Conversion Uninterruptible Power Supply
Bus 001 Device 005: ID 0424:7800 Standard Microsystems Corp.
Bus 001 Device 003: ID 0424:2514 Standard Microsystems Corp. USB 2.0 Hub
Bus 001 Device 002: ID 0424:2514 Standard Microsystems Corp. USB 2.0 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Install the apcupsd package
$ apt install apcupsd
Edit the apcupsd.conf file to configure the daemon with following parameters:
$ nano /etc/apcupsd/apcupsd.conf
| Parameter | Value | Comment |
|---|---|---|
| UPSTYPE | usb | Must be set to usb |
| UPSCABLE | usb | Must be set to usb |
| DEVICE | Should be blank | |
| NISIP | 192.168.1.26 | Must be a static IP address of the primary server |
| NISPORT | 3551 | Port that will be exposed by NIS on the primary server |
| UPSNAME | BX950U | Optional |
Edit apcupsd file and set ISCONFIGURED=yes
$ nano /etc/default/apcupsd
Start the service
$ service apcupsd restart
Verify that we can get the status of the UPS via USB using apcaccess command
$ apcaccess
APC : 001,036,0878
DATE : 2020-10-31 23:32:40 +0100
HOSTNAME : mainserver
VERSION : 3.14.14 (31 May 2016) debian
UPSNAME : BX950U
CABLE : USB Cable
DRIVER : USB UPS Driver
UPSMODE : Stand Alone
STARTTIME: 2020-10-31 23:32:38 +0100
MODEL : Back-UPS XS 950U
STATUS : ONLINE
LINEV : 238.0 Volts
LOADPCT : 9.0 Percent
BCHARGE : 100.0 Percent
TIMELEFT : 65.9 Minutes
MBATTCHG : 5 Percent
MINTIMEL : 3 Minutes
MAXTIME : 0 Seconds
SENSE : Medium
LOTRANS : 155.0 Volts
HITRANS : 280.0 Volts
ALARMDEL : 30 Seconds
BATTV : 13.4 Volts
LASTXFER : Unacceptable line voltage changes
NUMXFERS : 0
TONBATT : 0 Seconds
CUMONBATT: 0 Seconds
XOFFBATT : N/A
SELFTEST : NO
STATFLAG : 0x05000008
SERIALNO : 4B1952P18784
BATTDATE : 2019-12-28
NOMINV : 230 Volts
NOMBATTV : 12.0 Volts
NOMPOWER : 480 Watts
FIRMWARE : 925.T2 .I USB FW:T2
END APC : 2020-10-31 23:32:41 +0100
On the secondary server install the apcupsd package
$ apt install apcupsd
Edit the apcupsd.conf file to configure the daemon withe following parameters:
$ nano /etc/apcupsd/apcupsd.conf
| Parameter | Value | Comment |
|---|---|---|
| UPSCABLE | ether | Must be set to ether so we can connect to the primary sever |
| UPSTYPE | net | Must be set to net so we can connect to the primary sever |
| DEVICE | 192.168.1.26:3551 | <primary NIS IP>:<primary NIS PORT> |
| POLLTIME | 30 | Interval in seconds between polls |
$ nano /etc/default/apcupsd
Edit apcupsd file and set ISCONFIGURED=yes
$ service apcupsd restart
$ apcaccess
APC : 001,037,0929
DATE : 2020-10-31 23:57:27 +0100
HOSTNAME : pve1
VERSION : 3.14.14 (31 May 2016) debian
UPSNAME : BX950U
CABLE : Ethernet Link
DRIVER : NETWORK UPS Driver
UPSMODE : Stand Alone
STARTTIME: 2020-10-31 23:57:27 +0100
MASTERUPD: 2020-10-31 23:57:27 +0100
MASTER : 192.168.1.26:3551
MODEL : Back-UPS XS 950U
STATUS : ONLINE SLAVE
LINEV : 236.0 Volts
LOADPCT : 9.0 Percent
BCHARGE : 100.0 Percent
TIMELEFT : 65.9 Minutes
MBATTCHG : 5 Percent
MINTIMEL : 3 Minutes
MAXTIME : 0 Seconds
SENSE : Medium
LOTRANS : 155.0 Volts
HITRANS : 280.0 Volts
BATTV : 13.5 Volts
LASTXFER : Unacceptable line voltage changes
NUMXFERS : 0
TONBATT : 0 Seconds
CUMONBATT: 0 Seconds
XOFFBATT : N/A
SELFTEST : NO
STATFLAG : 0x05000408
SERIALNO : 4B1952P18784
BATTDATE : 2019-12-28
NOMINV : 230 Volts
NOMBATTV : 12.0 Volts
NOMPOWER : 480 Watts
FIRMWARE : 925.T2 .I USB FW:T2
END APC : 2020-10-31 23:57:37 +0100
Collecting Metrics
I have created a simple bash script to parse the metrics and send them to the influxdb server. The metrics I am interested in are the following
| Metric | Comment | Unit |
|---|---|---|
| LOADPCT | Percentage of UPS load capacity used | Percent |
| BATTV | Current battery voltage | Volts |
| TIMELEFT | Remaining runtime left on battery as estimated by UPS | Minutes |
| BCHARGE | Current battery capacity charge percentage | Percent |
| NOMPOWER | Nominal power output in watts | Watts |
| LINEV | Current input line voltage | Volts |
| STATUS | UPS status (online, charging, on battery etc) | N/A |
| CUMONBATT | Cumulative seconds on battery since apcupsd startup | Seconds |
| LOADWATT | Computed by multiplying load by the nominal power | Watts |
$ nano /bin/export_apc_metrics.sh
#!/bin/bash
INFLUX_URL="192.168.168.26:8086"
DB_NAME="UPS"
metrics=`apcaccess -u`
LOADPCT=$(echo "$metrics" | grep "LOADPCT" | awk -F ": " '{print $2}' | xargs)
BATTV=$(echo "$metrics" | grep "^BATTV" | awk -F ": " '{print $2}' | xargs)
TIMELEFT=$(echo "$metrics" | grep "TIMELEFT" | awk -F ": " '{print $2}' | xargs)
BCHARGE=$(echo "$metrics" | grep "BCHARGE" | awk -F ": " '{print $2}' | xargs)
NOMPOWER=$(echo "$metrics" | grep "NOMPOWER" | awk -F ": " '{print $2}' | xargs)
LINEV=$(echo "$metrics" | grep "LINEV" | awk -F ": " '{print $2}' | xargs)
STATUS=$(echo "$metrics" | grep "STATUS" | awk -F ": " '{print $2}' | xargs)
CUMONBATT=$(echo "$metrics" | grep "CUMONBATT" | awk -F ": " '{print $2}' | xargs)
LOADWATT=$(awk '{print $1*$2/100}' <<<"${LOADPCT} ${NOMPOWER}")
FLOAT_METRIC_NAMES="LOADPCT LINEV BATTV TIMELEFT BCHARGE NOMPOWER LOADWATT CUMONBATT"
STRING_METRIC_NAMES="STATUS"
for metricName in $FLOAT_METRIC_NAMES
do
echo "$metricName ${!metricName}"
curl -i "http://$INFLUX_URL/write?db=$DB_NAME" --data-binary "APC,host=rp $metricName=${!metricName}"
done
for metricName in $STRING_METRIC_NAMES
do
echo "$metricName ${!metricName}"
curl -i "http://$INFLUX_URL/write?db=$DB_NAME" --data-binary "APC,host=rp $metricName=\"${!metricName}\""
done
I have also made the file executable
$ chmod +x /bin/export_apc_metrics.sh
Now the only thing I need to do is setup a Cron job to launch my script on a regular basis. I this case it will run every minute.
...
...
* * * * * root /bin/export_apc_metrics.sh >/dev/null 2>&1
The Dashboard
The dashboard can be imported using the URL https://grafana.com/grafana/dashboards/13319 or the ID 13319. The data source has been parametrized for convenience.
This is obviously my first attempt at this dashboard and I still need to think more about it and what I actually will need to add, keep or remove. I will try to keep this post up to date with the changes.