Monitoring Homelab UPS with Grafana

Introduction

I have been running an uninterruptible power supply (UPS) for a few reasons.

  • I want my router to always have power and be able to shutdown gracefully my servers during power outages
  • I run a pihole on a raspberry pi that I want to always be running mostly to block tracking, phishing, malware, and ransom-ware sites
  • I don’t want my servers to shutdown while I am transferring data
  • I need surge protection from a voltage spike to avoid damaging any equipment

That said, I also want to be able to monitor the UPS’s battery, energy consumption and be able to compare with past data.

The Setup

My current setup is quite simple:

  • Main Server : raspberry pi 3b+ running pihole
  • Modem/Router providing Wifi and Lan
  • Secondary Server : Running multiple VMs on Proxmox VE (AMD Ryzen 3600X, 16GB of RAM for now, few terabytes of storage)
  • Maybe a third server in the future?
  • UPS : APC Back-UPS XS 950U (Costs around 100€ on Amazon FR, Amazon DE)

The UPS has 4 outputs so I still have the ability to add an other server and most importantly is that it have a USB data port for management.

So the goal is to connect my main server to the UPS via the USB data port and share the data readings with the rest of the servers. For APC UPSs apcupsd package can be and used to extract the data and also configure the servers to shutdown when the battery reaches a certain level of charge.

The UPS has one USB data port so only one device can be connected directly to the UPS. the good news is that the apcupsd comes with a NIS server so instances of upcupsd on other machines can get connected using a polling mechanism via ethernet.

Installing apcupsd

On the primary server We will have to verify that the USB connection is functional.

$ lsusb
Bus 001 Device 004: ID 051d:0002 American Power Conversion Uninterruptible Power Supply
Bus 001 Device 005: ID 0424:7800 Standard Microsystems Corp.
Bus 001 Device 003: ID 0424:2514 Standard Microsystems Corp. USB 2.0 Hub
Bus 001 Device 002: ID 0424:2514 Standard Microsystems Corp. USB 2.0 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Install the apcupsd package

$ apt install apcupsd

Edit the apcupsd.conf file to configure the daemon with following parameters:

$ nano /etc/apcupsd/apcupsd.conf
ParameterValueComment
UPSTYPEusbMust be set to usb
UPSCABLEusbMust be set to usb
DEVICEShould be blank
NISIP192.168.1.26Must be a static IP address of the primary server
NISPORT3551Port that will be exposed by NIS on the primary server
UPSNAMEBX950UOptional

Edit apcupsd file and set ISCONFIGURED=yes

$ nano /etc/default/apcupsd

Start the service

$ service apcupsd restart

Verify that we can get the status of the UPS via USB using apcaccess command

$ apcaccess
APC      : 001,036,0878
DATE     : 2020-10-31 23:32:40 +0100
HOSTNAME : mainserver
VERSION  : 3.14.14 (31 May 2016) debian
UPSNAME  : BX950U
CABLE    : USB Cable
DRIVER   : USB UPS Driver
UPSMODE  : Stand Alone
STARTTIME: 2020-10-31 23:32:38 +0100
MODEL    : Back-UPS XS 950U
STATUS   : ONLINE
LINEV    : 238.0 Volts
LOADPCT  : 9.0 Percent
BCHARGE  : 100.0 Percent
TIMELEFT : 65.9 Minutes
MBATTCHG : 5 Percent
MINTIMEL : 3 Minutes
MAXTIME  : 0 Seconds
SENSE    : Medium
LOTRANS  : 155.0 Volts
HITRANS  : 280.0 Volts
ALARMDEL : 30 Seconds
BATTV    : 13.4 Volts
LASTXFER : Unacceptable line voltage changes
NUMXFERS : 0
TONBATT  : 0 Seconds
CUMONBATT: 0 Seconds
XOFFBATT : N/A
SELFTEST : NO
STATFLAG : 0x05000008
SERIALNO : 4B1952P18784
BATTDATE : 2019-12-28
NOMINV   : 230 Volts
NOMBATTV : 12.0 Volts
NOMPOWER : 480 Watts
FIRMWARE : 925.T2 .I USB FW:T2
END APC  : 2020-10-31 23:32:41 +0100

Install the apcupsd package

$ apt install apcupsd

Edit the apcupsd.conf file to configure the daemon withe following parameters:

$ nano /etc/apcupsd/apcupsd.conf
ParameterValueComment
UPSCABLEetherMust be set to ether so we can connect to the primary sever
UPSTYPEnetMust be set to net so we can connect to the primary sever
DEVICE192.168.1.26:3551<primary NIS IP>:<primary NIS PORT>
POLLTIME30Interval in seconds between polls
$ nano /etc/default/apcupsd

Edit apcupsd file and set ISCONFIGURED=yes

$ service apcupsd restart
$ apcaccess
APC      : 001,037,0929
DATE     : 2020-10-31 23:57:27 +0100
HOSTNAME : pve1
VERSION  : 3.14.14 (31 May 2016) debian
UPSNAME  : BX950U
CABLE    : Ethernet Link
DRIVER   : NETWORK UPS Driver
UPSMODE  : Stand Alone
STARTTIME: 2020-10-31 23:57:27 +0100
MASTERUPD: 2020-10-31 23:57:27 +0100
MASTER   : 192.168.1.26:3551
MODEL    : Back-UPS XS 950U
STATUS   : ONLINE SLAVE
LINEV    : 236.0 Volts
LOADPCT  : 9.0 Percent
BCHARGE  : 100.0 Percent
TIMELEFT : 65.9 Minutes
MBATTCHG : 5 Percent
MINTIMEL : 3 Minutes
MAXTIME  : 0 Seconds
SENSE    : Medium
LOTRANS  : 155.0 Volts
HITRANS  : 280.0 Volts
BATTV    : 13.5 Volts
LASTXFER : Unacceptable line voltage changes
NUMXFERS : 0
TONBATT  : 0 Seconds
CUMONBATT: 0 Seconds
XOFFBATT : N/A
SELFTEST : NO
STATFLAG : 0x05000408
SERIALNO : 4B1952P18784
BATTDATE : 2019-12-28
NOMINV   : 230 Volts
NOMBATTV : 12.0 Volts
NOMPOWER : 480 Watts
FIRMWARE : 925.T2 .I USB FW:T2
END APC  : 2020-10-31 23:57:37 +0100

Collecting Metrics

I have created a simple bash script to parse the metrics and send them to the influxdb server. The metrics I am interested in are the following

MetricCommentUnit
LOADPCTPercentage of UPS load capacity usedPercent
BATTVCurrent battery voltageVolts
TIMELEFTRemaining runtime left on battery as estimated by UPSMinutes
BCHARGECurrent battery capacity charge percentagePercent
NOMPOWERNominal power output in wattsWatts
LINEVCurrent input line voltageVolts
STATUSUPS status (online, charging, on battery etc)N/A
CUMONBATTCumulative seconds on battery since apcupsd startupSeconds
LOADWATTComputed by multiplying load by the nominal powerWatts
$ nano /bin/export_apc_metrics.sh
#!/bin/bash

INFLUX_URL="192.168.168.26:8086"
DB_NAME="UPS"

metrics=`apcaccess -u`

LOADPCT=$(echo "$metrics" |  grep "LOADPCT"  | awk -F ": " '{print $2}' | xargs)
BATTV=$(echo "$metrics" |  grep "^BATTV"  | awk -F ": " '{print $2}' | xargs)
TIMELEFT=$(echo "$metrics" |  grep "TIMELEFT"  | awk -F ": " '{print $2}' | xargs)
BCHARGE=$(echo "$metrics" |  grep "BCHARGE"  | awk -F ": " '{print $2}' | xargs)
NOMPOWER=$(echo "$metrics" |  grep "NOMPOWER"  | awk -F ": " '{print $2}' | xargs)
LINEV=$(echo "$metrics" |  grep "LINEV"  | awk -F ": " '{print $2}' | xargs)
STATUS=$(echo "$metrics" |  grep "STATUS"  | awk -F ": " '{print $2}' | xargs)
CUMONBATT=$(echo "$metrics" |  grep "CUMONBATT"  | awk -F ": " '{print $2}' | xargs)
LOADWATT=$(awk '{print $1*$2/100}' <<<"${LOADPCT} ${NOMPOWER}")

FLOAT_METRIC_NAMES="LOADPCT LINEV BATTV TIMELEFT BCHARGE NOMPOWER LOADWATT CUMONBATT"
STRING_METRIC_NAMES="STATUS"

for metricName in $FLOAT_METRIC_NAMES
do
echo "$metricName ${!metricName}"
   curl -i "http://$INFLUX_URL/write?db=$DB_NAME" --data-binary "APC,host=rp $metricName=${!metricName}"
done

for metricName in $STRING_METRIC_NAMES
do
echo "$metricName ${!metricName}"
   curl -i "http://$INFLUX_URL/write?db=$DB_NAME" --data-binary "APC,host=rp $metricName=\"${!metricName}\""
done

I have also made the file executable

$ chmod +x /bin/export_apc_metrics.sh

Now the only thing I need to do is setup a Cron job to launch my script on a regular basis. I this case it will run every minute.

...
...

* * * * * root  /bin/export_apc_metrics.sh >/dev/null 2>&amp;1

The Dashboard

The dashboard can be imported using the URL https://grafana.com/grafana/dashboards/13319 or the ID 13319. The data source has been parametrized for convenience.

This is obviously my first attempt at this dashboard and I still need to think more about it and what I actually will need to add, keep or remove. I will try to keep this post up to date with the changes.