This week I ran into an issue with my reporting setup with Telegraf, Influxdb and Grafana on my Raspberry Pi 2.
All my dashboards in Grafana weren't able to recieve data from Influxdb. Every panel had error about wrong gateway. So I checked log of Influxdb which is the data source for my dashboards.
The Influxdb service was running but was restarting every 3 seconds. Influxdb crashed with a out of memory exception.
raspberrypi influxd[7774]: runtime: out of memory: cannot allocate 32192-byte block (33210 in use)
raspberrypi influxd[7774]: fatal error: out of memory
After a search on my favorit search, I found an issues thread about Influxdb not being able to start due to an out of memory exception.
jrbury mentioned that Influxdb uses an in memory index for all tags used in ones databases, which can cause that error.
So I had no choice and had to move all files of Influxdb to another box and hope for the best, that Influxdb will start there and give me the chance to deleted the some measurements or databases. Luckly I own a Ryzen powered box with 8 GB, which should be enough for start Influxdb databases of Raspberry Pi 2.
In order to create a copy of my Influxdb databases, I checked the service configuration for the paths to the meta data, data and wal data files.
[meta]
dir = "/var/lib/influxdb/meta"
[data]
dir = "/var/lib/influxdb/data"
wal-dir = "/var/lib/influxdb/wal"
All files are stored in /var/lib/influxdb
, so I
- Stop the Influxdb service
systemctl stop influxd
- copied all files into a seperate folder
mkdir $home/influxdb-rescue
cp -r /var/lib/influxdb $home/influxdb-rescue - change the ownership of the files to my user
chown -R masterlamps $home/influxdb-rescue
- copied the files the Ryzen box
scp -r $home/influxdb-rescue masterlamps@<ryzen-ip>:$home/influxdb-rescue
- delete local backup of influxdb files
rm $home/influxdb-rescue
- loged on to the Ryzen box and change the influxdb configuration to the folder with influxdb files
[meta]
dir = "/home/masterlamps/influxdb-rescue/meta"
[data]
dir = "/home/masterlamps/influxdb-rescue/data"
wal-dir = "/home/masterlamps/influxdb-rescue/wal" - start influxdb and delete unnessary measurements and databases
- copy the files back to Raspberry Pi 2.
scp -r $home/influxdb-rescue masterlamps@<raspberrypi2-ip>:$home/influxdb-rescue
- delete influxdb files on Ryzen box
rm $home/influxdb-rescue
- delete old files and copy new files from rescue folder to influxdb folder
rm /var/lib/influxdb
cp $home/influxdb-rescue /var/lib/influxdb - change own ship of the files in var/lib/influxdb to the influxdb user
chown -R influxdb /var/lib/influxdb
- start the Influxdb service
systemctl start influxd