Influxdb out of memory error

This week I ran into an issue with my reporting setup with Telegraf, Influxdb and Grafana on my Raspberry Pi 2.
All my dashboards in Grafana weren't able to recieve data from Influxdb. Every panel had error about wrong gateway. So I checked log of Influxdb which is the data source for my dashboards.
The Influxdb service was running but was restarting every 3 seconds. Influxdb crashed with a out of memory exception.
raspberrypi influxd[7774]: runtime: out of memory: cannot allocate 32192-byte block (33210 in use)
raspberrypi influxd[7774]: fatal error: out of memory

After a search on my favorit search, I found an issues thread about Influxdb not being able to start due to an out of memory exception.
jrbury mentioned that Influxdb uses an in memory index for all tags used in ones databases, which can cause that error.
So I had no choice and had to move all files of Influxdb to another box and hope for the best, that Influxdb will start there and give me the chance to deleted the some measurements or databases. Luckly I own a Ryzen powered box with 8 GB, which should be enough for start Influxdb databases of Raspberry Pi 2.
In order to create a copy of my Influxdb databases, I checked the service configuration for the paths to the meta data, data and wal data files.
[meta]
dir = "/var/lib/influxdb/meta"
[data]
dir = "/var/lib/influxdb/data"
wal-dir = "/var/lib/influxdb/wal"

All files are stored in /var/lib/influxdb, so I

  1. Stop the Influxdb service
    systemctl stop influxd
  2. copied all files into a seperate folder
    mkdir $home/influxdb-rescue
    cp -r /var/lib/influxdb $home/influxdb-rescue
  3. change the ownership of the files to my user
    chown -R masterlamps $home/influxdb-rescue
  4. copied the files the Ryzen box
    scp -r $home/influxdb-rescue masterlamps@<ryzen-ip>:$home/influxdb-rescue
  5. delete local backup of influxdb files
    rm $home/influxdb-rescue
  6. loged on to the Ryzen box and change the influxdb configuration to the folder with influxdb files
    [meta]
    dir = "/home/masterlamps/influxdb-rescue/meta"
    [data]
    dir = "/home/masterlamps/influxdb-rescue/data"
    wal-dir = "/home/masterlamps/influxdb-rescue/wal"
  7. start influxdb and delete unnessary measurements and databases
  8. copy the files back to Raspberry Pi 2.
    scp -r $home/influxdb-rescue masterlamps@<raspberrypi2-ip>:$home/influxdb-rescue
  9. delete influxdb files on Ryzen box
    rm $home/influxdb-rescue
  10. delete old files and copy new files from rescue folder to influxdb folder
    rm /var/lib/influxdb
    cp $home/influxdb-rescue /var/lib/influxdb
  11. change own ship of the files in var/lib/influxdb to the influxdb user
    chown -R influxdb /var/lib/influxdb
  12. start the Influxdb service
    systemctl start influxd
H2
H3
H4
3 columns
2 columns
1 column
1 Comment
Ecency