I did announce how my public HAF/HafSQL node was down in the last post and how it is going to be back up in 2 days.
Well, I was wrong. As it turns out the main server is broken somehow. I don't know what is wrong but it is way under performing. HafSQL should have taken 24 hours to finish syncing but after 4 days it is still around 20% done on this server.
I did switch to the backup server which was way faster to bring back online. It is running the newer code on newer HAF version. Although it is missing the communities data which I have not done that yet on the new code.
The backup server should work just fine until I replace the main server.
I'll be using ZFS snapshots going forward for faster recovery. With ZFS snapshots it should take minutes to bring back a server online.
It turns out there is a useful tool that can help with automatic snapshot taking and pruning. It is called Sanoid (thanks to @themarkymark)
I'll setup Sanoid and hopefully that should prevent disasters like this happening again.