My witness journey has seen many ups and downs, but I'm very happy with the destination I ended up at and am looking forward to improving even more!
A few months back, after we moved to 1.7.2, I downloaded a snapshot and restored it over a million and a half days, but when it finally started, no matter what I did, the blocks ahead would only increase and it would both not decrease and never got to 0. After struggling for a couple weeks, I literally gave up and shut down my witness. Depressing, no doubt.
When the light node was released, I thought I'd give it another shot and fortunately I spun up a working node extremely quickly and I was back online! Of course, when you shut down your node for a couple months you lose most of your votes so I've fallen down in the ranks quite a bit. I've had a bit to play with the light node and for awhile I was the highest ranking account running a light node so I've learned a lot there.
The issue started here:
Ugh, constantly is saying it softly, I was missing TONS of blocks on my shitty cloud server. This entire time, I have been using basic cloud servers online, which initially worked but I think we may have grown big enough to push past the baby leagues today so I've upgraded!
I've upgraded to a full dedicated server, running an Intel CPU with 8 cores and 16GB of memory. A quick snippet of top:
top - 06:12:32 up 22:15, 1 user, load average: 0.37, 0.33, 0.29
Tasks: 152 total, 1 running, 151 sleeping, 0 stopped, 0 zombie
%Cpu0 : 2.3 us, 0.3 sy, 0.0 ni, 97.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu1 : 0.7 us, 0.7 sy, 0.0 ni, 97.7 id, 1.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu2 : 5.0 us, 0.3 sy, 0.0 ni, 94.4 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
%Cpu3 : 25.8 us, 5.0 sy, 0.0 ni, 62.1 id, 7.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu4 : 12.0 us, 0.7 sy, 0.0 ni, 87.0 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
%Cpu5 : 0.7 us, 0.0 sy, 0.0 ni, 98.7 id, 0.7 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu6 : 0.3 us, 0.0 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu7 : 1.0 us, 1.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 38.0/16007.7 [|||||||||||||||||||||||||||||||||||||| ]
MiB Swap: 1.4/1463.0 [| ]
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
768 mongodb 20 0 7159760 5.2g 41972 S 33.9 33.1 246:21.56 mongod
891 root 20 0 825112 158004 32660 S 16.3 1.0 589:55.55 node
I was seeing loads of 3.0+ with literally ALL of my memory (and swap) taken up, so we've moved lightyears ahead.
I think this beast should be enough to hold out for awhile, or so I hope. The real test now is to see how soon I miss any blocks and if I can stay online now with way more resources.
I feel bad for running a shitty node for so long so I'm excited to get this optimized and running as well as possible for the long haul. And to be honest, if this turns out to be too small, I'm not afraid to upgrade again!
Of course, this brings up the obvious point: servers aren't cheap.
With that in mind, I respectfully ask for your vote to support my work as a witness so that I can continue to deploy reliable hardware to build a strong network. Thankfully, I'm not ranked super low so it wouldn't take a whole lot to push into the top 20, which is genuinely my goal. When Hive Engine first launched I was not only in the top 20, but like the top 5 and I'm sad that I fell out rank but I hope to earn your respect and climb back up.
With this in mind, would you consider voting for a second account I just created: 
I've also been thinking about doing some performance benchmarks to really test these deployments fully. If I can find a consistent way to measure performance, I want to deploy multiple nodes with different settings and/or versions of software to see what really performs best. We were recently debating which version of node to run and the officially supported version is Node 14 while many have been running Node 16 without issue. I'm interested to see which one actually performs better. And node just node, but also Mongo in particular and every other little lever that I can wiggle and test to find the perfect combination of settings.
One obvious improvement is to move from an EXT4 filesytem to XFS, Mongo will take a sigh of relief and literally thank you. I want to document this one better because I only see passing mentions to this and I don't think this is widely deployed, I'm almost certain most of you are running on EXT4 and that is not ideal.
In any case, I'm back up and running on good hardware for the long run! Thanks to everyone for your past and continued support, I sincerely appreciate it.