Hello fellow Hive community!
I am very excited and proud to announce that after unfortunately missing my first two potential blocks to be signed, as of about 7 hours ago I signed my first successful block as a Hive witness!!
The Witness Journey So Far
My witness journey certainly didn't come without some painstaking hours of troubleshooting and hunting through logs.. One of the purposes of this post is to note that becoming a Hive witness is not the easy money job it may seem at first glance. Some had asked me at first if the goal was just to "turn it on and get rich quick", and it certainly is not. The reason I became a witness in the first place was to do my part to help secure the blockchain and add the value generated back into the greater Hive community, and it certainly is not an easy task.
After enabling my witness about 40 days ago, I received my first block to sign after about 28 days. After waiting 4 weeks for that first block (impatiently I might add), I was pretty upset when I had checked my witness status to learn that I missed the block, and I had no idea why. I double, triple, and quadruple checked my config, and everything seemed to be in the right place. So next I decided to check out the generated logs of the witness, but alas after spending hours going through each line, I couldn't find anything that pointed to an issue. I was starting to feel defeated..
At that point I decided I needed some help. I reached out to a few folks/friends in the community who had helped me out in the past (which I greatly appreciate). After chatting with fellow witnesses @shmoogleosukami and @someguy123, I checked a few parameters of my config where mistakes were commonly made, but everything seemed to be in the right place. After that I was able to have a chat with @rishi556, who mentioned a few more common issues that could cause blocks to be missed, including what ended up being the issue all along, server clock time synchronization (but at that point I still hadn't quite fixed it yet, though it lead me in the right direction).
After feeling a little defeated, I upped my server resources well beyond what seemed necessary, rebuilt my witness node twice, including from source, and made sure my updates were in place for HF25. At this point, I was close to getting my second block to sign, which I missed... again. At this point I felt absolutely defeated, and felt as though I was letting my witness voters down. I had approximately 20 days before my next block to figure out a fix, and I was going to do everything in my power to figure it out.
I read article after article and every piece of documentation I could find that had been written about different node configurations and what not. At that point, I felt like I had a PhD in Hive witness configs, but I still had no idea why mine wasn't working. Then, just last night while having a beer for #beersaturday and just about to write a post for it (which I missed posting after focusing on my witness fix, sorry @detlev haha), I hit my eureka by accident.
Feeling fairly defeated and being so close to signing another block, I decided to take a step to get a backup server from the Privex team as I knew their all star support and knowledge would definitely make sure my witness was up to snuff for signing my blocks in case my primary witness node should fail again. After ordering my node, I received an email with a couple of attached articles, one of them being about editing the NTP server time synchronization to help keep you within 10s instead of 100s of milliseconds of blocks being added to the chain.
What?!? This seemed so simple, and what if it was my issue all along? I decided to give it one last shot, else I had planned to temporarily switch my infrastructure to a privex pre setup node for the time being. I setup the privex node as a backup and setup hive witness essentials to switch between nodes should one fail. Then it was time to apply the fix. EUREKA! Two lines added to a single config file, and next thing I know, I signed my first successful block!!
Guide for the quick Time Sync fix
If running/setting up your witness node on Ubuntu Focal (recommended, 20.04), you likely don't have NTP installed or configured with a high enough polling rate to keep in a proper time synchronization with blocks being added to the chain by default, which may be a big contributor to missed blocks for your witness, as the time sync seems to need to be under approximately 950 ms to ensure proper block signing (atleast from what I observed). Applying this fix solved this issue of lowering my block addition time sync from about 975 ms to 50-250 ms.
First, install NTP if you have not already:
sudo apt-get install ntp
Next, edit the config to ensure a more frequent polling rate:
sudo nano /etc/ntp.conf
and add to the end of the file:
minpoll 5 maxpoll 7
And save.
The above parameters are in powers of two, so this would cause the server to poll and synchronize its time every 25-49 seconds. THis is overkill in most cases, but semi necessary as a block producer.
After completing the above, enable and restart the process like so:
sudo systemctl enable ntp
sudo systemctl restart ntp
That's all, such a frustratingly easy fix for hours of troubleshooting haha.
Conclusion
Becoming a Hive witness is definitely not just an easy set-it-and-forget-it journey. It is one that takes careful configuration and maintenance in order to work properly in the way the community deserves. I want to thank everyone mentioned above for their help in getting me to this point and all of my followers/supporters who have moved my witness up to the 137th active witness spot!
My goal is to hopefully move into the top 100 active witnesses within the next 4 months. I don't expect this to happen without being earned, and I will continue to contribute value to the Hive community to help improve our wonderful blockchain ecosystem and earn a few more witness approval votes along the way. If you currently have witness votes to spare, consider voting for my @sicarius witness: https://vote.hive.uno/@sicarius! It would be greatly appreciated!
Also consider voting for the all star witness nodes below, whose help was greatly appreciated:
Until next time, Sicarius