Hello Steemians, it’s been a long day, but #NewSteem is now producing blocks once again. At 11:00 AM the Steem Witnesses began producing blocks in accordance with HF21. Block production appeared to proceed as intended for about 45 minutes at which point the blockchain halted during automated validation, preventing an unknown bug from impacting critical components of the blockchain like token balances and reward allocation.
At that point the Steemit Team along with the Witnesses and other community developers worked together to diagnose the offending bug and issue a fix. There are a number of validation steps that occur during transaction and block processing to ensure that activities on the chain are proceeding as expected. The bug caused one of these checks to fail, which resulted in the halt. To quote community member 
The blockchain performed its necessary checks and measures to ensure that the offending transaction did not make its way onto the ledger. This is a good thing because you would much rather have the system block a problematic transaction and halt so you can address the issue instead of going on unnoticed.
The blockchain resumed block production at 2:47 PM EST with the first block being produced by Witness 
While this certainly did not go as smoothly as we had hoped, we do feel that there are some positive takeaways and as of now, it has been a marked improvement over the rollout of HF20. Leading up to HF21 there was far more testing performed which unearthed problems which were fixed prior to the hardfork. In addition, the halt was due to a bug that would have been extremely difficult to produce in our testnet environment.
We are still investigating the incident to determine what further steps could have been taken. This will certainly be the headline topic of our next retrospective. We will provide more information about what happened, what we’ve learned from the experience, and what we intend to do in the future to ensure that this does not happen again.
Now that normal block production has resumed, the user experience on Steem applications appears largely unchanged, which was the desired outcome. Over time we expect to see rewards get distributed in subtly different ways. To learn more about the major changes to Steem that have now been added to the blockchain, check out this post.
While things appear to be going smoothly now, we are still on high alert and keeping all communication channels open so that in the event that any issues present themselves we are ready to respond.
If you do come across any bugs or other unusual behaviors on steemit.com or any other Steem application, please let us know in the comment section below. If you'd like to chat with other community members about the hardfork or any other technical issues, head on over to the SteemDevs Discord Channel.
The Steemit Team
