Hello Steemians, we apologize for the interruptions in service you experienced yesterday which are now resolved. Thanks to the blockchain’s built-in safety measures, at no time were funds at risk. The interruptions were due to an issue that arose out of the process of transitioning to Hardfork 20. In our last post we announced that we published our Hardfork 20 release and that witnesses and exchanges should begin running version 0.20.0 of steemd at their earliest convenience. When a witness runs the new code, it signals that they are voting in favor of those changes. If a super-majority of witnesses are running Hardfork 20 on September 25th, the blockchain will fork onto the new chain.

Blockchain Halting

Until the date of the Hardfork, both versions produce blocks on the same chain. This gives the witnesses time to test the code as well as cast their vote for the new changes by running the new version. Yesterday, that chain unexpectedly forked when the 0.20.0 nodes produced blocks that did not match the consensus of nodes running version 19 of steemd. This activated one of the many safeguards built into the blockchain which pause block production due to the presence of some unforeseen error. These safeguards guaranteed that no important information, including token balances, was at risk.

The Problem

The specific cause of the problem was a bug in the HF20 code. A constant was changed that should not have been, and this caused the initial fork. Your witnesses did a fantastic job of quickly spotting the suspicious behavior and began reverting to a stable build of the Steem blockchain. This exposed another bug that resided in our fork database logic that was present due to the infrequency that this logic is activated, and which caused the minority fork to halt as well.

Our blockchain developers worked extremely hard to identify the problems quickly and issue a patch that would enable the witnesses to resume block production. They assessed that the 19 fork (version 0.19.X of steemd) was not the problem and advised the witnesses to revert to that version and restart their nodes. 19 of the witnesses responded by restarting their nodes, returning stability to the network. Steemit Inc.’s production nodes are now fully functioning as is steemit.com.

Next Steps

We have already performed a post-mortem on the event and have drafted an attack plan for developing the tools and resources which will ensure that events like this will not happen again. But before we can get to work on those, we’ll be spending the rest of today investigating the fork database code in order to ensure that it’s prepared for the hardfork date of September 25th.

The Cost of Innovation

When we launched the Steem blockchain it was with the goal of creating a protocol that could power real applications that real people use every day. That goal, along with the governance mechanism we chose, has enabled us‒as a community‒to understand what changes need to be made and act on that information by upgrading the software through hardforks. So far we have successfully hardforked 19 times in just 2 years.

This rapid rate of iteration is one of Steem’s uniquely valuable attributes and is virtually unheard of in the blockchain space, which itself is still in a highly experimental phase of development. The cost of this rapid innovation is incidents like the one that occurred yesterday, and the fact that so many people were unable to use the apps they love, is a testament to how successful Steem has been at fostering the development of functional applications that real people use every single day. Yesterday also demonstrated, once again, that our community has the ability to work together to handle these issues so that we can continue to push blockchain technology forward at a more rapid pace.

Thank You

We want to express our gratitude to Steem’s amazing community of witnesses, developers, and users. Blockchain-halting events are pretty much as bad as it gets for a blockchain, but everyone worked together to ensure service was returned as rapidly as possible. It’s a testament to the power of this community that such an event was handled so effectively. Many people outside of Steemit Inc.‒especially the witnesses‒shared vital information that enabled us to respond more rapidly, and return the platform to proper operation.

You don’t lose if you get knocked down; you lose if you stay down. - Muhammad Ali

Steemit Team