The Steem blockchain recently stalled at block 23847548 due to an invalid transaction that was allowed into a previous block. The Steemit development team quickly identified the cause and issued a patch, which was then deployed by a majority of the witnesses. At this time, the Steem blockchain has resumed normal operation. At no point during the event were user accounts or tokens at risk.

It is important to emphasize that what took place was actually the result of a protection mechanism built into the blockchain preventing the invalid transaction from doing any real harm. While it is unfortunate that operations were suspended during this time, it is these protection mechanisms that help ensure that accounts remain safe and secure even in the face of unforeseen events.

Cause

Seven days ago, an account (@nijeah) attempted to submit a transaction that would have resulted in a negative balance of STEEM being powered down from their account. The blockchain has safety rules that forbid such a transaction from occurring, but these rules did not forbid the invalid transaction from being submitted, even though its attempted execution--which would occur seven days later--would not be allowed to occur by the blockchain.

When the scheduled power down occurred, witness nodes were unable to process the transaction--and all subsequent transactions--due to the aforementioned rules. This is what we refer to as “halting” (as opposed to something like “forking”). A code change was needed in order to define how to properly handle this behavior.

Solution

The Steemit development team, along with the assistance of several of the witnesses, was able to quickly identify the root cause of the problem. As soon as the cause was identified, a patch was issued and the rollout of the patch was coordinated with the top witnesses.

Within only a few hours of the issue occurring, the patch was applied by a majority of the witnesses, and the Steem blockchain resumed normal operation.

Instructions for node operators

This section contains instructions for node operators who still need to apply the patch.

All nodes running 0.19.3 should update to release version 0.19.5 to start receiving blocks again. The patch will not require a replay.

If you were running the AppBase release candidate (0.19.4), a new release candidate (0.19.10) will be made shortly. Alternatively, you can run the branch 20180702-fix-vesting-withdrawals-steemd to get the patch now.

Impact

Any transactions that were submitted during the time the blockchain was halted would have resulted in an error. Some pending transactions that were submitted just prior to the halt may not have been included in blocks, and would have expired. Affected transactions would need to be resubmitted, as they would not have been included in a block and are no longer valid.

Other than the period of time where no new transactions were allowed, there was no additional impact from the event. Everybody’s tokens remained safe, and accounts were not at risk of being hacked.

Conclusion

We want to thank everybody involved for their responsiveness during the event. It is a great testament to our amazing blockchain team and Steem witnesses that we were able get the blockchain back to operational status in such a short period of time.

Great job to everyone involved! Steem on!

Team Steemit