Hello Steemians, steemit.com is back up as are all of Steemit’s nodes. Over the weekend the chain halted due to a bug in the curations reward curve that was unrelated to the previous halt or the delegation bug. While all services have been restored, and operations appear stable, we wanted to take this opportunity to fill you in on what occurred. A more thorough investigation will be conducted and more information will be shared after our next retrospective.

We were able to develop a patch which did not necessitate a hardfork, however, it did require a replay of our nodes which are used by steemit.com and other Steem applications like Steem Monsters. Witnesses upgraded using the patch and resumed block production on September 2nd at 06:26:42 UTC. Around 13:00:00 UTC on September 3rd Steemit and the Steemit API services were restored after replays were completed.

The Bug

The bug was especially resistant to detection because the problem was not with the mathematics backing the curation reward curve, but subtleties in the integer based approximation of a square root. The bug was the result of a subtle behavior in the new curation rewards curve that lead to votes being able to have a curation reward weight of -1, which when stored in an unsigned field resulted in a computed value of 2^64 - 1.

When paid out, these votes were attempting to receive trillions in STEEM. This was caught by other assertions which instructed the blockchain to halt block production and prevent the unintended behavior from impacting any reward distribution.

The Fix

The new release creates a floor for the curation reward weight in such cases to 0. This can only happen when votes are allocating very few rshares and would not receive a reward, so forcing the value to 0 will not impact how rewards are allocated.

We apologize for the inconvenience. We will be conducting a retrospective and sharing the results of that retro on @steemitblog, so if you would like to learn more about this incident and what we have learned from it, be sure to follow this account.

The Steemit Team