I think it is time to sum up the week of mainnet issues now they seem resolved.
A little chronological history to understand what happened:
The Ark mainnet forks and it becomes obvious we have a problem on 9th October.
It is not too clear what the problem really is so we try restore from previous snapshot.
The restored blockchain seems to work but then the error pops up again.
Analysis shows that the network has problems to decide who's turn it is to forge. This leads to forks and the "generator error".
After analysing the error in more detail it looks like the problem happens as soon as 2 delegates are too close together vote wise.
Multiple delegates try to slowly sync the blockchain from 0 and find the same problem.
Somehow balances seem to be miscalculated when trying to decide which delegate is on which rank. This leads to forks as the network can't agree on the delegate that should forge.
We also try to synchronize the blockchain from 0 with an isolated V2 node and it seems to work better up to block 6026860.
It becomes obvious that the old "bignum" (big number) library of JS did have precision problems in older version and that the Ark v1 code is still using this old version.
Some developers fix the bignum issue and sync from 0 to verify that we are still on a clean blockchain.
It becomes clear that the blockchain is clean up to block height 6026860. This makes it necessary to create a snapshot at this height and restart the network from there. This also means that we have to roll back and replay 3 days.
Some prepared the clean snapshot and other started to download and prepare a script to replay all transactions after block 6026860.
We start up a small network of delegates and rebuilt to the height of 6026860 based on fixed bignum code.
We create pull request and code changes to fix the bignum.
Once the small network was stabilized we could fill up delegate wallets with some of the missing forging reward in order to replay all transaction on the network.
After having everything replayed we slowly opened up the network to other delegate making sure that every single delegate rebuilt from a clean snapshot.
Now we got back a clean Ark mainnet rebuilt from 0 without losing any transactions.
It is now 12th of October and we will try to keep it stable and alive until we can move over to v2. We will have to verify and validate the blockchain by synchronisation from 0 until we reach v2 as it became clear during the issue that v1 doesn't validate enough in normal operation.
I am sure i mixed some things up but I hope the rough timeline gives you an idea what happened behind the scenes.
There have been a lot of delegates and team member involved in creating this solution.
The following developers played a key role in the recovery and deserve special thanks:
Also thanks to outboard who had the initial idea to investigate the "bignum" code which lead to the code fix.
We had to put a lot of work into recovering Ark this week and most didn't get enough sleep.
To understand the efforts a bit better. We did all of the following in the last few days:
Analysing the network issues
9 commits to fix the code base
Multiple speed synchronisations from 0 which usually take 3 days with v1.
Multiple synchronisations from 0 to verify with v2 code.
Running a lot of additional nodes to stabilize the network.
Temporary rebuild to at least have a roughly working network.
Downloading all (around 2500) transactions of the last 3 days.
Code a script to replay all transactions to avoid losing any
Keep a small network stable and protected during replay
Create a script to fill up all balances to make replay possible
Coordinate 51 delegates to update and install the clean snapshot only
On top of the main work there was a lot of smaller contributions from the delegates, team and developers to fix our network.
In conclusion: We achieved to identify the code error and recover a clean and stable v1 network without losing transactions. We are all really looking forward to Ark V2. In the meantime we will have to synchronize from 0 now and then to make sure we stay on a clean blockchain. This is also necessary for other Ark based chains until they move to V2.
