Hi @richatvns, I will try to tackle these points one by one.
You have a run or catch fire attitude towards development with out the infrastructure to support an AGILE Development infrastructure.
We actually follow the AGILE software development methodology very closely, especially when it comes to blockchain development. We conduct regular daily standups, backlog grooming, story pointing, and engineering retrospectives (the post you read here was a conversion of the results of such a meeting). Our blockchain development backlog / project board is public and can be seen here.
There obviously was not a well thought out test plan.
This Hardfork was tested both internally and externally by our witnesses and developers for over two months. Issues that otherwise would not have been caught were found and addressed as a result of this testing. Of course there are some things to add to future testing which were mentioned in this post. Mistakes can happen, but it is very important to make sure you aren't making the same mistakes twice.
There obviously was no regression testing, a little unit testing maybe.
Steem has some pretty extensive tests that are regularly updated as part of our software development process. You can take a look and see for yourself here.
No one is looking at the overall system integration because things like reindexing times, can easily be calculated and planned for. And knowing how your initial new account staking works was not thought of.
And this reindexing time was planned for - what could not be planned for was the unexpected necessity of reindexing after a failure had occurred. In all cases we try to write bug fixes for the Steem blockchain that do not require a reindex when possible, but, unfortunately, this one required one. For the original rollout, it was very well coordinated (the release was available 30 days in advance and the requirement to upgrade was well communicated to all major node operators).
As mentioned in this post, some things have already been planned to prevent long reindex times and they are part of the spec for the next major update, SMTs.
The worst part was because you had no fall back procedures you implemented another fork patching stuff, instead of rolling back and testing.
What you are referring to would be a chain rollback. On immutable blockchains, that would be highly undesirable. Once a hardfork has occurred, and new transactions that are part of consensus have been added the ledger, it would be unethical to remove them unless there was not another option. This type of 'fix' should only be done in extreme circumstances. This is why 'Ethereum Classic' exists for example - I don't think anyone wants a 'Steem Classic' scenario. Further, ALL nodes would have to decide to also 'roll back' and ignore transactions that had already occurred - it would not just be us to make such a decision.
As far as the traditional front end and microservices development that we do, sure, in the event of a bug that makes it to production we can and do immediately rollback to the previous version. Blockchain development is quite a different thing though and cannot be treated the same way.
The blockchain entirely halting in the event of a bug that would break consensus logic or otherwise be detrimental to the chain is absolutely the correct behavior. The bug should be fixed, and the chain should be restarted and restored.
I could send you simple admins from companies in the 200-300 person level who don't know squat about this or any other blockchain, who would have had a better migration and they earn only about 120k a year!
No, I don't think that someone who doesn't know anything about any blockchain would do a better job for the reasons listed above. This is not a database migration or a simple deployment, this is a coordinated hard fork on an immutable blockchain. These things are not the same.
I think the tl;dr here is that a lot of your assumptions do not translate directly to blockchain development. If we were 'only' a traditional software development shop, I think some of your assumptions would be much more accurate. Anyway, I welcome criticism and wanted to address some of your points here today for the community. Thanks @richatvns
RE: Steemit Retro: August & HF21/22