Below is a list of Hive-related programming issues worked on by the BlockTrades team during the last week:
Hived work (blockchain node software)
Last week we’ve been focused on testing hived in preparation for a final feature freeze and fixing bugs as we find them. So far most of the bugs we’ve found are very old bugs that have been around for many years.
We fixed two bugs we found and discussed in last weeks report:
- Better warnings and less cpu-loading when node gets no peers at startup
https://gitlab.syncad.com/hive/hive/-/merge_requests/220 - Fix to beem to fix update_proposals test
https://gitlab.syncad.com/hive/beem/-/merge_requests/6
During benchmarking of hivemind sync on two similarly configured systems, we found one system was syncing much slower than the other one. After a fair amount of debugging effort, we ultimately were able to trace this back to intermittent errors on one of the disks in the raid array where hived stored it’s data, but more interestingly, this also exposed an error that was introduced over 5 years ago (ie. when this code was in the BitShares graphene repo).
This bug caused incoming blocks that are not yet linkable to the node’s head block to be discarded instead of being kept for future usage, once intervening blocks were received.
On normally functioning hardware, this was no big deal, but in the case where disk IO performance was strained (in this case due to intermittent disk errors), this error led to the node often dropping peers and generally straining to keep in sync with the rest of the p2p network.
Note that this is a practical real world problem, not just a theoretical one, as we believe this problem led to one commonly used exchange having problems maintaining their Hive wallet open for deposits. At the time, the problem was resolved by convincing the exchange to move their Hive wallet to another computer, but it’s obviously better to have Hive nodes more resistant to such issues.
We fixed this bug in these two merge requests:
https://gitlab.syncad.com/hive/hive/-/merge_requests/226
https://gitlab.syncad.com/hive/hive/-/merge_requests/227
As part of the above work, we had to dive pretty deep into profiling the performance of hived when processing transactions and blocks, and we identified several places where we would like to improve performance in the future, after the hard fork is completed. The majority of such changes can be released at any time, as they don’t require consensus changes.
To facilitate our profiling work, we also made changes to how timestamps are represented in hived logs. By default, timestamps are now stored as UTC times with milliseconds resolution in files and seconds resolution on the console, with the option to store the timestamps with microseconds resolution when needed for profiling: https://gitlab.syncad.com/hive/hive/-/merge_requests/229
We added a script that automatically code-generates the get_config api call, so that when new parameters are added to the blockchain, these parameters are automatically added to the output of this call. This script is executed by CMAKE at build time. https://gitlab.syncad.com/hive/hive/-/merge_requests/211
We fixed several longstanding errors with the account history plugin:
Under rare circumstances, the account history plugin could include duplicate records of an operation in an account’s history as a result of microforks because the notification of last irreversible block was sent to the plugin too early:
https://gitlab.syncad.com/hive/hive/-/merge_requests/205
The wrong number of operations could be returned by get_account_history because of miscounting of effective_comment_vote_operation:
https://gitlab.syncad.com/hive/hive/-/merge_requests/225
We also reviewed and merged in @howo’s changes for recurrent transfers today:
https://gitlab.syncad.com/hive/hive/-/merge_requests/150
Finally, we reviewed and approved @howo’s proposed implementation for resource credit (RC) delegations: https://gitlab.syncad.com/hive/hive/-/issues/152 We will be able to incorporate this change after the hardfork, as it doesn’t require any consensus changes.
Hivemind (2nd layer applications + social media middleware)
Our work in hivemind last week has also revolved around testing performance and fixing bugs.
We ran full hivemind syncs to test the new code that deep cleans the dictionaries to cut a few gigabytes of memory from hivemind’s memory footprint. The new code worked well, and without any apparent performance problems, but we’ll need to repeat the tests again to obtain final performance numbers, as we got distracted by the hived performance issue we found during this testing (discussed earlier in hived section of this post).
We made several fixes to hivemind:
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/493
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/494
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/496
We found another issue with hivemind that is still under investigation, so we’re not quite ready to make a new release candidate for hivemind yet, but I’m hopeful we’ll still be able to do so by the end of this week as originally planned. The timeframe will be tight, as we still need to merge in several of the above changes to the develop branch and do real-world performance testing on api.hive.blog.
Modular hivemind (framework for hive applications)
During documentation and discussion of the fork resolution code for hive using SQL shadow tables, some concerns were raised about issues where a modular-Hive based application could hold a transaction open, preventing the fork-resolving code from swapping to a new fork, so we’re looking at various ways to manage this issue.
We didn’t have time to make the changes for the sql serializer last week due to the time spent fixing the various issues discussed in this post, but we will likely complete that work in the coming week. Then we’ll be able to do a full performance test of the sql serializer in combination with a full sync of hivemind.
Condenser (code base for https://hive.blog)
We incorporated several fixes to condenser and deployed them on hive.blog:
Search fix by @quochuy: https://gitlab.syncad.com/hive/condenser/-/merge_requests/246
Iframe fix by @mahdiyari: https://gitlab.syncad.com/hive/condenser/-/merge_requests/249
Fix for 3speak.tv domain by @jes2850: https://gitlab.syncad.com/hive/condenser/-/merge_requests/251
Api node selection fix by @jes2850: https://gitlab.syncad.com/hive/condenser/-/merge_requests/254
Hived testnet
We’re planning to launch an updated testnet on Wednesday with the latest fixes to hived discussed above. We’ll be tagging this version of hived as an official release candidate.
That will be followed by the launch of an API node configured to draw data from the testnet (probably on Thursday). This API node will allow Hive applications to begin adding code changes to support new features added by the hardfork, such as vote expiration reporting.
Planned date for hardfork 25
Based on our testing experience so far, I’m currently projecting the hardfork will be in the latter part of June (about a week or so past my earlier “best case” projection).
Assuming no major problems are discovered in the coming week with the release candidate, we’ll next begin notifying exchanges in order to give them 30 days notice which should be plenty of time to update their wallets before the hardfork.