We’ve spent the past week analyzing the performance of Eclipse hived/hivemind in our API service (api.hive.blog) and making improvements based on real-world loading patterns.

It’s been a very productive week and we’ve resolved most of the outstanding issues reported and performance problems we observed. We’ve also identified and fixed several longstanding issues in the hived mutex locking code that prevented peak performance and led to potential data corruption issues.

Hived work (blockchain node software)

We made a number of non-consensus changes to improve hived performance and fix old bugs:

We reviewed the locking used to protect the chainbase database and found several serious longstanding bugs that showed up under high stress API loading of a hived node. These bugs could lead to data corruption and non-optimal performance. Those fixes are incorporated into this merge request:
https://gitlab.syncad.com/hive/hive/-/merge_requests/143

After fixing these problems, we ran several high stress tests using the siege tool and were unable to “break” hived’s operation anymore. We were also able to get hived to fully utilize all the server’s cores (48 on our test system), demonstrating that the locking is fairly optimal for realistic loads.

As a performance optimization, we enabled filtering of operations returned by one of the most time-consuming API calls, get_account_history:
https://gitlab.syncad.com/hive/hive/-/merge_requests/136
https://gitlab.syncad.com/hive/hive/-/merge_requests/137
https://gitlab.syncad.com/hive/hive/-/merge_requests/139

We made several changes to API responses returned by hived, mostly in response to reports from apps developers:
https://gitlab.syncad.com/hive/hive/-/merge_requests/125
https://gitlab.syncad.com/hive/hive/-/merge_requests/126
https://gitlab.syncad.com/hive/hive/-/merge_requests/133

We also limited the max number of operation of operations to that can be fetched by this call to 1k operations (previously it allowed up to 10k operations) to optimize hived throughput: https://gitlab.syncad.com/hive/hive/-/commit/9a05f4d780fc98b891f51ab4b88ebec89fde467b

We corrected account-impacting for several virtual operations created recently:
https://gitlab.syncad.com/hive/hive/-/merge_requests/134

Hived status

We started a full replay and a full sync early this morning to check all the above changes (full replay takes around 18 hours). We don’t expect any issues, but there is a very small chance for issues due to the mutex locking changes, even though no consensus changes were made.

With these changes, I think hived’s performance and reliability has been substantially improved over its past condition and it seems quite solid in the face of very substantial API traffic loading.

One minor issue still to be resolved is the longstanding issue with hived crashing if it receives API traffic when starting up, so we may tackle that next, since it can be an annoyance when updating several hiveds deployed inside of a API server node configuration.

As this version contains substantial operational improvements, we’ve tagged this version as v1.24.5, and it’s a strongly recommended upgrade for all API server nodes. However, there’s no need for exchanges or consensus nodes to upgrade to this version, since the performance improvements mostly come into play when a node is handling significant API traffic.

Hivemind

We made numerous optimizations and bug fixes in hivemind this past week:

Hivemind status (2nd layer social media microservice)

By late tomorrow I expect we’ll have resolved all the remaining big issues with hivemind responses and performance, other than reputation support (which we’ve intentionally left out until all other issues were resolved).

Condenser + Condenser wallet (open-source code for hive.blog)

Merged in support for decentralized lists:
https://gitlab.syncad.com/hive/condenser/-/merge_requests/135
This functionality is still waiting one last issue to be resolved in hivemind, so it won’t be fully testable until tomorrow when we deploy a new version of hivemind.

Proposals page changes:
https://gitlab.syncad.com/hive/wallet/-/merge_requests/47
https://gitlab.syncad.com/hive/wallet/-/merge_requests/49
https://gitlab.syncad.com/hive/wallet/-/merge_requests/51

Condenser status

Condenser is looking pretty good at this point, but we’ll continue to test and look for any remaining problems.

Hive-js improvements

In order to make hive.blog faster, we modified hive-js to support the new option to filter the operations fetched by the get_account_history call:
https://gitlab.syncad.com/hive/hive-js/-/merge_requests/12
https://gitlab.syncad.com/hive/hive-js/-/merge_requests/13
https://gitlab.syncad.com/hive/hive-js/-/merge_requests/14

What’s next for the week?

Tomorrow we’ll hopefully be deploying a new hivemind with all the performance improvements and fixes we made in the last week, after some real world traffic testing on our API node.

We’ll also be testing condenser and the wallet and looking for fixes and optimizations we can make.

Assuming deployment of latest hivemind goes well tomorrow, we’ll likely still need to make a few more fixes as issues get reported, but I expect that by the latter part of the coming week we can begin to analyze feasibility for various previously suggested features for HF25 and also intermediate changes that aren’t consensus changes that can be released earlier.

Update on BlockTrades Hive core coding