Below is a summary of the Hive core work done by the BlockTrades team last week and an update on hardfork 24 timing.

Hived (blockchain network software) testing and bug fixes

Earlier last week, @gtg found an intermittent error in the new snapshot functionality (non-consensus related feature). This error was fixed here:
https://gitlab.syncad.com/hive/hive/-/merge_requests/117

Yesterday, @good-karma began testing ecency against our Eclipse API service hosted by @gtg (https://beta.openhive.network/) and reported a problem: the API method find_account_names doesn’t return the post_json_metadata field.

We discovered that this was because this API call was previously handled by the fat node, which kept this data in memory. We no longer require a fat node in Eclipse-based API servers, so this call is now handled by the account history node (which is a low memory node, so it doesn’t collect the data to return this field in the response).

In theory, Eclipse-based apps should use the get_profile API method to get this field now, but as a temporary workaround to avoid requiring apps to update their code before the hardfork, we’ve added a compile flag COLLECT_ACCOUNT_METADATA to allow account history nodes for API services to collect this data:
https://gitlab.syncad.com/hive/hive/-/merge_requests/116

This change will require an 18 hour replay of our hived node, which was began about 10 hours ago, so the data won't be available on our Eclipse API service until the replay completes.

This is a temporary fix and will be deprecated in the coming weeks. In the long term, all apps should update to use get_profile instead to get post_json_metadata.

Note that the COLLECT_ACCOUNT_METADATA flag should only be set to ON when building an account history node that will be used as part of a API service configuration. This is a not a consensus-related change and there’s no need for update/replay by any other types of hived nodes. Also note that this flag is currently set to ON by default, so if you’re not building a node for a Hive API service, you will want to set it to OFF.

There are no currently known issues with hived.

Hivemind (social layer microservice) progress

Most of our effort last week was spent on further testing, bug fixing, and optimization of hivemind performance, with an emphasis on testing using condenser (hive.blog) and condenser’s wallet app to perform “real-world” style testing.

Changes that may potentially impact API method usage:
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/222
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/223

Fixes and improvements to notification methods (especially related to mentions):
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/185
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/213
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/212
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/224
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/225
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/220

Fix performance problem with get_profile reported by @good-karma:
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/233

The above problem was representative of one of the more tricky issues we often face with Postgres: on databases with different histories, we sometimes see Postgres selecting a bad query plan, which leads to bad performance. In this case, it performed well on our development database, but was slow on our testing database. We worked around the problem by eliminating the index that was being used for the bad plan. We don’t think that elimination of this index will impact performance of any other queries, but we’re enhancing our hivemind CI system to detect such potential regressions in performance.

Allow integration with pghero (SQL database profiling/monitoring tool):
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/231

Hive API service maintenance (api.hive.blog)

Currently, the API severs api.hive.blog and anyx.io handle the majority of Hive API network traffic.

Last week, the anyx.io API service was temporarily out of service due to problems in the datacenter where it’s hosted. And several other API services were also down for various reasons (some were updating to Eclipse).

This caused most of Hive’s API traffic to be handled by api.hive.blog (the production API service maintained by blocktrades) for several days. When this happened, we started to see “database timeouts” on our hived nodes due to overloading, which caused intermittent failures in transaction broadcasts by users.

In an attempt to alleviate this situation, we added some Eclipse-based hived nodes to our API service to spread the load. We used Eclipse-based nodes to do this because these consume much less memory, so we could fit them on our existing servers.

In the process, we discovered that neither condenser nor the web wallet had yet been updated to use the version of hive-js that is compatible with Eclipse. This caused the Transaction broadcast error: Obsolete form of transaction detected, update your wallet error that many hive.blog users experienced when posting or commenting.

As a temporary measure to resolve this problem, we reconfigured our jussi to only use the Eclipse nodes to serve blocks and use pre-Eclipse nodes for broadcasting transactions, until we had time to update condenser and the web wallet and test the changes.

In the long term, we will be looking into replacing chainbase, which is the root cause of these database timeouts, with a full database implementation such as rocksdb (i.e. effectively re-implementing MIRA more efficiently) because rocksdb has a more robust and efficient database locking mechanism.

Condenser (hive.blog) and web wallet (wallet.hive.blog) work

Per the above mentioned issues, we discovered that neither condenser nor the web wallet had been updated to the latest version of hive-js. We made the necessary changes here:
https://gitlab.syncad.com/hive/condenser/-/merge_requests/110
https://gitlab.syncad.com/hive/wallet/-/merge_requests/28

We’ve been doing limited testing of condenser and the web wallet starting early last week, but we began more intense efforts this weekend, after we discovered that hive-js hadn’t yet been updated. To simplify the testing of condenser and the web wallet in the develop branch against Eclipse, we’ve modified them to ease use of the Eclipse testing API service:
https://gitlab.syncad.com/hive/condenser/-/merge_requests/110
https://gitlab.syncad.com/hive/condenser/-/merge_requests/114
https://gitlab.syncad.com/hive/condenser/-/merge_requests/115
https://gitlab.syncad.com/hive/wallet/-/merge_requests/31
https://gitlab.syncad.com/hive/wallet/-/merge_requests/32

We also merged in one change to condenser that we couldn’t merge in until we were only testing against Eclipse nodes:
https://gitlab.syncad.com/hive/condenser/-/merge_requests/113

We’ve also seen some references to a hardcoded chain_id in the configuration of condenser and the web wallet. We don’t believe these references currently have any functional effects, so we’re going to remove them today and test to make sure there is no impact from the removal.

Call to arms for Hive Apps developers

We really need all Hive apps developers to begin testing their apps against our Eclipse API service as soon as possible.

We don’t want to execute HF24 until all major Hive apps are confirmed to work against Eclipse API services. @roelandp has created a spreadsheet tracking Hive apps where you can indicate the current state of your testing for Eclipse-compatibility:
https://docs.google.com/spreadsheets/d/10ahdZMR6AgYd6_dXQnioFN8FJAuXteEoWobhgClsGbI/edit?usp=sharing
If your Hive app is not listed already, please add it, it’s an open list.

Also, before you begin testing, please make sure you've made the changes described in this post at a minimum:
@mahdiyari/how-to-prepare-your-applications-for-hf24

And as a reminder, our testing node for the Eclipse API service is located at: https://beta.openhive.network

Plans for coming week

The top 20 witnesses are delaying the final hardfork date until all major Hive apps have signaled that they are Eclipse-compatible. Based on current apps testing levels, this means the hardfork will NOT happen on Oct 6th.

The current plan is for witnesses to review the status of apps compatibility on Oct 8th and decide at that point if most of the Hive apps are ready for HF24. This will require no changes to the hived blockchain software, as the hardfork time is ultimately controlled by when a super-majority (at least 16 of the top 20 witnesses required at minimum) signal for hardfork 24 to happen.

We’re also planning to create a dump file of our hivemind database, to allow other Hive API services to get their own Eclipse API services operational as quickly as possible (otherwise, they would need to do a full hivemind replay which takes 4 days on fast hardware).

Update on Hardfork 24 work by BlockTrades team