Thanks and appreciation to everyone who supported my DHF proposal to fund the work I am doing. It is now funded. I hope to provide value to Hive through the work I’m doing and see more features availed to the community through it.
In this report, I provide updates on the state of the two HAF projects I am currently working on: Plug & Play and Global Notification System (GNS).
TL;DR is at the end of this post
A Version 2 of Plug & Play is under way to:
custom_json operations to all HAF-supported operationsCurrent pull request: https://github.com/imwatsi/haf-plug-play/pull/20/files#
Note: the code is not suitable to run yet, as I am testing individual SQL functions on pgAdmin and updating them, before tying it all together. The server is also unavailable at the moment due to the instability caused by the issues detailed below, to avoid taking away development time by fixing the old version. (HAF replays, server holdups, etc)
I am moving code that handled the synchronisation of plugs from Python to PostgreSQL. This will take away most of the sync related logic from Python, leaving it to handle setup and new DB initialisation and use APIs for data retrieval.
This will improve stability, which was being affected repeatedly when an error occurred, necessitating a replay (taking upwards of 8 hours).
Other issues were encountered during an initial massive sync, where a problem with connection would mean having to re-sync all Plug & Play operations from the beginning.
When I moved most of the sync logic to PostgreSQL, the problems above were confirmed to be solved during tests.
I am making the plugs independent of each other by making them have their own HAF contexts, essentially making them separate “apps” in HAF terms. This means the individual plugs will maintain their own contexts and sync independently of each other. This contains errors within individual plugs and prevents them from cascading to the rest of the system.
I introduced the concept of “plug definitions” (adopted from my work on GNS) to make it possible to add properties to plugs that determine the following and more:
start_block: the block height at which the plug should start processing its dataops: the operation types and associated SQL function names to be used as triggers the plugTo expand on the concept of Plug & Play as a turnkey tool to extract, process and create APIs for specific types of data from the Hive blockchain, I am changing how plugs interact with operations to allow for more flexibility. The definitions mentioned in the section above help with that.
With all these changes, the documentation is being updated to include:
I expect to get a production deployment ready within a few days, barring unforeseen challenges.
Current pull request: https://github.com/imwatsi/hive-gns/pull/12/files#
Note: this code is also not suitable for deployment yet and the server is unstable for use at this point in time.
When I initially developed the GNS MVP, focus was on proof-of-concept and some aspects of the system were not suited for the scale and scope that is now planned.
I have been working on some core changes to how GNS works to make it scale and be able to support a wide range of Hive operations without complicating the implementation or making it hard to maintain or upgrade.
Since GNS only keeps an active dataset of 30 days to populate its notification database, when a new node is setup or when an existing one is reset, it will not have access to account preferences broadcasted before the 30-day window.
To rectify this, I developed a preload function that processes gns operations that were broadcasted since the first public release, to populate "account preferences" state, when none is found in the database.
I am currently working to add filtering support to all HAF operations, which improves how much you can customise notifications. For example, when fully implemented, the following will be possible:
custom_json_operation filtering by the JSON’s contents (key-value pairs for JSON object payloads; index-value pairs for array payloads)transfer_operation filtering by specifying matching rules (complete phrases or partial matches) in memo fieldI am creating groups of separate SQL functions to handle the various HAF operations available, instead of the somewhat singular approach currently implemented. This enables the system to scale with the addition of new notifications.
It also makes it easier to make system-wide changes to how specific Hive operations are handled without breaking other parts. In effect, there will be individual .sql files for each HAF operation type, with functions that define how to process that operation.
Work on this will be delayed a bit, since my current work on the above aspects blocks the development of additional notification types until it’s done.
I expect to finish up the core changes within a week, after which the focus will shift to expanding the notifications available on GNS.
Plug & Play:
custom_jsonGlobal Notification System (GNS):
Thanks for reading. Feel free to leave feedback.
To support my witness node:
Witness name: imwatsi