Hello Steemians! This is 
hivemind - the backend service which will soon introduce next-generation APIs and Communities for Steem. Now that it’s nearing completion, I would like to share a little information about how it works and what it will do.
Hivemind (or “hive”) will power Communities on Steem, but that’s only one important use case. It’s designed to be a developer-friendly layer on top of the Steem blockchain that will enable any developer to access a database containing all the social information stored on the Steem blockchain without having to rely on minimalistic core blockchain APIs. As with all the software we release, hive is fully open source. It is our goal that everyone developing applications powered by Steem will want to take advantage of hive, so that a consistent user experience can be maintained across the growing ecosystem of Steem applications.
Currently, most Steem-powered applications rely exclusively on steemd full nodes as the backend for information like trending lists, account profile information, reputation scores, comments, follows, and so on. This model has certain strengths which have gotten us quite far, but developers have outgrown it and need a more powerful solution. Relying on steemd nodes prevents frontend developers from iterating rapidly, as (1) all the APIs are written in C++, and (2) any (and all) code changes must go through a rigorous review process because they are close to the critical consensus layer. Thus, it’s quite clear the concerns of application developers (flexibility/simplicity) and blockchain developers (security/stability) need to be separated.
Much of the information that app developers want to leverage is not consensus-critical; that is, it either does not belong in a blockchain node or it doesn’t need to be accessed directly from one. Instead, we can store all the relevant information in a traditional database and keep it in sync with the blockchain, while adding new custom layers of logic and features. This is exactly what hive does; and because it’s written in Python, developers can modify its behavior and APIs with ease... and experiment to their heart’s content.
Despite Hive being “non-consensus” it is important that the information it serves is accurate and up to date. The main challenge here is fork resolution; witness nodes handle this automatically with their in-memory undo database, but it’s not so easy for an outside observer. We could avoid forks entirely by following the last irreversible block, but this would result in a delay of around 45 seconds. Instead, hive (by default) trails behind by just 6 seconds, allowing us to avoid the most common case of harmless microforks. These are trivial for witness nodes but would require extra mechanisms to properly handle externally. In the rare case hive encounters a fork more than 2 blocks deep, we’ve implemented a basic recovery routine. Developers who wish to live on the wild side have the option to run hive with a 0-second delay; those who would rather play it ultra-safe can specify a delay of up to 5 minutes (100 blocks).
Hive focuses on posts, relationships, social actions, custom operations, and derived states.
Hive does not track most blockchain operations. For anything to do with wallets, orders, escrow, keys, recovery, or account history, you’ll want to query SBDS or steemd.
Hive can be leveraged to create new features. Here are some that will be possible with hive:
It has a lot of potential I’m excited to see leveraged. That’s why I have dedicated so much time to making sure it works great: because hive will lay the foundation for a much richer and more social experience on the Steem blockchain, whether you’re using steemit.com or any other site which chooses to take advantage of this new open source layer.
The 1.0 release contains a MVP API which replicates all the relevant bits of get_state and get_discussions_* needed to support condenser. This is really quite boring, but it’s crucial to get steemit.com running against hive so it can be tested for load, reliability, and compatibility. The immediate goal is to take over certain steemd responsibilities while establishing the foundation for next-gen APIs.
The core indexer and API service are nearing completion. Because this is critical infrastructure, we need to ensure reliability and consistency. Currently, work is focused on launching the production environment and tuning performance. Then we begin integration and API testing. Once stable, condenser (powering steemit.com) will be switched over to the new hive endpoints; if everything is integrated properly, users will not notice a thing. It’s akin to upgrading an airplane engine mid-flight while making sure there are no hiccups and all controls work as usual.
Currently, we’re looking at the results of our first production reindex. As you can see in the chart below, we’ve had two large surges of activity on our blockchain. Recent blocks take up to 0.03 seconds to process. This may not sound like a lot, but it adds up.
If there exists a blockchain with anywhere near this quantity of rich and open social data, I’d be surprised (but do let me know). For those keeping score: Steem witnesses have produced over 20,900,000 blocks containing more than 285,000,000 transactions so far!
Hive’s full potential will not be unlocked until the new, modern APIs are finalized. We’ll post another update once we’re ready for beta testers and API feedback.
With the hard part out of the way, the fun begins with hive 1.5. Communities bring a whole new form of interaction to the Steem blockchain, and we are excited to see the ecosystem evolve once users can collaborate easily with like-minded individuals, curate topics of shared interest, and truly organize. We anticipate that the Steem blockchain will become the place to be to discuss everything and anything under the sun.
We know that everyone is as excited as us for Communities to finally launch. We still have some work to do before they are 100% ready, but we are very close to having 1.0 complete. We will continue to keep you up to date with our progress.
Stay tuned!
