NOTE: This outage did not affect the ability to process transactions on the blockchain, or to sign blocks. Only public facing HTTP API endpoints were affected.
Cypherglass, and many other block producers that provide full API access to the EOS community, experienced an outage of all of our API nodes at 2018-07-27 04:14UTC. These are the systems that sit behind our load balancers which provide our HTTP API service at api.cypherglass.com.
It appears that the outage was due to the recent spam on the EOS network creating ~73,000,000 transactions with no relevant data. Despite the fact that there is no relevant data, the BP nodes still have to process and store it.. This caused the NODEOS software to run out of memory, despite each BP’s node being set to support 64GB of EOSRAM and each system being configured with at least 72GB of system RAM. The result of running out of memory was that the NODEOS software crashed and corrupted it’s database. The outage impacted only those BP’s who provided unfiltered API nodes with the history plugin enabled. These are the only API nodes on the network that can provide data about any transaction ever performed on the blockchain.
This outage did not affect the ability to process transactions on the blockchain, or to sign blocks. Only public facing HTTP API endpoints were affected.
The Cypherglass API service was restored, although without the history_api enabled, at 2018-07-27 05:40UTC. This was due to the time it took to restore from backup; a node that did not have history_api and replay the chain.
To combat the spam we have been forced to limit the history we provide via our API node. We have chosen to use the standard adopted by several other Block Producers. Our thanks to the team at Greymass for this fine document
The history_api was enabled again, with the settings from the document above, at 2018-07-27 16:31UTC. Many block producers do not provide this service, but we believe it’s necessary for the EOS community to have access to the full blockchain.
Cypherglass’s API is fully functional again and we are pleased to provide this service at the following URL’s (Note these URL’s will not work in your browser, they are meant for application access):
This outage affected many Block Producers and it was amazing to see, once again, how the EOS community comes together when there is an issue. Technical support, advice and even snapshot files were traded between competing Block Producers in this time of crisis to make sure that we all provide the best service possible to the EOS community.
Example of one of the spam transactions that have flooded the network lately.
Thank you for your support EOS Community, to learn more about Cypherglass, check us out online here:
