Recently you might have noticed that if you had a sell order on NFTMart for Hive or HBD that your cards got sent back to you. We would like to explain what happened to cause your orders getting cancelled and what we are doing in order to ensure something like this doesn't happen again.
The Problem
Our database had crashed and wasn't starting back up. Our database stored all the prices set for the orders on Hive/HBD orders and without that up and running those parts weren't working on our site. After noticing it, we attempted to restart it multiple times as well as diagnosing the problem but none of that was working. We thought that we could just reinitialize it with the .idb files that MariaDB creates and be back up. Sadly, somewhere during the process of saving those files, they got corrupted and had nothing in them. Our recovery method had messed up and without that, we'd need to do a replay to get our database filled up again.
Replay Issues
Due to the current state of the public nodes, we didn't feel comfortable hammering one of the public nodes of one of our employees even with permission. Using a weakened version of our streamer would take too long to get back up and running and so we decided instead to cancel all current orders and return the cards to the owners ensuring that nothing was lost. We then reinitialized the database from scratch giving us a clean slate.
Ensuring This Doesn't Happen Again
We want to make sure that a problem like this won't affect our users in the future. We have enabled hourly backups of our server to be made and for them to be stored on both the server with the database an additional offsite server, with a different provider at a different datacenter.
Apology
We are extremely sorry to our users who and are working to ensure that this doesn't ever happen again. As an apology, for the next week, we have lowered the fees on sell orders down to 0.1%.