Hello !
For an upcoming project linked to steempress we needed to have a local db with all the posts on the blockchain that has been done for the last week. For speed of course but also to make complex queries in one go. Of couse you could argue that there is already solutions like steemdata or steemsql but steemdata is out of sync and steemsql is not free.
So I figured that building a custom parser would be the best way to go forward.
I spend a big amount of time on the code which can easily be modified for a lot of various use case. So I figured I'd release the source. Note that you still may find some bugs.
Source which you can now find on my github :
https://github.com/drov0/datasteem
datasteem is a tool that will fill a mysql database with infos about posts and users. more specifically :
for posts : block_id, author, title, date (unix timestamp god so much simpler to deal with those), text, permlink, image (if any), tag1,2,3,4,5, json_metadata, reward (in sbd), comment and upvotes number.
for the users : username, reputation, steem_posts, steem join date, followers, following count, steem power, delegated steem power.
It can fill the database in real time and in the case of a crash catch up if he missed a few blocks.
It's a typical node program with a mysql database behind it.
I stream the blocks via dsteem and do most of the queries via steemjs. This is because I'm way more familiar with steemjs for querying data and I find it more convenient. And dsteem's stream api is just great, hence the crossover.
I strongly recommend the use of pm2 to handle the process.
git pull [email protected]:drov0/datasteem.git
cd datasteem
npm i
install the database via the db.sql file
pm2 start datasteem.js
I plan on polishing the tool and db scheme to scale and handle more data/operations. For instance I would love to store all voting operations to perform some data analysis on how the global stake is moving.
Submit a pull request, comment your code or write it in a way so it read itself and good to go :D
