Fetching Steemmonsters' game data into the MongoDB

I use a similar approach at Steem Monster Explorer where I fetch custom JSON operations broadcasted by @steemmonsters constantly. (small sleep cycles) MongoDB works well for such small cases where you need to store and query dictionary (document) based data in a quick way.

Workflow

1. Fetch account_history and filter custom_json transactions
2. Ignore the follow plugin on custom jsons. (follow, unfollow, mute, reblog)
3. Insert the json data into the local mongodb instance

Script

import json
import pymongo
from steem.account import Account
import time

def import_sm_actions(mongo_collection, op_count=None):

        counter = 0
        print("Fetching @steemmonsters' history")
        account = Account('steemmonsters')
        for custom_json in account.history_reverse(filter_by=["custom_json"]):

            # stop at the max. limit
            if op_count and counter == op_count:
                break

            if custom_json["id"] == "follow":
                # do not sync follow, unfollow, mute, reblog ops.
                continue

            action_data = json.loads(custom_json["json"])
            action_data.update({
                "action_id": custom_json["id"],
                "timestamp": custom_json["timestamp"],
                "trx_id": custom_json["trx_id"],
            })

            mongo_collection.update(
                {'trx_id': custom_json["trx_id"]},
                {'$set': action_data},
                upsert=True
            )
            counter += 1

        print(f"Total of {counter} operations indexed.")


def main():
    mongo_collection = pymongo.MongoClient()["steem_monsters_db"]["actions"]
    while True:
        import_sm_actions(mongo_collection, op_count=100)
        time.sleep(3)


if __name__ == '__main__':
    main()

Notes

• Depending on your query behaviors you might need to add some indexes. (Otherwise full collection scan might hurt query times.)
• This code is POC. I have an enhanced version at Steem Monster Explorer. However, that's a good start if you're into playing with these kind of data.