utopian_data is a CLI application which fetches public data of Utopian into your Mongodb instance.
Supported collections:
- moderators (api/moderators)
- sponsors (api/sponsors)
- posts ( api/posts)
Why
I have been working on a separate statistics application (private) for a while to analyze Utopian data better. However, using a REST API for statistics is not practical.
You need a fast system where you can make complex queries. Syncing Utopian posts' data into your system (you can set a crontab and sync it in every 5 minutes) is much better.
Example Mongodb aggreegation Queries
Total Contribution Count
db.posts.find({}).count()
Top 3 moderators on blog category (moderation count)
db.posts.aggregate(
[
{"$match": {"json_metadata.type": "blog"}},
{"$group": {_id: "$moderator", count: {$sum : 1}}},
{"$sort": {"count": -1}},
{"$limit": 3},
]
)
Top 3 Contributors
db.posts.aggregate(
[
{"$group": {_id: "$author", count: {$sum : 1}}},
{"$sort": {"count": -1}},
{"$limit": 3},
]
)
And more... You can pretty much filter/group everything you want once
you learn how Mongodb aggregation queries work.
Installation and Usage
You need to install and configure golang first. After that:
$ go get github.com/emre/utopian_data
$ cd $GOPATH
$ go install github.com/emre/utopian_data
$ cd `$GOPATH`/bin
$ utopian_data --mongodb_uri=localhost
Output
Let me know if you encounter any issues.
Posted on Utopian.io - Rewarding Open Source Contributors