First release of marcelmaatkamp/steemit-data-analysis-platform

marcelmaatkamp(46)

Published in

#utopian-io

Words

380

Reading

2 min

Listen

Play

Release v1.0

https://github.com/marcelmaatkamp/steemit-data-analysis-platform/releases/tag/1.0

I hereby release the first version of my project where I ingest Steem data in JSON format, put it in RabbitMQ via and ingest the data via java Spring Boot into Neo4j where I can query the data to find interesting patterns. I focussed for the first version on the votes coming from Steem to analyse who is voting on what and made a screenshot available to show the first results of what I found in the data.

I do all this in my spare time so I hope that I can find some time to add more domain objects and make more sense of the data produced by the Steemit platform.

Documentation I added in this initial release:

Installation
Documentation on how/to ingest and display data
Added urls for external resources and
Screenshots added showing patterns in the data ( and thus mission v1.0 accomplished!)

The code so far:

is a fully working version
has examples in src/scratchbook
contains unit and integration tests and queries in src/test/java
contains a dockerfile and has
docker-compose file added to bootstrap the project

Once the ingest is running Neo4j can be queried to find patterns in the data:
graph_version_1.0.png

(The green nodes are "Authors", red nodes are "Voters" and grey nodes are "posts")

Plans to enhance this project could be:

To make the data accessible and interactive via a web-interface
Generate a docker container with this project and
Instantiate a working version on a platform like "Heroku".

Its a pity Steem does not let a author include interactive elements in its posts.

Prerequisites

To run this project the system needs the following locally installed:

docker and
docker-compose

Installation

Start the whole stack with just these 4 statements:

$ # Start proxy rabbitmq and neo4j services
$ git clone https://github.com/marcelmaatkamp/marcelmaatkamp/steemit-data-analysis-platform.git &&\
  cd marcelmaatkamp/steemit-data-analysis-platform &&\
  docker-compose up -d socksproxy rabbitmq neo4j
$ # Start the live data stream from steemit into rabbitmq
$ RABBITMQ_HOSTNAME=rabbitmq \
   RABBITMQ_EXCHANGE=steemit.api  \ 
   STEEMIT_ACCOUNT_NAME=<OUR_STEEMIT_NAME> \
   STEEMIT_ACCOUNT_ACTIVE_KEY=<YOUR_STEEMIT_ACTIVE_KEY> \
   docker-compose up steemit-live-data
$ # Ingest data from rabbitmq into neo4j
$  steemit_account_name=<STEEM_USERNAME> \
    steemit_account_postkey=<STEEM_POSTKEY> \
    steemit_account_activekey=STEEM_ACTIVEKEY> \
    docker-compose up --build steemit-database

Proxy

Set your browser proxy settings to connect to the services rabbitmq and neo4j via the following url:

socks5://localhost:9050`

RabbitMQ

Once the proxy is installed goto http://rabbitmq:15672 and login with username guest and password guest. The created exchange is steemit.api and the queue is steemit.api.votes.