I.T. Spices The LINUX Way

lightingmacsteem(54)

Published in

#blockchain

Words

364

Reading

2 min

Listen

Play

Python In The Shell: The STEEMIT Ecosystem – Post #94

EXTRACTING JSON LINES FROM THE STEEMIT BLOCKCHAIN

This python followup post took a while as I got very busy with the JOULESTEEM circuit finalizations. I need to go to specific lines in these python posts to enhance our learning experience, and in this post we will see how to extract some JSON lines from the blockchain.

I’m just glad I am again here finally.

Get Present Time

We need to get the present time and format it the way we want it to be. This time variable in the form of mmddyy-hhmmss (MonthDayYear-HourMinutesSeconds).

Any file attached to this numbers will surely be a unique name, like logs.

###GET TIME NOW
ddate = (datetime.datetime.now().strftime("%m%d%Y-%H%M%S"))

Define Temp Folders And Paths

It is much easier to manipulate results and scripts if the paths to any folders and files will be defined at the very start, this way we only need to change a few lines just in case our environment will change.

Any temp folder will be deleted after each run of this python script to avoid any confusion in the next run.

###FOLDERS
tempdir = '/dev/shm/insertjson'
###CREATE THE FOLDERS
#TEMP
if not os.path.exists(tempdir):
    os.mkdir(tempdir)
insertfile = ('/root/STEEM/steem.blockchain.json')
destfile = ('/dev/shm/steem.blockchain-incomplete_resume.json')

Last ID Record On The Database

We need to know the last line on the database using its ID. We need this count because this is a “resume” script, which means that this will be a multi-million lines records. In a record with multi-million lines, we need to start only at the line that is not yet recorded on the database.

We use the os module of python to make everything much easier; so python goes into the linux shell and execute the mysql client command from there.

This will serve as our pointer.

###WHAT IS THE LAST RECORD ON THE DB
###LAST LINE ID
c = ('mysql -h172.17.0.4 -u root -p1234567 -Dsteemit_blockchain -e ' + '"SELECT max(id) FROM data01"')
maxidnum = (os.popen(c).read()).splitlines()
if maxidnum[1] == "NULL":
    maxid = int(0)
    counter = int(maxid) + 1
elif int(maxidnum[1]) > 0:
    maxid = int(maxidnum[1])
    counter = int(maxid) + 1
else:
    print('MAXID is not found.....')

###BLOCK_ID OF LAST LINE
cc = ('mysql -h172.17.0.4 -u root -p1234567 -Dsteemit_blockchain -e ' + '"SELECT block_id FROM data01 WHERE id = ' + str(maxid) + '"')
blockid = (os.popen(cc).read()).splitlines()

There are many ways to execute such routines above, and as what I have said previously, this is the simplest even if I have to admit that it may not be the most efficient due to the os module.

We can do this in pure python, which may be the very reason of my future python blogs for comparison sake.

Have fun, and remember, whatever we learn will always stay with us for a lifetime!!!