I.T. Spices The LINUX Way

Python In The Shell: The STEEMIT Ecosystem – Post #101

BLOCKCHAIN AND DATABASE MANIPULATIONS USING PYTHON

In continuation as per the previous post here:

https://steemit.com/blockchain/@lightingmacsteem/58ebdn-i-t-spices-the-linux-way

We had arrived at a point where we need to format the JSON data into something easily understood by python using the json loads module:

17  
18      ###PYTHON
19      jason = json.loads(line)
20      if (jason["block_id"]) == []:
21          block_id = "NA"
22      else:
23          try:
24              block_id = (jason["block_id"])
25          except (OSError, TypeError):
26              block_id = "ERROR"
27      
28      if (jason["extensions"]) == []:
29          extensions = "NA"
30      else:
31          try:
32              extensions = (jason["extensions"])
33          except (OSError, TypeError):
34              extensions = "ERROR"
35      
36      if (jason["previous"]) == []:
37          previous = "NA"
38      else:
39          try:
40              previous = (jason["previous"])
41          except (OSError, TypeError):
42              previous = "ERROR"
43      
44      if (jason["signing_key"]) == []:
45          signing_key = "NA"
46      else:
47          try:
48              signing_key = (jason["signing_key"])
49          except (OSError, TypeError):
50              signing_key = "ERROR"
51      
52      if (jason["timestamp"]) == []:
53          timestamp = "NA"
54      else:
55          try:
56              timestamp = (jason["timestamp"])
57          except (OSError, TypeError):
58              timestamp = "ERROR"
59      
60      if (jason["transaction_merkle_root"]) == []:
61          transaction_merkle_root = "NA"
62      else:
63          try:
64              transaction_merkle_root = (jason["transaction_merkle_root"])
65          except (OSError, TypeError):
66              transaction_merkle_root = "ERROR"
67      
68      if (jason["witness"]) == []:
69          witness = "NA"
70      else:
71          try:
72              witness = (jason["witness"])
73          except (OSError, TypeError):
74              witness = "ERROR"
75      
76      if (jason["witness_signature"]) == []:
77          witness_signature = "NA"
78      else:
79          try:
80              witness_signature = (jason["witness_signature"])
81          except (OSError, TypeError):
82              witness_signature = "ERROR"
83      
84      if (jason["transactions"]) == []:
85          transactions = "NA"
86          operation_type = "NA"
87          transaction_ids = "NA"

Firstly, and to give us a better idea of what JSON data I am talking about here, an example screenshot of a steemit blockchain data is taken below:

JSON Lines Data Extraction Explained

Lines 19 converts the json formatted stream as given off by the line variable, making it into a python formatted single quoted stream. This is what the json.loads python command does, we made sure that whatever data extraction we will do after can be easily understood by our python script, and fast, of course. Actually very very fast.

As this is a repeating process of data extraction, I will just explain lines 20 to 26 in detail firstly.

Lines 20 to 26 is now the extraction of the block_id data. The if in combination with the try statements are put in place to eliminate any errors that will stall the python script. We are simply telling python what to do here for whatever the data it might encounter, including no data as represented by the blank [] characters.

If the block_id have data in it, then the else statements from lines 23 to 26 will take care of the data in memory. The jason["block_id"] explicitly extracts only the block_id info from the string of characters. It simply meant that the line variable has a block_id entry among many entries, I need you to get just that entry for me.

Lines 27 to 34 deals with the extraction of the extensions data of the blockchain.

Lines 35 to 42 deals with the extraction of the previous data of the blockchain.

Lines 43 to 50 deals with the extraction of the signing_key data of the blockchain.

Lines 51 to 58 deals with the extraction of the timestamp data of the blockchain.

Lines 59 to 66 deals with the extraction of the transaction_merkle_root data of the blockchain.
Lines 67 to 74 deals with the extraction of the witness data of the blockchain.

Lines 75 to 82 deals with the extraction of the witness_signature data of the blockchain.

Lines 83 to 87 deals with the extraction of the transaction_ids, operation_type and transactions data of the blockchain. These lines forces the routines after this to be only if the transactions data is [], or blank, meaning no data.

We need to take note that the blockchain data above is just one line, a block of data, among millions of lines and growing fast.

That is how important it is to use proper tools in extracting these. I am just glad python is there to assist us effectively.

We will discuss the lines that deals with the database insertion next.