Introduction

Hey it's a me again drifter1!

And with that, I'm basically mostly done with the project I had in mind! Other topics such as the explorer, wallet, light nodes etc. will be topics that we will covered in a monthly basis. As I will have to get into the thesis now, and I also don't want to leave the other series on the blog behind...

During the previous days, I got into the miner (finally) and also made the request bodies far smaller by introducing block headers instead of the complete block, and a transactions header instead of the transactions array. Therefore, transactions are basically retrieved one-by-one now, and not as a complete package. And instead of the complete block, the miner only posts the block header, and the full node then "reconstructs" the block using the local unconfirmed transactions.

There has been some cleanup as well, such as making a node update script and introducing more node settings to the common Node_Settings structure. The miner also comes with it's own Miner_Settings structure, as it only has to update the nodes, and contacts full nodes for the rest. Lastly, because of the miner, the testing.py script now simply acts as an example of posting a transaction.

So, without further ado, let's get more in-depth!

GitHub Repository

https://github.com/drifter1/blockchain

Transactions Header

The transactions header, is what is returned instead of the complete unconfirmed transactions array. So, it basically has a transaction_count field and an array of transaction headers. A transaction header is a stripped down version of a transaction with only the following fields:

timestamp
value
fee
hash

In JSON a transactions header looks like this:

{
    "transaction_count": transaction_count,
    "transaction_headers": [
        {
            "timestamp": timestamp1,
            "value": value1,
            "fee": fee1,
            "hash": hash1
        },
        {
            "timestamp": timestamp2,
            "value": value2,
            "fee": fee2,
            "hash": hash2
        },

        ...

    ]
}

Block Header

A block header is a stripped-down version of a block, so it contains all the fields of the original block, except the transactions. The transactions are replaced by transaction headers, leading to block headers of the form:

{
    "timestamp": timestamp,
    "height": height,
    "creator": creator,
    "reward": reward,
    "fees": fees,
    "nonce": nonce,
    "transaction_count": 1,
    "transaction_headers" : [
	{
            "timestamp" : timestamp1,
            "value" : value1,
            "fee" : fee1,
            "hash" : hash1,
	}
     ]
    "hash": hash,
    "prev_hash": prev_hash
}

So, when a full_node wants to reconstruct a block when validating and creating the block file, as well as when synchronizing with the network, it simply has to re-build the transactions and add them to the block header instead of the transaction headers. For that purpose, the following function has been defined:

def json_block_header_and_transactions_to_block(json_block_header: dict, json_block_transactions: dict):
    return {
        "timestamp": json_block_header["timestamp"],
        "height": json_block_header["height"],
        "creator": json_block_header["creator"],
        "reward": json_block_header["reward"],
        "fees": json_block_header["fees"],
        "nonce": json_block_header["nonce"],
        "transaction_count": json_block_header["transaction_count"],
        "transactions": json_block_transactions,
        "hash": json_block_header["hash"],
        "prev_hash": json_block_header["prev_hash"]
    }

Synchronizing Blocks

As we mentioned before, blocks and transactions need to be retrieved differently now, as headers are returned in their place. For example, take a look at a portion of the full node synchronization code:

# retrieve block header
json_block_header, status_code = general_retrieve_block_header(
    settings, json_node, height)

transaction_count = json_block_header["transaction_count"]

# retrieve transactions one-by-one
json_block_transactions = []
for tid in range(0, transaction_count):
    json_transaction, status_code = general_retrieve_block_transaction(
        settings, json_node, height, tid)

    json_block_transactions.append(json_transaction)

# construct block
json_block = json_block_header_and_transactions_to_block(
    json_block_header, json_block_transactions)

# create block file
json.dump(obj=json_block, fp=open(
    settings.block_file_path + str(json_block["height"]) + ".json", "w"))

First the block header is retrieved, and afterwards each transaction one-by-one. The two structures are then "combined" in order to get the complete block, and that block is stored to a local file.

Of course, the local endpoint can no longer be called, as the unconfirmed transactions are now needed. So, updating the utxo and updating the blockchain info, takes place exactly after the block has been created, and before retrieving the next one!

And after all blocks have been retrieved, all the unconfirmed transactions also have to be retrieved one-by-one, in order to finish synchronizing:

# retrieve unconfirmed transactions header
json_transactions_header, status_code = general_retrieve_transactions_header(
    settings, json_node)

transaction_count = json_transactions_header["transaction_count"]

# retrieve unconfirmed transactions one-by-one
json_transactions = []
for tid in range(0, transaction_count):
    json_transaction, status_code = general_retrieve_transaction(
        settings, json_node, tid)

    json_transactions.append(json_transaction)

# update local file
json.dump(obj=json_transactions, fp=open(
    settings.transactions_path, "w"))

Basic Miner

The miner retrieves a random known node (from a new endpoint for that purpose), and then start retrieving the necessary information for mining:

# retrieve blockchain info
json_blockchain_info, status_code = general_retrieve_blockchain_info(
    settings, json_node)

# retrieve unconfirmed transactions
json_transactions = retrieve_unconfirmed_transactions(
    settings, json_node)

# retrieve last block header
json_last_block_header, status_code = general_retrieve_last_block_header(
    settings, json_node)

Depending on whether it's the first block or not (status code will be 400 if the last block doesn't exist) some of the headers can now be setup:

# if not first block
if status_code == 200:
    block.height = json_last_block_header["height"] + 1
    block.prev_hash = json_last_block_header["hash"]
    block.reward = json_last_block_header["reward"]
else:
    block.reward = json_blockchain_info["block_reward"]

# headers
block.creator = settings.reward_address
block.transaction_count = len(json_transactions) + 1
block.fees = 0
try:
    if len(json_transactions) >= 0:
        for json_transaction in json_transactions:
            block.fees += json_transaction["fee"]
except:
    pass

After that, the reward transaction is created using a function for that purpose, and the transactions structure is created:

# reward transaction
reward_transaction = create_reward_transaction(
    settings, block.timestamp, block.reward, block.fees)

# add remaining transactions
block.transactions = []
block.transactions.append(reward_transaction)
for json_transaction in json_transactions:
    transaction = json_destruct_transaction(json_transaction)
    block.transactions.append(transaction)

And then comes the solving part. For now, the difficulty is static with 5 hexadecimal 0s (or twenty 0's in bit format). Solving is pretty straight-forward:

while True:
for nonce in range(0, 1 << 32):
    block.nonce = "{:08x}".format(nonce)

    if (nonce != 0) and (nonce % (1 << 20) == 0):
        print("Tried", nonce, "nonces...")

    # calculate hash
    calculate_block_hash(block)

    # check if smaller then target_hash
    if int(block.hash, 16) < int(target_hash, 16):
        return

try various nonces until the hash is less then the target hash.

If no nonce in that range works, then the timestamp is reset, which also leads to a recalculation of the reward transaction hash:

block.timestamp = int(time.time())

reward_transaction: Transaction = block.transactions[0]
reward_transaction.timestamp = block.timestamp
calculate_transaction_hash(reward_transaction)
block.transactions[0] = reward_transaction

and the previous procedure is repeated...

When the solution is found, then the json block and corresponding json block header is constructed and posted to all known full nodes:

# construct JSON block
json_block = json_construct_block(block)

# construct JSON block header
json_block_header = json_block_to_block_header(json_block)

# send block to all known nodes
json_nodes, status_code = local_retrieve_nodes(settings)

for json_node in json_nodes:
    json_ret, status_code = general_create_block(
        settings, json_node, json_block_header)

RESOURCES:

References

Images

https://pixabay.com/illustrations/blockchain-block-chain-technology-3019121/

The rest is screenshots or made using draw.io

Previous dev blogs of the series

Final words | Next up

And this is actually it for today's post!

Next week, I will give you a complete example run of the "finished" project. After that expect dev blogs about this project to come out more rarely!

Keep on drifting!

Wonderful sunset