Abstract
This document explains few terms like the SON heartbeat and gives sample logs in few common scenarios. This is expected to handy for operators to monitor their network.
.
The Graphene based blockchains (Bitshares, @Peerplays, Hive, EOS, Telos etc) doesn't have a decentralized mechanism connect to other blockchains. While chains like Polkadot is heavily focused on cross chain functionality from ground up and reaping rewards. The Sidechain technology created by Peerplays intends provide a solution for this particular problem. SONs aka Sidechain Operator Nodes provides decentralized, elected, DPoS like solution to address this problem.
The solution involves:
- Peerplays Blockchain
- Hive Blockchain
- Hive Public API nodes
- Sidechain Operator Nodes on the Peerplays chain
- A multisignature Account on the Hive blockchain
- A multisignature Account on the Peerplays Blockchain
How does it work ?
The Nodes
The Sidechain Operator nodes are similar to the witnesses and they run Peerplays blockchain software with sidechain_plugin enabled. We can elect these 15 nodes using the stake weighted voting & the nodes are rewarded for performing the cross chain transactions. The reward is propotional to the number of transactions each node processes on a daily (24 hour time period.)
This makes it extremly important for these SONs to be available all the time, communicte with each other over a reliable channel and have fast connectivity with the Hive blockchain.
The SONs performs periodic heartbeats using the Peerplays blockchain as an event bus to commuicate. This provides a reliable communication channel to maintain the network status.
list_sons command can be used from the cli_wallet to get their details.
unlocked >>> list_sons 0 100
list_sons 0 100
[[
"sonaccount01",
"1.33.0"
],[
"sonaccount02",
"1.33.1"
],[
"sonaccount03",
"1.33.2"
],[
"sonaccount04",
"1.33.3"
],[
"sonaccount05",
"1.33.4"
],[
"sonaccount06",
"1.33.5"
],[
"sonaccount07",
"1.33.6"
]
]
Example:
In this example, there are 7 SONs running on a TESTNET and connected to the Hive MAINNET
They are performing heartbeat in the following diagram.
Sample logs for SON heartbeat:
Oct 30 13:43:04 son-hive witness_node[27993]: 2584068ms th_a peerplays_sidechain_plugin.cpp:411 heartbeat_loop ] Sending heartbeat for SON 1.33.6
Oct 30 13:45:04 son-hive witness_node[27993]: 2704063ms th_a peerplays_sidechain_plugin.cpp:411 heartbeat_loop ] Sending heartbeat for SON 1.33.0
Oct 30 13:45:04 son-hive witness_node[27993]: 2704064ms th_a peerplays_sidechain_plugin.cpp:411 heartbeat_loop ] Sending heartbeat for SON 1.33.1
Oct 30 13:45:04 son-hive witness_node[27993]: 2704065ms th_a peerplays_sidechain_plugin.cpp:411 heartbeat_loop ] Sending heartbeat for SON 1.33.2
Oct 30 13:45:04 son-hive witness_node[27993]: 2704066ms th_a peerplays_sidechain_plugin.cpp:411 heartbeat_loop ] Sending heartbeat for SON 1.33.3
Oct 30 13:45:04 son-hive witness_node[27993]: 2704067ms th_a peerplays_sidechain_plugin.cpp:411 heartbeat_loop ] Sending heartbeat for SON 1.33.4
Oct 30 13:45:04 son-hive witness_node[27993]: 2704067ms th_a peerplays_sidechain_plugin.cpp:411 heartbeat_loop ] Sending heartbeat for SON 1.33.5
Oct 30 13:45:04 son-hive witness_node[27993]: 2704068ms th_a peerplays_sidechain_plugin.cpp:411 heartbeat_loop ] Sending heartbeat for SON 1.33.6
The Node configuration
Each node uses a configuration file very similar or same as the Hive or Peerplays blockchains, they have the account names (ids to be precise) & Public private key pairs of both Peerplays account and a corresponding Hive account.
hive-private-key = ["son1-hive","xxxxxxxxxxxx"]
# IDs of multiple SONs controlled by this node (e.g. ["1.33.5", "1.33.6"], quotes are required)
# SON id of son1-ppy
son-ids = ["1.33.0"]
# Hive node IP address
hive-node-ip = https://api.hive.blog
# Hive node RPC port
hive-node-rpc-port = 443
These private keys are used to create weighted multi sigunature wallet and collectively control the funds (on respective blockchains)
The cross chain magic
Since the cross chain communication is a financial transaction, its highly important that the SONs receives the relvant transactions as soon as they happen on the Hive blockchain. At this point the public nodes are the cheapest mechanism without needing additional means. Once we have a plugin to stream the chain similar to the sql_serializer that @blocktrades has mentioned, perhaps we can get faster performance.
The Hive - Peerplays Sidechain listener logic is outlined in : @bobinson/notes-hive-public-api-node-performance
Example:
In the following example, a SON is making query to the Hive API to find the details of the chain. (This example is shown as opposed to calling a block as this is smaller)
sample logs of database_api.get_dynamic_global_properties query:
Oct 30 13:19:30 son-hive witness_node[27993]: 1170510ms th_a peerplays_sidechain_plugin.cpp:465 son_processing ] Scheduled SON: 1.33.1 Now: 2021-10-30T16:19:27
Oct 30 13:19:31 son-hive witness_node[27993]: 1171292ms th_a rpc_client.cpp:476 send_post_request ] ### Request URL: https://51.79.10.214:443
Oct 30 13:19:31 son-hive witness_node[27993]: 1171292ms th_a rpc_client.cpp:477 send_post_request ] ### Request: { "jsonrpc": "2.0", "id": 115755, "method": "database_api.get_dynamic_global_properties" }
Oct 30 13:19:31 son-hive witness_node[27993]: 1171292ms th_a rpc_client.cpp:479 send_post_request ] ### Response: {"jsonrpc":"2.0","result":...
sample logs of fetching the latest block:
Oct 30 15:02:49 son-hive witness_node[27993]: 169167ms th_a rpc_client.cpp:477 send_post_request ] ### Request: { "jsonrpc": "2.0", "id": 124015, "method": "block_api.get_block", "params": { "block_num": 58744462 } }
Deposit operation
The process of sending HIVE or HBD from a Hive account to a Peerplays account is defined as deposit or cross chain deposit type transfer.
One of the many active SONs detects an incoming transaction. The following snippet is belong a transaction from @bobinson on Hive to alice-ppy. This is received by all the seven active nodes but not confirmed by anyone (SON).
240 Oct 31 09:35:42 son-hive witness_node[4103]: 2142404ms th_a sidechain_net_handler.cpp:155 sidechain_event_data ] sidechain_event_data:
241 Oct 31 09:35:42 son-hive witness_node[4103]: 2142404ms th_a sidechain_net_handler.cpp:156 sidechain_event_data ] timestamp: 2021-10-31T12:35:42
242 Oct 31 09:35:42 son-hive witness_node[4103]: 2142404ms th_a sidechain_net_handler.cpp:157 sidechain_event_data ] block_num: 13484
243 Oct 31 09:35:42 son-hive witness_node[4103]: 2142404ms th_a sidechain_net_handler.cpp:158 sidechain_event_data ] sidechain: hive
244 Oct 31 09:35:42 son-hive witness_node[4103]: 2142404ms th_a sidechain_net_handler.cpp:159 sidechain_event_data ] sidechain_uid: hive-c189bb3a979e0bd03cb95bf951db1206ffb4cf90-0
245 Oct 31 09:35:42 son-hive witness_node[4103]: 2142404ms th_a sidechain_net_handler.cpp:160 sidechain_event_data ] sidechain_transaction_id: c189bb3a979e0bd03cb95bf951db1206ffb4cf90
246 Oct 31 09:35:42 son-hive witness_node[4103]: 2142404ms th_a sidechain_net_handler.cpp:161 sidechain_event_data ] sidechain_from: alice-ppy
247 Oct 31 09:35:42 son-hive witness_node[4103]: 2142404ms th_a sidechain_net_handler.cpp:162 sidechain_event_data ] sidechain_to: son-account
248 Oct 31 09:35:42 son-hive witness_node[4103]: 2142404ms th_a sidechain_net_handler.cpp:163 sidechain_event_data ] sidechain_currency: HIVE
249 Oct 31 09:35:42 son-hive witness_node[4103]: 2142404ms th_a sidechain_net_handler.cpp:164 sidechain_event_data ] sidechain_amount: 1
250 Oct 31 09:35:42 son-hive witness_node[4103]: 2142404ms th_a sidechain_net_handler.cpp:165 sidechain_event_data ] peerplays_from: 1.2.27
251 Oct 31 09:35:42 son-hive witness_node[4103]: 2142404ms th_a sidechain_net_handler.cpp:166 sidechain_event_data ] peerplays_to: 1.2.19
252 Oct 31 09:35:42 son-hive witness_node[4103]: 2142405ms th_a sidechain_net_handler.cpp:167 sidechain_event_data ] peerplays_asset: {"amount":40,"asset_id":"1.3.0"}
253 Oct 31 09:35:42 son-hive witness_node[4103]: 2142518ms th_a peerplays_sidechain_plugin.cpp:465 son_processing ] Scheduled SON: 1.33.6 Now: 2021-10-31T12:35:39
254 Oct 31 09:35:42 son-hive witness_node[4103]: 2142518ms th_a sidechain_net_handler.cpp:431 operator() ] Deposit to process: {"id":"1.36.3","timestamp":"2021-10-31T12:35:42","block_num" :13484,"sidechain":"hive","sidechain_uid":"hive-c189bb3a979e0bd03cb95bf951db1206ffb4cf90-0","sidechain_transaction_id":"c189bb3a979e0bd03cb95bf951db1206ffb4cf90","sidechain_from":"alice-ppy","sidecha in_to":"son-account","sidechain_currency":"HIVE","sidechain_amount":1,"peerplays_from":"1.2.27","peerplays_to":"1.2.19","peerplays_asset":{"amount":40,"asset_id":"1.3.0"},"expected_reports":[["1.33.0 ",1],["1.33.1",1],["1.33.2",1],["1.33.3",1],["1.33.4",1],["1.33.5",1],["1.33.6",1]],"received_reports":["1.33.0","1.33.1","1.33.2","1.33.3","1.33.4","1.33.5","1.33.6"],"confirmed":true,"processed":fa lse}
The following log snipped shows sonaccount04 ( with id :1.33.1) detecting a transaction 1.39.5
Note the errors as the account_history_api.get_transaction fails for many of the calls. (This is not a cause for concern, but this is where a lot of the delay is happening in the cross chain transactions)
92 Oct 31 09:35:24 son-hive witness_node[4103]: 2124501ms th_a peerplays_sidechain_plugin.cpp:465 son_processing ] Scheduled SON: 1.33.1 Now: 2021-10-31T12:35:21
Oct 31 09:35:24 son-hive witness_node[4103]: 2124501ms th_a sidechain_net_handler.cpp:567 operator() ] Sidechain transaction to settle: 1.39.5
94 Oct 31 09:35:24 son-hive witness_node[4103]: 2124659ms th_a rpc_client.cpp:476 send_post_request ] ### Request URL: https://51.79.10.214:443
95 Oct 31 09:35:24 son-hive witness_node[4103]: 2124659ms th_a rpc_client.cpp:477 send_post_request ] ### Request: { "jsonrpc": "2.0", "id": 53409, "method": "account_history_ api.get_transaction", "params": { "id": "94a9469d80962462a9517d48f66f583ae1cbeca9" } }
96 Oct 31 09:35:24 son-hive witness_node[4103]: 2124659ms th_a rpc_client.cpp:479 send_post_request ] ### Response: {"jsonrpc":"2.0","error":{"code":-32003,"message":"Assert Ex ception:false: Unknown Transaction 94a9469d80962462a9517d48f66f583ae1cbeca9","data":{"code":10,"name":"assert_exception","message":"Assert Exception","stack":[{"context":{"level":"error","file":"acco unt_history_api.cpp","line":250,"method":"get_transaction","hostname":"","timestamp":"2021-10-31T12:35:24"},"format":"false: Unknown Transaction ${t}","data":{"t":"94a9469d80962462a9517d48f66f583ae1c beca9"}}]}},"id":53409}
Finally a SON gets a block detail with relevant transaction, ie a transaction with son-account as the recipient
561 Oct 31 09:36:27 son-hive witness_node[4103]: 2187501ms th_a peerplays_sidechain_plugin.cpp:465 son_processing ] Scheduled SON: 1.33.5 Now: 2021-10-31T12:36:24
562 Oct 31 09:36:27 son-hive witness_node[4103]: 2187655ms th_a rpc_client.cpp:476 send_post_request ] ### Request URL: https://51.79.10.214:443
563 Oct 31 09:36:27 son-hive witness_node[4103]: 2187656ms th_a rpc_client.cpp:477 send_post_request ] ### Request: { "jsonrpc": "2.0", "id": 53514, "method": "account_history_ api.get_transaction", "params": { "id": "c189bb3a979e0bd03cb95bf951db1206ffb4cf90" } }
564 Oct 31 09:36:27 son-hive witness_node[4103]: 2187656ms th_a rpc_client.cpp:479 send_post_request ] ### Response: {"jsonrpc":"2.0","result":{"ref_block_num":46401,"ref_block_ prefix":701377099,"expiration":"2021-10-31T12:45:36","operations":[{"type":"transfer_operation","value":{"from":"bobinson","to":"son-account","amount":{"amount":"1","precision":3,"nai":"@@000000021"} ,"memo":"alice-ppy"}}],"extensions":[],"signatures":["20170afef1e7c9ad21e28e123bbd328927b6a8db1bf4e09333c3d2697b2d0ad2c775e55bb64c7c0da16df8dc405761b52fa30e2fc4b87c9a2b567fe8f56b08c311"],"transaction _id":"c189bb3a979e0bd03cb95bf951db1206ffb4cf90","block_num":58766676,"transaction_num":77},"id":53514}
Sidechain transaction to settle: 1.39.5 (bobinson - alice-ppy) is picked by multiple SON nodes and they all verify the transaction independently. Using different Hive API nodes is a good idea.
Oct 31 09:36:42 son-hive witness_node[4103]: 2202547ms th_a peerplays_sidechain_plugin.cpp:465 son_processing ] Scheduled SON: 1.33.1 Now: 2021-10-31T12:36:39
699 Oct 31 09:36:42 son-hive witness_node[4103]: 2202547ms th_a sidechain_net_handler.cpp:567 operator() ] Sidechain transaction to settle: 1.39.5
Once all the active SON nodes verifies the transaction independently verifies the transaction, it is settled.
log to be added.
block_api.get_block API call is used to fetch the latest, blocks and processing is started to ensure that the SONs complete required evaluations by the time the transaction is finalised on the Hive Blockchain. A method of looking for the LIB can also be done but it will be much slower. Once the processing is completed, relevant deposit transfers are processed and the HIVE/HBD is locked away in a multi signature wallet on the Hive Blockchain and the receipient on the Peerplays blockchain is issues equivalent HIVE/HBD.
The Multisignature wallet and example transfers will be shared in another post.
Special thanks to @hiveio & @spknetwork to make this happen