![witness update log 26.png](https://files.peakd.com/file/peakd-hive/techcoderx/23tbRUi7ywgRx8MD1wQqtcboCczhMZoEqwm1gpigBXLUJyJeP8itHXpfvBSsMgT1bWkS8.png) After 6 days of [cleaning the server](https://files.peakd.com/file/peakd-hive/techcoderx/EogPCzKUkkZhsTWrrNHT8rsbNLifAGtkRtLFRCecLEJoYAVWBGM9Guxc2zj18ccTU7q.JPG), moving files and replaying, techcoderx.com is now running on HAF enabled hived v1.26 along with HAfAH for account history APIs. Both are used to sync Hivemind on the latest stable release which are currently not HAF-based. ![Screenshot 2022-07-29 at 10.10.38 AM.png](https://files.peakd.com/file/peakd-hive/techcoderx/23tkmusiEFaCdpVjEQSyfw1iT54iJNFp3WciXP52CGghkNnpPynzoZpsb69fyfaqvJ6ZA.png) ^{`sql_serializer` plugin in action} Many have probably asked about how to run a node like this. As I'm writing this, the docs for them are all over the place and that makes it hard to follow along. So I will be documenting the whole process I went through here for anyone to follow. Certain parts of the guide may change over time as it isn't a final release yet, but most of this should be valid for the next 7 days from the publication of this post (most Hive posts are rarely viewed past the post payout anyways). # Hardware requirements * At least 6 core CPU, preferably scores >1200 single-core and >6000 multi-core on Geekbench 5 * 32GB RAM (64GB works better due to ZFS caching) * 2x 2TB NVMe SSD in RAID 0 for Postgres DB. I will explain why later. * 1TB SATA/NVMe SSD as boot drive and to store block_log file * No GPU is required * 20Mbps bandwidth. More is required for high traffic node. If running on residential internet connection behind CGNAT, a nearby VPS is required for a public node. Anything $5/month or less should work as long as it has sufficient bandwidth. # Dependency setup For this guide, we will be using Ubuntu Server 22.04 LTS which is what I used on techcoderx.com node. #### Essentials First of all, install some essential packages (including ZFS): ``` sudo apt-get update && sudo apt-get upgrade sudo apt-get install wget curl iotop iftop htop jq zfsutils-linux ``` #### Python 3.8 As it comes with Python 3.10 by default, if you wish to run Hivemind on it you would need to install Python 3.8 until it gets updated to support the latest Python version. You may find the original tutorial [here](https://www.linuxcapable.com/how-to-install-python-3-8-on-ubuntu-22-04-lts/), but to summarize it in a few commands: ``` sudo apt-get install software-properties-common -y sudo add-apt-repository ppa:deadsnakes/ppa -y sudo apt-get update sudo apt install -y \ python3.8 \ python3.8-dev \ python3.8-venv \ python3.8-distutils \ python3.8-lib2to3 \ python3.8-gdbm wget https://bootstrap.pypa.io/get-pip.py python3.8 get-pip.py python3.8 -m pip install --upgrade pip ``` You may also need to replace every instance of `python3` commands in this guide with `python3.8` to use the above installation. #### hived and Postgresql ``` sudo apt-get install -y \ autoconf \ automake \ cmake \ g++ \ git \ zlib1g-dev \ libbz2-dev \ libsnappy-dev \ libssl-dev \ libtool \ make \ pkg-config \ doxygen \ libncurses5-dev \ libreadline-dev \ perl \ python3 \ python3-jinja2 \ libboost-all-dev \ libpqxx-dev \ postgresql \ postgresql-server-dev-14 ``` # ZFS pool setup Running the node without compression will fill up the drives very quickly and you will need a lot more NVMe storage, so it is much more practical to place the postgresql database in a ZFS pool with lz4 compression enabled. Ideally ZFS pools would consist of at least 2 drives, hence the dual NVMe requirement listed above. ZFS will also handle striping the drives to mimic a RAID 0 setup. A single-drive ZFS pool is totally possible and it is currently what I'm using, but the performance of a single PCIe 3.0 NVMe SSD isn't enough to sync both hived and Hivemind at reasonable speeds (but can still catch up with the network). PCIe Gen 4 might work better, but I would say it is more cost effective over time to go for a RAID 0 setup. If you only have one 2TB NVMe SSD like me, you can probably get away by using a very recent Hivemind dump (no more than 1 month old) for the next 6-12 months. I have used [this tutorial](https://blog.programster.org/zfs-create-disk-pools) to set this up, but to summarize in a few commands: #### List all disk identifiers To create any ZFS pools, you need to know the identifiers for the drives to be used first. ``` $ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS loop0 7:0 0 62M 1 loop /snap/core20/1581 loop1 7:1 0 62M 1 loop /snap/core20/1587 loop2 7:2 0 79.9M 1 loop /snap/lxd/22923 loop3 7:3 0 44.7M 1 loop /snap/snapd/15534 loop4 7:4 0 47M 1 loop /snap/snapd/16292 nvme0n1 259:0 0 465.8G 0 disk ├─nvme0n1p1 259:1 0 1G 0 part /boot/efi └─nvme0n1p2 259:2 0 464.7G 0 part / nvme1n1 259:3 0 1.9T 0 disk ├─nvme1n1p1 259:4 0 1.9T 0 part └─nvme1n1p9 259:5 0 8M 0 part ``` In my example above, I'm using `/dev/nvme1n1` for my single drive ZFS pool. Yours may vary, take note of the identifiers of each drive to be used in the pool. #### Create pool Command: ``` sudo zpool create -f ... ``` Example for single-drive pool: ``` sudo zpool create -f haf /dev/nvme1n1 ``` Example for two-drive pool: ``` sudo zpool create -f haf /dev/nvme1n1 /dev/nvme2n1 ``` #### Get pool status ``` sudo zpool status ``` #### Enable compression ``` sudo zfs set compression=on ``` #### Set ARC RAM usage limit Only needed for 32GB RAM systems. In this command, we are setting it to 8GB. ``` echo "options zfs zfs_arc_max=8589934592" | sudo tee -a /etc/modprobe.d/zfs.conf ``` You should now have the pool mounted at `/`. For more useful commands, refer to my [hiverpc-resources](https://github.com/techcoderx/hiverpc-resources/blob/main/zfs.md) notes on ZFS. # Postgresql setup By default, the postgresql database is stored in the boot drive. We want it to be in the ZFS pool created previously. Create the required directory with appropriate permissions (assuming `haf` is the pool name): ``` cd /haf mkdir -p /db/14/main sudo chown -R postgres:postgres db sudo chmod -R 700 db ``` Set the data directory in `/etc/postgresql/14/main/postgresql.conf`: ``` data_directory = '/haf/db/14/main' ``` And tune the database as such in the same config file (below values are for 32GB RAM, scale according to what you have): ``` effective_cache_size = 24GB shared_buffers = 6GB work_mem = 512MB synchronous_commit = off checkpoint_timeout = 20min max_wal_size = 3GB ``` Restart postgresql to reflect the changes. ``` sudo service postgresql restart ``` # Compile hived ``` git clone https://gitlab.syncad.com/hive/haf cd haf git submodule update --init --recursive cd hive git checkout cd .. mkdir build cd build cmake -DCMAKE_BUILD_TYPE=Release -DCOLLECT_ACCOUNT_METADATA=ON .. make -j$(nproc) sudo make install ``` `sudo make install` installs the required extensions for `sql_serializer` plugin to work with postgresql. To install `hived` and `cli_wallet`, move the executables found in `hive/programs/hived/hived` and `hive/programs/cli_wallet/cli_wallet` respectively to `/usr/bin`. # Setup HAF db Enter the `psql` shell as `postgres` user: ``` sudo -i -u postgres psql ``` Create the db with permissions and extensions: ``` # Roles CREATE ROLE hived_group WITH NOLOGIN; CREATE ROLE hive_applications_group WITH NOLOGIN; CREATE ROLE hived LOGIN PASSWORD 'hivedpass' INHERIT IN ROLE hived_group; CREATE ROLE application LOGIN PASSWORD 'applicationpass' INHERIT IN ROLE hive_applications_group; CREATE ROLE haf_app_admin WITH LOGIN CREATEROLE INHERIT IN ROLE hive_applications_group; # Set password if needed ALTER ROLE role_name WITH PASSWORD 'rolepass'; # Database used by hived CREATE DATABASE block_log; # Use database \c block_log # sql_serializer plugin on db CREATE EXTENSION hive_fork_manager CASCADE; ``` # Block Log As of writing this, there isn't a compressed `block_log` file to download, so if not already you can download an up-to-date uncompressed `block_log`: ``` wget -c https://gtg.openhive.network/get/blockchain/block_log ``` #### Compress block_log To generate a compressed `block_log`, use the `compress_block_log` tool found in `hive/programs/util/compress_block_log` in the build directory. ``` ./compress_block_log -j $(nproc) -i /src/uncompressed/blocklog/folder -o /dest/blocklog/folder ``` # Create ramdisk 98% of the time the replay is bottlenecked by disk I/O if the `shared_memory.bin` file is located on disk. Move it to ramdisk for replay purposes (hence the memory requirements above) and move it back to disk once done. Replace `25G` with the size of the ramdisk desired. ``` cd ~ mkdir ramdisk sudo mount -t tmpfs -o rw,size=25G tmpfs ~/ramdisk ``` # Configure hived Generate a hived config file if not already: ``` hived --dump-config ``` By default, the data directory is `~/.hived` but you may also store it in another directory if you wish. ``` mkdir -p ~/.hived/blockchain ``` Move the resulting compressed block_log and block_log.artifacts files to the `blockchain` folder of the data directory. Then edit the config.ini file in the data directory as follows (find the key and modify its value, add them if it is not there): #### Plugins ``` # Basic plugin = witness webserver p2p json_rpc database_api network_broadcast_api condenser_api block_api rc_api # Reputation plugin = reputation reputation_api # Get accounts/witness plugin = account_by_key account_by_key_api # Internal market plugin = market_history market_history_api # Transaction status plugin = transaction_status transaction_status_api # State snapshot plugin = state_snapshot # SQL serializer plugin = sql_serializer ``` #### SQL serializer ``` psql-url = dbname=block_log user=postgres password=pass hostaddr=127.0.0.1 port=5432 psql-index-threshold = 1000000 psql-operations-threads-number = 5 psql-transactions-threads-number = 2 psql-account-operations-threads-number = 2 psql-enable-account-operations-dump = true psql-force-open-inconsistent = false psql-livesync-threshold = 10000 ``` See [sql_serializer plugin documentation](https://gitlab.syncad.com/hive/haf/-/blob/develop/src/sql_serializer/README.md) on details of each parameter and configuring relevant filters if desired. #### Shared file directory Don't forget to move the shared_memory.bin file back to disk and update this value once done. ``` shared-file-dir = "/home//ramdisk" ``` #### Shared file size At least 24GB is needed (currently 20GB), this value will increase over time. ``` shared-file-size = 24G ``` #### Endpoints Configure the IP and port of the endpoints to be made available for seed and API calls. ``` p2p-endpoint = 0.0.0.0:2001 webserver-http-endpoint = 0.0.0.0:8091 ``` # Replay! Finally we are ready to replay the blockchain. Start a `screen` or `tmux` session and run the following: ``` hived --replay-blockchain ``` If your hived data directory is not the default `~/.hived`, specify it in the above command with `-d /path/to/data/dir`. For HAF nodes, this process is split into 5 stages: 1. Normal replay. `hived` reads the blockchain and rebuilds chain state. 2. Initial network sync, which fetches the latest blocks so far from the network. 3. Indexes and foreign keys creation. 4. Normal network sync to catch up after stage 3 above. 5. Live sync, where your node is fully synced. You should see something like the first screenshot of this post above in your logs at this point. This entire process took under 48 hours for me to replay 66.4M Hive blocks, ~5-10% faster than the previous version. # HAfAH For this guide, we will be using the Python version which worked well for me. #### Installation ``` git clone https://gitlab.syncad.com/hive/HAfAH hafah cd hafah python3 -m pip install --user -r requirements.txt ``` #### Setup required permissions Run in `psql` console in `block_log` database: ``` GRANT USAGE ON SCHEMA hafah_python TO haf_app_admin; GRANT SELECT ON ALL TABLES IN SCHEMA hafah_python TO haf_app_admin; GRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA hafah_python TO haf_app_admin; GRANT USAGE ON SCHEMA hive TO haf_app_admin; GRANT SELECT ON ALL TABLES IN SCHEMA hive TO haf_app_admin; ``` #### Install schemas For some reason, the `--recreate-schema` flag did not work for me, so I manually ran the pgsql script which worked for me. ``` psql -f queries/ah_schema_functions.pgsql block_log ``` #### Start HAfAH server Replace `` with the port you wish to use. I prefer to use the next one available, that is 8093. ``` python3 main.py -p postgresql://haf_app_admin:@127.0.0.1:5432/block_log -n ``` #### Test API call To check that it works, try out an API call to the HAfAH server. ``` curl -s --data '{"jsonrpc":"2.0", "method":"condenser_api.get_ops_in_block", "params":[60000000,false], "id":1}' http://localhost: | jq ``` # Hivemind I'm currently on the master branch which isn't HAF-based. It is likely that this section may be updated when a HAF-based release is out. #### Create hivemind db Run in `psql` shell: ``` CREATE DATABASE hive; \c hive CREATE EXTENSION IF NOT EXISTS intarray; ``` #### Installation ``` python3.8 -m pip install --upgrade pip setuptools wheel git clone --recurse-submodules https://gitlab.syncad.com/hive/hivemind.git cd hivemind python3.8 -m pip install --no-cache-dir --verbose --user . 2>&1 | tee pip_install.log ``` #### Configure At the end of the `~/.bashrc` file, define the env vars: ``` export DATABASE_URL=postgresql://:@localhost:5432/hive export HTTP_SERVER_PORT=8092 export STEEMD_URL='{"default":"http://hivedurl:8091","get_ops_in_block":"http://hafahurl:8093","enum_virtual_ops":"http://hafahurl:8093"}' ``` Notice that we have defined separate endpoints for certain methods that uses HAfAH in `STEEMD_URL` (which for some reason isn't renamed to something like `HIVED_URL` till now). #### Restore dump I might serve a dump file myself, will update here once it is available. ``` pg_restore -j $(nproc) -d hive hivemind.dump ``` #### Sync the db In `screen` or `tmux` session: ``` hive sync ``` In my single drive ZFS pool, the disk I/O is the bottleneck for 90% of the time and it took me over 2 hours to sync 6 days of blocks and left it running overnight to build the indexes. `hived` struggled a bit but still managed to not go out of sync. #### Start server In `screen` or `tmux` session: ``` hive server ``` #### Troubleshooting hive.db_head_state ![Screenshot 2022-07-29 at 4.01.53 PM.png](https://files.peakd.com/file/peakd-hive/techcoderx/23tw9kF8omDAYZVQaQFzbQ1WJfsJGA6Zan3VvKFs1wnhKdaMdLGGcWVGHFEz8dnT3U873.png) ![Screenshot 2022-07-29 at 3.54.38 PM.png](https://files.peakd.com/file/peakd-hive/techcoderx/23tv8HJp61Nf6k3cQ9yXcNxenAisRuv7VvRskkgb5suxzV9yLBzG6HnENJoDhoy2BD11S.png) Open the file specified in the last line of traceback (above db_head_age), find that line in L55 and edit it as such: ``` db_head_age=int(time.time() - float(row['ts']))) ``` Restart `hive server` to reflect changes. # Reverse proxy At this point, you should have the required pieces to serve a public Hive node. To put them together into a single endpoint, we need a reverse proxy such as [jussi](https://gitlab.syncad.com/hive/jussi) or [rpc-proxy](https://github.com/ecency/rpc-proxy) (please do not use the one in my hiverpc-resources repo, it is only there if either doesn't work for me). If your node is on residential internet behind CGNAT, setup the VPN and forward the ports (hived, HAfAH, hivemind server) [here](https://whattheserver.com/openvpn-server-with-port-forwarding). For those who need a jussi config, you can find them [here](https://github.com/techcoderx/hiverpc-resources/blob/main/jussi_config.json). Replace `10.8.0.2` with the actual URL to the respective services. The easiest way to run jussi is through docker: ``` git clone https://gitlab.syncad.com/hive/jussi cd jussi docker build -t="$USER/jussi:$(git rev-parse --abbrev-ref HEAD)" . docker run -itp 8094:8080 -d --name jussi --env JUSSI_UPSTREAM_CONFIG_FILE=/app/config.json -v /home/username/jussi/jussi_config.json:/app/config.json "username/jussi:master" ``` Replace `/home/username/jussi/jussi_config.json` with path to jussi config file. At this point, you should now have a Hive API node accessible at port 8094 which can be put behind nginx and be accessible by the public. # Conclusion As of now, it is good to see that running such Hive node is totally possible with mainstream hardware (e.g. my barebones B350 motherboard) today. The disk requirements may look very demanding on its own, but it is nothing compared to the node requirements of popular blockchains such as [BSC](https://docs.bnbchain.org/docs/validator/fullnode#suggested-requirements) or [SQLana](https://docs.solana.com/running-validator/validator-reqs). Even though we now store the data as compressed format, the actual disk usage after compression remains the same at 1.9TB total compared to the previous node setup. Details below. As for the RAM usage, `hived` have consumed less memory than previously as seen in the smaller size of shared_memory.bin file that is 2GB less. # Server resource statistics #### hived (v1.25, all plugins, as of block 66365717) `block_log` file size: 620 GB `shared_memory.bin` file size: 22 GB `account-history-rocksdb-storage` folder size: 755 GB `blockchain` folder size: 1,398 GB #### hived (v1.26, d821f1aa, all plugins) `block_log` file size (compressed): 317 GB `block_log.artifacts` file size: 1.5 GB `shared_memory.bin` file size: 20 GB `blockchain` folder size: 338 GB #### HAF db Output of `SELECT pg_size_pretty( pg_database_size('block_log') );` Database size: 2.9 TB Compressed: 1.2 TB #### hivemind (v1.25.3, a68d8dd4) Output of `SELECT pg_size_pretty( pg_database_size('hive') );` Database size: 528 GB Compressed: 204 GB RAM usage: 4.2 GB

[![Hive witness footer 2.png](https://ipfs.io/ipfs/QmVTNReuWzvLtY5sucfunQ2m9JpLWroYrXrcztpVsZq3sG)](https://hivesigner.com/sign/account-witness-vote?witness=techcoderx&approve=1)