Vidulum Validator Tweak Guide

Hello, fellow validators! I am sharing this guide in hopes you find it helpful and that you also see the same benefits that I have been seeing after I've made the changes that I have listed below.

I recommend that if you do not know what these steps are doing or are unsure about what the configuration setting does, google it and decide if it still will fit your needs.

Advanced User guide!

Vidulum Daemon Tweaks

For most Validators, there are features that you can and probably will want to disable. These will free up resources and allow the daemon to perform better.

The changes that I will be guiding you through will be:

• Enable unix sockets where applicable
• Turn logging down
• Disable unused features
• Adjusting log level output and tx/rx rates
• Adding sysctl & rc.local tweaks
• Using BFQ Scheduling

In nano:

• Use ctrl+k and ctrl+u to cut and paste a copy of the original configuration setting.
• Change your configuration to match the examples for your setup.
• Save and Quit.

Important consideration

Because you are changing to unix sockets, if the daemon crashes unexpectedly, I have seen where the socket files are left behind.
This is something you will need to be aware of if you restart your daemon and you don't see it working. It could be that the socket files weren't deleted after the daemon halted.

Changes in config.toml

First, let's make sure we fix the way how most validators were originally configured. Get rid of your persistent-peers list.
You just need the seeds now.

seeds = "883ec7d5af7222c206674c20c997ccc5c242b38b@ec2-3-82-120-39.compute-1.amazonaws.com:26656,eed11fff15b1eca8016c6a0194d86e4a60a65f9b@apollo.erialos.me:26656"

This will help you better in finding the right peers to connect to. DO NOT PUT THE SEEDS IN YOUR PERSISTENT-PEERS!

Convert to Unix sockets

Changing what we can that is used by the daemon from a tcp to a unix socket will improve performance AND provides a more secure way to reverse proxy the rpc ports.

Below you will see the example configurations. I have made a copy and commented out the original section and replaced it with the unix://... socket. You will want to make the same changes according to your path/configuration.

#######################################################################
###                   Main Base Config Options                      ###
#######################################################################

# TCP or UNIX socket address of the ABCI application,
# or the name of an ABCI application compiled in with the Tendermint binary
proxy_app = "unix:///home/vidulum/.vidulum/abci.socket"
#proxy_app = "tcp://127.0.0.1:26658"
...
....

AND...

..
...
#######################################################
###       RPC Server Configuration Options          ###
#######################################################
[rpc]

# TCP or UNIX socket address for the RPC server to listen on
laddr = "unix:///home/vidulum/.vidulum/trpc.socket"
#laddr = "tcp://127.0.0.1:26657"
...
..

optionally you can also increase your max send/receive and tune logging

..
...

# Output level for logging, including package level options
log_level = "warn"
...
..

..
...
# Rate at which packets can be sent, in bytes/second
send_rate = 55120000

# Rate at which packets can be received, in bytes/second
recv_rate = 55120000
...
..

Centera posted an alternative method, but the idea is the same. Skip a few blocks to ensure you do not double sign! You will most likely notice that the daemon start's almost instantly after the unix socket changes. Corey suggested being careful, as it could cause a double sign. The following should prevent that within.

# How many blocks to look back to check the existence of the node's consensus votes before joining consensus
# When non-zero, the node will panic upon restart
# if the same consensus key was used to sign {double_sign_check_height} last blocks.
# So, validators should stop the state machine, wait for some blocks, and then restart the state machine to avoid panic.
double_sign_check_height = 1

Changes in client.toml

node = "unix:///home/vidulum/.vidulum/trpc.socket"
#node = "tcp://localhost:26657"

I also suggest that you set the chain-id and the keyring backend so you don't need to pass it as an argument each time as well!

Changes in app.toml

Find and disable unused features(set to false):

..
...
[gRPC]
...
..

..
...
[grpc-web]
...
..

the rest api and telemetry should already be disabled.

Sysctl Tweaks

Here are my current list of sysctl.conf settings that I have been running with for the last couple of months. Some increase the log sizes, some memory limits, I recommend that if you don't know what it is, to google it and see if it's right for your configuration. YMMV!!!

## My custom sysctl ##
vm.dirty_background_ratio = 5
vm.dirty_ratio = 10
vm.swappiness = 3
vm.dirty_expire_centisecs = 500
vm.dirty_writeback_centisecs = 500
fs.file-max = 10000
sysctl net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
net.core.somaxconn = 3000
net.core.netdev_max_backlog = 5000
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 0
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
net.ipv4.tcp_sack = 1
net.ipv4.tcp_low_latency = 1
net.ipv4.tcp_adv_win_scale = 1

I scoured the web to find articles that were recent that spoke about improving performance, specifically storage drive and network.

If you find that changing these to some other value, or adding other sysctl features, please share below! #ValidatorFam!

I don't claim to be an expert on any of this, I just do what I consider enough research/RTFM and set a plan and apply it. I'm human and make mistakes. :D

rc.local Tweaks

Ubuntu doesn't come with it's rc.local enabled by default. I will leave that to you to figure out how to enable.

However, I do set a few things on every reboot. Most notably I hard set the read-ahead for all my drives.

Prior to copy and pasting this, you should run the command:

ethtool -g eth0

Use this information to appropriately set the values in your rc.local below!

#!/bin/bash
set -x
##Set read ahead on all block devices
## Change as needed, you may have more, or other named devices
blockdev --setra 4096 /dev/sda
blockdev --setra 4096 /dev/sdb
blockdev --setra 4096 /dev/dm-0

# Increase your transaction queue length
# and increase the eth device tx/rx buffers
# Does not work with linode para-virt, must use full-virt
ip link set dev eth0 txqueuelen 3000
ethtool -G eth0 tx 4096 rx 4096

#BFQ Tunables
#/sys/block/sd{a,b,c,d,e,f,g,h}/queue/iosched/
for i in a b c d e f g h
do
        blockdev --setra 4096 /dev/sd$i

        echo 0 > /sys/block/sd$i/queue/iosched/low_latency
        echo 0 > /sys/block/sd$i/queue/iosched/slice_idle
        echo 0 > /sys/block/sd$i/queue/iosched/timeout_sync
done

exit 0

Use the following command to test your drive speed. It will be best to not have the daemon running when you do this to get a better reading.

Run this a 2-3 times to get a good idea of the readings

hdparm -tT /dev/sda #change sda to your blockchain storage device

As you can see, there are several ways you can set sysctl values.
The other notable change here is increasing the transaction queue length of your ethernet device.

Update systemd service file

I've added Sockets=... to mine like:

[Unit]
Description=Vidulum Validator
After=network.target

[Service]
Group=vidulum
User=vidulum
WorkingDirectory=/home/vidulum
ExecStart=/home/vidulum/.local/bin/vidulumd start
Sockets=/home/vidulum/.vidulum/sockets/trpc.socket
Restart=on-failure
RestartSec=3
LimitNOFILE=10000

[Install]
WantedBy=multi-user.target

If need be, replace it with the proper locations for your configuration.

If you are exposing the rpc ports, nginx will need to be able to access them.

• Create a folder in .vidulum called run
• Set permissions chmod 2774 run/
• Set acl setfacl -d -m u:vidulum:rwX,g:www-data:rwX,o::- run/
• Add a entry in your service file ExecStartPost=/PATH/TO/SCRIPT
• Create a script that waits for the sockets to be created and gives them the proper permissions.

Example script:

#!/bin/bash
set -x

until [ -e /home/vidulum/.vidulum/run/trpc.socket -a -e /home/vidulum/.vidulum/run/restapi.socket ]
do
        sleep 5
done

sleep 3
chmod g+w /home/vidulum/.vidulum/run/*

exit 0

Change to BFQ Scheduling

See what scheduler your drives are currently using:

grep . /sys/block/sd*/queue/scheduler

Before updating, I recommend googling about BFQ Scheduling before implementing it. That probably should be said about everything in this guide! :D

Enable BFQ Scheduling:

modprobe  "bfq"
echo "bfq" > /sys/block/sda/queue/scheduler
echo "bfq" > /etc/modules-load.d/bfq.conf
echo 'ACTION=="add|change", KERNEL=="sd*[!0-9]|sr*", ATTR{queue/scheduler}="bfq"' > /etc/udev/rules.d/60-scheduler.rules

Edit grub, nano /etc/default/grub, and on the line GRUB_CMDLINE_LINUX, append this to the end:

scsi_mod.use_blk_mq=1

IE:
GRUB_CMDLINE_LINUX="quiet vt.global_cursor_default=0 scsi_mod.use_blk_mq=1"

Update grub:

update-grub

Reboot and make sure the changes worked since you just edited your boot loader.
Lastly, rerun and make sure it's enabled. [bfq]

grep . /sys/block/sd*/queue/scheduler

Example output:

/sys/block/sda/queue/scheduler:mq-deadline [bfq] none
/sys/block/sdb/queue/scheduler:mq-deadline [bfq] none
/sys/block/sdc/queue/scheduler:mq-deadline [bfq] none

Conclusion

Once you are done and have saved all your configuration settings, I found it helpful to start the daemon by 'hand' first before doing it through systemd/systemctl. If you do start it by hand, you will most likely need to remove the socket files created before attempting to start it from the systemd/systemctl method.

You should see a noticeable difference in how the daemon now starts. You should expect to see a better response from issuing cli commands to query the blockchain as well. Prior to me using unix sockets, there were times that my queries would lock up my validator and I'd miss a couple of blocks.

Next time, I plan on sharing how to go through and set up a Sentry network to protect your validator even more.

Staking or Donating

If you found this helpful and you are feeling generous, here are some addresses if you'd like to donate and as always, choosing to stake with Moneta:

• $VDL - vdl1zvalykc0hp3jxteasnz626djt724fxkyv73p6k
• $ATOM - cosmos1vwd80092nvgffy7zad0c508pj9ajmaqhznlhfd
• $OSMO - osmo1vwd80092nvgffy7zad0c508pj9ajmaqh2gv8ll
• Stake your Vidulum with Moneta!