The Smart Approach To Backup And Recovery - Post #36
Flowers that bear chocolate fruits……...
JUST A BRIEF PROFESSIONAL BACKGROUND
It may be best that you read my introductory post as well, link below.
As this is my offer of help to our respected witnesses that are tech savvies, it may be best if I will introduce myself first in the area of my profession. I am an Engineer for more than 20 years working on the Telecomms sector (first 3 years) and shifted gears towards the I.T. sector since 2001. My main forte is in Designing and Architecting systems. What differentiates me from my colleagues is my “insistence” to always try my designs first before I would even let any vendors touch it. A small I.T lab at the office is always my “initial” request in every company that I work with, and this has been my way professionally in helping companies achieve solutions as I hands-on with System solutions.
My present company is in the banking sector with nationwide presence in this part of the Earth as the Philippines, and also presently, the main System that I had the privilege of tinkering is IBM’s Power8 ppc64le machines.
It is my honest to goodness intent here at STEEMIT to just share in whatever good way I can, and I think I can give beneficially in the area of I.T Systems as far as the STEEMIT platform is concerned.
This post will be one way of showing my salute to our respected witnesses.
KNOW THE PROBLEM BEFORE SOLVING
Lately I have been very busy studying the STEEMIT environment and intricacies as a way of effectively moving forward. So far, one of the most important part of the STEEMIT platform that I find to be very crucial are the works and dedication of the witnesses. As per 
I will give my support in this post in the area of effectively solving the needs to “Backup and Recover If Needed” any running linux server, and when I say backup and recover, I mean the easiest, the safest and the fastest way possible. I can guarantee every witness that this approach even surpasses the performance of all the costly commercial backup systems currently advertised.
I designed and applied this approach myself as per the huge need (actually so so huge!) of the bank I am currently serving.
BACKUP EVERYTHING EVEN IF YOUR SERVER IS RUNNING PROD
One of the secrets of backing up (for the purpose of restoring it back at the fastest way possible) any linux server carrying critical services is that WE SHOULD BACKUP EVERYTHING EVEN IF THE SERVER IS RUNNING. The all-caps letters as emphasized can clearly tell us that this is exactly the backup method for a witness server because the STEEM blockchain is working 24 hours a day.
I am bombarded by the usual oppositions (as with any radical idea) when I presented my solution to the bank’s Management for a truly seamless backup method the opensource way. As per their description, “this may cause too much strain on the running server”. If only I have not proven this effective backup method to them at the lab, I myself could have gone over to their side very easily. As I write this, all of our servers are backed up in a state of RUNNING PRODUCTION, and not once did it fail us after more than three months in auto-action thru the cron jobs.
Without any further delay, here it is, the steps and hints for the tech-savvy witnesses to use and improve as well for BACKUP AND RECOVERY:
TO BACKUP EVERYTHING INCLUDING THE RUNNING OS
-1. Preliminary hints:
*Make sure that all your server instances (the host server) have the same disk partitioning layout and the same platform (64bit x86 for all nodes for example)
*Make sure to have a streamlined disk size for all server node instances; 50 Gb for all OS disk size as an example, no matter the purpose of the server (witness node, seed node etc)
*Be sure to separate the data partition from the root partition during installation; the data partition can be on a separate disk for example.
*Keep the root partition as small as possible, a 50 Gb disk is very much enough for the root (/), swap (/swap) and boot (/boot) partitions. With the root partition and the data partition separate, be sure that their corresponding backups are also on separate folders.
*The target server to be backed-up should be reachable thru ssh from the backup server.
-2. Ready a backup server on linux with at least 4Gb memory and lots of disk space for the backup files storage. This server will serve as the backup and recovery start point. This description alone will give us the idea that the said server can be our home desktop with lots of disk space, no need for elaborate and costly VPS servers as the backup server.
-3. When step #2 is finished, we now have a backup server (the linux desktop at your home maybe) and a target server (the witness and the seed servers for example). Issue this one liner at the linux terminal of the backup server to backup everything snapshot-style (even if the target server is running prod!!!):
"rsync -aAXv --progress --exclude={/dev/,/proc/,/sys/,/tmp/,/run/,/mnt/,/media/,/lost+found,ANY-UNIQUE-NAMED-DATA-FOLDER-HERE*} root@TARGET-SERVER-IP-HERE:/ /LOCAL-FOLDER-PATH-HERE"
--
*The rsync command above will “copy everything” from the target server’s filesystem and place it at the local folder of the backup server, even if the target server is running. Also, the rsync command can be added with a bandwidth limiter option (example: --bwlimit=1200) if you really want to be very sure that the target server will not experience any hiccups. Also, the exclude option should include in the list the DATA folder because we only need to backup the root (/) partition at this point that includes /swap and /boot but not including the folder where huge data is placed.
*The running Operating System of the target server will be copied whole like a snapshot, including all the logs and applications that are installed in it at the time of the backup. So if the target server’s approach to the witness node setup is thru docker, then the DOCKER program and all its needed filesystem will be included as well. The DATA folder should not be included, it should be on another disk to start with to make all backups and recoveries much easier and faster.
*There will be no noticeable and heavy memory and CPU usage on the target server during the backup; the backup server is the one that will have a heavier CPU and memory usage, no problem as the backup server is just your desktop computer at home for example.
*All files and folders as backed-up and placed at a folder locally can be “recovered” into any server instance, provided that their disks are formatted “exactly the same” and their platform is also the same (32bit for 32bit x86, 64bit for 64bit x86, ppc64le for ppc64le, etc).
*If a witness keeps the backup files this way (snapshot style), all that needs to be done is to keep a disk image of only one running server as the base recovery disk of the root filesystem partitioning (details in the next steps)
WE NEED A ROOT DISK IMAGE OF A WITNESS SERVER
-1. This can be done “immediately after” the OS install on the VPS; no need to wait for the final programs and the needed updates to be installed. All you need here at this point is the exact partitioning layout of the base disk image for all future recovery.
-2. Boot the witness server into a linux rescue CD; the UBUNTU installation iso is a perfect tool for recovery in this part of the backups. Another perfect tool for this is the SysRecCD utility which is intended only for rescue and recovery. Be sure to place the same public IP address, with sshd service up and can be logged in as the root user as per the booted recovery CD.
-3. Once step #2 done, be sure to know the disk where the OS is installed (if there are multiple disks), otherwise if only one disk then the disk is surely designated as “/dev/sda”.
-4. From the backup server (your desktop at home perhaps), open the linux terminal and type this command to produce a disk image of the target witness server VPS instance:
"ssh root@TARGET-SERVER-IP-HERE “dd if=/dev/sda bs=12M” | dd of=/LOCAL-FOLDER-HERE/IMAGE-FILENAME-HERE.img"
--
The command above creates a disk image as executed on the remote target server using the “dd” utility; however, the disk image file (with the extension .img) that is produced is “transferred over the network” so that it is written at the local disk of the backup server and not on the disk of the remote target server.
TO DO A VERY FAST AND SECURE RECOVERY
Once the snapshot-style backup files and the disk image file is created, you as a witness can recover any witness node very fast and securely as connections will be encrypted over SSH. Just do these steps:
-1. Let us say your witness server’s disk is “damaged” beyond repair and can not boot by itself anymore, so a replacement and a new disk is installed by the VPS provider. First step for a full recovery is to boot the said VPS into a recovery linux CD (UBUNTU for example). Be sure public IP address is configured together with sshd service and networking.
-2. From your backup server where both the .img disk image file and the snapshot-style backup files are, issue the following command to recover the exact root partition files:
"dd if=/LOCAL-FOLDER-HERE/IMAGE-FILENAME-HERE.img bs=12M | ssh root@TARGET-SERVER-IP-HERE “dd of=/dev/sda”"
--
The command above will clone (copy exactly the same) the .img disk image as previously backed up; this is to have a uniform partitioning at any target server for recovery. UNIFORMITY will always make recovery much faster, hence the basic hints earlier.
-3. Once #2 is finished, go over to the target remote server thru ssh as root and mount the root partition. If the partition is logical, make sure to “lvscan” first and mount the logical root partition if it is ACTIVE already, if not, make it active first before mounting.
-4. ERASE all files on the remote server’s root partition as mounted; don’t worry as this is not your intended root files, all you need from the disk image .img file is the exact partitioning. After erasing the root files and folders, leave the root disk mounted for the next step.
-5. Copy the snapshot-style backup files as previously backed-up, the direction of sync is from your local desktop (the backup server) to the remote target server; issue the command below to do this:
"rsync -aAXv --progress /LOCAL-SNAPSHOT-BACKUP-FOLDER-HERE/ root@TARGET-SERVER-IP-HERE:/MOUNTED-ROOT-FOLDER-HERE"
--
The command above will copy all files and folders (exactly the same like a snapshot) as backed up previously into the new and recovered instance of the target server. Keep in mind that you “erased” all of the files and folders on the remote server as mounted, this will give us the idea that we do not really need the root files of the original base disk image as we will overwrite everything anyway using the snapshot-style backup as per the command above.
IN CONCLUSION
To conclude:
-These steps are just explained extensively, but with closer examination these are very simple repetitive steps as it can be easily replicated by putting it all as a crontab script. Any tech-savvy STEEMIAN can easily say that if a task is repetitive, then it can be easily programmed.
-The files that a witness needs locally in order for an easy and full recovery are only these:
*A snapshot-style backup of every running witness server placed in separate folders locally (one folder per witness server) for easy identification and deployment during emergencies
*A disk image file that is produced thru the command “dd” in combination with “ssh”; we only need one disk image file for all if all of our installs have the same partitioning layout and the OS are also the same platform (all OS for example are 64bit x86)
-The approach stated here can still be upgraded and be made much more efficient by any tech-savvy witness. For example, if there is an easy way to just “partition” a disk manually and exactly the same as a given and existing layout, then the step involving the .img disk image file restoration will not be needed anymore, further reducing the time to recover a witness server or any server for that matter.
-The shortcut description of the backup and recovery steps:
*Backup thru RSYNC the running witness server “snapshot-style” not including the DATA folder
*Backup the full disk image of the witness server by creating a disk image of the same thru “dd over ssh”; this is only one time and can be used for all recovery needs if UNIFORMITY of witness node installation is followed
*Restore and recover any witness server anywhere by restoring the .img disk image followed by the copying of all snapshot-style backup files
Hope this can help our witnesses, and again, my hats off!!!
Previous Posts Link:
https://steemit.com/introduceyourself/@lightingmacsteem/lighting-for-everyone-the-steemit-way