One of the best things about using Unix-like operating systems such as Linux, Linux Subsystem for Windows, and MacOS, is the ability to use shell scripts to automate so much.
I do, however, realize that the thing that gives the shell so much power also makes it feel impenetrable for many people more used to mousing around.
This is why whenever I have the opportunity, I try to (gently) introduce people to some simple but incredibly powerful aspects of Shell Scripting.
Namely, Pipes.
Pipes (|) in Unix-like systems let you use the output of one command directly as the input to another. They allow chaining commands together to process data in steps without creating intermediate files.
This takes simple command line instructions and allows you to think of them almost like programming functions instead.
Pipes eliminate the need for temporary files or intermediate variables.
How this works is a feature called "Standard In/Standard Out". The textual output that ordinarily is sent to your screen can be instead sent to the next command in a chain of commands.
Although there are a few ways to string applications together in a graphical environment, there are none as simple and as powerful as this simply because there are as yet few standard ways to grab the output from a desktop app and plumb it in as input to the next.
The easiest way to get your head around this is by example.
Say you want to count how many .JPG files you have in your photos directory and all its subdirectories. There are a number of ways to do this, but one way might be to use find command:
find -type f -name "*.jpg"
There is one problem, this just lists all the files that match, we need to use wc (Word Count) to do the counting. This is where pipes come in - we send the output of find into wc and tell wc we want to count lines instead of words!
find -type f -name "*.jpg" | wc -l
How does this help with automation?
Consider a case where you host many websites, the number of which changes reasonably often, but you still need daily website backups.
A couple of options spring to mind, including having a database of clients that is read by a Python script, but instead here is my actual solution that makes use of pipes and loops:
#!/bin/bash
# Base directory where your WordPress sites are stored
BASE_DIR="/path/to/sites"
# Backup destination directory
BACKUP_DIR="$HOME/backups"
mkdir -p "$BACKUP_DIR"
# Find all website directories but exclude those with 'staging.' in the path
find "$BASE_DIR" -type d -path "*/app/public" -not -path "*/staging.*/*" | while read -r WP_PATH; do
# Extract the domain name from the path (e.g., "example.com")
DOMAIN=$(basename "$(dirname "$(dirname "$WP_PATH")")")
# Set the output tar.gz file path
BACKUP_FILE="$BACKUP_DIR/${DOMAIN}_$(date +%F).tar.gz"
# Compress the directory into a tar.gz file
tar -czf "$BACKUP_FILE" -C "$(dirname "$WP_PATH")" "$(basename "$WP_PATH")"
echo "Backup created for $DOMAIN: $BACKUP_FILE"
done
My script uses the fact that each website has a standard structure
Here’s how it works:
-not -path "*/staging.*/*" Ensures that directories containing staging. in their paths are excluded from the search results.while read -r WP_PATH loop reads this output line-by-line.DOMAIN=$(basename "$(dirname "$(dirname "$WP_PATH")")") extracts the domain name (e.g., example.com) from the directory path stored in the variable WP_PATH.
dirname command removes the last component of a path.basename command extracts the last component of a path.DOMAIN=$(echo "$WP_PATH" | dirname | dirname | basename)Using this script I can run a command on every website that is NOT a staging site, without having to copy and paste unique paths into the script.
Set and forget!