One of the best things about using Unix-like operating systems such as Linux, Linux Subsystem for Windows, and MacOS, is the ability to use shell scripts to automate so much.
I do, however, realize that the thing that gives the shell so much power also makes it feel impenetrable for many people more used to mousing around.
This is why whenever I have the opportunity, I try to (gently) introduce people to some simple but incredibly powerful aspects of Shell Scripting.
Namely, Pipes.
What are pipes and why should I care?
Pipes (|) in Unix-like systems let you use the output of one command directly as the input to another. They allow chaining commands together to process data in steps without creating intermediate files.
This takes simple command line instructions and allows you to think of them almost like programming functions instead.
Pipes eliminate the need for temporary files or intermediate variables.
How this works is a feature called "Standard In/Standard Out". The textual output that ordinarily is sent to your screen can be instead sent to the next command in a chain of commands.
Although there are a few ways to string applications together in a graphical environment, there are none as simple and as powerful as this simply because there are as yet few standard ways to grab the output from a desktop app and plumb it in as input to the next.
Example - Count Your Photographs
The easiest way to get your head around this is by example.
Say you want to count how many .JPG files you have in your photos directory and all its subdirectories. There are a number of ways to do this, but one way might be to use find command:
find -type f -name "*.jpg"
There is one problem, this just lists all the files that match, we need to use wc (Word Count) to do the counting. This is where pipes come in - we send the output of find into wc and tell wc we want to count lines instead of words!
find -type f -name "*.jpg" | wc -l
More Useful Example - Backing Up All Your Websites
How does this help with automation?
Consider a case where you host many websites, the number of which changes reasonably often, but you still need daily website backups.
A couple of options spring to mind, including having a database of clients that is read by a Python script, but instead here is my actual solution that makes use of pipes and loops:
#!/bin/bash
# Base directory where your WordPress sites are stored
BASE_DIR="/path/to/sites"
# Backup destination directory
BACKUP_DIR="$HOME/backups"
mkdir -p "$BACKUP_DIR"
# Find all website directories but exclude those with 'staging.' in the path
find "$BASE_DIR" -type d -path "*/app/public" -not -path "*/staging.*/*" | while read -r WP_PATH; do
# Extract the domain name from the path (e.g., "example.com")
DOMAIN=$(basename "$(dirname "$(dirname "$WP_PATH")")")
# Set the output tar.gz file path
BACKUP_FILE="$BACKUP_DIR/${DOMAIN}_$(date +%F).tar.gz"
# Compress the directory into a tar.gz file
tar -czf "$BACKUP_FILE" -C "$(dirname "$WP_PATH")" "$(basename "$WP_PATH")"
echo "Backup created for $DOMAIN: $BACKUP_FILE"
done
My script uses the fact that each website has a standard structure
Here’s how it works:
- The find command outputs a list of paths, with one path per line.
-not -path "*/staging.*/*"Ensures that directories containing staging. in their paths are excluded from the search results.- The
while read -r WP_PATHloop reads this output line-by-line. - Each line (delimited by \n) is assigned to the variable WP_PATH.
- The -r flag prevents backslash escapes () from being interpreted.
- The loop processes each path individually within the WHILE block.
DOMAIN=$(basename "$(dirname "$(dirname "$WP_PATH")")")extracts the domain name (e.g., example.com) from the directory path stored in the variable WP_PATH.dirnamecommand removes the last component of a path.basenamecommand extracts the last component of a path.- Another way to do this would be with pipes!
DOMAIN=$(echo "$WP_PATH" | dirname | dirname | basename)
Using this script I can run a command on every website that is NOT a staging site, without having to copy and paste unique paths into the script.
Set and forget!
I really ought to do more scripting. I know how powerful it can be. These days I mostly code in Python, but bash can be simpler for manipulating files.
Yeah Python is awesome but sometimes a lot more complex and overhead than necessary :)
Interesting post. I am a programming enthusiast and a Linux fan, and finding these things generates many ideas in my head. Thanks for sharing it.
Scripting is always fun and I appreciate this post! I use linux in my other devices and I look forward trying out your tricks 😊
Thanks for your contribution to the STEMsocial community. Feel free to join us on discord to get to know the rest of us!
Please consider delegating to the @stemsocial account (85% of the curation rewards are returned).
You may also include @stemsocial as a beneficiary of the rewards of this post to get a stronger support.