Peace of Mind: An Automated Backup Script for Server Roots

• Thu, Jun 24, 2021
• 4-minute read

This blog post introduces a script I developed to backup the root of my servers to local or remote storage. Simple yet efficient, the script ensures you never miss a backup and keeps you informed.

Backup is not just about securing data; it’s about securing peace of mind. Whether you are a system administrator or managing your own server, the data is invaluable. The script shared in this post is my small contribution towards making the backup process seamless and fool-proof.

Purpose of the Script

This script primarily serves to automatically backup the root directory of a server to either local storage on the same server or remote storage on a different server. It’s designed to run as a cron job, which ensures that backups are made regularly without manual intervention. Moreover, in the unfortunate event that a backup fails, the script will ensure that you are promptly notified via email.

One of the worst things that can happen to any backup process is failure that goes unnoticed. To mitigate this, I’ve integrated a healthcheck which ensures that your script doesn’t fail silently.

Script Dependencies

To run this script, you need to have sendmail set up on your server for email notifications. Additionally, you’ll need to have an account with Healthchecks.io as the script uses their service to perform health checks. If you prefer a self-hosted alternative to Healthchecks.io, you can replace it with Uptime Kuma, a reliable, open-source status monitoring solution.

Implementing the Script

I usually place this script in the /etc/cron.daily directory which ensures it’s executed daily. If you wish to run the script manually, it’s smart enough to detect the interactive shell and display live progress.

The script creates a lock file to ensure that only one instance runs at a time. If it’s already running, it won’t execute a new instance. All the activities are logged, and in case of any failure, the script sends an email with the details of the failure. It utilizes rsync for the actual backup, and it’s tailored to exclude certain directories like dev, proc, sys, tmp, run, mnt, media, and lost+found. Source: https://wiki.archlinux.org/title/Rsync#Full_system_backup

The script contains a unique Healthchecks.io key. In case of script failure, the health check still runs but you will receive a warning email. This is a crucial feature as it ensures you are aware of any issues as soon as they occur.

Script

I’m sharing the script below, however, be sure to replace the Healthchecks.io key with your own and set the appropriate email address for notifications:

#!/bin/bash
set -x

# Script variables
SCRIPT_NAME=$(basename -- "$0")
DATE=$(date +"%F_%T")
EMAILSUBJECT="$(hostname) - $SCRIPT_NAME"
HEALTHCHECKIO="<HEALTHCHECK_ID>"
LOCK_FILE="/tmp/$SCRIPT_NAME.lockfile"
LOG_DIRECTORY="/path/to/log/directory"
LOG_FILE="$LOG_DIRECTORY/$SCRIPT_NAME.log"

failfunction_rsync() {
    if [ "$1" != 0 ]; then
        echo -e "Subject: FAILURE cron - $EMAILSUBJECT" "\n\n\n One command failed: $SCRIPT_NAME failfunction_rsync" | sendmail <RECIPIENT_EMAIL>
    fi
}

# Check if running in interactive shell
if tty -s; then
    RSYNC_OPTIONS="--stats -h -aAXH --info=progress2 --exclude={\"/dev/*\",\"/proc/*\",\"/sys/*\",\"/tmp/*\",\"/run/*\",\"/mnt/*\",\"/media/*\",\"/lost+found\"}"
    INTERACTIVE_SHELL=true
else
    RSYNC_OPTIONS="--stats -h -aAXH --exclude={\"/dev/*\",\"/proc/*\",\"/sys/*\",\"/tmp/*\",\"/run/*\",\"/mnt/*\",\"/media/*\",\"/lost+found\"}"
    INTERACTIVE_SHELL=false
    # Check if log directory exists, if not create it
    [ -d "$LOG_DIRECTORY" ] || mkdir -p "$LOG_DIRECTORY"
    # Redirect output to log file
    exec >"$LOG_FILE" 2>&1
fi

# Rsync root
if [ ! -e "$LOCK_FILE" ]; then
    touch "$LOCK_FILE"
    rsync $RSYNC_OPTIONS /path/to/source /path/to/destination
    # Only run if not in an interactive shell
    if [ "$INTERACTIVE_SHELL" = false ]; then
        failfunction_rsync "$?"
        curl -fsS -m 10 --retry 5 -o /dev/null "https://hc-ping.com/$HEALTHCHECKIO"
        # Send log via email
        mail -s "log cron - $EMAILSUBJECT" <RECIPIENT_EMAIL> < "$LOG_FILE"
    fi
    rm "$LOCK_FILE"
else
    echo "Script is still running"
    echo -e "Subject: FAILURE cron - $EMAILSUBJECT still running" "\n\n\n $SCRIPT_NAME is still running" | sendmail <RECIPIENT_EMAIL>
fi

Wrapping Up

This script provides a simple but robust solution to a critical aspect of server management - backups. By automating the process and incorporating a health check, you can ensure that your data is not only backed up regularly but that you’re also the first to know if something goes wrong.

Remember, securing your data means securing peace of mind. Happy backing up!