CLI for rsync backups on external SSD
using justfile and bash
Backups - the problem:
Backups are important but too often shrouded in anxiety and manual error prone workflows.
The most common way to backup data is manually dragging and dropping files from a source (e.g., your computer or phone) to a destination (e.g., an external hard drive or other computer).
While common, manual drag-n-drop has several cons:
- Manual errors from “slips” or “typos”
- Needs deleting and complete re-copy for updates
- No helper for iterative backup (which files have changed?)
- Minimal feedback (what happened: files, memory, deletions etc.)
Commercial solutions exists
There are of course solutions out there - but they are not for free…
Carbon Copy Cloner (CCC) seems like a great solution, however, it’s not for free. As I haven’t tried it, I cannot recommend it. But it seems as the solution I would pick from the market at the moment of writing.
Time Machine on Macs is of course the native and simple solution. However, the TM algorithm currently backs up the entire OS which requires significantly more memory and time than just backing up a couple of folders that you might need. Besides, many people find themselves with the unhappy surprise that TMs often copy the OS bug as well and you’ll have to try-and-error when the bug first occurred.
rsync - the CLI solution
Luckily for us, Linux made a command line interface (CLI) that solved the issue, called rsync. rsync was first released in 1996 and is still actively maintained. It’s written in C and
rsync is a…
rsync 2.6.9 released under the GPL 2.0 license, with the somewhat more restrictive GPL 3.0 license restricting Apple from using any newer version. Therefore, Apple switched to openrsync. openrsync is compatible with rsync 2.6.9, but lack some of the newer features. It is actively maintained and is capable of everything we’ll be covering in this blog.
Due to serious AI-slop criticism on the rsync project, I’ve deliberately decided on using openrsync instead of the original rsync. See also here.
I’m running Mac 26.5.2 with openrsync: protocol version 29:
rsync --version
openrsync: protocol version 29
rsync version 2.6.9 compatibleAs a vertue of it’s +30 year long life, it has a robust and versetile set of options that can be called using flags. Below is a list of the flags I’ll be covering in this blog.
| Flag | Meaning |
|---|---|
-r |
Recursive mode. Copies directories and their contents recursively. |
-l |
Copies symbolic links as symbolic links instead of copying the files they point to. |
-p |
Preserves file permissions. |
-t |
Preserves file modification times (timestamps). |
-g |
Preserves group ownership. |
-o |
Preserves file owner (requires appropriate privileges, typically root). |
-D |
Preserves device files and special files (--devices --specials). |
-a |
Archive mode (-rlptgoD). Recursively copies files while preserving symbolic links, permissions, ownership, group, device files, and timestamps. |
-v |
Verbose output. Shows the progress of the sync process. |
-h |
Human-readable output. Formats file sizes in KB, MB, GB instead of raw bytes. |
--dry-run |
The most important flag. Simulates a sync without making any changes. Great for testing. |
--delete |
Deletes files in the destination that no longer exist in the source. Use to make the destination mirror the source. |
--progress |
Displays progress information for each file being transferred. |
--info=progress2 |
Displays overall transfer progress instead of per-file progress. |
--stats |
Prints a detailed summary of the transfer, including bytes sent, speed, and number of files transferred. |
--modify-window=1 |
Allows timestamps to differ by up to 1 second when comparing files. Useful when syncing with filesystems that have lower timestamp precision (e.g., FAT or some network filesystems). |
--exclude |
Skips files or directories that match the specified pattern. |
--exclude-from=rsyncignore.txt |
Reads exclude patterns from the file rsyncignore.txt instead of specifying them on the command line. |
So in practice, it’s as easy as:
rsync -a my_source my_destination/As seen from the table, the -a (archive) flag is a shorthand for the seven flags -rlptgoD. This is the standard format for UNIX/linux/mac native formatting, which brings us to the next section.
Cross-platform compatibility and the exFAT format
Although I prefer linux-like systems (Ubunto or MacOS), I also value cross-platform compatibility for storing my data. Therefore, I used exFAT for my Samsung T5 external SSD (2T memory).
exFAT is great for many reasons:
- cross-platform compatibility
- unlike the older FAT format, it has no size limits on individual files.
- ??
But as always, there is no free lunch:
- permissions, commonly used by UNIX systems, are not supported
- the format is not native to either Windows, Linux, or Mac
Therefore, we need to adjust our rsync flags in order to safely copy our data.
https://stackoverflow.com/questions/32682694/rsync-backup-to-external-hard-disk-exfat-fails
https://nelsonslog.wordpress.com/2017/02/06/rsync-to-exfat-usb/
https://www.disfinder.com/docs/notes/rsync-to-exfat/
Instead of -a, it’s therefore recommended using -vrltD and omitting -gop. As seen from the flag table, -gop is fro permissions (chmod bits),group ownership, and user ownership (requires root). As these are not supported by FAT or exFAT, we will have to skip them.
Note that this is a limitation of cross-platform compatibility and not recommended for pure Mac or Linux systems compatibility.
recap:
- The rsync -a (archive) flag is a shorthand for -rlptgoD:
- Recommended to exFAT is
rsync -vrltD --progress --stats(so without pgo and adding v(erbose)) - pgo is
- Permissions (chmod bits)
- Group ownership
- User ownership (requires root)
We now have:
rsync -vrltD my_source my_destination/Fixing timing issues with windows
https://superuser.com/questions/763366/why-does-rsync-seem-to-overwrite-an-already-existing-file-on-exfat
recap: –inplace: Modifies the file in place. This is useful for large files (like disk images) on SSDs but can be less efficient for small changes on remote networks. –modify-window=1: Essential for exFAT. exFAT rounds timestamps differently than Linux file systems. This flag allows a 1-second tolerance, preventing rsync from repeatedly re-copying files that are actually unchanged.
You need –modify-window=1.
Quoted below from man rsync:
Specifying 1 is useful for copies to/from MS Windows FAT filesystems, because FAT represents times with a 2-second resolution (allowing times to differ from the original by up to 1 second).
Therefore, we add:
rsync -vrltD --modify-window=1 my_source my_destination/Making a justfile CLI
Just is an amazing way to make a CLI fast and without any unnecessary wrapper code. It’s not really a program, but rather a mapping of code snippits executed from the commandline. The power lies in it’s ability to host a full workflow with its own tests, helper functions, and user-facing functions.
Adding style to it
First, we make variables for color coding our echo text. This is pure aesthetics.
# Color codes
GREEN := '\033[0;32m'
RED := '\033[0;31m'
YELLOW := '\033[1;33m'
BLUE := '\033[0;34m'
NC := '\033[0m' # No Color
Config of the setup
Next, we make some configurations specific to our setup.
# Configuration
SOURCES := '$HOME/code/ $HOME/uni-non-code/'
DEST := '/Volumes/Samsung_T5/backup-ssd'
RSYNC := '/usr/bin/rsync'
RSYNC_BASE_FLAGS := '-vrltDh --delete --info=progress2 --stats --modify-window=1 --exclude-from=rsyncignore.txt'
RSYNC_DRY_FLAGS := '{{RSYNC_BASE_FLAGS}} --dry-run' # all base flags + dry-run!
We hardcode variables:
SOURCES— the folder(s) we want backupDEST— the place to backup and store it allRSYNC— wherersyncis (usr/bin, brew, or other)RSYNC_BASE_FLAGS— all the flags we want to useRSYNC_DRY_FLAGS— adding--dry-runto our base flags
Note that the base flags are:
-v— Verbose-r— Recursive-l— the-t— Timestamp-D— the-h— Human-readable
--delete— Mirror function--info=progress2— Progress info on entire backup--stats— Memory and time feedback
--modify-window=1— more loose timing for FAT format to work--exclude-from=rsyncignore.txt— exclude pattern for files
The rsyncignore.txt file is a simple list of all patterns to ignore.
These include .DS_Store, .quarto, *.tmp and .httr-oauth etc.
A help command
It’s good practice to provide overview by writing a help command. In this case, we just list the four just commands in the terminal.
help:
@echo "Backup to Samsung T5 SSD - Available commands:\n"
@echo " just help - Show this help message"
@echo " just preview - Dry run: show what would be synced (no changes)"
@echo " just backup - Run the mirror backup (with deletion confirmation)"
@echo " just status - Quick health check"
As we just use echo, theres no need for me to explain further.
The status command
The status command is the simplest of the three. Here, we just check whether the external SSD is mounted or not.
status:
@echo "{{BLUE}}=== Backup Status ==={{NC}}"
@if [ -d "{{DEST}}" ]; then \
echo "{{GREEN}}Destination exists: {{DEST}}{{NC}}"; \
else \
echo "{{RED}}Destination NOT found: {{DEST}}{{NC}}"; \
echo "{{YELLOW}}Please ensure Samsung T5 SSD is connected and mounted.{{NC}}"; \
exit 0; \
fi
@if [ -d "/Volumes/Samsung_T5" ]; then \
echo "{{GREEN}}Samsung T5 is mounted at /Volumes/Samsung_T5{{NC}}"; \
DF_OUTPUT=$(df -h /Volumes/Samsung_T5 2>/dev/null | tail -1); \
echo "{{BLUE}}$DF_OUTPUT{{NC}}"; \
else \
echo "{{RED}}Samsung T5 is NOT mounted{{NC}}"; \
fiThe function has two if-else statements:
- Feedback on whether the destination directory within the SSD exists (
-d "{{DEST}}") usingecho. - Feedback on the memory state and space left on the SSD, if it’s mounted (
-d "/Volumes/Samsung_T5"). We get this feedback usingdf -h /Volumes/Samsung_T5 2>/dev/null | tail -1.
Here’s the breakdown of the last code call:
dfdisplays filesystem disk space usage.-hformats the sizes in a human-readable way (e.g. 500G, 1.2T).2>/dev/nullRedirects stderr (file descriptor 2) to /dev/null, effectively hiding any error messages. For example, if the drive isn’t mounted,dfwould normally print an error, but this suppresses it.| tail -1Pipes the output to tail. tail -1 prints only the last line, which is the filesystem information rather than the header.
The preview command
We’ve already covered most of this command. It looks like this:
preview:
@echo "{{YELLOW}}=== DRY RUN: No changes will be made ==={{NC}}\n"
@{{RSYNC}} {{RSYNC_DRY_FLAGS}} {{SOURCES}} {{DEST}}/
@echo "{{BLUE}}\nPreview complete. Use 'just backup' to execute.{{NC}}"So we basically call 4 variables that define our rsync, flags, sources, and destination: {RSYNC} {{RSYNC_DRY_FLAGS}} {{SOURCES}} {{DEST}}. Easy piecy!
The backup command
backup:
@echo "{{YELLOW}}Checking for changes...{{NC}}"
@DRY_OUTPUT=$({{RSYNC}} {{RSYNC_DRY_FLAGS}} -i {{SOURCES}} {{DEST}}/ 2>&1); \
HAS_DELETIONS=$(echo "$DRY_OUTPUT" | grep -q '^\*deleting' && echo "yes" || echo "no"); \
if [ "$HAS_DELETIONS" = "yes" ]; then \
echo "{{RED}}WARNING: Deletions detected in dry run!{{NC}}"; \
echo "$DRY_OUTPUT"; \
echo "\n{{RED}}Files will be DELETED from the backup destination.{{NC}}"; \
read -p "Continue with backup? [y/N] " -n 1 -r; \
echo; \
if [ "$REPLY" != "y" ] && [ "$REPLY" != "Y" ]; then \
echo "{{YELLOW}}Backup cancelled.{{NC}}"; \
exit 0; \
fi; \
else \
echo "{{GREEN}}No deletions detected.{{NC}}"; \
echo "$DRY_OUTPUT"; \
read -p "Continue with backup? [Y/n] " -n 1 -r; \
echo; \
if [ "$REPLY" = "n" ] || [ "$REPLY" = "N" ]; then \
echo "{{YELLOW}}Backup cancelled.{{NC}}"; \
exit 0; \
fi; \
fi; \
{{RSYNC}} {{RSYNC_BASE_FLAGS}} {{SOURCES}} {{DEST}}/ && \
echo "{{GREEN}}Backup completed successfully!{{NC}}" || \
echo "{{RED}}Backup failed!{{NC}}"1. Dry-run test
DRY_OUTPUT=$({{RSYNC}} {{RSYNC_DRY_FLAGS}} -i {{SOURCES}} {{DEST}}/ 2>&1)
HAS_DELETIONS=$(
echo "$DRY_OUTPUT" |
grep -q '^\*deleting' &&
echo "yes" ||
echo "no"
)Here’s what each part does:
DRY_OUTPUT=$(...)
Executes the rsync command (with placeholders like {{RSYNC}}, {{RSYNC_DRY_FLAGS}}, etc.). 2>&1 redirects standard error to standard output so all output is captured. The entire output is stored in the DRY_OUTPUT variable.
grep -q '^\*deleting'
Searches the captured output for lines beginning with *deleting. -q makes grep quiet so it only returns an exit status.
^\*deletingmatches rsync’s itemized deletion lines, for example:
*deleting old-file.txt&& echo "yes" || echo "no"
If grep finds a match (exit status 0), HAS_DELETIONS is set to “yes”. Otherwise it is set to “no”.
2. Feedback on deleted files
read -p "Continue with backup? [y/N] " -n 1 -rHere’s what each option does:
-p "Continue with backup? [y/N] "
Displays the prompt before reading input. The [y/N] convention indicates that No is the default if the user simply presses Enter.
-n 1
Reads exactly one character without requiring the user to press Enter (on most terminals).
-r
Prevents backslashes () from being treated as escape characters, so the input is read literally.
The character entered is stored in the default variable REPLY unless another variable name is provided.
3. exit or backup
Depending on the user response, the script either
exit 0;: terminates the program early or{RSYNC} {{RSYNC_BASE_FLAGS}} {{SOURCES}} {{DEST}}/ &&: performing the backup.
The && at the end means the next command will run only if this rsync command succeeds (returns an exit status of 0).If the backup completes successfully, execution continues with the next command in the script. If rsync encounters an error (such as a missing source directory or permission issue), the chain stops at this point because of the && operator.
Good to go
There you have it.
We have:
- Discussed the issue of backups and why you should use code for this tedious task
- Discussed the pros and cons of
rsyncandopenrsync - Cross-platform compatibility using exFAT
- Timing issues in FAT formats
- Walkthrough of justfile for personal backup CLI using
rsync
All code can be found in the Github repo.