Notes

Occasional write-ups on systems, networks, and tooling.

Backups that survive the person who made them

2026-09-12

The most common backup problem is not corruption. It is that nobody ever tried to restore, so the backup encodes the same misunderstanding as the original setup.

Restore drills

Schedule a restore into a scratch directory once a quarter. It does not need to be fast, it needs to actually run end to end.

tar xzf /backup/etc-2026-09-12.tar.gz -C /tmp/restore-test
diff -r /tmp/restore-test/nginx /etc/nginx

If the diff output surprises you, the backup is not capturing what you assumed.

Off-site, not just off-directory

Backups on the same disk protect against deleted files. They do nothing against a failed disk, a wiped host, or a provider mistake. At minimum, copy to a second machine.

Keep more than one generation

Three generations covers most accidental deletion cases: today, last week, last month. Seven daily copies all containing the same bad config change are worth very little.

Record how to restore

A backup with no restore instructions is a liability. Write the steps down somewhere that does not live on the machine being backed up — the whole point is that you may be reading it while that machine is gone.

Keep it to a page: where the archives are, what the decryption key is, and the three commands needed to get a service running again.