An unverified backup is not a backup

Too many times: the backup job is green every day, and on the day it matters the archive is empty, the key is gone, or the database was never included.

Verify in three layers. The file: tar -tzf backup.tar.gz > /dev/null, gzip -t. The contents: extract a random file and compare sha256sum with production. The restore: once a quarter, rebuild on a clean machine and time it.

The restore is the step everyone skips and the one that finds the real problem — often the key needed to decrypt the backup lives on the machine being backed up.

Write the restore down as a runbook and keep it next to the backup script. Nobody improvises commands at 3 a.m.