My Backup Strategy Failed—Here's What I Do Now
💥 The Disaster
It was 2 AM on a Sunday. My server crashed. No big deal, I thought. I have backups. Except when I tried to restore, the backups were corrupted. Three months of work, gone.
The lesson: Having backups isn't enough. You need backups that actually work when you need them.
❌ What Went Wrong
1. Single Backup Location
I was backing up to a single external drive. When that drive failed, I had nothing.
2. No Backup Verification
I never tested restoring from backups. I assumed they worked. They didn't.
3. Manual Process
Backups were manual. I'd forget, or skip them when busy. Inconsistent backups are worse than no backups.
4. No Offsite Backup
Everything was local. Fire, theft, or hardware failure would destroy everything.
✅ My New Strategy
1. 3-2-1 Rule
- 3 copies of important data
- 2 different media (local disk + cloud)
- 1 offsite backup
2. Automated with Restic
# Automated daily backup script
#!/bin/bash
restic -r /backup/local backup /important/data
restic -r s3:s3.amazonaws.com/my-bucket backup /important/data
restic check # Verify backups work
3. Automated Verification
# Weekly restore test
restic -r /backup/local restore latest --target /tmp/restore-test
# Verify files are correct
rm -rf /tmp/restore-test
4. Monitoring and Alerts
I get notified if backups fail, if verification fails, or if backups haven't run in 24 hours.
📊 Current Setup
| Backup Type | Frequency | Retention |
|---|---|---|
| Local (Restic) | Daily | 30 days |
| S3 (Restic) | Daily | 90 days |
| ZFS Snapshots | Hourly | 7 days |
💡 Key Lessons
- Test your backups regularly. A backup that can't restore is useless.
- Automate everything. Manual processes fail when you're busy or forget.
- Follow the 3-2-1 rule. Multiple copies, different media, offsite.
- Monitor and alert. Know immediately if backups fail.
- Document your restore process. When disaster strikes, you won't remember the steps.
🎯 Conclusion
Losing data taught me that backups aren't optional—they're critical infrastructure. My new strategy isn't perfect, but it's tested, automated, and redundant. That's what matters.