The backup you have never tested

Everyone has backups. Almost nobody has restores. The four backups that quietly do not work, and a twenty-minute drill that tells you which kind you have.

Ask a business owner whether they have backups and the answer is almost always yes. Ask when someone last restored one and the room goes quiet.

That gap is the whole subject. A backup is a file; a restore is a capability. Nobody has ever been saved by a file — they were saved by the ability to turn it back into a working system, on a bad day, quickly, with someone available who knows how. Until you have done that once, what you own is a hope with a filename.

Two numbers decide everything

Before tools, agree on two numbers with whoever runs the business:

  • How long can we be down? An hour? A working day? Three days? This sets how fast the restore has to be — and therefore what it costs.
  • How much data can we afford to lose? If backups run nightly at 2am and the server dies at 5pm, you have lost a day of orders. Is that survivable, or does it need to be hourly?

Most businesses have never said these out loud, which is why backup arguments go in circles. Once they are written down the technical choices mostly answer themselves — and you find out quickly whether the current arrangement matches what the business assumed.

Four backups that do not work

  1. The snapshot on the same server. If the disk fails, the fire happens, or the account is deleted, the backup goes with it. A copy in the same place as the original is a convenience feature, not a backup.
  2. The one someone does manually. It runs when a person remembers, which means it ran daily for three weeks and then stopped in March when they got busy. Nobody noticed, because nothing fails when a backup does not happen.
  3. The one nobody can open. Encrypted with a password held by an employee who left, or in a format the current system will not import. Technically present, practically gone.
  4. The one that is only a file copy of a live database. Copying database files while they are being written produces a file that looks fine and restores into a corrupt, half-consistent state. Databases need a proper dump or a snapshot the database engine itself is aware of.

3-2-1, in a form a small business can actually keep

The rule of thumb is three copies, on two kinds of storage, one of them off-site. In practice, for a company running a site, a database and a mailbox:

WhatHow oftenWhereKept for
DatabaseDaily, plus before any changeHost backup + a second location30 days
Uploaded files and mediaDailySame as above30 days
MailboxesDailyProvider retention + export before a migration30 days
Site code and configOn every changeVersion control, not a backup at allForever
Full server imageWeeklyOff-site4 weeks

Note the fourth row: code does not belong in backups at all. It belongs in version control with its history intact — a repository you control, like PearlGit, is not a backup of your code, it is your code.

Off-site means a different failure domain

"Off-site" is not a folder on the same provider under the same login. The point is to survive a category of event, and the categories are worth naming: hardware failure, an account compromise, a deletion made in good faith by someone with permission, theft or fire at the office, and a provider you fall out with.

An external drive that lives in the same office as the server fails the fire test. A copy on the same hosting account fails the account-compromise test — an intruder with your password deletes the backups as casually as the data. At least one copy should require a different credential to reach.

The power question, locally

Where the grid cuts several times a day, one failure mode is far more common than dramatic disasters: a database that was mid-write when the power went. It usually recovers. Occasionally it does not, and the damage is not obvious for days.

Two consequences. Keep enough retention to go back past the moment corruption started, because a single overnight copy will faithfully back up the broken state and overwrite your last good one. And put the database server on protected power, which we argued more broadly in power cuts are a design input.

The twenty-minute drill

Once a quarter, book twenty minutes and actually do this:

  1. Pick yesterday's backup. Not a special one — the ordinary one.
  2. Restore it somewhere that is not production: a scratch server, a local machine, a staging site.
  3. Open the restored system and check three real things: the newest order is present, an uploaded image loads, someone can log in.
  4. Write down how long it took and every step you needed, including where the credentials came from.
  5. Note what was missing. There is always something the first time — an environment file, a certificate, a cron job, a DNS record.

That document is worth more than the backup itself. On the bad day, the person restoring is stressed, possibly not the person who set it up, and possibly working from a phone. A written, timed, previously-executed procedure is what turns a disaster into an inconvenience.

What to check this week

  • Do backups run automatically, and does someone get told when one fails?
  • Is at least one copy somewhere a compromise of the main account cannot reach?
  • Is the database dumped properly rather than file-copied?
  • How far back can you go — and is that further back than you would notice a problem?
  • Who can perform a restore, and is that more than one person?
  • When was the last successful restore, and where is it written down?

If the last question has no answer, that is the finding. Everything else is guesswork until it does.

What we do

Every site on our hosting is backed up automatically, off the machine it runs on, with a retention window long enough to step back past a problem rather than just to yesterday — and restores are a support ticket, handled by the team that runs the platform, not a self-service maze.

Which still leaves the twenty minutes. Do the drill anyway, on your own systems, this quarter. The first one is always the most educational, and it is much better to learn on a Tuesday afternoon than at 11pm on the night something breaks.

More from the blog

All articles