A restore test is an exercise in which a team recovers a real system from its backups, against a clock, and verifies that the recovered system works. Until that exercise has been run, a backup is a hypothesis: there is a job, the job reports success, and everything else about it is assumption. The backup industry has trained us to check the wrong thing. A green backup job proves a job ran. It does not prove the data is complete, that the restore path exists, that anyone has permission to run it, or that the restored system comes back in an order that works.
This post is about closing that gap: what recovery objectives actually promise, why untested restores fail, and how to run a drill that produces a number you can put in front of a board or an auditor.
What RTO and RPO actually promise
Two numbers describe recovery, and they point in opposite directions from the moment of failure.
RPO, the recovery point objective, is how much data you have agreed to lose. If your database is snapshotted every six hours, your RPO cannot be better than six hours no matter how fast the restore runs. RTO, the recovery time objective, is how long you have agreed to be unavailable, measured from the failure to verified working service — not to “the restore command finished”.
Both numbers are set by the business and delivered by engineering, and the most common failure in the conversation is that only one side ever says a number out loud. An RPO of fifteen minutes and an RTO of an hour are perfectly reasonable requests, and they imply a specific, costed set of mechanisms. Write both down for each system that matters, then check what you actually have.
| Mechanism | Typical RPO | Typical RTO | What it does not protect against |
|---|---|---|---|
| Synchronous replica | Near zero | Minutes | Deletion, bad migration, ransomware — the replica copies them |
| Daily snapshot | Up to 24 hours | Hours | Anything that happened since the snapshot |
| Point-in-time recovery | Minutes | Hours | Loss of the storage account or the key |
| Object versioning | Near zero per object | Hours to days at scale | A bulk delete that also removes versions |
| Rebuild from code | Not applicable — infrastructure carries no data | Hours, if rehearsed | Application state, which still needs its own copy |
The last row is the one teams forget. Recovering data and recovering the environment the data lives in are different problems with different mechanisms, and a real incident often needs both.
The four ways an untested restore fails
Across postmortems, failed restores cluster into four causes. None of them are exotic.
The backup was incomplete. The database was captured; the object store holding user uploads was not. The application comes back with a consistent database and broken references. This is the most common finding in a first drill, and the only reliable way to find it is to restore and then use the system.
The restore path was undocumented. Someone knew how to do it in the old architecture. The current architecture has different encryption, a different network path and a different bootstrap order. The knowledge did not survive the migration.
A dependency was missing. The restore needs a KMS key that lived in the account that was deleted, a DNS record that no longer exists, or a secret that was rotated after the backup was taken. Encrypted backups are only as recoverable as the key material, and key material has its own lifecycle.
Nobody had permission. In a well-governed environment, production is read-only for almost everyone. That is correct, until the one person with the restore role is on a plane. A break-glass path that has never been exercised is not a path.
Running the drill
A restore drill is not a tabletop exercise. Somebody types the commands.
- Pick a real target and a real clock. Choose the system whose loss would stop the business. Announce a window, then start a timer that does not stop until someone other than the operator confirms the system works.
- Restore into a separate, isolated environment. A sandbox account or subscription, not production and not staging-that-someone-depends-on. The point is to prove the path, not to create a second incident.
- Restore everything the system needs, in order. Data store, object storage, secrets, configuration, DNS, and whatever else the application reads at startup. Write down the order you discovered, because next time that order is the runbook.
- Verify with a business transaction, not a health check. A health endpoint returning 200 proves the process started. Place an order, run a report, log in as a real user. That is the definition of “restored”.
- Record the number you actually achieved. Not the target. The achieved figure is the only honest input to the next planning conversation.
- Fix one thing. Every drill surfaces more than one problem. Pick the one that most reduces the achieved RTO and fix it before the next drill, rather than filing all of them and doing none.
The first drill will be slow and slightly humiliating. That is the point: it is the cheapest possible version of the bad day. Teams routinely find that a promised four-hour RTO is a nine-hour RTO the first time, and a three-hour RTO the third time.
What the drill produces
The output of a drill is not a feeling of relief. It is three artifacts.
- A runbook written during the exercise, by the person doing the work, in the order the work actually happened. Runbooks written in advance describe an imagined system; runbooks written during a drill describe the real one.
- An achieved RTO and RPO for that system, dated. Two consecutive drills give you a trend, which is far more useful than a single number.
- Evidence. Most control frameworks ask whether recovery has been tested and when. A dated record naming the system, the operator, the duration and the outcome answers that question directly. Compliance evidence produced as a by-product of engineering work is the only kind that does not require a scramble later — the same principle applies to SOC 2 evidence from infrastructure.
Rebuilding the environment is also recovery
Data restore gets the attention, but a large share of real incidents are not “we lost the data”. They are “we lost the environment”: an account was misconfigured beyond repair, a region became unusable, a change cascaded. In that case the question is not what your backup software can do. It is whether the environment exists anywhere other than in the cloud console.
Infrastructure defined entirely in code changes the shape of that answer. When every network, role, policy and cluster is a reviewed file in a repository, rebuilding is an apply, not an archaeology project. It is not free — you still need the data, the DNS, the certificates and the secrets — but it converts the most improvisational part of a recovery into something a second engineer can rehearse.
That is the property the BuiltForProd Standard calls recoverable: recovery is engineered and tested, not assumed. In the AWS Enterprise Baseline, the foundation itself is 100% infrastructure as code, the audit log archive lives in a separate account that the organization’s guardrails prevent anyone from stopping, and drift detection runs on a schedule so the code and the cloud do not quietly diverge between drills. None of that replaces a data backup strategy, and we are explicit with customers that it does not. It removes one large source of improvisation from the worst hour of the year.
A calendar you can actually keep
Ambitious recovery programs die in month two. A modest one survives:
| Cadence | Exercise | Time cost |
|---|---|---|
| Quarterly | Full restore of the one system that would stop the business | Half a day |
| Quarterly | Break-glass access exercised by someone who does not normally use it | 30 minutes |
| On change | Re-test after any change to backup configuration, storage location or keys | 1 hour |
| Annually | Environment rebuild from code into a clean account | One day |
Four entries. Put them in the calendar with names attached, not team names. An unowned drill does not happen.
Where to start this week
Pick one system. Ask three questions and insist on dated answers: When did we last restore this? How long did it take? Who did it? If any answer is “I don’t know” or “we haven’t”, you have found your first drill, and you have learned something important before it cost you anything.
If you want an outside read on where the gaps are, that is exactly what a BuiltForProd Assessment is for — and if your systems are already in better shape than you feared, we will tell you that too.