Operability is the property that a competent engineer who did not build a system can understand its state, change it safely and recover it — alone, at an hour when nobody wants to be called, without escalating to one specific person. It is the last of the eight properties in the BuiltForProd Standard and the one most often assumed rather than designed.
Everything about operability is decided before the incident. At 03:00 you cannot add observability, you cannot write a runbook, and you cannot grant yourself access you do not have. You are running the system that was built, with the information it was built to emit. The work is all upstream.
The only measurement that matters
#Every mature incident practice eventually converges on the same metric, and it is not mean time to recovery. It is the time between the page and the responder being able to say what is wrong.
That interval is where the damage happens, because during it nobody is fixing anything. And when teams time it honestly, most of it turns out to be navigation rather than diagnosis: finding which dashboard covers this service, remembering which account it runs in, hunting the runbook, searching chat for the last time this happened, working out who to ask.
None of that is a skill problem. It is a system that did not tell its operator where to look.
What a system owes its operator
#Six properties decide how that interval goes. They are not tooling choices; each one can be satisfied several ways.
1. The alert says what is wrong, not what is unusual
#A page should name a user-visible condition and the action it expects. “Checkout error rate above 2% for five minutes” is actionable. “CPU above 80%” is a fact that may or may not matter and trains people to dismiss alerts.
The hard rule: page only on conditions a human must act on now. Everything else goes to a dashboard or a daily report. A team receiving four hundred alerts a day is receiving none, because the only workable response to that volume is filtering by instinct, and instinct filters out the rare, important one.
| Signal | Where it belongs | Why |
|---|---|---|
| Customers cannot complete a core action | Page, immediately | A person must act now, and the impact is already happening |
| Error budget burning fast enough to exhaust within hours | Page, immediately | Acting later means acting after the commitment is broken |
| Root account used, guardrail policy changed, audit logging stopped | Page, immediately | Either an emergency or an intrusion; both need a human now |
| Capacity trending toward a limit in days | Ticket with an owner | Real, but nobody should be woken for a trend |
| A single instance is unhealthy behind a healthy service | Dashboard | The system is already handling it; the alert adds noise |
| Drift detected outside the pipeline | Tracked issue, triaged weekly | Needs a decision, not a reflex |
| Backup job failed | Ticket, and a page if it fails twice | One failure is an event; two is a pattern that threatens recovery |
2. State is visible from one place
#Centralized logs, metrics and traces are not a maturity badge. They are a navigation shortcut. When a responder has to log into four accounts to assemble a picture, the picture arrives late and incomplete.
In the AWS Enterprise Baseline, cross-account observability is wired to a sink in the security account so metrics, logs and traces from every account are readable in one place, and the audit archive is separate with its own retention and lifecycle. Those specifics describe the AWS edition; the property — one place to look — belongs to every edition.
3. The runbook exists and is not fiction
#A runbook written in advance describes the system the author imagined. A runbook written during a real exercise, by the person doing the work, describes the one that exists. That is why the restore drill in backups you haven’t restored are assumptions produces a runbook as its main artifact: the drill is the only reliable way to discover the actual order of operations.
Keep runbooks in the repository next to the code they describe. Documentation in a separate wiki decays silently; documentation beside the code fails review when someone changes the thing it documents.
4. The responder has the access they need, before they need it
#Production should be read-only for almost everyone. That is correct and it creates an obligation: an explicit, rehearsed path for the person on call to get the access an incident requires.
A break-glass path that has never been exercised is not a path. Make it explicit, time-boxed, loudly logged, and use it once a quarter on purpose so the first time is not during the outage.
5. Changes are traceable to a person and a moment
#“What changed?” is the first useful question in most incidents. It should be answerable in under a minute, and the answer should come from the same place for infrastructure and application: a merge, a reviewer, a timestamp, a pipeline run.
This is a side effect of routing every change through review, as described in if deployment requires a hero. It is also why undetected drift hurts so much at 03:00 — the change log is accurate only if changes cannot happen outside it.
6. The failure mode was considered before it happened
#Blast radius is a design decision. Whether a failure in one environment can reach another is settled by the network topology and the account boundaries, long before the failure. So is whether a noisy neighbor can exhaust a shared resource, and whether a bad deployment to one workload can affect a different one.
The responder at 03:00 inherits those decisions. They cannot renegotiate them.
Operability is verified by handover, not by documentation
#The most common self-deception in this area is believing that written documentation proves operability. It does not. Documentation proves that someone wrote something down.
The verification is a handover: a second engineer performs the real tasks, unaided, while the person who normally does them watches without helping. Every hesitation is a defect. Every “oh, you also need to…” is a missing line in a runbook.
That is how we close a deployment: the customer’s own engineers perform a defined set of tasks themselves — making a change through the pipeline, reviewing and applying it, promoting between environments, resolving a drift report, rotating a secret, adding a team member, walking the escalation path — before the engagement ends. Not a presentation. A working session where the customer’s team does the work while we watch. In the AWS edition that set is eleven tasks, and the documentation they refer to runs to hundreds of pages, including roughly a hundred and sixty symptom-based troubleshooting entries written as cause, fix and prevention.
The reason for the format is simple: a system is not operable because it is documented. It is operable because someone other than its author has operated it.
The bus-factor audit
#An hour, once a quarter, that consistently produces a better backlog than most planning sessions:
- List the operational tasks that matter: deploy, roll back, restore, rotate a credential, add an account, respond to the three most likely alerts, grant and revoke access.
- For each, name everyone who could do it unaided, today.
- Every task with exactly one name is a defect.
- Fix the defects by having the second person do the task, not by writing more documentation.
Teams are almost always surprised by the result, and the surprise is the point. The tasks with one name are rarely the ones people expected.
The integration points, briefly
#Operability usually involves tools your team already pays for. Those connections should be deliberate attachment points rather than assumptions: a notification topic, an event bus, an observability sink, the audit archive, and the pipeline’s own identity. Each is documented as a place to attach Datadog, PagerDuty, Slack, Jira or whatever you actually use — an integration you configure, not a native feature we claim. Naming them as attachment points keeps the boundary honest and keeps your tooling choices yours.
Where to start tonight
#Pick the most likely alert for your most important service. Ask the person on call this week to walk you through what they would do when it fires — out loud, without opening anything first. Then let them actually do it and time the gap until they can say what is wrong.
Whatever slowed them down is your first operability defect, and it will be something smaller and more fixable than you expect. It usually is.
If your team is carrying production without the capacity to do this work, BuiltForProd Managed does it through your own pull requests and your own access — and we will tell you if what you need is something else entirely.