Infrastructure drift is any difference between the infrastructure described in your code and the infrastructure that actually exists. A security group has an extra rule. An instance is a size larger than the code says. A policy gained a statement during an incident three weeks ago and nobody went back.
Drift is commonly treated as a moral failing — evidence that a team is not disciplined enough about its own process. That framing is wrong, and it is expensive, because it makes teams hide drift instead of looking for it. Drift is a measurement. It tells you that something happened to your environment that your process did not record. The only real failure is not knowing.
Why drift happens in well-run environments
#Every source of drift below occurs in environments run by people who know exactly what they are doing.
Emergency change. Production is failing at 02:00. Someone widens a limit, adds a rule, scales a service. The alternative was a longer outage. The change was correct; the follow-up got lost in the morning.
A provider changed a default. A new resource attribute appears with a default value, or a deprecated field starts reporting differently. Nothing in your organization changed; the ground did.
Another system is doing its job. An autoscaler adjusts capacity. A certificate manager rotates material. A cluster controller writes tags. These are not drift in a meaningful sense, but they show up in the same report unless they are excluded on purpose.
A human used the console. Sometimes because it was faster, sometimes because they did not know the resource was managed, and occasionally because they should not have had that access at all.
The code is wrong. The code describes something the cloud will not accept and quietly normalizes, so every plan shows a difference that never resolves.
Five causes, five different correct responses. Any process that treats them identically will be wrong four times out of five.
The real risk is silence
#A drifted resource is usually harmless. An undetected drifted resource is a change that reached production without review — which means nobody checked it, nobody approved it, and nobody can say when it happened.
That has three consequences worth naming:
- Recovery becomes unreliable. If the environment is no longer what the code says, then rebuilding from code produces something other than what you had. The recovery path you rehearsed does not lead where you think. That connects directly to the point made in backups you haven’t restored are assumptions: a recovery plan is only as good as the accuracy of the thing it recreates.
- Change management evidence stops being true. If your audit story is “every change goes through a reviewed pull request”, undetected drift makes that statement false in a way an auditor can find.
- The next planned change gets riskier. The following apply will either fight the drift or silently erase it, and neither is a decision anyone made.
Detect on a schedule, not on a feeling
#Detection is a scheduled run of the same plan your pipeline computes, against real state, with nothing applied. It answers one question: would applying the code right now change anything?
Two parameters matter.
Frequency should follow consequence. Accounts where an unreviewed change would be serious — the organization root, security, identity, network, production — are worth a daily check. Everything else is fine weekly. Running everything hourly produces a stream nobody reads, which is the same as not running it.
Output must be trackable. A drift report that posts into a chat channel is a notification. A drift report that opens an issue with an owner is a task. Only one of them can be closed, counted or audited.
In the AWS Enterprise Baseline, drift detection runs at 01:00 UTC daily against the critical accounts and weekly across all fourteen, and it opens a tracked issue when it finds something. Nothing is reverted automatically. That is a deliberate decision, and it is the next section.
Never auto-revert
#Automatic reconciliation is an appealing idea and a bad default for infrastructure.
Consider the emergency change from 02:00. The engineer widened a connection limit to stop a cascading failure. At 01:00 the following night, an automatic reverter sets it back. The cascade returns, at the hour with the thinnest coverage, caused by the tool that was supposed to be keeping things consistent.
Or consider the second case: the drift was introduced by an attacker. Auto-reverting it destroys the evidence and tells the attacker their change was noticed, without removing their access.
The correct model is detect automatically, decide deliberately. Reconciliation is a pull request like any other, and the interesting question is always which direction it goes: does the code change to match reality, or does reality change to match the code? A human is well placed to answer that. A cron job is not.
Triage in four classes
#Every drift finding is one of four things. Naming the class first makes the response obvious.
| Class | What it looks like | Correct response | Time frame |
|---|---|---|---|
| Emergency change | A person fixed something real, outside the pipeline | Codify the fix, open a pull request, apply it properly | Same week |
| External actor | A change nobody on the team can account for | Treat as a security event before anything else: who, when, from where | Immediately |
| Code defect | The same difference reappears after every apply | Fix the code so the plan converges; suppress nothing until it does | Next sprint |
| Legitimate exception | Another system owns this attribute by design | Exclude it explicitly, in code, with a comment saying why | Next sprint |
Two rules make this work. First, “external actor” is the default assumption until someone accounts for the change. It is cheap to be wrong in that direction and expensive to be wrong in the other. Second, an exception must be written down as an exception. Silently ignoring a resource produces an environment where the report is clean and the environment is not, which is worse than having no report.
Make drift rarer without pretending it will be zero
#Detection is the control. These reduce the volume:
- Production read-only for almost everyone. Most console drift is possible because the console can be used. In the AWS edition, production is read-only for everyone except the platform and DevOps leads, which removes the opportunity rather than discouraging it.
- A break-glass path that is obviously a break-glass path. People improvise when the legitimate emergency route is unclear. Make it explicit, time-boxed and loudly logged, and the emergency change arrives already attributed.
- A pipeline fast enough to use during an incident. If applying a fix properly takes forty minutes, someone will use the console. That is a performance problem wearing a governance costume — the same failure mode described in if deployment requires a hero.
- Immutable-first patterns where they fit. A resource that is replaced rather than edited has fewer attributes anyone can quietly adjust.
Notice what is not on the list: telling people to stop touching production. That instruction has never worked anywhere.
A workable weekly rhythm
#You do not need a program. You need a standing half-hour:
- Read the drift issues opened since last week.
- Assign each a class.
- Turn the emergency changes into pull requests.
- Escalate anything unaccounted for.
- Close what is resolved, and note anything that has appeared twice.
Item five is where the value compounds. Recurring drift is a design signal: the same attribute drifting every month means the code and the operational reality disagree about who owns it, and that disagreement is worth thirty minutes of thought now rather than an outage later.
Drift will never reach zero, and a team that reports zero drift usually has detection that is not working. The goal is a short list you understand, reviewed regularly, with nothing on it older than a couple of weeks. That is what operable means in practice — and if keeping that rhythm is the part your team does not have capacity for, it is exactly the kind of work BuiltForProd Managed does through your own pull requests.