Skip to content
On this page 7

Terraform state at enterprise scale

Terraform state is a production system: one key per unit, a real lock, versioning, and a restore drill. Here is what to do when state and reality diverge.

Terraform state — the same file whether you run Terraform or OpenTofu — is the map between the resources in your code and the resources that exist in your cloud. It records that this database block owns that database. Lose it and your code no longer describes your infrastructure; it describes a second copy of it, which a plan will offer to create.

Most teams treat state as plumbing: configured once during the first week, never revisited, and discovered properly during the first incident that involves it. It deserves the treatment you give a production database, and this post is about what that treatment consists of.

What is actually in the file?

#

A state file is JSON. For every resource it holds the address in your code, the provider that manages it, the real identifier in the cloud, and a snapshot of the attributes as last observed. It also holds a serial number and a lineage identifier, which is how the backend detects that two people are writing different histories of the same thing.

Three consequences follow, and they set the whole design.

It is authoritative, not derivable. You cannot regenerate state from the cloud, because the cloud does not know which of your code blocks meant which resource. The mapping exists only in the file.

It contains secrets. Any sensitive attribute a provider returns is written into the file. A generated password, a connection string, a private key material field — in plain text, inside the JSON. The state bucket is therefore a credential store whether you intended one or not.

Every apply is a read-modify-write. Two applies against one state at the same time do not merge. One of them wins and the other writes a history in which the first never happened.

Why one state key per unit is the decision that matters

#

The state boundary is the blast radius. Everything else follows from where you drew it.

A single state for the whole estate is easy on day one and expensive on day four hundred. Every plan refreshes every resource, so plans get slower until people stop running them. One lock covers everything, so one long apply blocks the entire organization. One corrupted write puts every environment in the same recovery. And the identity that runs the apply needs permission over everything, which means there is no way to give a team the ability to change their own service without giving them the ability to change production networking.

State layoutPlan readsA lock blocksA corruption affectsWho can apply
One state for everythingThe whole estateEveryone, for the length of the longest applyEvery environment at onceOne identity with permission over all of it
One state per environmentOne environmentEveryone working in that environmentThat environmentOne identity per environment
One state per unitOne unitOnly work touching that unitOne unit’s resourcesA least-privilege role scoped to that unit

The third row is what Terragrunt units give you by construction: one directory, one configuration, one state key, generated from a single root configuration so nobody hand-writes a backend block and nobody mistypes a key. In the AWS Enterprise Baseline, 52 unit definitions generate 88 units across 15 accounts, each with its own state key and its own least-privilege deployer role, and the key is derived from the unit’s path rather than typed — which is the detail that makes the convention hold when a repository has eighty of them.

The state backend is also the first thing that exists. It is created by a bootstrap script before the organization does, then adopted into code with import blocks so that the thing holding every other unit’s state is itself under management. That ordering is not an accident: the bucket has to exist, and be versioned, before anything writes to it.

Figure 1
Shared state versus one state key per unit

On the left, one state file covers network, security and three workload environments, so a single apply locks and can damage all of them. On the right, each unit has its own state key, so an apply affects only that unit.

one shared stateterraform.tfstatenetworksecurityprodone lock, one corruption,one permission set for all of itone key per unitnetworksecurityprodkeykeykeythe apply touching prod locks prodand nothing else
The state boundary is the blast radius. With one shared state, an apply locks and can damage everything; with one key per unit, the failure is contained to the unit being changed.

Locking, versioning, encryption: three controls, three failures

#

They get configured in the same ten minutes and they protect against completely different things.

Locking stops two writers. A lock item in a table was the long-standing pattern; current versions can instead hold a lock file in the state bucket itself using conditional writes, which is what the Baseline uses — the lock is simply an object sitting beside the state file, and there is no separate table to provision, pay for or forget. What is not acceptable is a backend with no lock, or a lock that only the pipeline respects while an engineer’s laptop can write around it. The failure without it is a state file describing a history that never happened, and it usually shows up as a plan proposing to create resources that already exist.

There is a second, quieter control that removes most lock contention before it happens: a concurrency group on the apply workflow, set to queue rather than cancel. Two merges in quick succession then run one after the other instead of racing. After that, a lock error nearly always means a human was in the loop, which is a much more useful signal than “somebody is deploying”.

Versioning makes corruption reversible. Object versioning on the state bucket turns “the state is wrong” into “restore the previous version”, which is a two-minute operation. Without it, a bad write is permanent and the recovery is the slow one at the end of this post. Turn on versioning, block public access, and deny deletion of the bucket in policy rather than trusting that nobody will. Expire noncurrent versions on a schedule — ninety days is a reasonable choice — so the bucket does not become an unbounded archive of every plan anyone ever applied.

Encryption covers the secrets that are in the file whether you like it or not. Server-side encryption with a key you manage, plus a bucket policy that denies any request that is not over a current TLS version, is the baseline. OpenTofu can additionally encrypt state client-side, which is worth considering when the people who can read the bucket are not the same people who should read production secrets.

The better move, where the provider supports it, is to keep the secret out of state entirely. The Web App Blueprint sets its DocumentDB password through a write-only argument, so the value is sent to the API and never returned into state. That is one fewer credential in one fewer place, and it is the kind of detail that only exists if someone decided it should.

Who can read the bucket?

#

Ask the question directly, because the answer is often “more people than can read production”.

Three properties are worth enforcing. The state bucket lives in an account that is not the account the resources live in, so compromising a workload does not hand over the map to everything. Read access is limited to the deployer roles and a small number of named humans, not to a broad “engineering” group. And access is logged in an audit trail that the people with access cannot edit — the same separation described in the multi-account landing zone.

The mechanism that makes the second property practical is a dedicated state-access role in the account that owns the bucket. Every deployer, in every other account, assumes that one role to reach state, and that role can read, write, delete and list objects in the state bucket and nothing else. The alternative — granting each deployer role direct cross-account access to the bucket — works, and then quietly accumulates exceptions until nobody can enumerate who can read state.

What to do when state and reality diverge

#

There are four cases. They look similar in a plan and they have different correct responses, and choosing the wrong one is how a discrepancy becomes an incident.

What you seeWhat it meansThe correct moveWhat makes it go wrong
A plan wants to create something that already existsThe resource exists in the cloud but not in stateImport it into state at the right address, then plan again and expect no changes. Declaring the import in configuration rather than running a command makes it a reviewable pull requestApplying instead, which creates a duplicate — or, with a fixed name, fails halfway and leaves the unit stuck
A whole unit wants to create everything it already hasThe state key moved — usually because a directory was renamed, or a copied root configuration derives the key differentlyPause the apply pipeline, copy the state object from the old key to the new one, plan to verify it is empty, then delete the old keyApplying, which duplicates an entire unit’s worth of infrastructure
A plan wants to create something that was deleted outside the pipelineThe resource is in state but not in realityConfirm why it was deleted, then let the apply recreate it, or remove it from state if it was decommissioned on purposeRemoving it from state when it was deleted by an attacker, which destroys the evidence
A plan wants to change an attribute nobody changedReal drift, or a provider default that movedTriage it as drift before touching it: emergency change, external actor, code defect or legitimate exceptionReverting automatically, when the drift may be the thing holding production up
The state file is unreadable, empty or the wrong lineageA corrupted or clobbered writeRestore the previous object version and re-planRe-initializing and letting the tool adopt an empty state, which orphans every resource it managed
An apply reports the state is locked and nobody is runningA crashed or cancelled run left the lock behindConfirm no run is in flight, then clear the stale lock deliberately — with an object lock, that is deleting one object with the credentials that own the bucketClearing it reflexively during a real concurrent apply, which produces exactly the corruption the lock existed to prevent

Row three is a whole discipline of its own, and drift is a fact, not a failure covers the triage. The rule that ties the table together: never let an apply resolve a divergence you do not understand. A plan is a question. The answer to a surprising question is an investigation, not an apply.

There is one more prevention worth naming. Refresh-only runs on a schedule tell you about cases one to three before a human is mid-change, which is exactly what scheduled drift detection is. In the AWS edition that runs at 01:00 UTC daily against the critical accounts and weekly across all fourteen, opening a tracked issue rather than reverting anything.

The recovery nobody has rehearsed

#

Here is the drill. Pick a non-production unit. Delete its state object — you have versioning, so this is safe — and restore it from the previous version. That takes a few minutes and teaches you the mechanics and the permissions.

Then do the harder one, which is the one that matters: rebuild a unit’s state from nothing.

  1. Confirm the resources are still live. They are; only the map is gone.
  2. Run a plan. Everything appears as a create. This is the moment to be sure the apply is gated and that nobody runs it.
  3. Import each resource at its code address, from the smallest dependency outward. Recent versions let you declare imports in configuration rather than as commands, which makes this a reviewable pull request instead of a shell session.
  4. Plan again after each group. You are finished when the plan is empty.
  5. Write down how long it took and what was ambiguous. That number is your real recovery time for this failure, and it is usually several hours for a substantial unit.

Most teams have never done step three, and it is the step that determines whether losing a state file is an afternoon or a week. Smaller units make it dramatically faster, which is the blast-radius argument arriving again from a different direction.

This is the same principle as backups you haven’t restored are assumptions, applied to the one file that is not in your backup plan because nobody thought of it as data.

A short checklist you can run today

#
  • Every unit has its own state key, and no key is written by hand.
  • The state bucket has versioning on, public access blocked, deletion denied in policy, and encryption with a customer-managed key.
  • Locking is enforced for every apply path, including the one an engineer could run locally — or local applies are impossible, which is better.
  • The state bucket lives in a different account from the workloads it describes, and its access is logged where the accessors cannot edit the log.
  • Sensitive values use write-only arguments wherever the provider offers them.
  • Somebody has restored a state file from a version, this quarter, and written down what it took.
  • Somebody has rebuilt one unit’s state by import, ever.

The last two are the ones that separate a design from a system. Everything above them is configuration; those two are evidence.

One more thing worth doing while you are in there: every service that is added later inherits whatever state convention exists on the day it is created, which is why the state key belongs in the golden path rather than in someone’s memory. A repository whose backend block is generated cannot have a mistyped key in it.

If you want that evidence produced as part of a foundation rather than as a project you schedule and then postpone, the BuiltForProd Baseline deploys the state backend before the organization exists and hands over the runbooks for all of the above, the BuiltForProd Blueprints carry their own state and their own least-privilege deployer roles on top of it, and the BuiltForProd Assessment is the short engagement that tells you which of the seven lines is currently false.

Questions

What is Terraform state and why does it matter so much?

State is a file that maps each resource in your code to the real resource it created, along with the attributes last observed. It is how the tool knows that this database block means that database rather than a new one. Without it, a plan cannot tell the difference between changing a resource and creating a second one.

Should we use one state file or many?

Many. One state key per unit of deployment, where a unit is a directory with its own configuration. A single shared state means every plan reads everything, one lock blocks everyone, and a corruption affects the whole estate. Splitting by unit keeps the blast radius equal to the thing you were changing.

Do we still need a DynamoDB table for locking?

Not necessarily. A lock table was the long-standing pattern, and current versions of OpenTofu and Terraform can instead hold a lock file in the state bucket itself using conditional writes, which removes a whole resource from the design. What matters is that a lock exists, that it is enforced for every apply including manual ones, and that clearing a stale one is a deliberate act.

What do we do if the state file is lost or corrupted?

Restore the previous object version from the state bucket, which is why versioning is not optional. If no version exists, the recovery is to rebuild state by importing the live resources one by one, which is slow, manual and entirely possible. Rehearse it once on a non-production unit so the first time is not during an incident.

Are secrets stored in Terraform state?

Historically, yes: any sensitive attribute a provider returns is written to state in plain text inside the state file, which is why the state bucket is encrypted and tightly restricted. Write-only arguments, where a provider accepts a value and never returns it, remove whole categories of secret from state, and they are worth using wherever the provider supports them.

This is what we build.

BuiltForProd deploys a production foundation into your own cloud accounts, hands over the repositories and the documentation, and, if you want it, keeps running it with your team. Tell us what you are running and what is coming next. The first call is 30 minutes, with an engineer who scopes engagements, not a salesperson.

The calendar invite and the video link go to this address. Prefer to write? Send a message instead.

Drift is a fact, not a failure

Infrastructure drift is not a sign of a broken team. It is information about what your environment needs. Detect it, triage it, and never auto-revert it.