Skip to content
On this page 8

Secrets management without a vault cluster

Encrypted secrets in Git, one key per stage, and a sync to parameter storage: why repository write access still does not let anyone read production.

Encrypted secrets in Git is a design where secret values are committed as ciphertext, encrypted with a cloud-managed key, and synchronized into parameter storage when a change is merged. Nothing readable is ever in the repository. The permission to decrypt lives in the key policy, not in the version control system, which means write access to the code and access to the production secret become two separate decisions.

That separation is the whole idea, and it is worth stating in its strongest form: a developer with full write access to the repository still cannot read a production secret, because the decrypt call goes to the key service and the key service says no.

The shape of the design

#

Four moving parts, and none of them is a server you run.

Encrypted files in the repository. One file per application per stage — sandbox, development, staging, production. Values are encrypted individually, so a diff still shows you which key changed even though it cannot show you the value. The file is otherwise ordinary source code: reviewed, versioned, branched, and subject to the same code ownership rules as everything else.

A key per stage, in the security account. Each stage has its own customer-managed encryption key, and the key policy names who may decrypt with it. This is the access control. It lives in an account most engineers have no administrative rights in, and it is applied by the cloud provider at the moment of the call.

A sync on merge. When a change is merged, a pipeline role decrypts the file and writes the values into parameter storage as encrypted strings. That role is the only identity that performs the write, and no human uses it interactively.

Delivery to workloads. Applications do not read the repository. They read parameter storage through their own workload identity — in Kubernetes, through an operator that projects parameters into the objects a pod consumes. The application never holds a long-lived credential to a secret store; it holds an identity, and the identity is what the policy grants.

Figure 1
The write path and the read path for an encrypted secret

An engineer edits an encrypted secrets file and opens a pull request. A code owner approves it and an environment reviewer gates the merge. A pipeline syncer role then decrypts the file with the production key and writes the values into parameter storage, from which the workload reads them through its own identity. Separately, a developer attempting to decrypt the production file directly reaches the key service and is refused, and the refusal is recorded in the audit trail.

write pathpull requestciphertext difftwo approvalscode owner · reviewersyncer roledecrypt · writeparameter storeencrypted stringsproduction keypolicy names who may decryptpermittedread by workloadworkload identityread attempt, developergit clonehas the ciphertextdecrypt → refusedand recorded in the audit trail
Two paths, one gate. Writing a secret goes through review, approval and a pipeline role; reading it requires a decrypt call that the per-stage key policy either permits or refuses. Repository access decides neither.

Why repository access is not secret access

#

The claim deserves its mechanism spelled out, because it sounds like a slogan.

When someone runs the decrypt, the ciphertext contains a reference to the key that encrypted it. The tooling calls the cloud key service and asks it to decrypt a small data key. That call is authenticated as whoever made it, and it is authorized by the key policy — a document in the security account that lists the principals allowed to decrypt with that key. A developer’s role is on that list for the development key. It is not on the list for the production key.

So the sequence is: clone succeeds, file opens, decrypt call goes out, refusal comes back. The developer holds a file they cannot read. And because the call was made, there is now a record of it in the audit trail — an access denial with a principal, a time and a key, which is one of the more useful signals a security team can have.

Three consequences follow that are easy to miss:

  • Compromising the repository is not compromising production. A leaked personal access token, a malicious dependency in the build, a contractor who keeps their laptop: all of them yield ciphertext.
  • The blast radius of a key is one stage. Key compromise is a serious incident, but it is a serious incident about one environment rather than all of them.
  • The control is testable. You can prove the property by trying it, which is a better assurance than reading a policy document.

The separation-of-duties flow

#

For non-production stages, the flow should be short: edit, review, merge. Friction there buys nothing and teaches people to route around the process.

For staging and production, five steps, each doing a distinct job:

  1. A lead writes the change. They can encrypt with the production key — writing is a different permission from reading, and in many setups the author does not need decrypt rights at all.
  2. A second code owner approves the pull request. Code ownership on the secrets path is what makes this automatic rather than remembered.
  3. An environment reviewer approves the deployment. This is a separate gate from the code review and usually a different person, which is the part that makes it separation of duties rather than two reviews.
  4. The syncer role writes the value. A pipeline identity, federated per job, with permission to decrypt and to write parameters and nothing else. No human holds it.
  5. A read-only role consumes it. The workload’s identity can read the parameter and decrypt it at read time; it cannot write.

A useful check on any secrets process: name the person who can change a production secret without anyone else being involved. If that name exists, the process has a gap, and it is the same gap regardless of which tool is underneath.

What you give up, honestly

#

No design here is free. The comparison that matters is not “which is most secure” but “which capabilities do you actually need”.

Encrypted files in GitManaged secret storage aloneA vault cluster you runCI provider’s secret store
Where the value livesCiphertext in Git, plaintext only in parameter storageManaged storeThe cluster’s own storageThe CI system
Who can read productionWhoever the key policy names, which is testableWhoever holds the read permissionWhoever the policy engine allowsAnyone who can edit a pipeline definition
Change controlA pull request with a diff, review and historyAn API call, usually unreviewedPolicy-dependent; often an API callAn edit in a settings page
RotationA reviewed change; manualAutomatic for supported resource typesDynamic credentials per requestManual
Dynamic credentialsNoFor supported databasesYes, its core strengthNo
Cross-cloud and on-premisesNoNoYesNo
Infrastructure to operateNoneNoneA highly available cluster with its own unseal, backup and upgrade storyNone
Failure mode on outageValues already in parameter storage keep servingStore outage affects readsCluster outage can block workloadsPipeline outage blocks deployments

Read the last two rows together. A vault cluster is genuinely the most capable option on this list, and it is also a distributed system with an availability requirement equal to production, an unseal procedure, a backup story and an upgrade path. Teams who need dynamic credentials or a policy engine spanning clouds should pay that cost with open eyes. Teams whose actual requirement is “applications get their configuration secrets and changes are reviewed” are buying an operational burden for a capability they will not use.

The bottom-right cell is worth noting too, because it is where most real-world secret sprawl lives: secrets pasted into the CI provider’s settings, readable by anyone who can edit a pipeline, with no diff and no history. That is the status quo this design most often replaces, and the comparison is not close.

Rotation is a reviewed change, not a cron job

#

This is the limitation to state plainly rather than bury: in this design, rotation is manual. There is no automated rotation loop. Changing a secret means rotating the value at its provider, updating the encrypted file, opening a pull request, getting it approved, merging, and confirming that the old value no longer works.

Three things make that acceptable in practice, and one thing makes it a genuine constraint.

It is acceptable because the procedure is written down and rehearsable, because the change is reviewed like any other and therefore visible, and because most configuration secrets do not benefit much from frequent scheduled rotation compared with the things that actually reduce risk — removing long-lived credentials, scoping permissions, and detecting misuse.

It is a genuine constraint because rotation you have to remember is rotation that slips. If your obligations require demonstrated rotation on a schedule, put the dates on a calendar with an owner, or use a managed rotation capability for the resource types that support it — database credentials being the obvious case. That is a sensible hybrid, not a contradiction.

One rule is absolute: a secret that was ever committed in plaintext must be rotated at its source. Git history is permanent, forks and clones are everywhere, and re-encrypting a value that was once readable changes nothing at all.

The other limits worth knowing

#
  • Key names are visible. Values are encrypted; the structure of the file is not. That is deliberate, because it is what makes diffs reviewable, and it does mean the names of your integrations are in the repository.
  • The syncer role is a high-value target. It can decrypt and write. Federate it per job, scope it to one stage, and watch its usage — the same argument as removing long-lived credentials from CI entirely.
  • Placeholders are a real failure mode. A file merged with a placeholder still in it produces a confusing outage. Fail the pull request on the pattern rather than trusting review to catch it.
  • Plaintext still gets committed by accident. A pre-commit hook that detects private keys and obvious secret material is cheap and catches the common cases before they reach history.
  • No dynamic secrets. Every value is static until someone changes it. If you need credentials that exist for an hour, this is the wrong layer.

What ships in the Secrets Blueprint

#

The Secrets Blueprint — one of the BuiltForProd Blueprints — implements exactly this design on AWS: SOPS with AWS KMS encrypting files in Git, one file per application per stage across sandbox, development, staging and production, per-stage keys held in the security account, and a sync into SSM Parameter Store as encrypted strings on merge. External Secrets Operator delivers them into workloads. The five-step separation of duties above is the staging and production flow; a pull request check fails on any remaining placeholder. It ships with an incident playbook covering five scenarios, including plaintext committed to history and key compromise, and a seven-step rotation procedure.

The characteristic worth pointing at is what it does not add: no cluster, no new service to run, no additional availability dependency. It is a workflow and a key policy on top of services the foundation already has. The property underneath — that no person holds standing access to a production secret — belongs to the BuiltForProd Standard rather than to any one cloud, and the security page sets out where it sits among the other controls.

Where to start this week

#
  1. Find the plaintext. Search your repositories and your CI configuration for the obvious patterns. Whatever you find must be rotated at the source, not deleted from a file.
  2. Name who can read production secrets today. If the list is “anyone with repository access” or “anyone who can edit a pipeline”, that is the gap.
  3. Split the keys by stage. One key per environment is the change that makes every other control in this design mean something.
  4. Put a second approval on the production path. A code owner on the secrets directory plus an environment reviewer is two settings and it is the whole separation of duties.
  5. Write the rotation procedure and put a date on it. Since it is not automated, it needs an owner and a calendar entry, and pretending otherwise is how it fails.

If you would rather inherit the whole flow than assemble it, the BuiltForProd Blueprints are deployed into your own accounts with the repositories handed over. And if the wider question is who can reach production at all rather than which values they can read, that is zero trust access for engineers.

Questions

Is it safe to store secrets in Git?

Encrypted values are safe to store in Git; plaintext never is. With envelope encryption against a cloud key service, what is committed is ciphertext that is useless without a decrypt call the key policy must permit. The repository becomes a distribution mechanism and an audit trail, not a place where secrets are readable.

How does SOPS with KMS differ from putting secrets in Secrets Manager directly?

The secret ends up in cloud-managed storage either way. The difference is the workflow: with encrypted files in Git, a change to a secret is a pull request with a diff, a reviewer and a history, and the promotion between stages uses the same pipeline as everything else. Managed secret storage alone has no review step; whoever holds write permission changes production silently.

Why one key per stage instead of one key for everything?

Because the key is the access control. If one key decrypts everything, then anyone who can decrypt development can decrypt production, and the only remaining boundary is a convention about which file people open. Separate keys let the same repository be fully readable to a developer for development values and cryptographically opaque for production ones.

What happens when a secret is exposed?

You rotate the secret at its source, not just the key. Git history is permanent, so a value that was ever committed in plaintext must be treated as compromised forever, and re-encrypting it changes nothing. Rotate at the provider, update the encrypted file, sync, and then confirm that the old value is rejected.

When do you actually need a vault cluster?

When you need dynamic, short-lived credentials generated per request — database users that exist for an hour, certificates issued on demand — or a secrets layer spanning several clouds and on-premises systems under one policy engine. Those are real requirements. If your need is "applications get their configuration secrets safely and changes are reviewed", running a highly available cluster to get it is a large operational cost for a capability you are not using.

This is what we build.

BuiltForProd deploys a production foundation into your own cloud accounts, hands over the repositories and the documentation, and, if you want it, keeps running it with your team. Tell us what you are running and what is coming next. The first call is 30 minutes, with an engineer who scopes engagements, not a salesperson.

The calendar invite and the video link go to this address. Prefer to write? Send a message instead.

Zero trust access for engineers

How people should reach production: federated sessions instead of users, no long-lived keys anywhere, and access that ends the day someone leaves.

Nine security services, one account

What GuardDuty, Security Hub, Config, CloudTrail and their siblings each actually catch, what they cost, and the order to switch them on.