Cloud cost is decided in architecture and paid in arrears. By the time an invoice explains what happened, the decision that caused it is months old, the person who made it has moved on to something else, and the only available response is a meeting. That is why cost programs built on reports get politely ignored: they deliver information to the wrong place, at the wrong time, to the wrong person.
Cost governance that engineers accept has three properties. The number is visible at the moment of the decision. Attribution happens automatically, without anyone filling in a field. And it informs rather than blocks, because a control that stops deploys gets routed around within a quarter.
Why the monthly report does not work
#Picture the sequence. An engineer adds a managed service in a pull request. It merges on a Tuesday. Six weeks later, a finance partner circulates a spreadsheet showing a line item rose. Somebody is asked to investigate. They find the pull request, read the diff, and discover the decision was reasonable given what was known at the time.
Nothing in that sequence is anyone’s fault, and nothing in it changes behavior. The information existed when the diff was written — the price of that service was knowable to the minute — and it was simply not in the room.
Moving the number earlier is most of the work. Everything below is a way of doing that.
What makes a tag survive contact with a hurry?
#Attribution is the foundation, and tagging schemes fail for one reason: they depend on somebody remembering during a busy afternoon.
Three mechanisms make tags hold.
Tags are applied by the code, not by the person. In a Terragrunt repository the root configuration is generated into every unit, so default tags are set once and every resource in every account inherits them. There is no path where a resource is created by the pipeline without tags, because the pipeline does not have a path that skips the root configuration. That also means a new service inherits the tags on the day it is created, provided the scaffolding it copies is the golden path rather than the last team’s repository.
An organization tag policy defines the vocabulary. The AWS Enterprise Baseline ships a tag policy with six required tags, running in report-only mode by default and switching to enforcement with one flag. Report-only first is deliberate: turning on enforcement in an estate with existing untagged resources breaks deploys on a Monday and teaches everyone that the cost team causes outages.
Untagged resources are a tracked list, not a feeling. A weekly query for resources missing required tags, with an owner attached, is the whole program. The Resource Tagging standard in Security Hub is one way to produce it; a scheduled query is another.
The priced switch
#This is the pattern worth stealing whatever you are building on.
Expensive capabilities in the AWS Enterprise Baseline ship off, behind a single flag, and the comment next to the flag states the price. Cheap resilience is on by default; anything with a meaningful monthly bill is a decision you make with the number in front of you, in the file where you make it.
| Capability | Default | Approximate list price, as written beside the switch |
|---|---|---|
| Network Firewall egress inspection | Off | About $0.395 per endpoint-hour, so roughly $288 a month per availability zone and about $865 for three, plus $0.065 per GB inspected |
| Shield Advanced | Off | $3,000 a month, organization-wide, on an annual commitment |
| Client VPN, integrated with single sign-on | Off | About $73 a month per availability zone for the subnet association, plus per-connection hours; a full-tunnel configuration for a twenty-person team lands in the hundreds once egress is counted |
| Transit Gateway inter-region peering | Off | About $36 a month per peering attachment, plus inter-region transfer |
| Inspector, Macie, Shield Advanced | Off | Priced per instance, per bucket and per gigabyte inspected; the switch names the dimension |
| A second and third NAT gateway, one per availability zone | On | About $32 a month each. Turning them off saves roughly $65 a month and means an availability zone failure stops egress for every spoke |
| SOC 2 baseline conformance pack | On | Config rule evaluations only, the cheapest of the 29 pack templates |
| Twenty-eight further conformance packs | Off | Priced by rule count, which the switch states |
Prices are approximate AWS list prices and vary by region; treat them as the order of magnitude that lets you decide, not as a quote.
Look at the sixth row for a moment, because it is the interesting one. It is not a feature — it is resilience, priced. Somebody can read that line and decide that a development environment does not need egress to survive the loss of an availability zone, and that production does. That is a good decision, made in ten seconds, by the person best placed to make it, and it is only possible because the price and the consequence are written in the same comment.
What the pattern actually changes is the conversation. “Do we want egress inspection” is an architecture debate that can run for two meetings. “Do we want egress inspection for about $865 a month” is a decision someone can make in four minutes, and it is a decision the person who will pay for it can make correctly.
The same discipline applies to the things that look free. Three availability zones instead of two costs money. A 365-day log retention costs money. Both are often right; both should be visible.
What NAT gateways and endpoints actually cost
#This is where multi-account networks quietly spend, and where the largest structural saving usually hides.
A NAT gateway costs roughly $0.045 per hour, about $32 a month, plus roughly $0.045 per gigabyte processed. Highly available means one per availability zone. Three zones per VPC, four workload VPCs, and you are paying for twelve NAT gateways before any traffic moves.
Centralized egress collapses that. The AWS Enterprise Baseline runs a hub-and-spoke network over a Transit Gateway with NAT in the hub across three availability zones, and the spoke VPCs have no NAT at all. Twelve becomes three.
Be honest about the trade. Transit Gateway charges roughly $0.02 per gigabyte per attachment, so centralizing swaps a fixed cost you always pay for a variable cost that grows with traffic. For several spokes with moderate egress, centralizing wins clearly and also buys you one place to inspect traffic. For a single very chatty VPC, local NAT can be cheaper. Do the arithmetic for your traffic rather than adopting the pattern because it is the pattern.
Then remove traffic from NAT entirely where you can:
| Path | Fixed cost | Per-GB cost | When it wins |
|---|---|---|---|
| Through NAT | About $32 per gateway per month | About $0.045 | Never, for a service that has a gateway endpoint |
| S3 or DynamoDB gateway endpoint | None | None | Always. There is no threshold to calculate |
| Interface endpoint | About $7 a month per availability zone, so roughly $21 across three | About $0.01 | Above roughly 600 GB a month to that service, and immediately if you need the traffic off the public path anyway |
The break-even for an interface endpoint is the fixed cost divided by the per-gigabyte saving: about $21 divided by $0.035, which is around 600 gigabytes a month. Below that, NAT is cheaper and you may still want the endpoint for network-path reasons. Above it, the endpoint pays for itself and keeps paying.
Gateway endpoints for S3 are in the Baseline by default, because a control with no hourly charge and no per-gigabyte charge is not a decision. Interface endpoints are not: the unit that would create them ships with an empty list, because which ones are worth $21 a month depends entirely on what your workloads call. That is the priced-switch pattern applied honestly — the default is the one that costs nothing, and the arithmetic is left where you can do it.
Karpenter, spot, and the ceiling that matters more
#In the Web App Blueprint, one of the BuiltForProd Blueprints, cluster capacity comes from a small managed node group plus Karpenter, frequently on spot capacity, with the Cluster Autoscaler alongside it.
Three separate savings are at work, and they are worth separating because teams usually adopt the least important one first.
Right-sizing. Karpenter provisions nodes shaped for the pods that are actually pending, rather than adding another copy of whatever instance type someone chose a year ago. It also consolidates: when workloads can be packed onto fewer nodes, it replaces them.
Purchase option. Spot capacity is dramatically cheaper and can be reclaimed at short notice. Stateless application pods tolerate that. System components — the ingress controller, the GitOps controller, the secrets operator — belong on the on-demand node group, so the thing that reschedules your workloads is not itself being rescheduled.
The ceiling. The Blueprint caps Karpenter at 32 vCPUs in development, 64 in staging and 128 in production. This is the control that protects the invoice, and it is underrated because it is boring. An alert tells you spend passed a threshold, after it passed. A ceiling means a misconfigured horizontal pod autoscaler, a retry storm or a bad rollout scales to a bounded number of nodes and then stops, with the excess pods left pending for someone to find in the morning. Pair it with per-stage sizing — the Blueprint runs staging and production with high availability and development minimally, and keeps logs for 30, 90 and 365 days respectively — and the bounded case is bounded everywhere.
Retention is the other lever nobody pulls. Tiering the audit archive to infrequent access at 90 days and to archival storage at a year, expiring old state versions, and keeping only release images plus the last few dozen builds in the registry are each worth less than one architectural decision and together they compound quietly.
The same “scale to zero” thinking shows up elsewhere in the foundation: the optional self-hosted CI runners are ephemeral, on spot, and scale to zero when idle, and the Data and ETL Blueprint has no always-on compute at all. There is more on the cluster side of this in an EKS platform without a platform team.
Cost on the pull request
#Everything above is design. This is the feedback loop.
An optional Infracost step posts an estimated monthly cost difference as a comment on every foundation pull request, generated from the same tree the plan used. The reviewer sees the diff and the number together. Nobody has to request an estimate, and nobody has to know which service is expensive this year.
Four details decide whether it works:
- It is a comment, not a gate. A cost check that fails a build gets an override, and the override becomes the habit. Every step is marked to continue on error, so cost tooling can never block a plan.
- It updates in place. One comment per pull request, rewritten on each push, rather than a new comment on every commit. A thread of twelve stale estimates is read by nobody.
- It sits next to the change, not in a dashboard. The only place cost information reliably reaches an engineer is the place they are already looking. In practice a reviewer sees the number three times: in the priced comment beside the switch in the diff, in the plan, and in the estimate.
- It is an estimate and is labeled as one. Usage-based services are guesses. The value is in the direction and the order of magnitude: “this adds about a hundred dollars a month” changes a review, and being wrong by fifteen percent does not.
It also does not need to be everywhere. The estimate runs on the foundation repository, where a pull request can add a firewall or a gateway; the workload repositories carry their prices as comments on the stage values instead, because what changes there is a node count or an instance size whose price is already written beside it.
The cultural effect is the point. After a few weeks, people start estimating before they push, which is the behavior every cost program says it wants and almost none of them produce.
What we do not ship, and what you should turn on
#Two things this foundation deliberately does not include, said plainly rather than implied.
No AWS Budgets and no Cost Anomaly Detection. Neither is configured. Both are inexpensive, both take under an hour, and both are worth having — Budgets for the threshold you want to know about, Anomaly Detection for the spend shape you did not predict. Set them up. We would rather name the gap than leave you to discover it from an invoice.
No cost allocation beyond tags. The tags make showback possible; the reporting layer is yours to choose, and the FinOps tooling market is genuinely good.
If your problem is primarily “we do not know where the money goes”, a cost platform will help you faster than a foundation will. If your problem is “the architecture generates the spend and we cannot change it safely”, that is the other way round, and it is the problem this post is about.
A week of work that pays
#- Turn on a budget and an anomaly monitor. An hour, and you stop finding out from the invoice.
- Count your NAT gateways. Multiply by $32. Then look at whether your spokes need their own.
- Add S3 gateway endpoints everywhere they are missing. There is no arithmetic to do.
- Find your top three interface-endpoint candidates by NAT data-processing volume and run the 600-gigabyte check.
- Add a cost comment to your infrastructure pipeline. One workflow step.
- Write the price in the comment next to the next expensive thing you make optional. Then do it for the rest.
Items two through four are usually the largest single structural saving in a multi-account estate, and none of them require anyone to be careful.
If the architecture itself is the cost problem — the flat account, the per-VPC NAT, the cluster that has never been right-sized — the BuiltForProd Assessment measures it against the BuiltForProd Standard, and the BuiltForProd Baseline ships with these defaults already chosen and every paid switch showing its price.