An EKS platform is not a cluster. It is the set of decisions that have to be made before application teams can use a cluster safely: where capacity comes from, how changes reach it, how secrets arrive, what a pod is allowed to do, and what has to happen before something runs in production. A platform team is the usual way those decisions get made. A well-built golden path is the other way.
This post is a worked example, not a customer story: the Web App Blueprint, one of the BuiltForProd Blueprints, deployed into the workload accounts of an existing foundation. Seventeen AWS services and eight Kubernetes components per stage, as 14 modules, 16 units and five application workflows, written in OpenTofu and Terragrunt like everything else in the foundation. Its sibling, the Data and ETL Blueprint, makes the opposite trade — no cluster at all — and the comparison at the end of this post is partly about when that is the better answer. The interesting part is not the component list — it is which decision each component represents, and the honest accounting at the end of what you take on by choosing Kubernetes at all.
Where does capacity come from?
#Two autoscalers, with separate jobs. This is the arrangement that keeps working, and the reasoning is worth having.
A small managed node group provides a predictable base — in the Blueprint, t3.medium instances across three private availability zones, two to three nodes in development and three to four in staging and production. It carries a taint so that ordinary application pods do not schedule onto it, which leaves it for the controllers: the load balancer controller, the GitOps controller, the secrets operator, the DNS controller, and Karpenter itself. The application chart deliberately sets no tolerations, and that absence is the mechanism rather than an oversight.
Karpenter provisions everything else. It watches for pods that cannot be scheduled and launches nodes shaped for them, rather than adding another copy of whatever instance type was chosen a year ago.
| Setting | What it is in the Blueprint | Why it is set that way |
|---|---|---|
| Instance families | A short list of current general-purpose families | A longer list usually lowers cost, because it gives the provisioner more offerings to price against. Anything not listed, it cannot buy |
| Capacity types | Spot and on-demand | It takes spot whenever spot capacity exists for a type that fits, and falls back to on-demand. Dropping spot gives an all-on-demand pool |
| Node lifetime | 30 days | Nodes are replaced on a schedule, so the cluster never accumulates long-lived instances with drifting images |
| Consolidation | After one minute of a node being under-used | Repacking recovers the capacity that scaling up leaves behind |
| Disruption budget | Ten percent of the pool at a time | Replacement is continuous, and it should never look like an outage |
| vCPU ceiling | 32 in development, 64 in staging, 128 in production | The one setting that bounds the bill. At the ceiling, provisioning stops and excess pods stay pending |
The ceiling deserves its own sentence. An alert tells you that spend passed a threshold, after it passed. A ceiling means a misconfigured autoscaler, a retry storm or a bad rollout scales to a bounded number of nodes and then stops, at three in the morning, without anyone deciding anything. That is the difference between a notification and a control, and it is the same argument made at length in cost governance that engineers accept.
How do changes reach the cluster?
#Through Git, and through exactly one path.
ArgoCD watches a GitOps repository and reconciles the cluster to it. The application definition itself is created by the infrastructure code rather than by a manifest, so adding a new application is a pull request against the platform repository — the registration model described in the golden path for new services.
The decision that matters is the sync policy, and it is different per stage.
- Development and staging sync automatically, with pruning and self-healing on. Merge, and the cluster follows. Speed is the point.
- Production does not. There is no automation block, so ArgoCD reports that the application is out of date and applies nothing until a person presses sync.
That asymmetry is deliberate and it is worth defending against the obvious objection. Self-healing production sounds strictly better: the cluster returns to the declared state on its own. In practice it means an emergency change made during an incident is silently reverted by a controller while the incident is still running, and it means the approval you configured earlier is not actually a gate, because the merge is the deploy. Leaving production manual makes the approval real.
It also has a consequence you should accept knowingly: because production does not self-heal, a change made directly in the production cluster stays there until someone notices. That is drift, and drift is a fact rather than a failure — detect it on a schedule and triage it deliberately.
How do secrets reach a pod?
#Not through Git, and not through a pipeline variable.
Infrastructure code creates the database and the cache and writes their connection details to the cloud parameter store, encrypted. The application chart declares which parameters it needs. The External Secrets Operator, authenticating with a pod-level cloud identity rather than a stored key, reads those parameters and writes a Kubernetes secret that the application mounts.
Two properties fall out of that arrangement. Nothing in Git holds a secret value, and no pipeline ever passes one. And the operator’s permissions are scoped by path — the development operator’s role can read development parameters and nothing else, so an accidental request for a production parameter fails rather than succeeds quietly.
There is a matching discipline on the infrastructure side. The database password is set through a write-only argument, so the value is sent to the API and never returned into state — one fewer credential in the state file, which is a place most teams forget holds secrets at all.
Build once, promote by approval
#The delivery rule is one sentence: the artifact that reaches production is the artifact that was built and tested, not a rebuild of it.
The mechanics are worth spelling out because the details carry the guarantee.
A pull request builds the image, runs the tests, and scans it — a vulnerability gate that fails on high and critical findings — and pushes nothing. Merging to the main branch builds and pushes with an immutable tag derived from the commit, then opens a deploy pull request that updates the development image reference and merges itself. There is no tag called latest anywhere in the system, and a second push of an existing tag is rejected by the registry.
Publishing a release re-tags the same digest for staging. Production is a separate, manually started workflow that takes a version, waits for an environment approval from a required reviewer, verifies that the tag exists in the registry and prints its digest, push time and tags, and then opens a deploy pull request that does not merge itself. A person merges it. A person then syncs ArgoCD. Rolling back is the same workflow run with the previous version — which is the property that makes a rollback a procedure rather than an improvisation, the point argued in if deployment requires a hero, it isn’t automated.
What a pod is allowed to do
#These are the defaults a team would otherwise argue about for a month, and they are not per-stage: development runs what production runs.
Every pod runs as a non-root user with a fixed numeric identity, with the default seccomp profile applied. Every container has privilege escalation disabled, a read-only root filesystem, and all Linux capabilities dropped. The single writable path is a small in-memory volume mounted at the temporary directory, because the application server needs one — which is the right shape for an exception: narrow, deliberate, and documented next to the code that needs it.
Network access is default-deny. A network policy in the chart denies everything and then allows five flows: inbound application traffic from the VPC ranges, DNS to the cluster resolver, outbound to the database and cache ports, outbound HTTPS, and one easily forgotten rule — the link-local address of the pod identity agent. Miss that last one and every cloud API call from the pod fails with a credentials error that does not mention networking.
Two operational notes that will save someone a day. A network policy object is accepted by Kubernetes whether or not anything enforces it, so confirm the enforcement agent is enabled; otherwise you have policies and no policy. And if the allowed inbound range does not include the load balancer and the kubelet, the readiness probe starts failing the moment the policy takes effect.
Identity is per workload. Seven pod-level identity associations exist in the Blueprint — one each for the load balancer controller, the two DNS controllers, the secrets operator, the application, Karpenter and the cluster autoscaler — with the association held in the cluster API next to the cluster rather than in each role’s trust policy. The cluster endpoint is private, instance metadata requires session tokens, and nothing in the delivery path holds a long-lived cloud key.
What you are signing up for
#This is the section that most Kubernetes write-ups leave out, and it is the one a CTO should read twice.
Cluster upgrades are a standing obligation. Kubernetes releases move, support windows end, and an upgrade is a coordinated change across the control plane, the node images, the add-ons and the custom resource definitions that your controllers install. It is scheduled work, several times a year, forever. A team that treats it as an interruption falls behind the support window, and catching up from two versions behind is dramatically harder than staying current.
Node churn is continuous by design. A 30-day node lifetime and active consolidation mean instances are replaced constantly. That is good — it prevents long-lived nodes with drifting images — and it requires that your application tolerates being moved, which means real readiness probes, graceful shutdown and pod disruption budgets rather than hopeful ones.
A cluster is a shared blast radius. Namespaces, network policies and per-workload identity narrow it substantially, and it remains true that a control plane problem or a bad add-on upgrade affects everything on the cluster.
There is a fixed cost before any workload runs. An EKS control plane is roughly $73 a month per stage, before nodes. With three stages and a base node group, you are paying a few hundred dollars a month for the privilege of having a platform. For a large estate that is negligible. For two services, it is not.
Someone has to like it. The honest predictor of whether a small team runs Kubernetes well is whether at least one person is interested in owning it. Encoding the decisions removes the design work, not the operational relationship.
Should you choose Kubernetes at all?
#Often, no. The comparison that matters is not “Kubernetes versus nothing”, it is “Kubernetes versus the smallest thing that meets the requirement”.
| Your situation | The better choice | Why |
|---|---|---|
| One or two services, straightforward scaling, no appetite for a cluster | A managed container service with no cluster, such as a serverless container runtime | Removes cluster upgrades, node management and the controller ecosystem entirely. The trade is less scheduling control and a smaller ecosystem |
| Event-driven work with bursty, short-lived execution | Functions | No capacity to think about at all, and cost that follows use |
| A single application team who want to ship, not to operate | A managed application platform | Faster to a running service. The trade is that you inherit that platform’s opinions and its ceiling |
| Many services, real scheduling requirements, controllers you genuinely use, or a compliance need for controls you place yourself | Kubernetes | The ecosystem and the scheduling model are the product. Everything in this post is the cost of admission |
| You are already running Kubernetes badly and want it to stop hurting | Kubernetes, with the decisions remade | Migrating away and running it properly are both real projects; pick the one your team will finish |
If you are in row four and the cost of admission is the part you cannot staff, that is the gap the Blueprint is shaped to fill: the decisions arrive made, as code, in your repositories and your accounts.
Where to start
#- Write down your five platform decisions — capacity, delivery, secrets, network policy, promotion — as they exist today. If any of them is “it depends who set it up”, that is your first piece of work.
- Put a ceiling on your autoscaling. One setting, today, and it bounds the worst case.
- Check whether your network policies are enforced, not merely present.
- Stop rebuilding on promotion. Promote the digest. This is usually a one-day change and it removes a whole category of “it worked in staging”.
- Decide production’s sync policy deliberately, and write down why.
If you would rather start from a working system than a design document, the Web App Blueprint is deployed into your own accounts on top of the BuiltForProd Baseline, with the repositories handed over to you. And if the honest answer after reading this is that Kubernetes is not what you need, the BuiltForProd Assessment exists to reach that conclusion before you build, measured against the BuiltForProd Standard.