Skip to content
On this page 8

An EKS platform without a platform team

A worked EKS golden path: a managed node group with Karpenter, ArgoCD, External Secrets, promotion by approval, and what Kubernetes costs you to operate.

An EKS platform is not a cluster. It is the set of decisions that have to be made before application teams can use a cluster safely: where capacity comes from, how changes reach it, how secrets arrive, what a pod is allowed to do, and what has to happen before something runs in production. A platform team is the usual way those decisions get made. A well-built golden path is the other way.

This post is a worked example, not a customer story: the Web App Blueprint, one of the BuiltForProd Blueprints, deployed into the workload accounts of an existing foundation. Seventeen AWS services and eight Kubernetes components per stage, as 14 modules, 16 units and five application workflows, written in OpenTofu and Terragrunt like everything else in the foundation. Its sibling, the Data and ETL Blueprint, makes the opposite trade — no cluster at all — and the comparison at the end of this post is partly about when that is the better answer. The interesting part is not the component list — it is which decision each component represents, and the honest accounting at the end of what you take on by choosing Kubernetes at all.

Where does capacity come from?

#

Two autoscalers, with separate jobs. This is the arrangement that keeps working, and the reasoning is worth having.

A small managed node group provides a predictable base — in the Blueprint, t3.medium instances across three private availability zones, two to three nodes in development and three to four in staging and production. It carries a taint so that ordinary application pods do not schedule onto it, which leaves it for the controllers: the load balancer controller, the GitOps controller, the secrets operator, the DNS controller, and Karpenter itself. The application chart deliberately sets no tolerations, and that absence is the mechanism rather than an oversight.

Karpenter provisions everything else. It watches for pods that cannot be scheduled and launches nodes shaped for them, rather than adding another copy of whatever instance type was chosen a year ago.

SettingWhat it is in the BlueprintWhy it is set that way
Instance familiesA short list of current general-purpose familiesA longer list usually lowers cost, because it gives the provisioner more offerings to price against. Anything not listed, it cannot buy
Capacity typesSpot and on-demandIt takes spot whenever spot capacity exists for a type that fits, and falls back to on-demand. Dropping spot gives an all-on-demand pool
Node lifetime30 daysNodes are replaced on a schedule, so the cluster never accumulates long-lived instances with drifting images
ConsolidationAfter one minute of a node being under-usedRepacking recovers the capacity that scaling up leaves behind
Disruption budgetTen percent of the pool at a timeReplacement is continuous, and it should never look like an outage
vCPU ceiling32 in development, 64 in staging, 128 in productionThe one setting that bounds the bill. At the ceiling, provisioning stops and excess pods stay pending

The ceiling deserves its own sentence. An alert tells you that spend passed a threshold, after it passed. A ceiling means a misconfigured autoscaler, a retry storm or a bad rollout scales to a bounded number of nodes and then stops, at three in the morning, without anyone deciding anything. That is the difference between a notification and a control, and it is the same argument made at length in cost governance that engineers accept.

How do changes reach the cluster?

#

Through Git, and through exactly one path.

ArgoCD watches a GitOps repository and reconciles the cluster to it. The application definition itself is created by the infrastructure code rather than by a manifest, so adding a new application is a pull request against the platform repository — the registration model described in the golden path for new services.

The decision that matters is the sync policy, and it is different per stage.

  • Development and staging sync automatically, with pruning and self-healing on. Merge, and the cluster follows. Speed is the point.
  • Production does not. There is no automation block, so ArgoCD reports that the application is out of date and applies nothing until a person presses sync.

That asymmetry is deliberate and it is worth defending against the obvious objection. Self-healing production sounds strictly better: the cluster returns to the declared state on its own. In practice it means an emergency change made during an incident is silently reverted by a controller while the incident is still running, and it means the approval you configured earlier is not actually a gate, because the merge is the deploy. Leaving production manual makes the approval real.

It also has a consequence you should accept knowingly: because production does not self-heal, a change made directly in the production cluster stays there until someone notices. That is drift, and drift is a fact rather than a failure — detect it on a schedule and triage it deliberately.

How do secrets reach a pod?

#

Not through Git, and not through a pipeline variable.

Infrastructure code creates the database and the cache and writes their connection details to the cloud parameter store, encrypted. The application chart declares which parameters it needs. The External Secrets Operator, authenticating with a pod-level cloud identity rather than a stored key, reads those parameters and writes a Kubernetes secret that the application mounts.

Two properties fall out of that arrangement. Nothing in Git holds a secret value, and no pipeline ever passes one. And the operator’s permissions are scoped by path — the development operator’s role can read development parameters and nothing else, so an accidental request for a production parameter fails rather than succeeds quietly.

There is a matching discipline on the infrastructure side. The database password is set through a write-only argument, so the value is sent to the API and never returned into state — one fewer credential in the state file, which is a place most teams forget holds secrets at all.

Build once, promote by approval

#

The delivery rule is one sentence: the artifact that reaches production is the artifact that was built and tested, not a rebuild of it.

Figure 1
Build once, promote the same image by approval

A pull request builds, tests and scans an image but pushes nothing. A merge to main pushes the image to an immutable registry and deploys it to development automatically. Publishing a release promotes the same digest to staging. Production requires an environment approval, a merged deploy pull request and a manual sync.

pull requestbuild · test · scanmergeregistryimmutable tagsdevauto-syncreleasestagingauto-syncproduction: three human decisions, in order1. approve the runenvironment reviewer2. merge the deploy PRdigest verified first3. sync in ArgoCDproduction never self-heals
One image, built once, promoted by approval. A pull request builds and scans without pushing; merge deploys to development; a release promotes to staging; production requires three human decisions in sequence.

The mechanics are worth spelling out because the details carry the guarantee.

A pull request builds the image, runs the tests, and scans it — a vulnerability gate that fails on high and critical findings — and pushes nothing. Merging to the main branch builds and pushes with an immutable tag derived from the commit, then opens a deploy pull request that updates the development image reference and merges itself. There is no tag called latest anywhere in the system, and a second push of an existing tag is rejected by the registry.

Publishing a release re-tags the same digest for staging. Production is a separate, manually started workflow that takes a version, waits for an environment approval from a required reviewer, verifies that the tag exists in the registry and prints its digest, push time and tags, and then opens a deploy pull request that does not merge itself. A person merges it. A person then syncs ArgoCD. Rolling back is the same workflow run with the previous version — which is the property that makes a rollback a procedure rather than an improvisation, the point argued in if deployment requires a hero, it isn’t automated.

What a pod is allowed to do

#

These are the defaults a team would otherwise argue about for a month, and they are not per-stage: development runs what production runs.

Every pod runs as a non-root user with a fixed numeric identity, with the default seccomp profile applied. Every container has privilege escalation disabled, a read-only root filesystem, and all Linux capabilities dropped. The single writable path is a small in-memory volume mounted at the temporary directory, because the application server needs one — which is the right shape for an exception: narrow, deliberate, and documented next to the code that needs it.

Network access is default-deny. A network policy in the chart denies everything and then allows five flows: inbound application traffic from the VPC ranges, DNS to the cluster resolver, outbound to the database and cache ports, outbound HTTPS, and one easily forgotten rule — the link-local address of the pod identity agent. Miss that last one and every cloud API call from the pod fails with a credentials error that does not mention networking.

Two operational notes that will save someone a day. A network policy object is accepted by Kubernetes whether or not anything enforces it, so confirm the enforcement agent is enabled; otherwise you have policies and no policy. And if the allowed inbound range does not include the load balancer and the kubelet, the readiness probe starts failing the moment the policy takes effect.

Identity is per workload. Seven pod-level identity associations exist in the Blueprint — one each for the load balancer controller, the two DNS controllers, the secrets operator, the application, Karpenter and the cluster autoscaler — with the association held in the cluster API next to the cluster rather than in each role’s trust policy. The cluster endpoint is private, instance metadata requires session tokens, and nothing in the delivery path holds a long-lived cloud key.

What you are signing up for

#

This is the section that most Kubernetes write-ups leave out, and it is the one a CTO should read twice.

Cluster upgrades are a standing obligation. Kubernetes releases move, support windows end, and an upgrade is a coordinated change across the control plane, the node images, the add-ons and the custom resource definitions that your controllers install. It is scheduled work, several times a year, forever. A team that treats it as an interruption falls behind the support window, and catching up from two versions behind is dramatically harder than staying current.

Node churn is continuous by design. A 30-day node lifetime and active consolidation mean instances are replaced constantly. That is good — it prevents long-lived nodes with drifting images — and it requires that your application tolerates being moved, which means real readiness probes, graceful shutdown and pod disruption budgets rather than hopeful ones.

A cluster is a shared blast radius. Namespaces, network policies and per-workload identity narrow it substantially, and it remains true that a control plane problem or a bad add-on upgrade affects everything on the cluster.

There is a fixed cost before any workload runs. An EKS control plane is roughly $73 a month per stage, before nodes. With three stages and a base node group, you are paying a few hundred dollars a month for the privilege of having a platform. For a large estate that is negligible. For two services, it is not.

Someone has to like it. The honest predictor of whether a small team runs Kubernetes well is whether at least one person is interested in owning it. Encoding the decisions removes the design work, not the operational relationship.

Should you choose Kubernetes at all?

#

Often, no. The comparison that matters is not “Kubernetes versus nothing”, it is “Kubernetes versus the smallest thing that meets the requirement”.

Your situationThe better choiceWhy
One or two services, straightforward scaling, no appetite for a clusterA managed container service with no cluster, such as a serverless container runtimeRemoves cluster upgrades, node management and the controller ecosystem entirely. The trade is less scheduling control and a smaller ecosystem
Event-driven work with bursty, short-lived executionFunctionsNo capacity to think about at all, and cost that follows use
A single application team who want to ship, not to operateA managed application platformFaster to a running service. The trade is that you inherit that platform’s opinions and its ceiling
Many services, real scheduling requirements, controllers you genuinely use, or a compliance need for controls you place yourselfKubernetesThe ecosystem and the scheduling model are the product. Everything in this post is the cost of admission
You are already running Kubernetes badly and want it to stop hurtingKubernetes, with the decisions remadeMigrating away and running it properly are both real projects; pick the one your team will finish

If you are in row four and the cost of admission is the part you cannot staff, that is the gap the Blueprint is shaped to fill: the decisions arrive made, as code, in your repositories and your accounts.

Where to start

#
  1. Write down your five platform decisions — capacity, delivery, secrets, network policy, promotion — as they exist today. If any of them is “it depends who set it up”, that is your first piece of work.
  2. Put a ceiling on your autoscaling. One setting, today, and it bounds the worst case.
  3. Check whether your network policies are enforced, not merely present.
  4. Stop rebuilding on promotion. Promote the digest. This is usually a one-day change and it removes a whole category of “it worked in staging”.
  5. Decide production’s sync policy deliberately, and write down why.

If you would rather start from a working system than a design document, the Web App Blueprint is deployed into your own accounts on top of the BuiltForProd Baseline, with the repositories handed over to you. And if the honest answer after reading this is that Kubernetes is not what you need, the BuiltForProd Assessment exists to reach that conclusion before you build, measured against the BuiltForProd Standard.

Questions

Can a small team run Amazon EKS responsibly?

Yes, if the platform decisions are made once and encoded rather than made repeatedly under pressure. The recurring work is real: cluster version upgrades, add-on and CRD compatibility, node image churn and the delivery tooling. A team that treats those as scheduled maintenance can run a cluster; a team that treats them as interruptions will fall behind the support window.

Why run both a managed node group and Karpenter?

They have different jobs. The managed node group is a small, predictable base that carries the controllers — the ingress controller, the GitOps controller, the secrets operator, and Karpenter itself — usually behind a taint so application pods do not land there. Karpenter provisions everything else, shaped to the pods that are actually pending. Neither autoscaler runs on capacity the other could take away.

Should ArgoCD auto-sync production?

No. Automatic sync with self-healing is right for development and staging, where speed matters more than ceremony. In production, leaving out the automation block means the application reports that it is out of date and applies nothing until a person syncs it, which is what turns the approval into a real gate rather than a formality.

How do secrets reach pods without appearing in Git or a pipeline?

Infrastructure code writes the values to a parameter store; the application chart declares which parameters it needs; the External Secrets Operator, authenticating with a pod-level cloud identity, reads them and writes a Kubernetes secret. Git holds the names, never the values, and no pipeline ever handles the secret.

When is Kubernetes the wrong choice?

When you have one or two services, no need for custom controllers or complex scheduling, and no team member who wants to own a cluster. A managed container service with no cluster to upgrade removes an entire category of recurring work. Kubernetes earns its cost when you have many services, real scheduling requirements, or an ecosystem of controllers you genuinely use.

This is what we build.

BuiltForProd deploys a production foundation into your own cloud accounts, hands over the repositories and the documentation, and, if you want it, keeps running it with your team. Tell us what you are running and what is coming next. The first call is 30 minutes, with an engineer who scopes engagements, not a salesperson.

The calendar invite and the video link go to this address. Prefer to write? Send a message instead.

The golden path for new services

A golden path turns a platform team from a ticket queue into a paved road: what it must include, the new-repository flow, and the 18 pitfalls to design out.

Cost governance that engineers accept

Cloud cost is an architectural property, not a monthly report. Tags that hold, priced switches, endpoint arithmetic and a cost comment on every pull request.