Data governance becomes enforceable at the moment permissions stop describing storage and start describing meaning. A bucket policy can say who reads a prefix. It cannot say that analysts may read the customers table without the email column. AWS Lake Formation moves the decision into the catalog, where the objects have names, columns and tags — and where a grant can outlive the table it was written for.
This post is a worked example, not a customer story: the Data and ETL Blueprint, one of the BuiltForProd Blueprints, deployed into the workload accounts of an existing foundation. It is a small system deliberately, because the governance argument is clearer when there is not much else in the picture.
Why prefix-based access stops working
#The first version of every data lake governs with IAM and bucket policies, and it works for about a year.
The failure is arithmetic. Policies are written per role per prefix, so the policy count grows with teams multiplied by datasets. Nobody can answer “who can read this table” without reading every policy in every account. There is no column, so protecting one field means physically splitting the data into two locations and keeping them in step. And revocation is a search: when a team’s access should end, somebody has to find every policy that mentioned them.
The deeper problem is that the unit of governance is wrong. Your compliance obligation is about personal data, not about s3://lake-processed/orders/year=2027/. When the control names the path and the obligation names the meaning, a translation step exists, and translation steps are maintained by hand until they are not.
| Access model | The unit you grant on | What happens when a new table appears | Column-level control |
|---|---|---|---|
| IAM and bucket policies | A prefix in a bucket | Somebody edits a policy, or the data is unreadable, or it was already covered by a wildcard nobody intended | Not possible without splitting the data |
| Lake Formation, named resources | A specific database or table | Somebody grants on the new table, every time | Yes, per table, per principal |
| Lake Formation, LF-Tags | A tag expression such as sensitivity equals restricted | It inherits the grants that match its tags, with no edit | Yes, and the column tag travels with the column |
The third row is the one that survives growth, and it is the reason to adopt Lake Formation rather than a tidier arrangement of bucket policies.
What the three zones are for
#The Blueprint’s lake has three S3 zones — raw, processed and curated — each with SSE-KMS encryption, versioning, and lifecycle rules that move older objects to infrequent access and then to Glacier storage. Each zone has its own database in the AWS Glue Data Catalog.
Three zones is not about having three copies. It is about having three different access rules that do not have to argue with each other.
| Zone | What lives there | Format | Who reads it | Why the rule differs |
|---|---|---|---|---|
| Raw | Exactly what arrived, unmodified | Source format, for example JSON | The pipeline role, and named engineers for debugging | It still contains everything the source sent, including fields nobody has classified yet |
| Processed | Cleaned, deduplicated, partitioned | Parquet | The pipeline role and data engineers | Schema is known, so sensitivity can be assigned per column |
| Curated | Aggregates and models built for consumption | Parquet | Analysts and downstream services | The audience is broad, so the data must already be safe for a broad audience |
Keeping raw immutable and unmodified is also what makes reprocessing possible: when a transformation turns out to be wrong, the fix is to run it again, not to reconstruct what the source sent six months ago.
How LF-Tags make grants outlive the tables
#The Blueprint tags catalog objects on three keys: zone, sensitivity and domain. Zone says which of the three stages an object belongs to. Sensitivity says how careful you must be with it. Domain says which part of the business owns it.
Grants are then written against tag expressions. “Data engineers may select where zone is processed.” “Analysts may select where zone is curated and sensitivity is not restricted.” Those sentences are the policy, and they are short enough to review in a meeting.
The property that matters arrives next quarter. A new table lands in the curated zone. It inherits the database’s tags, the existing grants already match it, and nobody edited anything. Compare that with named-resource grants, where the same event produces a ticket, and where the ticket not being filed looks exactly like correct behavior until an analyst reports that a dashboard is empty.
Column tags do the same job one level down. Tag the email column as restricted and every grant that excludes restricted data excludes that column, in every query, for every engine that goes through the catalog. The analyst sees the table. They do not see the column. No second copy of the table exists.
The policy that makes it real
#Lake Formation governs the catalog path. An identity with s3:GetObject on the bucket does not use the catalog path. It reads the files.
That means a data lake with beautiful column-level grants and an ordinary bucket policy has no column-level security at all — it has a suggestion. The control that closes the gap is a bucket policy that denies every principal except the Lake Formation service role and the pipeline roles. After that, there is no direct read. Analysts query through an engine that asks Lake Formation for permission and receives scoped temporary credentials for exactly the objects and columns they were granted.
Two things follow that are easy to miss. Auditing gets simpler, because access to data now produces catalog-level records that name a table and a principal instead of object-level records that name a key. And the blast radius of a leaked role shrinks, because the role’s data access is whatever Lake Formation grants it at query time rather than whatever the bucket policy allowed it in general.
Event-driven, with nothing always on
#The Blueprint’s pipeline has no cluster. An object lands under input/ in the raw zone. The event invokes a Lambda function — packaged as a container image, running in private subnets, traced with X-Ray — which starts a PySpark job on AWS Glue. The job cleans, deduplicates and partitions the data, writes Parquet into the processed zone, and a crawler updates the catalog so the new partitions are queryable. Upload a JSON file and the pipeline runs end to end. Between uploads, nothing runs.
The cost shape follows: the demo sizing sits under five dollars a month at rest, and a Glue run costs roughly forty-four cents. That is a worked example of a small pipeline, not a promise about your data volumes — your cost is a function of how much data you process and how often, and the number that matters is the one Infracost prints on your own pull request.
The governance benefit of “nothing always on” is less obvious and more durable. A long-lived cluster is an identity with standing access to the lake, a patching obligation, and a place where credentials sit. A job that exists for four minutes with a scoped role is none of those things. This is the same argument as the one in cost governance that engineers accept, arriving from the security side: the cheapest component and the most defensible component are frequently the same component.
What stays true as the lake grows
#Growth is where governance designs are actually tested. Four properties carry the weight.
- Grants reference tags, so new tables are governed on creation rather than when somebody remembers.
- The zones keep their rules because the rule is attached to the zone tag, not to a list of table names that someone maintains.
- The infrastructure is code. The Blueprint is 6 modules, 6 units and 3 workflows, with its own state and its own least-privilege deployer role, and it reads network and account facts from Parameter Store rather than duplicating what the landing zone owns. A new domain is a pull request with a plan attached — the same registration model as the golden path for new services, applied to data instead of to applications.
- The direct path stays closed. The deny policy is in the same repository as everything else, so removing it is a reviewable change rather than a console click.
The honest limits are worth stating too. Lake Formation’s column and row controls apply to engines that integrate with its credential vending; confirm support before promising it to a reviewer. Cross-account sharing works and adds real complexity. And automated classification — deciding which columns are sensitive in the first place — is a separate problem: the AWS Enterprise Baseline ships Macie as one of the switches that is off by default with its price written next to it, because the right answer depends on how much data you have and how much of it you already understand.
Where the compliance conversation lands
#An auditor asking about data access wants three things: who can read what, evidence that the restriction operated, and evidence that access changed when people changed. A tag-driven catalog answers the first as a short list of grants. Configuration history and the audit trail answer the second and third. That is the readiness half of the work — and readiness is not certification, which stays your auditor’s decision, as we set out in what your infrastructure has to produce for SOC 2 and on the compliance page.
If you are standing up a lake, do these three things first, in this order, whether or not you ever talk to us:
- Put the deny-direct-S3 policy in place before there is data worth reading. Retrofitting it means finding every job that reads the bucket directly, and every one you miss is an outage.
- Decide your tag keys now. Three is a good number. Zone, sensitivity and domain have earned their place.
- Write the grants as tag expressions from the first table, even when there is only one table and a named grant would be simpler.
If you would rather start from a working system than a design document, the Data and ETL Blueprint is that system, deployed into your own accounts on top of the BuiltForProd Baseline, with the repositories handed over to you.