Skip to content
On this page 7

Data platform governance with Lake Formation

A three-zone lake, LF-Tags and a deny-direct-S3 policy turn data governance into something enforced at query time instead of described in a spreadsheet.

Data governance becomes enforceable at the moment permissions stop describing storage and start describing meaning. A bucket policy can say who reads a prefix. It cannot say that analysts may read the customers table without the email column. AWS Lake Formation moves the decision into the catalog, where the objects have names, columns and tags — and where a grant can outlive the table it was written for.

This post is a worked example, not a customer story: the Data and ETL Blueprint, one of the BuiltForProd Blueprints, deployed into the workload accounts of an existing foundation. It is a small system deliberately, because the governance argument is clearer when there is not much else in the picture.

Why prefix-based access stops working

#

The first version of every data lake governs with IAM and bucket policies, and it works for about a year.

The failure is arithmetic. Policies are written per role per prefix, so the policy count grows with teams multiplied by datasets. Nobody can answer “who can read this table” without reading every policy in every account. There is no column, so protecting one field means physically splitting the data into two locations and keeping them in step. And revocation is a search: when a team’s access should end, somebody has to find every policy that mentioned them.

The deeper problem is that the unit of governance is wrong. Your compliance obligation is about personal data, not about s3://lake-processed/orders/year=2027/. When the control names the path and the obligation names the meaning, a translation step exists, and translation steps are maintained by hand until they are not.

Access modelThe unit you grant onWhat happens when a new table appearsColumn-level control
IAM and bucket policiesA prefix in a bucketSomebody edits a policy, or the data is unreadable, or it was already covered by a wildcard nobody intendedNot possible without splitting the data
Lake Formation, named resourcesA specific database or tableSomebody grants on the new table, every timeYes, per table, per principal
Lake Formation, LF-TagsA tag expression such as sensitivity equals restrictedIt inherits the grants that match its tags, with no editYes, and the column tag travels with the column

The third row is the one that survives growth, and it is the reason to adopt Lake Formation rather than a tidier arrangement of bucket policies.

What the three zones are for

#

The Blueprint’s lake has three S3 zones — raw, processed and curated — each with SSE-KMS encryption, versioning, and lifecycle rules that move older objects to infrequent access and then to Glacier storage. Each zone has its own database in the AWS Glue Data Catalog.

Three zones is not about having three copies. It is about having three different access rules that do not have to argue with each other.

ZoneWhat lives thereFormatWho reads itWhy the rule differs
RawExactly what arrived, unmodifiedSource format, for example JSONThe pipeline role, and named engineers for debuggingIt still contains everything the source sent, including fields nobody has classified yet
ProcessedCleaned, deduplicated, partitionedParquetThe pipeline role and data engineersSchema is known, so sensitivity can be assigned per column
CuratedAggregates and models built for consumptionParquetAnalysts and downstream servicesThe audience is broad, so the data must already be safe for a broad audience

Keeping raw immutable and unmodified is also what makes reprocessing possible: when a transformation turns out to be wrong, the fix is to run it again, not to reconstruct what the source sent six months ago.

How LF-Tags make grants outlive the tables

#

The Blueprint tags catalog objects on three keys: zone, sensitivity and domain. Zone says which of the three stages an object belongs to. Sensitivity says how careful you must be with it. Domain says which part of the business owns it.

Grants are then written against tag expressions. “Data engineers may select where zone is processed.” “Analysts may select where zone is curated and sensitivity is not restricted.” Those sentences are the policy, and they are short enough to review in a meeting.

The property that matters arrives next quarter. A new table lands in the curated zone. It inherits the database’s tags, the existing grants already match it, and nobody edited anything. Compare that with named-resource grants, where the same event produces a ticket, and where the ticket not being filed looks exactly like correct behavior until an analyst reports that a dashboard is empty.

Column tags do the same job one level down. Tag the email column as restricted and every grant that excludes restricted data excludes that column, in every query, for every engine that goes through the catalog. The analyst sees the table. They do not see the column. No second copy of the table exists.

The policy that makes it real

#

Lake Formation governs the catalog path. An identity with s3:GetObject on the bucket does not use the catalog path. It reads the files.

That means a data lake with beautiful column-level grants and an ordinary bucket policy has no column-level security at all — it has a suggestion. The control that closes the gap is a bucket policy that denies every principal except the Lake Formation service role and the pipeline roles. After that, there is no direct read. Analysts query through an engine that asks Lake Formation for permission and receives scoped temporary credentials for exactly the objects and columns they were granted.

Figure 1
The governed access path and the denied direct path

An analyst’s query goes to a query engine, which asks Lake Formation to check the grant against LF-Tags in the Glue Data Catalog. Lake Formation issues scoped temporary credentials and the engine reads only the permitted columns from the S3 zone. A separate arrow from the analyst directly to the bucket is shown as denied by the bucket policy.

analystSSO rolequeryquery enginecatalog pathcheck grantLake FormationLF-Tag grantsGlue Data Catalogscoped credentialsS3 zoneraw · processed · curateddenied by bucket policy
Lake Formation vends scoped credentials only after checking the grant. The bucket policy denies every other principal, so the governed path is the only path to the data.

Two things follow that are easy to miss. Auditing gets simpler, because access to data now produces catalog-level records that name a table and a principal instead of object-level records that name a key. And the blast radius of a leaked role shrinks, because the role’s data access is whatever Lake Formation grants it at query time rather than whatever the bucket policy allowed it in general.

Event-driven, with nothing always on

#

The Blueprint’s pipeline has no cluster. An object lands under input/ in the raw zone. The event invokes a Lambda function — packaged as a container image, running in private subnets, traced with X-Ray — which starts a PySpark job on AWS Glue. The job cleans, deduplicates and partitions the data, writes Parquet into the processed zone, and a crawler updates the catalog so the new partitions are queryable. Upload a JSON file and the pipeline runs end to end. Between uploads, nothing runs.

The cost shape follows: the demo sizing sits under five dollars a month at rest, and a Glue run costs roughly forty-four cents. That is a worked example of a small pipeline, not a promise about your data volumes — your cost is a function of how much data you process and how often, and the number that matters is the one Infracost prints on your own pull request.

The governance benefit of “nothing always on” is less obvious and more durable. A long-lived cluster is an identity with standing access to the lake, a patching obligation, and a place where credentials sit. A job that exists for four minutes with a scoped role is none of those things. This is the same argument as the one in cost governance that engineers accept, arriving from the security side: the cheapest component and the most defensible component are frequently the same component.

What stays true as the lake grows

#

Growth is where governance designs are actually tested. Four properties carry the weight.

  1. Grants reference tags, so new tables are governed on creation rather than when somebody remembers.
  2. The zones keep their rules because the rule is attached to the zone tag, not to a list of table names that someone maintains.
  3. The infrastructure is code. The Blueprint is 6 modules, 6 units and 3 workflows, with its own state and its own least-privilege deployer role, and it reads network and account facts from Parameter Store rather than duplicating what the landing zone owns. A new domain is a pull request with a plan attached — the same registration model as the golden path for new services, applied to data instead of to applications.
  4. The direct path stays closed. The deny policy is in the same repository as everything else, so removing it is a reviewable change rather than a console click.

The honest limits are worth stating too. Lake Formation’s column and row controls apply to engines that integrate with its credential vending; confirm support before promising it to a reviewer. Cross-account sharing works and adds real complexity. And automated classification — deciding which columns are sensitive in the first place — is a separate problem: the AWS Enterprise Baseline ships Macie as one of the switches that is off by default with its price written next to it, because the right answer depends on how much data you have and how much of it you already understand.

Where the compliance conversation lands

#

An auditor asking about data access wants three things: who can read what, evidence that the restriction operated, and evidence that access changed when people changed. A tag-driven catalog answers the first as a short list of grants. Configuration history and the audit trail answer the second and third. That is the readiness half of the work — and readiness is not certification, which stays your auditor’s decision, as we set out in what your infrastructure has to produce for SOC 2 and on the compliance page.

If you are standing up a lake, do these three things first, in this order, whether or not you ever talk to us:

  1. Put the deny-direct-S3 policy in place before there is data worth reading. Retrofitting it means finding every job that reads the bucket directly, and every one you miss is an outage.
  2. Decide your tag keys now. Three is a good number. Zone, sensitivity and domain have earned their place.
  3. Write the grants as tag expressions from the first table, even when there is only one table and a named grant would be simpler.

If you would rather start from a working system than a design document, the Data and ETL Blueprint is that system, deployed into your own accounts on top of the BuiltForProd Baseline, with the repositories handed over to you.

Questions

What does AWS Lake Formation actually enforce?

Lake Formation holds permissions against the Glue Data Catalog — databases, tables, columns and rows — and mediates access for engines that integrate with it, such as Athena, Glue and Redshift Spectrum. When a query runs, Lake Formation checks the grant and issues scoped temporary credentials for only the underlying data the caller is allowed to read.

What is an LF-Tag and why use tags instead of table names?

An LF-Tag is a key and value attached to a catalog object, for example a zone, a sensitivity level or a business domain. Grants are written against tag expressions rather than named tables, so a table created next quarter that carries the same tags inherits the existing grants automatically. Named-resource grants require an edit for every new table, which is the mechanism that quietly stops being maintained.

Can someone bypass Lake Formation by reading the S3 bucket directly?

Yes, unless you stop it. Lake Formation governs the catalog path, so any identity with direct s3:GetObject on the bucket reads the files without a permission check. The fix is a bucket policy that denies every principal except the Lake Formation service role and the pipeline roles, which forces all human and analytical access through the governed path.

Do we need an always-on cluster for a data lake?

No. An object landing in a bucket can raise an event, invoke a function, and start a serverless job that writes partitioned columnar output and updates the catalog. Nothing runs between uploads. That removes the largest fixed cost in most data platforms and also removes a long-lived compute surface that would otherwise need patching and access control.

Does column-level security work for every query engine?

It works for the engines that integrate with Lake Formation credential vending. An engine reading the files directly with its own credentials sees whatever the file contains, which is why the deny-direct-S3 policy matters and why you should confirm integration support before promising column-level protection to a compliance reviewer.

This is what we build.

BuiltForProd deploys a production foundation into your own cloud accounts, hands over the repositories and the documentation, and, if you want it, keeps running it with your team. Tell us what you are running and what is coming next. The first call is 30 minutes, with an engineer who scopes engagements, not a salesperson.

The calendar invite and the video link go to this address. Prefer to write? Send a message instead.

An EKS platform without a platform team

A worked EKS golden path: a managed node group with Karpenter, ArgoCD, External Secrets, promotion by approval, and what Kubernetes costs you to operate.

Cost governance that engineers accept

Cloud cost is an architectural property, not a monthly report. Tags that hold, priced switches, endpoint arithmetic and a cost comment on every pull request.

The golden path for new services

A golden path turns a platform team from a ticket queue into a paved road: what it must include, the new-repository flow, and the 18 pitfalls to design out.