So when a pipeline owner leaves and three engineers spend a Tuesday reverse-engineering what stg_orders.status_v2column actually means, the lesson everyone draws is the wrong one: we need to care more about documentation. You will add it to the Definition of Done. And six weeks later you will have a table where every column description reads like customer_id — the id of the customer.
That’s not a failure of culture. That’s a system working exactly as designed. If you make documentation expensive to produce and give the producer no immediate return, you get the cheapest legal compliance your engineers can get away with.
Stop Asking People to Care
Declaring “we’re a documentation-first team now” changes what people say, not what they do. A mandate may increase documentation coverage, but it often produces the bare minimum: descriptions that satisfy the rule without helping anyone understand the system. Documentation for documentation’s sake.
The wiki is the classic trap. It lives in a different system from the data it describes, so it’s never open when you’re writing the transform, it’s never reviewed when you’re approving the PR, and it rots silently because nothing in your platform knows it’s wrong. And the quarterly documentation sprint produces a beautiful snapshot that is stale before the sprint retro is over.
These all share one assumption: that documentation is something engineers write. As long as that’s true, you’re fighting day-to-day business incentives with a backlog ticket, and the business incentives always win.
So flip it.
Documentation should be something the work produces, not something a person writes. A byproduct of shipping, not a task bolted onto it. The job of a platform isn’t to convince engineers to author docs, but to make documentation fall out of the code they were already going to commit. Get the cost of producing it close enough to zero and the willingness question disappears, because there’s nothing left to be unwilling about.
The Docs Don’t Drift Because Anyone Got Lazy
Now suppose your team does write the documentation. Real intent, the why and the what, the good stuff. Six months later it’s wrong anyway, and again, nobody did anything wrong to make it wrong.
Processes evolve continuously. A transform grows a new branch. A column quietly changes meaning. A join gets added, a status value is repurposed, a “temporary” filter becomes persistent. Each of those changes ships through code review on its own merits. The logic is sound, the tests pass, the PR gets approved. Meanwhile the description of that process is sitting in a different system, three folders deep in a wiki, on a clock of its own, and nothing in the platform so much as points at it when the code moves. The pipeline changed. The doc didn’t. The gap between them only ever widens, and it never closes on its own.
Then we call the result “stale documentation” and quietly file it under developer negligence. It isn’t negligence. It’s the guaranteed output of an architecture that keeps a process in one place and the description of that process somewhere else entirely, with no mechanical link between the two. You built two artifacts in two systems and connected them with a human’s memory. Memory will always loose. Asking an engineer to keep two uncoupled systems in sync by hand and then blaming them when they drift apart is blaming the person for a decision the data platform made.
Take the stg_orders.status_v2 column from the opening. The platform can automatically record its source, lineage and freshness. Its description lives next to the model, where an LLM can draft it and the owner can confirm it. CI can make sure the description exists before the model appears in the catalog. When the column changes, its code and description change in the same pull request.
That is how you prevent documentation from drifting: keep it next to the work it describes. It will not guarantee that every description is correct, but it makes missing or outdated information easier to spot during review.
There are four moves here, and the cheapest one doesn’t involve a developer at all.
1. Make the Platform Document Itself
Some documentation should never be written by a person. It should be emitted by the platform as a side effect of running, the way a log line is.
The cleanest example is one you may already have: the technical columns a good ingestion layer stamps onto every row the moment a source is onboarded. Columns like _ingested_at, _source_system, _source_file, _batch_id, a row hash, or whatever your platform standardizes on. Nobody writes them, nobody maintains them. They mean the same across the whole platform and they’re added by convention when the source comes online and written by the pipeline on every run, which means they document where a row came from, when it landed, and how to replay it.
Then take it one step ahead. Generate lineage from SQL, capture freshness and volume on every run, snapshot schemas at load time and read ownership from CODEOWNERS. The platform already knows these facts. Surface them instead of asking people to rewrite them.
The leverage point is onboarding. When a new source or model is created the scaffolding should already be there: the metadata columns wired in, the description stubs in place, pre-filled with everything the platform can infer, with only the genuine blanks left for the human. You document at creation, while the context is in your head, instead of six months later, in an incident, from memory you no longer have.
So before any of the human moves: emit everything a machine already knows. Whatever’s left over is the only thing worth a person’s attention.
2. Move the Docs to Where the Work Already Lives
Documentation about data belongs next to the definition of that data – in the model, the transform, the pipeline spec, the contract. Not in a separate place a human has to remember to open and update.
The mechanics vary by stack and the tool genuinely doesn’t matter: a description block in a dbt model, a field comment on a data contract, an annotation on a pipeline task, a docstring on the asset. Pick whatever your platform already version-controls. What matters is that the description ships in the same diff as the schema it describes.
The moment docs live in the pull request, three things become much cheaper. They can be reviewed alongside the code, by the same person, in the same pass. They version with the data, so a column rename and its new meaning can land together instead of drifting apart. And fixing a wrong description becomes a one-line diff someone can make in the moment, not a Confluence expedition they’ll never schedule. Co-location is the core of the design. A wiki is documentation physically separated from the thing it describes and then asked to stay accurate; it rarely does so reliably. A description travelling in the same diff is harder to overlook and easier to challenge.
And you only pay for each description once. A column described where it is first defined – in staging, say – should inherit that description into every downstream model that selects it (without a transform), instead of being re-typed by hand at every hop. Describe order_status once at the edge and it carries its meaning all the way to the mart. This matters more than it looks: the cheapest column to document is the one whose description you wrote four layers upstream and never touched again, and the less surface you have to cover, the more of it actually ends up covered. Most “low documentation coverage” is really the same fact written a hundred times in a hundred places, none of them linked.
A wiki can still have a job, just not as a second handwritten copy of the code. Cross-team specifications, product context and decisions spanning several systems may belong there because their audience is wider. An ADR or implementation spec tied to one codebase usually belongs beside that code, where it can be versioned and reviewed with the change it governs. The boundary is not “wiki bad, repository good”; it is lifecycle and ownership. Put a document where the people responsible for keeping it true already work, choose one canonical source, and make every other surface a link or generated projection of it.
3. Let the Machine Write the First Draft
Point an LLM at the transformation logic, the upstream column descriptions, a sample of real values, and the commit message, and it will propose a description for every column and every model, plus a plain-English summary of what the pipeline does. The developer’s job changes from authoring to reviewing. Reviewing is cheaper. You haven’t asked the engineers to do more work. You’ve replaced their writing work with more tolerable reviewing one.
But there’s a line you have to draw carefully, because not every sentence produced by a machine is accurate.
Some documentation is observed: schema, lineage, runtime, freshness, test results. Publish those facts directly from the systems that measure them; no LLM is needed. Some is inferred: this expression casts epoch-millis to UTC; this model left-joins the dimension table. A model can draft that explanation and point to the code or runtime evidence behind each claim. And some is asserted: the meaning the team intends consumers to rely on. Even a sentence that sounds mechanical may contain business knowledge the code cannot prove. The machine can draft, but evidence and reviewer should own the truth.
LLMs cannot be trusted to invent the why. We keep soft-deleted rows because finance reconciles against them every quarter. This filter exists because of one bad vendor in 2022. This status code is technically wrong but downstream depends on it, so don’t fix it. The why is institutional knowledge. It is often absent from the code. It’s the reason the code looks the way it does and it’s precisely what walks out the door when someone leaves your company. Capturing and validating it is the most valuable use of your teammates’ attention.
Here’s the part people miss, though: the why is often already written down somewhere. It’s in the PR description, in the commit message or in the ticket description. It’s in the review thread where someone asked “wait, why this filter?” and got a straight answer that then evaporated into Git history. The same model that drafts the what can harvest candidate explanations from decisions your team already recorded, instead of making them explain everything again to a wiki nobody will read. But provenance matters: show the PR, JIRA ticket or ADR behind the claim. If sources conflict or no evidence exists, the system should ask a person rather than manufacture an answer. Don’t ask people to repeat the why. Ask them to confirm the why the machine found in a source they can inspect.
So the system publishes observed facts, drafts what it can infer and spends focused human judgment on intent.
One warning: automation that’s too helpful stops being trustworthy. An unreviewed AI description is a just confident guess. A catalog full of auto-generated descriptions nobody checked is worse than an empty one, because an empty field tells you you don’t know something, while a plausible wrong field tells you a lie you’ll act on during an incident. Generation without review doesn’t give you documentation. It scales your wrongness and dresses it up. The review step is not the part you optimize away to save time. It is the entire point.
4. Enforce Presence and Consistency, Not Prose Quality
CI can block a pull request that adds a column without a description. That check is cheap, deterministic, and unarguable. Do it.
But the most effective gate isn’t the one on the commit. It’s the one on the thing the developer actually wants. An undocumented model doesn’t get promoted into the gold layer. An undocumented source doesn’t appear in the catalog the analysts browse. A gate on a property the platform can mechanically check, attached to the outcome the author cares about. People will happily skip documentation that only protects you. They will not skip the documentation that stands between their model and the people who are supposed to use it.
Do not ask a linter to grade semantic quality. Shallow prose rules only define the cheapest passing answer: TODO, see above, or the column name rewritten as a sentence. Enforce what the platform can prove, e.g. broken links, missing owners and schema contradictions, naming conventions violations and leave meaning to human review.
Semantic quality comes primarily from two places instead. It comes from the review step in move two – a description checked by the person accountable when it’s wrong is more likely to be a description that’s right. And it comes from a quieter economic fact: if the AI draft is good, accepting or correcting it is less effort than cheating it. Typing a fake placeholder is now more work than doing the useful thing. That’s the win condition for any process aimed at humans under deadline: make the correct path the easy path. Stop relying on people choosing extra work, and arrange things so the useful outcome falls out of the workflow they already follow.
And where you can, skip the prose altogether and make the documentation executable. A description that claims a column is never null, backed by a not-null test, is either automatically true or automatically loud. A data contract is a doc that breaks the build the moment reality stops matching it. The closer a piece of documentation sits to being a test, the less room it has to lie quietly. The parts of your docs that can be expressed as a check should be checks.Prose is what you fall back to for the things you can’t yet enforce, not the default for things you can.
The Pipeline Is an Asset Too
Everything so far has been about the data. But the thing that produces the data is an asset in its own right, and during an incident it’s usually the one you’re desperate to understand rather than the data. A column description tells you what a value means. It tells you nothing about why the table is six hours stale, what it depends on, or how to safely re-run it at 2 AM. That’s pipeline documentation, and most platforms have even less of it than they have of the data kind.
The same four moves apply, because the orchestrator knows more about a pipeline than the warehouse knows about a column.
The first move in this article emits most of it. The dependency graph is already in your DAG. The schedule is in the config. Last run, duration, success rate and output freshness are recorded on every execution. Ownership should come from a declared field such as CODEOWNERS, commit history may suggest experts, but it cannot assign pager duty. Move two keeps the pipeline description and runbook beside the DAG or contract, so an operational change and its explanation travel through the same pull request. Move three drafts the rest of the mechanical account from task definitions and config – reads X from Y, transforms through Z, lands in W, runs hourly. That provides a complete lineage, use it. The recoverable what of a pipeline should be cheaper to review than to write.
Then the why, which is where pipelines keep their worst surprises, because operational knowledge is almost entirely why. Why does this run at 04:00 and not on the hour? Why is the backfill window seven days? Why is the retry count so high? What actually breaks this thing, how do you notice, and what’s the runbook to recover? Is it idempotent, or will a careless re-run double-count the day? None of that is in the DAG, and all of it is what turns a 3 AM page into ten minutes instead of three hours of searching for answers. Same rule as before: the machine can’t invent it, so harvest it from the incident write-ups and review threads where it was already said once, and ask a human only to confirm.
And enforce it the same way. No production-critical pipeline reaches production without an owner, an SLA, and a recovery path proportionate to its risk. Make it gated, not requested, and its operational promises will become executable. A low-risk internal job may need less ceremony than a revenue-critical feed, but neither should depend on tribal memory. “Lands by 06:00 AM” backed by a freshness check, is documentation that pages you the moment it stops being true, instead of quietly lying until someone three layers downstream notices for you.
Close the Loop, or It All Rots Anyway
Documentation decays because systems move. If nobody encounters it in the flow of work, that drift creates no feedback and the wrong description survives. So surface documentation where people already look.
And generate those surfaces; never maintain them as separate sources. The data catalog, the lineage graph and the docs site are projections of descriptions and facts held in their canonical systems, rendered fresh on every build. Nobody hand-updates the portal, because the portal is a view, not a second source of truth. The moment a human keeps another copy of the documentation in sync by hand, you have rebuilt the exact separation this whole piece exists to delete: two artifacts, two clocks, predictable drift. One fact, one canonical source, many views.
The first time an engineer hits a wrong description during an incident and can correct it with a one-line PR on the spot, the docs become a living thing instead of an archive. Decay is the default state. A short, in-the-flow correction path is the only thing that fights it.
Good documentation should let someone use an asset correctly, change it safely and recover it when it failswithout reconstructing its history from Slack and old pull requests. For a dataset or a pipeline, it should answer:
- What is this for, and who relies on it?
- What does one record, output or run actually represent?
- Where do the inputs come from, and what happens to them before they become the output?
- What guarantees can consumers rely on – schema, grain, freshness, completeness, quality – and what is explicitly not guaranteed?
- Which non-obvious decisions, exceptions or constraints shape its behaviour, and why do they exist?
- What depends on it, what counts as a breaking change, and how can it be changed safely?
- How do we know when it is wrong, who owns the response, and how do we recover or reprocess it?
A column’s null semantics, a pipeline’s retry policy and a dataset’s freshness target are different answers to the same underlying questions: what does this promise, where can that promise fail, and what must a person know before trusting or changing it?
If your documentation can’t answer those, you don’t need more of it. You need to stop generating the kind that can’t.
Standardize the Questions, Not the Form
Notice what that test standardizes and what it leaves alone. It fixes the questions every asset must answer – where it came from, why it exists, who’s paged, what breaks, when it’s due and also a small machine-readable envelope: owner, dependencies, guarantees and recovery path. That minimum structure is what lets CI gate presence and the platform generate useful projections. What it does not prescribe is one narrative template. A streaming source, a nightly batch, a reverse-ETL job and an ML feature pipeline don’t share a shape, and forcing them into the same prose form can produce a lot of “Not Applicable” values alongside real concerns. You should focus on standardizing the contract and its evidence; let each process explain the rest in the form its reality demands.
But here’s the part worth sitting with. If two processes genuinely share an answer – the same scheduling semantics, the same retry policy, the same lineage shape – then that shared part was never really documentation in the first place. It was structure. And structure that repeats is structure you should declare, not describe. The common ground between pipelines is precisely the surface you push down into a declarative spec the platform understands: state what a pipeline depends on, when it runs, what it produces and what its SLA is, and let the platform document it, all from the one declaration. That closes a neat loop: the more of your platform you can make declarative, the more of its documentation writes itself, and the smaller portion of the docs a human ever has to touch. The form varies because processes vary. The moment it stops varying, that’s your signal to stop documenting that part and start declaring it instead.
Stop trying to make engineers care about documentation. They already care about the things that matter – shipping, not getting paged, not explaining the same pipeline for the third time this quarter. You’re not going to win a fight against those priorities, and you shouldn’t want to. The job was never to change what they value.
The job is to make documentation a byproduct of the work they’re already doing, draft the recoverable parts with a machine, and focus human judgment on the context a machine cannot supply.
Make the correct path cheaper than skipping it, and useful documentation will keep moving with the work.