You still write code, and now you’re also answerable for what three other people ship. Monday opens with a Slack thread about a number that moved. You put two of your reports on it and lose three days. The sprint work slips, and the engineer you brought on to build models spends her week doing forensics on a schema change nobody announced. And your customers want more, and more, and more.
You didn’t take the job to run an incident desk. Neither did they. So the question isn’t whether your team needs DataOps. The question is narrower and more annoying: what do you do first, what do you hand your reports to read, and in what order?
We’ve been pushing DataOps for over a dozen years. We started DataKitchen in 2013, wrote the DataOps Manifesto, and led this movement from the beginning, back when explaining the idea took 20 minutes and half the room thought you meant DevOps with a typo. Everything below is what we’ve learned and published since.
This is that answer. Four sections. The first one gives you and your reports a shared set of ideas. The second gives you one thing your team can do this week without asking anybody for budget. The third deals with AI, because your reports now generate pipeline code faster than you can review it, and you are the reviewer. The last one is for whoever on your team came from software and can’t figure out why everything is so weird here.
Learn the ideas first, because tools won’t save you
DataOps is a way of working, and you can't buy a way of working.
People buy a data quality tool, point it at the warehouse, and wonder why nothing is better six months later. The tool was fine. The problem was the process around it, and no vendor ships that.
Start with the manifesto. It’s 18 points and takes 10 minutes. Read it yourself, send it to your reports, and see which points start an argument. That argument is useful. It usually lands on the part of your process that’s broken and that everyone has agreed not to talk about. From there, the order that works is the seven steps for what to actually change, the fundamentals course so your whole team means the same thing by the word “test,” and the maturity model to find out which weakness to attack first. A score beats your opinion when you’re asking your manager for time to fix something.
The “why” is worth more than it sounds. DataOps borrows from Lean manufacturing, Agile, and DevOps because those disciplines have already solved your problem: shipping reliable work with a handful of engineers and constant interruptions. Sprints look different when half the week is someone walking up to a desk with a broken dashboard, and the Agile webinar is honest about which parts don’t transfer cleanly.
One of these is for later. Recipes for DataOps Success is about running this across an organization, and you don’t own the org chart. What you own is a team that can produce a result other groups want, which is how this actually spreads. Read it when people outside your team start asking how you did it.
| Resource | What it is | Reach for it when |
|---|---|---|
| The DataOps Manifesto | Manifesto | You want the shared starting point, and an argument worth having |
| 7 Steps to Implement DataOps | White paper | You need the sequence of changes, from first principles to measurable results |
| What is DataOps? | Explainer page | Your manager asks what you're proposing and you don't want to write it at 11pm |
| Agile, DataOps and Data Team Excellence | Webinar | Agile keeps half-working on your team and you want to know why |
| The DataOps Cookbook | Free book | You want the long version, or the engineer who keeps saying "we should automate this" needs a plan |
| DataOps Fundamentals | Free course | You want you and your reports to finish with the same vocabulary |
| DataOps Maturity Model | Assessment | You need a score instead of an opinion about where the team stands |
| Recipes for DataOps Success | Free book | People outside your group start asking how you did it |
| DataOps consulting, assessments, and training | Services | Your manager asks who could help |
Where to start today, with data testing
You don't get trust back until the errors stop reaching your customers.
Pick one place in your current setup and improve it. Not a platform migration, which you couldn’t get approved anyway. One place.
That place is testing. Every other DataOps improvement compounds on top of it. Deployment automation is great, but it can make bad data arrive faster if you skip this step. So you check the data in development, before your team merges, and in production, so you catch the vendor who changed their file format over the weekend.
The question your reports will hit in the first hour is where to put the tests, and the four points post is the most useful thing in this section. There are four places. Source data, so you can push problems back to whoever owns them. Ingestion, so you catch what arrives late, short, or reshaped. Between transforms, where a tripwire stops bad data before it reaches gold. And CI/CD, so a renamed column doesn’t break a report three layers down. The same null check does a different job at each one. Run it in a single place, and you’re covered for one failure mode and blind to the other three.
You’ll also get an objection from whoever already wrote a few dbt tests. Do the math with them. A bronze layer of 100 tables averaging 10 columns needs roughly 200 table tests and 2,000 column tests, and silver and gold each want their own. At 30 minutes per test, that’s months of typing, which is exactly why nobody does it. TestGen profiles the data and generates about 80 percent of them. Your team spends its time on the 20 percent that encodes business logic no profiler can infer. Docker compose up, point it at a schema, and the queries run inside your database, so no data leaves your perimeter and your security review gets shorter.
One distinction worth making before your team starts. Data quality tells you the values are wrong. Process observability tells you the job that produces those values ran late, ran twice, or didn’t run at all. Most teams instrument the first and skip the second, which is how you end up with data that is perfectly valid and eight hours stale.
| Resource | What it is | Reach for it when |
|---|---|---|
| The four points where data testing really matters | Blog post | Your team asks where the tests go. Read this one first |
| What is DataOps Data Quality? | Explainer page | You want data quality framed as engineering work, not a quarterly audit |
| What is DataOps Process Observability? | Explainer page | Your data is right and your pipeline still let you down |
| The DataOps Way to Data Quality and Data Observability | Free book | You want testing and observability covered together |
| Data Observability and Data Quality Testing | Free course | This is somebody's actual job, not a side project |
| Why we open sourced our tools | Blog post | You want to know why a company gives away what it used to sell |
| Install open source TestGen | Software | You're done reading and want results this afternoon |
AI and DataOps
AI didn't change what good data engineering means. It changed how fast bad data engineering happens.
Your reports can generate a new transformation in 90 seconds, and it lands in review faster than you can read it. The bottleneck moved, and it moved onto your desk. Review catches the code that throws a stack trace. It doesn’t catch the join that fanned out on duplicate keys and made the revenue number 4 percent higher. Nothing failed. The dashboard rendered. Somebody made a decision.
Three things to take from the reading below. Idempotency stopped being good practice and became mandatory: a human who watches a load fail stops halfway and looks at the target table, while an agent sees a non-zero exit code and runs the command again, turning 1,000 rows into 2,000 with nobody watching the row count. Test coverage has to grow at the same rate as generated code, or you get 4,000 new lines of SQL sitting behind 40 old tests. And an agent never grades its own homework, which is the rule holding all seven sins together. The piece most teams miss is context. Your AI is only as good as what it knows about your schemas, your conventions, and the reason that one table has a column called flag_2_final.
| Resource | What it is | Reach for it when |
|---|---|---|
| DataOps + FITT + Data Testing = 10x with AI | Blog post | You want the architecture: functional, idempotent, tested, two-stage |
| The Seven Deadly Sins of AI-Generated Data Pipelines | Blog post | You want the failure modes named, with the fix for each |
| Stop Clicking, Start Asking: The AI Playbook for Data Quality | Webinar | You want AI doing the data quality work instead of creating more of it |
| The Equation for AI Success: DT + DX + CTX = 10x | Blog post | Your agent writes syntactically perfect SQL against semantics it invented |
What changes when the errors stop
Reliability that depends on who is paying attention isn't reliability. It's luck with a rotation.
Right now your pipelines work partly because certain people are watching them. You know which tables are fragile. Your senior engineer knows which vendor file arrives malformed twice a quarter. That knowledge lives in heads, which means your error rate is a function of who is awake and who is on vacation. Heroics feel like competence from the inside. From the outside, they look like a system nobody can predict.
Everything in this post is one idea applied repeatedly: move that knowledge out of heads and into the system. It’s borrowed from manufacturing and from software, and it holds in data too. Quality is a property of the process, not the vigilance of whoever is standing at the end of the line.
The economics are the 1:10:100 rule, which George Labovitz and Yu Sang Chang put a name to in 1992. A problem caught in development costs about a dollar. The same problem caught in production costs ten. Caught by your customer, in a meeting, on a slide with your team’s name on it, it costs a hundred, and most of that hundred is paid in trust you spend the next two quarters earning back.
The return shows up on your team’s calendar. Fewer days lost to forensics. Releases nobody has to stay late and babysit. Here’s the marker of a high-performing data team that I like best: you can hand a 22-year-old the keys, let them rename a column, and a red light goes off telling them they broke a downstream report before it ever reaches production. That engineer learns in an hour what used to take an outage to teach. You review with less dread. Nobody apologizes to a VP.
Your team stops explaining and starts building. Every test you write is a gift to your future self, and when you have reports, to theirs.
Appendix: if somebody on your team came from software
In software, you control your inputs. In data, you don't.
Somebody on your team came from DevOps or application development, and this world is strange to them. The deploys are weird. The tests are weird. Production breaks because of data nobody on the team wrote, from a vendor none of you have met.
All of us who founded DataKitchen made this same move about 20 years ago, so they have our sympathy, and we’ve collected the shortest path we know.
The thing that has to land before anything else makes sense: your code can be perfect and your pipeline still fails, because the data changed underneath it. Every practice in this post comes from that one fact.
| Resource | What it is | Reach for it when |
|---|---|---|
| Your Data Nerd Friends Need You (slides) | Keynote | Day one, aimed squarely at the software person on your team |
| The Unicorn Project excerpt | Book excerpt | Gene Kim covers the data side in language they already speak |
| DataOps is Not Just DevOps for Data | White paper | They're applying the DevOps playbook straight and it keeps not working |
FAQ
What are the key points in this blog?
Learn the ideas first, because DataOps is a way of working and no tool ships one. Then improve one place in your current setup, and make that place data testing, since every other DataOps practice compounds on top of it. Review AI-generated pipeline code with tests rather than eyeballs. The payoff is a team whose reliability no longer depends on who is paying attention.
Where should a data team start with DataOps?
Start with data testing in your current pipelines, not a platform migration. Check the data in development, before your team merges, and in production, so you catch the vendor who changed a file format over the weekend. Before that, read the 18-point DataOps Manifesto with your team and notice which points start an argument. That argument usually lands on the part of your process everyone has agreed not to talk about.
Where should data quality tests go in a pipeline?
In four places: source data, so you can push problems back to whoever owns them; ingestion, so you catch what arrives late, short, or reshaped; between transforms, where a tripwire stops bad data before it reaches gold; and CI/CD, so a renamed column doesn’t break a report three layers down. The same null check does a different job at each point, so testing in one place covers one failure mode and misses three.
How many data quality tests does a medallion architecture need?
More than anyone writes by hand. A bronze layer of 100 tables averaging 10 columns needs roughly 200 table tests and 2,000 column tests, and silver and gold each want their own set. At 30 minutes per test that is months of typing. DataOps TestGen profiles the data and generates about 80 percent of them, leaving your team the 20 percent that encodes business logic no profiler can infer.
How does AI-generated code change data engineering?
It moves the bottleneck onto the reviewer. Code review catches the transformation that throws a stack trace, not the join that fanned out on duplicate keys and made revenue 4 percent high. Three things follow: idempotency becomes mandatory, because an agent that sees a non-zero exit code reruns the load; test coverage has to grow as fast as generated code; and an agent never grades its own homework.
What is the 1:10:100 rule in data quality?
The 1:10:100 rule, from George Labovitz and Yu Sang Chang’s 1992 quality work, says a problem caught in development costs about a dollar, the same problem caught in production costs ten, and one caught by your customer costs a hundred. In data teams most of that hundred is paid in trust, which takes the next two quarters to earn back. It is the economic case for testing early.
