When DataOps can’t make them coordinate
After the Civil War, two families on either side of the Tug Fork, on the border between West Virginia and Kentucky, feuded with each other for decades. The Hatfields and the McCoys. The feud outlived its cause.
Your data organization has a version of this: the groups behave like warring tribes. I wrote about it in Warring Tribes into Winning Teams, and the cause is structural. The groups report to different management chains. They compete for limited resources. They sit in different locations. Above all, they use different tools, and tools set a group’s iteration cadence, its approach to problems, and its tolerance for risk. A BI team and an ETL team can both do excellent work and still disagree on what “done” means.
Management science has a name for this: low relational coordination. Relational coordination is how well people in different roles communicate and relate while they work on the same task. Brandeis professor Jody Hoffer Gittell studied flight departures at major US airlines and wrote it up in The Southwest Airlines Way (2003). Where relational coordination was higher, planes turned around faster, staff were more productive, and there were fewer customer complaints, lost bags, and late arrivals. The pattern held even between two sites of the same airline. In a low-coordination organization, communication between groups is inadequate, inaccurate, and focused on blame. A pizza party doesn’t fix that. The barriers are built into the workflow, and nobody brings pizza to a workflow. If your teams live in the left column of the table below, you are a good candidate for a data contract.
Blame culture is one of The Five Organizational Sins of Data Teams. Development blames QA, QA blames operations, operations blames the data providers, and everyone says “garbage in, garbage out” from outside the blast radius. Blame has a price: when people fear it, they stop looking for problems. The Seven Deadly Sins of Data Quality puts shame first, because the person who finds a data error inherits it.
We have written books on how to get these teams to coordinate: The DataOps Cookbook and Recipes for DataOps Success. DataOps works when you can change the workflow. Sometimes you can’t. The teams sit in separate organizations with separate goals: a supplier in another company, a business unit with its own budget, a customer who is never going to join your standup. Across those lines, no workflow change will stop the arguments. You need a way to settle them.
What do you do then?
The problem underneath is communication. It is the hardest part of working with data, for data teams, their customers, and their suppliers alike. How do you communicate with people you rarely meet, people you may never see?
The value chain is split across all of them: one team supplies the data, an ETL or data team moves it, data scientists use it, and BI teams present it. The data team sits in the middle. Suppliers send bad data, and consumers want everything yesterday.
Most organizations settle this with a spreadsheet. A file interchange agreement (FIA) or data interchange agreement (DIA) lists the columns. It has no checks and no meaning, and it says nothing about what happens when something breaks. When something does, you have an argument and no evidence.
What if you used a data contract when DataOps fails?
What a data contract is
Data contracts are a communication vehicle for teams who dislike each other.
A data contract is a written, versioned agreement between the team that produces a data set and the teams that use it. Think of it as a schema description with proof points and meaning. In the Open Data Contract Standard (ODCS), it is a YAML file that holds the schema, the semantics, the quality rules, the ownership, and the delivery timing.
You can check a contract. It gives you “evidence instead of finger-pointing. Here’s the contract, you agreed to it, here’s the three terms that aren’t true. Can you fix it?”
Manufacturing has a rule of thumb, the 1:10:100 rule. A problem caught at the source costs 1. Fixed after it is created, it costs 10. Fixed after it reaches a customer, it costs 100. A contract moves the check toward the source, where it is cheapest. (We covered how this ships in open source TestGen, and why the contract is your receipt.)
Four use cases follow. Each one is about people, not pipelines.
Use case 1: protect the input and hold your suppliers accountable
The communication challenge: your supplier works in another company and has never seen your downstream. They send the data and move on. “My rule is data suppliers forget that you exist.” You find their mistakes after the data lands, so the only time they hear from you is when you call to complain. They don’t trust you, you don’t trust them, and nothing written down says who is right.
Your supplier sends you a data interchange agreement (DIA). Turn it into a contract and send it back with a link to TestGen, which is open source. The supplier installs it and runs the contract against the data before it leaves their building, on every delivery.
When a term fails at ingest, you stop the line so bad data never reaches bronze, silver, and gold, and you send the supplier the failed terms. Run the contract as a step in your orchestrator, right after the data lands. If any term fails, the step fails, and the load into the next layer never starts. That is the promise with consequences. You save time running your data factory, and the supplier saves time because they find the problem before you phone them. Many suppliers never check their own data before they send it. With a contract, you stop absorbing their mistakes.
In my experience, our customers have 50 to 100 data sources and about a dozen suppliers. If a supplier doesn’t like TestGen, they can use another contract tool, because the contract is a standard ODCS file. The schema and terms travel; the checks may need translating.
A dozen suppliers also means a dozen parties who can say no. People ask me, “what if my data supplier refuses to run it?” My answer: you just run it for them. With the contract on your side, you still check the data when it arrives, on ingest. If you end up running it for them, you could ask why that beats the tests you already run on ingest. The difference is who can see it: your ingest tests are private, while a contract is a standard file both sides can read, version, and argue over. When a term fails, you send the supplier the term they agreed to, not a screenshot of your test log. Ownership of the contract is shared between producer and consumer. It is a point of collaboration, not your contract or mine.
This is where data contracts will deliver the most value. The supplier is where problems start, and by the 1:10:100 rule of thumb, a problem caught at the source is the cheapest one you will ever fix.
Use case 2: protect the output and prove you’re awesome
The communication challenge: your customers find a problem in the data and decide all of it is wrong. Often they are right. Sometimes they are wrong, and you can’t show it, because the only record of what you checked lives in your own logs.
You and the people who consume your data stand on opposite sides. They don’t trust you because the last bad number is still fresh. You don’t trust them because they report a problem before they check it. So deliver the data with a contract. Here is the description. Here are the hundred or thousand tests I ran. Here are the results. A passing contract is a green stamp of approval, and a failing test sits on the page where you can’t ignore it.
It runs on every delivery. A passing contract proves the written terms held when you handed the data over, and the run results are the record you point to. Over the years, my teams have said to customers: “I ran 10,000 checks on this data. I know you think the data is wrong. Tell me exactly what’s wrong.” That turns “your data is garbage” into a specific term that either held or didn’t.
Because ODCS is a standard, a data catalog, a governance tool, or however you package data products can ingest your contract. You write it once and it travels. The schema, the meaning, the ownership, and the delivery terms carry cleanly between tools. The executable quality checks are the part that may need translating, because each tool runs its own tests.
Use case 3: a shared interface for stewards and business owners
The communication challenge: data stewards, business owners, and partners speak in business terms. The data team speaks in tables and pipeline code. Each side suspects the other of not listening, and when something breaks, there is no shared place to agree on what the rule should have been.
The contract adds a layer of governance. It shows the terms and the latest results in language the business reads, and it keeps the pipeline’s complexity underneath. For this, a contract tool needs a UI, so people who don’t read YAML can see the terms.
Love your errors. An error is a term the contract has yet to include. When you have a problem, show it to your customer, and add a new term to the contract together so it doesn’t happen again. The alternative is sticking your head in the ground like an ostrich.
Data teams are often so busy they lose touch with what the business needs. Writing a contract together fixes that. The steward can then use the same contract with their own customers. One file, passed down the line.
Use case 4: prove the middle is awesome
The communication challenge: your engineers do the work in the middle, and nobody outside the team can see it. Customers see the input and the output. Managers hear about the middle only when it breaks. So nobody credits the work, and nobody can tell which step failed.
I have managed a lot of data engineers, so I look at this one as a manager. How do I know each step works? Did the data arrive on time, at the right frequency and volume? Did the schema change? Did a transformation break? Am I delivering it right? Did my CI/CD pipeline introduce a regression?
Build a contract from the tests in your ETL steps. Run it on every pipeline run and every CI/CD build. Now the customer can see the value of the work between raw input and delivery. Run development with yesterday’s data and today’s code, and a passing contract shows the written terms held on that run. The results are the record you point to.
As a manager, you also see what your team is doing and how good or bad your data suppliers are. Bring that to the next supplier meeting.
Where to start
My favorite starting point is one contract at the supplier and one on delivery. If you can’t get the supplier to run it, check it yourself on ingest and on delivery. Measure a contract first by whether it passed and what failed, then by pass rate over time and coverage.
You don’t have to write it by hand. In our demo on the Jaffle Shop sample data (six tables, 26 columns, about 154,000 rows), open source TestGen profiled the tables and generated the contract and its tests in about 60 seconds. That was about 80% of the tests in the finished contract. An LLM working through the TestGen MCP server added more tests in about five minutes, which got it toward 90%. Business users added the last rules in the UI, because they know the rules that no profile can see. Those figures come from that demo, not from a guarantee about your data. Install TestGen or watch the webinar.
Feuds don’t end because people start liking each other. They end when both sides agree on what happened.
FAQ
What are the key points in this blog?
Some data teams will never coordinate, because they sit in separate companies or business units with separate goals. A data contract gives them a written, versioned agreement with a test behind every term, so they settle arguments with evidence instead of blame. Use it four ways: at the supplier, on delivery to customers, as a shared interface for stewards and business owners, and to show the work in the middle of the pipeline.
What is a data contract?
A data contract is a written, versioned agreement between the team that produces a data set and the teams that use it. In the Open Data Contract Standard, it describes the schema, the meaning of the data, quality rules, ownership, and delivery timing, and it carries tests that show whether the agreement held on each delivery.
When should I use a data contract instead of DataOps?
Use a data contract when the teams that share data cannot share a workflow: a supplier in another company, a business unit with its own budget, or a customer outside your organization. DataOps fixes coordination when you can change how the work is done. A contract works across those lines, because both sides read the same terms and the same test results.
What is low relational coordination?
Low relational coordination means teams that depend on each other share neither goals nor knowledge, and they communicate late, inaccurately, and with blame. Jody Hoffer Gittell’s airline research found that higher relational coordination went with faster turnarounds and fewer complaints, lost bags, and late arrivals. Data teams with low relational coordination are good candidates for a data contract.
What if my data supplier refuses to run the contract?
Run it for them. With the contract on your side, you check the data the moment it arrives, on ingest, and stop the load when a term fails. Then you send the supplier the term they agreed to, not a screenshot of your test log. Ownership of the contract is shared between producer and consumer.
How much of a data contract can TestGen write for me?
In our demo on the Jaffle Shop sample data, six tables and about 154,000 rows, open source TestGen generated about 80% of the finished contract’s tests in about 60 seconds. An LLM working through the TestGen MCP server took it toward 90% in about five minutes. Business users added the last rules in the UI. These figures come from that demo, not a guarantee.