On-Demand Webinar · 44 min
How to Do Data Contracts
Chris Bergh covers what a data contract is, why a contract without tests is only a description, and how open source TestGen gets you a working contract fast: 80% of the tests in 60 seconds, 90% with an LLM and the TestGen MCP server, and the last 10% from your business users in the UI. Then the four places to run one: at the supplier, on ingest, on delivery, and in CI/CD.
What you'll learn 8 points
- A data contract is a written agreement between the team that produces a data set and the people who use it: a schema description with proof points and meaning. Its biggest benefit is that it can be checked, so it tells you whether it is being met rather than only describing the data.
- Most file interchange and data interchange agreements are spreadsheets that describe the columns but carry no checks and say nothing about what happens when something breaks. A contract adds the tests and the consequence.
- Contracts serve four groups of people: suppliers, who can run the contract before they send data; your own team, proving each delivery is right; data stewards and business owners, who get a shared language; and data engineering managers, who can see whether each step of the pipeline works.
- TestGen generates about 80% of the tests a contract needs in 60 seconds from profiling. An LLM working through the TestGen MCP server, with context about your business, takes it toward 90%. The last 10% lives in people's heads, so business users add it in the TestGen UI.
- Contracts in TestGen are versioned. You can freeze a version, compare versions with a diff view, deactivate a test you don't want, export the contract as ODCS YAML, and import a contract someone sends you and run it.
- There are four places to run a contract: at the supplier before the file leaves their building, on ingest the moment data lands, on delivery before the final data goes out, and in development or CI/CD with yesterday's data and today's code.
- Chris's favorite places to start are at the supplier and on delivery, bookending the pipeline. If the supplier won't run it, run it yourself the instant the data arrives.
- Measure a contract by whether it passed or failed and what failed, then track the pass rate over time and the test coverage relative to your tables.
Prefer to read it? The written version is in On-Demand Webinar: How to Do Data Contracts.
Slides
Transcript
Show chapters and dialogue 24 chapters · 7,108 words
- 0:00 Welcome and agenda
- 1:56 What a data contract is
- 3:28 Why schema descriptions and interface agreements fall short
- 4:52 Why data contracts: catch problems at the source
- 5:41 Four ways people use data contracts
- 10:08 Why we added data contracts to TestGen
- 12:01 Demo: 80% of the tests in 60 seconds, then the LLM and the MCP
- 15:02 Demo: the Jaffle Shop contract in the TestGen UI
- 19:02 Demo: Claude adds v2 tests and contract metadata
- 21:41 Demo: an AdventureWorks contract with 650 tests from one sentence
- 24:09 Importing a contract someone sends you
- 25:22 Where to do data contracts: four gates
- 27:10 Gate 1: at the supplier
- 28:37 Gate 2: on ingest
- 29:23 Gate 3: on delivery
- 30:56 Gate 4: in development and CI/CD
- 32:08 Where to start: bookend your pipelines
- 32:47 Measuring contracts: pass or fail, and sharing the results
- 34:46 Turning descriptions into tests: contract test suites
- 36:54 Contract metrics: pass rate and coverage
- 38:00 Who finds value in data contracts
- 38:46 How to start, and the two editions
- 40:04 TestGen compared with dbt contracts and tests
- 41:01 Questions: suppliers who refuse, ownership, where to start
00:00:00 Welcome and agenda
Chris Bergh: And so just some logistics who I am. My name is Chris Bergh. I'm a data person and software person from way back. I'm the CEO at DataKitchen. And just some logistics, we're going to share the slides and the recording and the transcript after this. Please feel free to ask questions. The comment box on Google Meet is in the lower right. And feel free to put your questions in there. I promise to get to them maybe during the presentation or definitely at the end. And so we'll leave time. And so today's webinar should be on the order of 45 to 50 minutes. So if that makes sense, I'm going to start. So what are we going to talk about today? So we're going to talk about the what and the whys and the where's and the hows and the who's of data contracts. And so and in the middle we're going to have a demo of TestGen's new data contract feature and why we're why we're so excited about it. So I in doing research and building this feature there's just been a lot of talk about data contracts the searches in data contracts and you know we're just as a company we've been focused on helping data teams become more efficient and the linchpin in making data teams more efficient is data quality testing and testing not just of source data quality but all over the value chain and how you do data and analytics. And so in a lot of ways, data contract is a natural extension of that focus on helping data teams be productive through data ops and then helping them do that through open source TestGen and by making the creation and the utilization of data quality checks everywhere in your analytics fast and easy. And so what is a data contract?
00:01:56 What a data contract is
Chris Bergh: Think of it as a written version agreement between a team that gets a data set or produces a data set and someone who gets it or uses it. It's sort of a promise. Here's what I'm going to give you. And I think what's really challenging in data and analytic teams and their customers and their suppliers is always communication. How do you communicate with people you may not meet very often, people you may not see? Your data suppliers, your data customers between team members. The value chain in data engineering is often separate. You may, you know, you may have a team that gives you the data. You may have an ETL or a data team. You may have data scientists. You may have BI teams. And so data contracts is a way to help your communication by providing an industry standard that allow you to communicate sort of what the expectations are. And so we followed the a great standard called the open data contract standard. And what is it? Well, think of it as a text file, a YAML file that sort of describes the data, its schema, its semantics, and then describes how you should check the data to prove that it's true. Those are quality rules. And then just some more other things like ownership and delivery timing. So it's a little bit more than what you would expect in just a schema description, right?
00:03:28 Why schema descriptions and interface agreements fall short
Chris Bergh: Is a schema description with, proof points and meaning on it. And so, why is that important? Well, you know, over the years, we've gotten a lot of data because part of our company does data engineering. I've done data engineering. And when you get a file, like sometimes you'll get another file associated with it, a schema description, or sometimes you'll get a file interchange agreement or a data interchange agreement. Usually they're these Excel spreadsheets that kind of describe what goes on and like here's what you mean. And it's it's those are much better than just getting a file where you've got to like look at the columns and determine the types. And as a data engineer or someone who's dealt with data, I'm sure all of us have kind of taken that. I've got a CSV and I have no idea what's in it. So, the metadata or the file next to the file is good, right? Because it it's helpful. But often times it's just sort of descriptive, right? It doesn't have any checks to prove that what the data gave you and it doesn't really have any meaning. It just sort of describes the columns and then it doesn't tell you what happens if something breaks. And so, schema descriptions, you know, FIAs, DIAs, they're useful, but I think it's great that we've got something more. And so, one of the biggest benefits we see in contracts is the ability to check to prove that they're actually valid.
00:04:52 Why data contracts: catch problems at the source
Chris Bergh: So, a contract is not a description, but it's a description that tells you whether it's working or not. And so we're just, as I said, a big believer in checking data, checking when it arrives, checking it when it's processing, checking data during regression testing. And all these characteristics have the same problem is that if you find a problem sooner, it's going to save you a lot of work and a lot of time and a lot of headache. And this sort of 1:10:100 rule that comes from manufacturing, I think really matters. And so what also it does is it the contract provides evidence instead of fingerpointing. It's like here's the contract, you agreed to it, here's the three terms that aren't true. Can you fix it? And so it's again clear interfaces make good neighbors.
00:05:41 Four ways people use data contracts
Chris Bergh: And so the same thing in software engineering if you have a well-defined interface and API that makes it clear and it helps communication. And so, a contract allows you to find problems before they cause trouble, but also clearly communicate your expectations of what you want or what you're giving someone. And so let's just talk through kind of the human use cases on data contracts. So in this diagram in the next four slides, you've got a supplier, you've got data ingestion, transformation, delivery, and customers. Very simplified. A great case to use a data contract is with your supplier. So maybe they are giving you a data interface agreement. Well, put some teeth in it. Have some description in it. Create a contract and give it to the supplier and say, "Hey, why don't you run this?" we're very excited about that capability. And then every time the supplier sends you data, ask them to run it beforehand. Make sure that they are doing what they said. And this can save you time as a running sort of a data factory, but also can save the supplier time. And you know dealing with data most of our customers have 50 to 100 different data sources maybe a dozen different data suppliers you know my rule is data suppliers forget that you exist and one way to have a contract that goes gone beyond just a description of data is that it can help them make sure that their delivery to you is correct and so that's one great use case on the people's side. The second is sort of I think of this as sort of prove your pro prove you're awesome, right? Your customers are always finding problems in the data. Well, what if you delivered your data with a contract saying here I ran these tests. I've proved that it's right. Here's the description of the data. Here's the hundred or thousand tests that I've done. And it kind of shows that you're awesome. And I think it becomes kind of a stamp of approval. I've met the contract. And so this is a great way to show value in all the work that you've done. And because it's a standard, it can be ingested in lots of different tools. Maybe into your own data catalog or your own data governance tool or however you format data products in your organization. And a third case is there's always lots of people in the data value chain. There's data stewards, there's business owners, there's partners. It becomes a shared language to talk about things and I think this is really important because we've guided in our own work and our customers and our consulting that the idea of loving your errors and so what is an error and an error is a term in a contract that's yet to be written and so if you have problems show that to your customers say oh we had a problem and now we're going to together add a new term to the contract maybe you know the business logic data steward or business owner. I don't let's collaborate and add a term to the data contract so this doesn't happen again. And again, communication's important. Shared terminology is important. And sort of this idea that instead of sticking your head in the ground like an ostrich, avoiding errors, loving your errors and using contracts as a way to have a dialogue with your customer to how to avoid it. And so, one of the challenges a lot of data teams have is they're they're so busy they are not in touch with what the business or the organization needs. And trying to collaborate on a contract we think is a really important way to help move forward and improve communication and delivery among teams. And then the last is sort of prove the middle's awesome. I think of this almost from a person who's managed a lot of data engineers. You know, how do I know each substeps working when I've ingested the data? Did it arrive on time? Is it the right frequency, the right volume? Did the schema change? Did the contract break? Did something happen in my transformations? Am I delivering it right? And even in my CI and CD pipeline, have I introduced any regressions? And so almost from I'm a data engineering manager, I can use contracts as a way to do this.
00:10:08 Why we added data contracts to TestGen
Chris Bergh: And so these three core use cases we think are a great way. And so why do we add contracts to TestGen? Well, we've already got this tool that generates a lot of contract generates a lot of tests and we've got ways to export it in CSV and in JSON, you've got an MCP. But that's all great. But what we think is that it provides a way in a larger organization, maybe a political organization, to be able to share these contracts around in a more standard way. And standards are good. Standards are a great way to communicate. And so instead of somebody having to look at a JSON that they've never seen before, well, they can go read the contracts back and and do it themselves. And also, we've we found it valuable to our for ourselves. Here's a file to our data suppliers. And so it's just a great way because we're just a big believer in data testing across the value chain and data and analytics. So that's why we added data contracts to TestGen. And so, and there's a lot of tools out there that do, open data contracts. And so I think ODCS won the format war. And to me, I look at it as, okay, it's a file format, but it's really about automating the whole contract creation, execution, and management process. And it's really about communication. How can you communicate on the contract with people? And to me, I think you need a UI. You need to be able to sort of send the tool with the contract. And we have both those in TestGen. You can just say here's the contract, here's a tool. You don't understand the term, let's go look in the UI. And also because we include our a full MCP on our open source being able to collaborate with a tool that can help you write an additional business level tests and we'll we'll discuss that.
00:12:01 Demo: 80% of the tests in 60 seconds, then the LLM and the MCP
Chris Bergh: So, I'm going to go into sort of a demo of data contracts, and this should take about five or 10 minutes, but just kind of give you an idea of how it works. And so you know our view is that we built TestGen trying to get 80% of the tests that you need in a smart way based on profiling data based on a semantic data model and based on sort of the decades of history our team has in in testing data. So our idea is that you can get 80% of the tests you need in 60 seconds with TestGen. And so it gets much harder to get to that extra 20%. And so, having an LLM with some better context about your business, about, how the data is used can get there. And so, I'll show you that. And then the last 10% I just don't think, you know, maybe AI with better context, you know, it's the question of like how far AI can take the world. And so I still think that it the last 10% the last miles you're still going to need to collaborate with your business because there's so much contextual knowledge that's in people's heads. So I'm going to do this with an MCP. And so the first thing that TestGen does is you can point it at a few tables and you can profile them and generate tests. And what happens in TestGen is you just go and create those tests. And so what I did in Claude is I downloaded the Jaffle Shop data which is a demo data from dbt and it's got I don't know six tables you know 150,000 rows and I said okay here's the data and so it ended up in a database on my machine and then in Claude I said let's create a data contract for the Jaffle Shop data all the tables using TestGen and then create tests in TestGen and add them to the contract. So, what happened then is it went through and did it created 67 tests and it used these TestGen MCP tools. And so, I can show you that, in another data set. So, what I'm going to do is show you in this window. So, this window is actually my, my Claude window. And so, here I've got Claude running and I've got some AdventureWorks data in it. And so I I'm going to go run Claude. If you can see it here, it says create a data contract for adventureworks_copy, which is a copy of a different open data set that people use to demonstrate, make tests, run them, and version it. So what's going to happen here is Claude's going to talk to TestGen MCP and go off and create it. And so how does that look in TestGen? Well, I'm going to show you if I go and share something else. I'm going to share TestGen here. So, here I'm into the TestGen product.
00:15:02 Demo: the Jaffle Shop contract in the TestGen UI
Chris Bergh: And so into TestGen I've got a Jaffle Shop and in there something called the Jaffle Shop contract. And so I can see that it's got six tables, 26 columns, a bunch of rows and a bunch of different data points. I profiled it and from that profiling I learned some hygiene issues which are kind of the first level of checking, right? It identifies PII. It says there's some names that have interesting problems in the data. So, here's some cases where, there's some record counts with something that looks similar but not. And so, I've profiled the data and then I've created a test suite and in our way of working and I've created a in TestGen, we profile the data automatically. There's 51 different profiling characteristics. We have about two dozen data hygiene which is sort of checks on static data and then we've got I don't know 40 50 or so tests that run and so we built a suite of tests and so here I built that suite of tests and I can go in and look at my data contract and so here I can see the data contract here's all the tests that it's generated I can see the contract metadata So here for each table I can go in and see all the quality rules or all the quality items in that contract. I can see some metadata that was added and I can edit it. There's overview, there's team metadata, there's roles, there's support, there's service level. All this metadata can get added either by you clicking edit and editing yourself or going in and putting it from the MCP. And so you can actually go in and drill into the quality rules for this table. And here you can see all the quality rules that I have. And some of them here are actually generated automatically. So I've got things like a reference match which it goes in and checks this was something that was created by Claude. It checks that the raw_orders and raw_stores that the store ID and the ID match each other. And so this is something that Claude figured out from the data and put in a custom test. And so what's exciting about this also is that you can if you want to add a test or you want to edit a test, you can just go in and edit it here. You and your you can do it yourself or your business customer. Likewise, you can actually see the history of it. So we have a version of the contract. So I could go look and see what version one was. I could compare it to version three. And here I've added these quality rules and I've added this contract metadata. So, keeping track of it and actually you can look at it from a, if you're used to sort of a git diff, here's a here's a diff view of it. So again, a great way to manage this contract in one place with diffing, seeing the changes, etc. So this is important because contracts having a version, you're going to want to look at what happened to it and how we edited it. And so think of a contract as a series of data quality tests with a export where I can take the contract and I can export it as a version of the contract here. This is in YAML and I can take this and deliver it to someone. I can mail it. I can share it with my business partners etc. So here's the standard ODCS format of a data contract. I could, if I want, import a contract into this and make a new one. So, if I keep going through this, I've got a data contract, I've got tests, I've got contract metadata, I've got versioning, and if I go back to
00:19:02 Demo: Claude adds v2 tests and contract metadata
Chris Bergh: My demonstration, I can see, for instance, in this Claude, I can go in and it created a new version by freezing it, it put v2 in force and then Claude went in and created 81 extra tests. And so, here's the test that Claude created by looking at the data. And so, in our next release, we're putting we're having updating our skills, and adding more metadata to make even the MCP test generation process even better and faster. And so, I could then go in and I added some metadata. So I said add some contract metadata to v2 and export it to YAML. And so it went through and created and added more metadata. So I don't know if you are like me but I'm sort of doing you know half my work in Claude nowadays to be able to from things to write to code to do marketing and sales activities. And so having a robust MCP with contracts is really important. And so if I go back to the last part, the last 10% you know, having a UI that you can collaborate with your business customer, being able to send them a link saying, "What do you think of this test? We just had an issue. I'm going to add a new test. Can you review it? Should this be 5%? Should this be 2%." I think are a great way for you to improve communication over the contract. So instead of having Hatfields and McCoys or you know I hate my data providers, they're idiots or I hate my customers, they're idiots, it provides a very concrete way for you to collaborate both with your customers and with an AI and your metadata context to improve the tests and create less errors. And so here's what I did. I said in Claude I said let's create a data contract for Jaffle Shop data all tables using TestGen create test and TestGen and add to the contract freeze then create a v2 with those new tests put v2 in force add some contract metadata and export YAML so this whole process it really didn't take that long so I'm going to go and share a different I'm going to share a different tab here.
00:21:41 Demo: an AdventureWorks contract with 650 tests from one sentence
Chris Bergh: So, let's go back and see how this happened with this window. So, you can see what I did is I went through it. It on this AdventureWorks copy data. I if we look it had a v1, it generated 485 tests automatically and schema checks and then it created I asked it added 137 tests by looking at the profiling data and the semantic data types and so the way that looks in TestGen is that was just one sentence it went in and did all this work and if I want to go back and share how that looks in TestGen, I can say that here was TestGen going back to the project dashboard. I can see now I've got AdventureWorks data. I can look at it in my data catalog and say wow I've got a lot of tables in my AdventureWorks data. So I can see here's dimcustomer. I can see the table the rows. I can see it found a bunch of hygiene issues in there and I it can find a bunch of tests. And so, that's great. Now, if I look at, again, going back to my test suites, it created a contract test suite with 650 test definitions. So, this happened just sort of behind the scenes in five minutes. And so one of the things I've really enjoyed using TestGen is just sort of downloading random data, generating test suites, learning about it, learning about it is it's super fast and super easy. And so here I've got if I go into the contract I can say I've got all these quality rules and here these ones some of them are are automatically generated. So and some of them are generated by TestGen some of them are generated by Claude and so I can have so for instance it generated a match row count test which says that the row count at baseline is 99. So, it's very formally saying that this is what the contract should be. And maybe you don't like this. Maybe you in your business customer doesn't think this is good. You can just deactivate it in the contract and turn it off so it's not there anymore. And so and if I don't like that one, I can make it a version three. I can freeze it and I can say this is for the demo and I can freeze the version.
00:24:09 Importing a contract someone sends you
Chris Bergh: So now I've got a new frozen version of the contract that I can use and share with my customers. So kind of getting back to the presentation, the last thing that you want to do is sort of is what happens if someone gives you a data contract email? Well, you just import it into TestGen and run it. It's that simple. And I think that's what's great about having these types of things. And so one of the challenges with the open data contract standard as it is it has a sort of minimal express expressivity in the contract. So a lot of the contracts that a lot of the tests that we have in the data contract are custom tests that are unique to TestGen where we could we tried to make them plain old SQL. But you know there's you we've got a whole bunch of tests that go into TestGen and trying to have that sort of scope of testing is kind of outside what the current data contract standard has. And so yeah, here's a case of what it did. Just, you know, import the YAML, run it, preview it, and it just does that all through MCP or through our UI.
00:25:22 Where to do data contracts: four gates
Chris Bergh: So, let's talk about where to do data contracts. So we've kind of gone through what is a data contract. It's a communication vehicle that describes data and the describes it with teeth says what it actually should be by checks. Why data contracts? Well proving your worth is great. Checking other worth is great. Trying to understand what your team's doing great. It has a lot of value. And a lot of it value from our perspective just comes from testing. And so where to do data contracts. So we're going to talk about four places to do them. So in this ideal pipeline where you go from suppliers sorry I forgot I was not sharing my screen. So so you go from suppliers on this slide. So there's suppliers here. That's where the data comes from in your data team. That's sort of your landing zone, your three parts of your medallion if you will. Going out and then finally to your consumer, your apps, your models, your dashboards, and then your code changes, right? Where and this is our classic data ops T diagram where data goes from the left to your customers and then code goes up into production. And so these are the four places you could use a data contract at the supplier to check it before it lands. On ingest. Maybe your suppliers don't aren't using contracts, maybe you don't trust them, or maybe you just want to make sure that the delivery rate is happening on time.
00:27:10 Gate 1: at the supplier
Chris Bergh: And then the third case is on your delivery to prove that you're awesome. And then the fourth case is in your development cycle to check for regressions or impact. So let's go through each one of these now. So gate one of four. So I have a data supplier. They're giving me data once a day or once an hour. Well, I want to check their data. And oftentimes, it's amazing how many data suppliers just don't even check to see if their data is right. So, create a contract, send it to them, send them the link to TestGen and say, "Here, run this every time before you give it to me. Run this data contract." That way, I can prove that we're we're both checking the same thing. And if you find problems before I do, you can then fix them. And so this is very exciting for us in our services organization. Sending them the contract and the tool at the same time. Here, go run this. And if they don't like our tool, they can pick another contract tool. And so you kind of stop playing goalie for someone's mistakes. The supplier is hitting data to you. You're like, "Oh, something's wrong. You got to contact them." So the first step is just get your suppliers to run a contract. And I think that's to me I think that's where the biggest value in data contracts are going to be.
00:28:37 Gate 2: on ingest
Chris Bergh: The second is your data team on ingest. Well, it's a second line of defense, right? Test the data with a contract the moment before it lands. Maybe you're using the same contract you gave to your supplier. Or maybe you can test not only data quality but delivery monitoring the data observability features of freshness, volume and schema. And then when you find something you can send it to your supplier or it can sort of stop the line to make sure that things don't go into production and you're sort of bronze, silver and gold and then you can send your supplier the failed terms. So now you've gone through your bronze, silver, medallion. You could use it on delivery and so you could say every delivery.
00:29:23 Gate 3: on delivery
Chris Bergh: So you could have contracts at the end of every level. You could say your bronze level, silver level, gold level if you wish. That might be a great way to do it. I actually like the final way of looking at it to make sure that your data as it's integrated in customer ready form makes sense. And so, and it also becomes a great way for your data customers to say, "Okay, here's the contract. I've proven it. I've run 300 tests, 10,000 tests to prove that it's right." And so, over the years, we've often found our in our customers, both of our tools and our software business have always said, "Look, I ran 10,000 checks on this data. I know you think the data is wrong. Well, tell me exactly what's wrong. And yeah, it's kind of a way to shut up your customers when they're complaining, but also it becomes a great way to prove that you've done things. And lastly, on these on-delivery contracts, you always are going to make errors. And so to love your errors, it gives you an opportunity to collaborate with your consumer saying, "Yeah, we did make a mistake. We need to add a few more tests to our contracts. Can we work together to figure out what those tests are? Or can you review them and have them look at it in the UI? So, let's add a new term to the contract together to catch it. And this becomes the benefit of collaborating with your customers because a lot of data teams feel they're sort of caught between a rock and a hard place between their data suppliers giving them crappy data and you know their data consumers wanting everything
00:30:56 Gate 4: in development and CI/CD
Chris Bergh: Yesterday. So the last use case is really in development. So if I have a working pipeline, I'm getting data. Everything's green. All my contracts are working great. Your customer says, I want to change. So you've modify a schema. Maybe you add in a new data set. Maybe you change a calculation. So how do you prove that both that works and that you haven't had a regression? And so you know we think the best way to test a code change is with realistic data. So for instance run your pipeline in development with yesterday's data and today's code and the data contract proves that they work together. And so finding these problems in development against real data is a great way to make sure that you don't have regressions that your customers don't find problems and the contract especially it's tests not so much the description but the sort of corpus of tests that you have in the contract can really help you prove that you don't have any regressions.
00:32:08 Where to start: bookend your pipelines
Chris Bergh: So that was the sort of four areas sort of at the supplier on ingest on delivery and in development or CI and CD. Could there be more places? Could you have more in your you know bronze, silver, gold, titanium levels? Perhaps. Yeah. But these are a great place to start. My sort of favorite way to use data contracts honestly is one at the supplier and two on delivery. Those are great places to start. Get your supplier to do it. And if you can't get your supplier to do it, start doing it on it. Start doing number two and number three on ingest and on delivery.
00:32:47 Measuring contracts: pass or fail, and sharing the results
Chris Bergh: Sort of bookend your pipelines with contracts. So going to the next one, how do you measure data contracts? So, I've got a contract. I'm understanding why it's helping me communicate with my suppliers or my customers or my developers. What do I do? Well, the first measurement is did the contract passed or fail. It's as simple as that. Run the contract and then share the results. And so and in our product, the way that we have several ways to share the results and you can give them you know, you can give them an access, send them the URL and a login to TestGen to be able to look at the failed result. You could go in and let me go back to an example here that we have of our default data. So let's here we've got some faults in it. So, I can say here's here's what's failed. So, there's a pattern match in the credit card. I can actually go in and either send them the link. I could then take this and generate what we call an issue report, which is a PDF of exactly what the problem is. And so, here it has the table and the threshold values and some SQL on to be able to recreate it. Or you could take all these test results and just export all of them as and here this ends up being just a big old Excel file of all those. So there's lots of ways for you to share the test results back into the either give them a link, give them an issue report or actually just give them a spreadsheet. And communicating again is very important. So going back to my deck here is that's the really the first place to measure contract compliance.
00:34:46 Turning descriptions into tests: contract test suites
Chris Bergh: Did it pass, fail? What pass? What didn't pass? And so one of the challenges with contracts is that there's a lot of terms that are descriptive. And so you need to having just a description is great. It's informative, but it doesn't have any teeth. So turning those descriptions into tests is really important. And so as a result with our TestGen, we actually have a special what we call a generation suite. So if I want to go to test suites and I want to add a test suite and add a contract, I could say okay, I'm going to add a contract for default. I'm going to name it demo. I'm going to hit next. And we have sets of suites that we test suites that are have varying degrees of tightness or looseness. So there's a contract test suite that has one a standard one a table comparison that is actually very close to say you if I want to compare a database with one table to another table. And so being able to go in or I can just upload a test suite here. So, being able to being able to show that. Oops, I for I guess I forgot to show I forgot to show that in TestGen. So, let me just go back and kind of take you through what I just described. Sorry about that. I want to add a test suite. I want to add a contract test suite. I'm going to name it demo. And then I'm going to go and hit for table group default. Hit next. And here's are the different ways that we sort of corral or generate tests. Some which are varying degrees of tightness on how close you want to test. A standard one is some looseness table comparison is very exact contract isn't so one of the point I was trying to make in this is that the open data contract has a description part and then a test part and it's important that you have enough test coverage to make sure that you can enforce the contract.
00:36:54 Contract metrics: pass rate and coverage
Chris Bergh: So that's the first way you can measure going back to this tab. The second way is kind of looking at metrics. What's sort of the pass rate and the coverage. And so, what you want to think about is like I have you want to look at sort of pass rate over time. If I have a contract on my supplier and last week they missed 10 terms, this week they missed five terms, what are they going to do next week? Sort of keeping track of contract metrics, sort of the pass rate, and then keeping track of the coverage of how many tests are in the contract compared to the tables that you have. And so both of these things are available in TestGen. We don't have a specific UI for them right now, but you can get them through export. And actually you can get them pretty directly that the pass rate just by looking right at our UI. And so these are important metrics. Did it pass fail? What passed and what failed? And then kind of looking at the pass rate over time and the amount of test coverage that you have.
00:38:00 Who finds value in data contracts
Chris Bergh: These are important metrics to look at how good your contracts are. And then lastly, who finds value in data contracts? Well, I think everybody on data teams in the greater data and analytic value ecosystem has got communication issues. And so if I'm a data engineer, helps me communicate to my supplier, helps me communicate to my customer, can stop blame, can stop fingerpointing. If I'm a manager, the contracts can give me some visibility into what my team's doing. How good or bad the data suppliers are. If I'm a data steward, it can give me some visibility into can I trust the data that I'm giving. Right? If I've got a data product and I've put all this work in describing what it is, do I trust it?
00:38:46 How to start, and the two editions
Chris Bergh: And do I trust this specific version? And so I think everyone in the greater data in the greater data ecosystem can improve a communication and to that end lots of tests and contracts can help that. And so how do you start? Well, you know, Like I said, you download TestGen. Takes about five minutes to install. Grab a data set, connect it, write the contract, and then run it and then share it. I think it's really easy to start with one contract, as I showed with using MCP. You just have to type a few words to make it happen on your favorite data. It it just is really slick and really can get you somewhere very quickly with your data suppliers. And so the data contract feature ships in both editions. So it's the full contract engine. All the tests run Apache 2 with the MCP. The enterprise is the same engine but it's got multiple user roles and multiple user connections and a great flat price per connection. And so you know we're we're here one of the reasons that we built TestGen is that we believe that this collaboration problem over testing is a challenge for every team.
00:40:04 TestGen compared with dbt contracts and tests
Chris Bergh: So having a place where you can keep all your tests where you can share them have a UI to monitor them have an MCP to interact with them have a whole library of tests of 100 tests and the associated profiling data. So I think there was a question of what's the what's the value of TestGen over dbt contract and dbt tests for us we it really think it just comes down to that the point of contracts are to collaborate and so being able to collaborate in a UI being able to automatically generate tests being able to have enough coverage. So I think if you're using dbt contract and dbt tests, you're in for a lot of YAML writing and editing. Whereas TestGen, because it has the profiling information, it can autogenerate the tests and because it comes with the UI in the open source version, you're you're automatically ready to collaborate.
00:41:01 Questions: suppliers who refuse, ownership, where to start
Chris Bergh: And so I've reached the end. I've got a couple of questions that I think people would be interested in. If you've got any more questions, please add them into the chat. So how about what if my data supplier refuses to run it? Well, my answer is you just run it for them. And you know, you can't sometimes you know, you can bring a cow to water, but you can't make them drink. Well, if you've got a contract on your side, at least you're going to check it the instant the data arrives. And so, who owns the contract, the producer or the consumer? I think that's a good question because I think you should think of it as a shared ownership. If the consumer can have the ability to edit it or version it to improve the quality, why wouldn't a producer want that? So, I think instead of it's your contract or my contract, I think of it as a collaborative a point of collaboration to be able to work on it. And where should I start? And I think I mentioned this. You know, I'm a big believer in trying to check your data when it arrives. So, having a contract that's shared with your supplier is the obvious first place. And then how does this help communication? Well, I I you know, I think that's really my theme in all of this is that communication's hard. Having a standard to communicate over is good. Getting out of the blame game and the shame game in data and analytics is a really an important way to drive productivity to drive customer success and so by having a well-defined format that people can edit that they can use that they can see in a UI I think that really helps and so you know there's this old adage good fences make good neighbors and so let you know let's let your contract be your fence And so that's it for today. Any other questions that I can answer? All right, I don't see any more. Give it an install. We're really excited about this feature. And try it out on some data and give us any feedback you have. Thank you much.
Machine-generated transcript, lightly edited: filler words and repeated words removed, product and speaker names corrected, and one attendee's name removed. The recording opens at the logistics introduction; chapter times are aligned to the published video.
Questions from this session
What is a data contract?
A written agreement between a team that produces a data set and someone who gets it or uses it: a promise of what they are going to get. Chris Bergh describes it as a schema description with proof points and meaning on it. Unlike a file interchange agreement, a contract carries checks, so it tells you whether it is working, and it helps the two sides communicate their expectations.
Why isn't a schema description or data interchange agreement enough?
They are useful, and much better than a file with nothing next to it, but they are usually descriptive spreadsheets. They have no checks to prove the data matches, little meaning, and nothing about what happens when something breaks. A contract adds the tests, so it provides evidence instead of finger-pointing: here are the terms you agreed to, and here are the three that aren't true.
What is the value of TestGen over dbt contracts and dbt tests?
Chris's answer is that the point of a contract is collaboration. With dbt contracts and dbt tests you are in for a lot of YAML writing and editing. TestGen has the profiling information, so it can generate the tests automatically, and its open source version includes a UI, so you are ready to collaborate with suppliers and business users from the start.
What if my data supplier refuses to run the contract?
Run it for them. You can't make a supplier use it, but if you have the contract on your side you can at least check the data the instant it arrives, which is the second gate, on ingest.
Who owns the contract, the producer or the consumer?
Think of it as shared ownership. If the consumer can edit or version the contract to improve quality, there is no reason a producer wouldn't want that. Chris treats the contract as a point of collaboration rather than your contract or my contract.
Where should I start with data contracts?
Check your data when it arrives. A contract shared with your supplier is the obvious first place. Chris's favorite pair is a contract at the supplier and one on delivery, bookending the pipeline. If your supplier won't run one, start with contracts on ingest and on delivery.
How do data contracts help communication?
Communication between data teams, their suppliers, and their customers is hard. A standard, well-defined format that people can edit, run, and see in a UI gets everyone out of the blame game. When something goes wrong, you add a new term to the contract together so it doesn't happen again. As Chris put it, good fences make good neighbors, so let your contract be your fence.
How do I measure a data contract?
First, did it pass or fail, and which terms failed. Share the results with a link to TestGen, an issue report PDF with the threshold and the SQL to reproduce the failure, or an export of all the results. Then track two metrics over time: the pass rate, such as a supplier going from 10 missed terms one week to five the next, and test coverage, meaning how many tests the contract has relative to its tables.
Where to go next
- Install open-source TestGen Apache 2.0, runs in your own database. Docker Compose to a first quality score in about 15 minutes.
- Data contracts ship in open source TestGen The November 12 release: create, version, export, and import ODCS contracts in both editions.
- The data contract is your receipt Why a contract earns its keep only when its terms run as tests.
- Every on-demand webinar The full recording library.