The Seven Deadly Sins of AI-Generated Data Pipelines

00:02:02 Welcome and housekeeping

Chris Bergh: We'll start in one minute. Wait for a few more people to arrive. Okay. Well, we we'll start. My name is Chris Bergh. I'm CEO of DataKitchen and I'll be your host of the webinar today. Before we start, welcome. Our slides and recording and transcription will be shared this week. In the lower right hand corner of your window is a chat window from from Google and put your questions in there as they as you go along. I'll try to answer them and we'll have time for questions. Questions at the end and we've targeted for 45 minutes for our presentation and if you have time we're going to do a demonstration at the end. So what's our agenda? I I think at a really high level, you know, people are using AI to write code and lots of software engineers, lots of data engineers and people in data and analytics are creating code. And so how do you want to engineer your data transformations, your data pipeline code for success?

00:03:34 The seven sins, and what this session covers

Chris Bergh: That's really the the theme here. So how do you get successful? Well, we're going to talk through kind of what are seven problems in doing that with a metaphor of seven deadly sins. So, we'll talk about gluttony and pride and greed and envy and sloth and vain glory and lust. And then we'll give a a few things at the end to summarize. So, the the challenge is you know Claude Code got started to get really good about November last year. And then everyone started to notice, hey, I can do this to write SQL. I can do this to write Python. And it started to know more about DBT and about Airflow. And so it became possible then for you to sort of create and edit and modify whole data pipelines and data transformations with just a few words and in fact you could talk to it. And so that got fast. And the problem is what didn't change was your data engineer sort of understanding of the problem, understanding of the business context, and the fact that like you're creating a lot of code in some ways means your code knowledge goes down.

00:04:46 Producing code got cheap; verifying it did not

Chris Bergh: And so producing code got cheap, but in some ways verifying that it's the right code or verifying that it is doing the right thing didn't. And so and one of the biggest problems with software is or with data and analytics is sometimes the difference between running something cleanly and getting an error is hard to see. And so as a result some stats are you know there's a stack overflow developer survey and this might have mixed software developers and data engineers but I think I think it's pretty clear that a lot of data people are using a lot of AI tools. I know in our company in our consulting organization where we do data work almost everyone is using Claude Code on a daily basis. But the problem is you know the trust goes down. And so you may have this sort of big hunk of code and then you don't know if it's quite right. And so that's one of the challenges. And then in researching this sort of looked at our data engineering and some research.

00:05:50 What engineers are saying about cheap code

Chris Bergh: And there's some themes that came up, right? One engineer in a data engineering post basically said code became a cheap commodity. And that obviously wasn't true even a year ago, right? So, it's it's pretty easy to go into Claude or your favorite LLM and create a thousand lines of code or 3,000 lines of code in an hour. And that used to take days of work. And so, as a result, you're iterating more. You know co your tool is producing code but then you're checking it and iterating it and updating it and it becomes easy then to sort of do whole scale refactors on your code and so like I said a lot of times with data code it runs and it runs fine it's just is the data that it produced right did it cause a regression in other data sets and then a lot of problems with data systems are that the meaning of them is not inherent in either the data itself or in the transforms. So, we've all seen cases where there's status or what's blank and you just don't know what that means.

00:07:02 Why AI makes an old problem harder

Chris Bergh: And so, that's always been a problem in data engineering and data and analytic systems, but it becomes more challenging when your AI is trying to write your code for you. And so, but one of the problems in data is like it's different than writing software code. You write your code, it either runs or it doesn't, and your tests run or it doesn't. And a pipeline, it can all run, but the data, it could run perfectly, but your data could be wrong. And so, how do you get that right? How do you know that your pipeline ran? Not just so it didn't throw an error, but that your business users are going to actually give you the right are not going to yell at you for having the wrong data. And that's gotten harder when you can kind of burp up a 10,000 lines of code in an afternoon. And so, you know, you can just do a lot, you can iterate a lot. And so what we're going to talk a lot here is about how to make sure that when you do that, you're getting things that actually work and meet the customer's needs.

00:08:07 When code generation is free, verification is the constraint

Chris Bergh: And so, you know, kind of a meta theme of this is when code generation is free, data ver data verification becomes the constraint. So, we're going to talk through this now. We're going to go through each one of these seven sins and we're going to start first with kind of this sort of how do you construct a pipeline of a series of steps a DAG so that this works so you can live with this. So the first thing is that if you've ever used tools like Claude Code iterating is really fast. You sit there and use your whisper dot, you know, you use your desktop whisper program and talk to your code, dictate a few things. It builds a bunch of stuff and then it's like not quite right and then you want to run it over again and run it over and over over again. And what happens if you build your pipelines? In a non what's called idempotent way. If you run them over again you get an you get additive results.

00:09:06 Idempotent pipelines and re-running safely

Chris Bergh: So the first run here in this diagram you get a thousand rows but the second run you get a thousand more. So an idempotent transaction means that you've run it over and again and again and you get the same table every time. So iteration is incredibly fast in AI tools. Trying things is incredibly fast. But you don't want to have these unwanted side effects. So, building pipelines that are idempotent are really important. And so, and so what happens, and here's a bunch of text on it, that talks about it, but really the idea is, can I success? Can I try something, try it again? Did the LLM hear me just right? And I want to keep looking at it. And we've written about a pattern that we use called functional idempotent tested and two-stage as an architecture pattern. We've written about a couple of that and done some blogs. We use it in our own consulting pack practice. But the key thing I think is is item potency.

00:10:12 Why an agent needs to re-run the pipeline

Chris Bergh: Being able to run the pipeline over and over again because your AI tool is going to want to do that because it's not going to hear you the first time. And there's techniques to get around that, right? Have write the spec first talk to it, review it, but you're still the mental model I have of AI is a very eager, very smart, you know, 21-year-old intern who really kind of knows how to produce code but doesn't know what it means. And so, you're going to have to check the answers. Coding in AI is a lot more like management than it is actually doing individual contributor work. And so the second thing here pride and and the pride is is kind of you you understand what's going on I think as an engineer and so well if I understand what's going on doesn't my AI model and if I type up a bunch of good things or I talk to it enough will it be able to figure out what what it means? And as anyone who's done data knows, there's just a bunch of hidden context in your actual data itself.

00:11:19 Context as part of the process

Chris Bergh: And so that's the key here. You need to have context as part of your process. And it could be definitions of terms. It could be join conditions. It could be example queries. It could be a whole bunch of things. And we've written and had a a whole webinar on what context means in data and analytics. And there's kind of in in our work we've been learning there's kind of two kinds of contexts that are almost intersecting but not quite the same. One is the context that we're talking about here for the data engineer to be able to create pipelines and create transformations. There's another context that has even more in it for an analyst to actually query the data to get the right right results and they're related but they're not exactly the same. And so for instance the data engineer needs to know a lot more about the process steps that go on whereas the person who's just doing queries of the data to get insight doesn't need to know very much of that at all.

00:12:18 Context as a deliverable

Chris Bergh: And so building context as another deliverable that goes with your data engineering is is also important and in order to feed your model so you don't have the pride that the model's going to be so smart it's going to do things. So in some ways it's it's helping your model give things correct. So now we're going to go on from how you construct a kind of testing and reliability. And so, we're going to talk about greed and envy and sloth. So, greed, well, you're writing more code. Did, you know, how do you know it's right? And so, test coverage, specifically data test coverage across your data pipeline is really important. And how do you get we've we've had a number of discussions and blogs and written articles about what test coverage means in a data and analytic system. But first of all it means really if you change something I think of it the biggest principle is non-locality. If you change something small in one part you can tell if something broke in another part and that you've actually covered enough columns and tables.

00:13:34 The coverage rule of thumb

Chris Bergh: And a rule of thumb that we use is every column should have at least two tests. Every table should have at least three or four tests. And so you the numbers increase quite rapidly as the number of columns and tables go up. And so getting enough tests to make sure that you that principle of non-locality, you change something in some place, you can tell if it broke in another is really important again because AI can write code quick. So it can change things quickly. And so if you look at it from everyone's favorite architecture paradigm, medallion, you want to be able to say as each layer, your bronze, silver, and gold, you want to cover all the tables in each one of these layers to make sure that you've got enough coverage. And then you also want to cover your tools, right? Because if airflow throws an error, that is a problem, right? And so, you want to make sure that you cover all your tables and cover all your tools to make sure you've got adequate testing in this very complicated world of medallion architecture.

00:14:42 Data lineage is not enough

Chris Bergh: And so, here's maybe a controversial one. Data lineage is not enough in this world. And so, the way I look at it is data lineage is a is a blueprint. And if this column in this table it touches is in some ways all these other tables and columns and that's great, right? It touches it has some relationship with and the kind of abstract syntax tree they use to create lineage pulls all the queries and all the evidence and kind of gives you almost a reverse blue blueprint of what happens. But it I think in this case you don't need that because what your AI model needs to do is say something changed here on the left hand side but something on the right broke, not everything else. And so if your lineage says this one column touches five tables, you don't want to have to manually check five tables. You want to know that the one out of the five is the one that broke. So from our perspective, full coverage means you need to have Ss across every table so that when you make a small change, you can see its effects and the AI model can see an effect.

00:15:59 The fire alarm analogy

Chris Bergh: And lineage is nice. But if you have, here's an example. If you have a fire in your building, the first thing that you're not going to do is pull out a a blueprint. You're going to actually go to the fire alarm co control panel. And that's really what test coverage is both in development when AI is making changes and in production when when the codes change and they're both useful because in in production you could have a case where the data changes but the code is fixed and in development of course your data is fixed and your code changes. And so we've written about that and have a couple of talks about how to get test coverage and how to compare and contrast to lineage. And so as my own career as a data engineer, I almost rarely ever use lineage. And our customers or at least my customers had very very very few errors. And so lineage is a helps you understand and I can understand its its use.

00:17:01 Coverage rather than lineage

Chris Bergh: But in the case of AI, you really need test coverage. Lineage is a is sort of a an artifact that's produced, but not doesn't help your AI model iterate to get the answer. And then sin five, sloth. Well, a lot of people are very busy in data and analytic teams. And so in in my mind, a lot of people have this crunch like their business customers are asking for 20 things. They don't trust their data providers. And so they're just focus on getting through the day. They focus on their task list. And so, creating these tests is hard. And so, as a result, a lot of data engineers don't do it because number one, they don't know what tests to write. And then number two, they don't have enough context of the data to know. And so, what we think is that 80% of the tests are sort of syntactic. You can let them machine write those. But there's 20% of your tests that actually really have to do with the context of your business.

00:18:04 Freshness, volume, and the tests only people can write

Chris Bergh: And those need to be written I by your data engineer or in context with other people. And so things like freshness and volume, distribution, schema drift, value range, we've got a list of of dozens of tests that can automatically be created. And however, there's ones that have to do with what your business needs. When your fiscal year is what do what is actually an equivalized unit mean and what is too much what is a big change and these things really take a village to get these sort of business rights you don't know as a data engineer you don't understand what's going on in your business it's a rare data engineer or data scientists that do that it becomes kind of a community thing in that the context think of it at like A lot of people are touching the elephant and in this because AI can create a lot of code because lineage is a blueprint and doesn't give it you need test coverage. So you need a lot of tasks you need two types of tasks ones that can be syntactically created and then ones that have to do with the semantics of your business but that semantics of your business actually fits in a whole bunch of people's heads.

00:19:22 Sharing test knowledge, and a UI to manage it

Chris Bergh: And so we believe that the the data test knowledge needs to be shared and you need a UI to manage it and a team to help manage those tests and help you work through those. And this is very a very different idea than most people think about data testing because you know it's if in a lot of companies if testing gets done at all data testing gets all it's a data engineer but in my experience tweaking the tests parameters what tests it takes a dialogue with people and having kind of a database of record of all those tests is it makes it very effective for people to have these two buckets of tests where you know the machine writes 80% but you've got these contextualized organization specific that are in a bunch of people's heads and you need a UI to help manage that. So, and then you know it's also hard to build a system like the current design of Medallion where you've got source data, bronze, silver, gold. It hops through three steps. You know, you want to be able to do tests against your source data to improve data quality.

00:20:31 Where tests sit across the pipeline

Chris Bergh: As your data is ingested, you want to do things like frequency, volume, schema. As it's going through production, you want to put TWIP crip wires to see if anything breaks across all your tables with full coverage. And then of course you want to do a lot of those tests in development. And so there's multiple places to to put these trip wires and put these data tests to make sure things are right. And so going on to going on to part three. And again, if there's any questions, feel free to jump in and and join us. Put some put some things in here. So, we're going to talk about the last two parts here. One is vain glory or being vain and the other one is lust. Attendee, you have a you have a question.

Attendee: Yes.

Chris Bergh: You can feel free to or talk.

Attendee: Yeah. Yeah. Sorry. So, when we talk about test coverage, how do you write test coverage for tables which are like huge like very wide 200 odd tables and like billions and billions of rows.

00:21:43 Q&A: coverage on very wide, very deep tables

Chris Bergh: Yeah. Well, what I mean by test coverage is sort of invariant of the number of rows in the tables. The way we calculate it is really about the number of columns in the tables and the number of tables itself. So we've got a sort of rule of thumb on how we think about having a couple of tests on every column and then a couple of tests on every table. And so the tests themselves are usually pretty quick to run. And so in our tool we autogenerate those tests, those syntactic tests. But yeah, that's that's one of the reasons why we built the tool is because we were doing the same tests over and over again in our consulting and our engagement. It's like why do we want to why do we want to recreate the wheel? And and from an execution test those those run fairly

Attendee: Yeah, I asked this because so what happens is like when we inference a table or like it

Chris Bergh: Quickly.

00:22:45 Q&A: append-mode tables and write-audit-publish

Attendee: Inference like thousand rows of a table and like we write the test coverage on top of that but sometimes what happens is like over and like over a time period when you build a table like in an append mode or like majorly in append mode. Sometimes at the source the like one of the columns data type is changed and at that point of time when you append it it fails straight like it fails in the next step like like it will append the file but when you process it one of the file will have a different kind of data type and it will fail.

Chris Bergh: Yeah.

Attendee: So how you like handle these kind of issues?

Chris Bergh: Well, it becomes less. I guess the rule of thumb is find the problem first before it gets in. And there's two architectural patterns that you can use to do that. There's a pattern called write audit publish where you do the append, you do the write, you audit, i.e. Test it and then only it's after it's tested you push it in. So your append, do you test your append before you actually do the append?

00:23:59 Q&A: finding problems before they land

Chris Bergh: End.

Attendee: Okay.

Chris Bergh: And so that's one pattern.

Attendee: God.

Chris Bergh: We do a very similar thing in our our FITT architecture where we do that sort of that as a big we take that and sort of step it up one chunk. But the idea is you want to find problems before they get into production.

Attendee: Okay.

Chris Bergh: So and you can do that either by in the case of this is this this discussion is really about AI generating code and your AI

Attendee: Yeah.

Chris Bergh: Is generated code already, right? This is more about what happens in development, but in production the the rule still

Attendee: Okay.

Chris Bergh: Applies.

Attendee: Thank you. Thank you.

Chris Bergh: And Attendee says, "Can these be used with the dbt agent?" And by these, do you mean tests? I'm What do you mean by these? I'm not sure what what what that refers to.

Attendee: So these seven these deadly sins as you have you put them is there a way that they can work with the chaos agent like if it can test for let's say trip wires or like we're trying to build a program to improve data analyst into Alex engineers.

00:25:12 Q&A: chaos agents and the sins

Attendee: So it' be a good way to test where their skills at instead of relying on AI. So the chaos agent will actually give them the wrong answer. They'll know what's wrong and it would be anchored against one of these sins. Do you think that'd be possible to replicate easily?

Chris Bergh: As a you know I can't say yes or say no I have to learn a I mean I'm familiar with chaos agent testing from software and so if you're throwing a bunch of random data at a data pipeline the you know the the idea is that the

Attendee: What's

Chris Bergh: Tests will catch errors in that data and the the coverage will will catch it and the and if you put the

Attendee: The

Chris Bergh: Tests in in the right place, you'll see that the trip wires will stop. So theoretically, yes, but I, you know, I'm certainly willing to talk more when when we get done about this. I'm not I'd have to learn a little bit more about the chaos agent, how it creates test data and how it runs the profiles, etc. Or how it runs the how it runs dbt.

00:26:09 Q&A: recording logistics

Chris Bergh: But in in in principle, yes.

Attendee: That's excellent. And are we getting a recording of this at the end of the day or the end of the week, just for my own

Chris Bergh: Yeah,

Attendee: Reference?

Chris Bergh: I'll I'll probably we'll probably produce it tomorrow. It should be up.

Attendee: Perfect. Thank you.

Chris Bergh: So, we got a lot. So, again, this is, you know, the the question is, you know, you've got an AI tool in development that's writing a lot of code, right? And so one of the interesting things about using code and using AI is that it it iterates a lot. It sort of and it grades your homework and there's this sort of sycopency or giving you words that you want to hear like, oh, you're really smart. Oh, that sounds great. And so one of the challenges I find in data and analytic systems is how do you get over that sick of fancy? And so the challenge here is how do you when you run it, how do you know that it's right?

00:27:11 Unit, regression, and production tests

Chris Bergh: And there's different levels of tests and data and analytic systems that happen in development. One test is called a unit test, which is I make up a little bit of fake data. I run the the one part of my transform and it passes or fails. And and those are useful. They're very fast. But we're talking about something a little bit more consequential where you know, you're you can, as any person knows, your unit test could be all perfect, but you need more end to end functional testing that is based upon good test data. Like in our practice, we sort of run yesterday's data with today's code and to see if it's right. And so that's what you need this sort of independent harness with good test data. You need to kind of think outside in on what happens with your system to be prove that it's right. And it's not that unit tests are useless. It's just that you need to be able to because your agent is trying different things.

00:28:12 Sub-agents trying different approaches

Chris Bergh: You create sub agents that try different ways. So I've coded times with half a dozen different sub agents running and doing something at the same time and you end up picking the winner. You need to really have this sort of outside inspector that can that can tell. And likewise, when it's fairly easy to kind of go from your prompt to like look, I wrote some code. Look, it worked on my machine. I'm going to go check it in and my CI is going to run and hey, I've I've done this fast. And so that's always been the case, right? Or people that patch pipelines, people who who know what's going who think they know what's going on but don't. And so that does work for some people because some people have the whole code knowledge in their head. But what happens with AI is that knowledge of all the code is getting lost because people are using AI to write the code and they don't have the full system in their head.

00:29:05 Letting the agent write, run, and see failures

Chris Bergh: So you actually need to be able during the development to have the agent write, run some tests, see what failed, and then try it again. And don't just trust unit tests or don't just trust that it ran green. And so that sort of lust to get things in production is something that you have to hold off and you've got to, you know, put tests in. And one of the ways that we think testing is really important is to have you know our opinion of the world is that you know tests should be part of the pipeline and so some of the tests can be embedded in the pipeline like there's dbt tests or our orchestrator has tests in them and some of them can be put out in a separate database as a trip wire especially the ones that are complicated that have to do with business knowledge and So, that's really where we think about this. And so, MCP makes this pos possible to have sort of a a separate test engine outside of your pipeline.

00:30:09 Summary of best practices

Chris Bergh: And so, what's what does this all mean? So, I'm going to I'm going to finish up here and then we'll answer some more questions if people need. So, the first part is, you know, we're in development. We're talking to a a model. It's making changes. You're creating sub agents to do the work. Things are happening fast, right? And they're happening over and over again. So, you need item potency. You need to be able to run a pipeline over and over and over again and not and and be able to kind of start from the beginning. And then second, for the a the agent to work at all, you need sort of data and pipeline context specific to the data engineering task separate from the analyst task related but separate. And the third is you need test coverage. You can't trust a lineage. You need test coverage across all your tables. And some of those tests have to be business context tests which should be shared with your business user.

00:31:09 No open loops

Chris Bergh: And then fourth is sort of no open loops like you you know don't because you can do things fast doesn't mean you actually should. And you know verification is engineering work, testing is engineering work and teamwork. And just because your AI is giving you compliments left and right, don't trust it. And so that's our sort of view of the world. And the last two slides are more of an advertisement. So we happen to have an engine that is does automated testing has a UI for business users, tries to get you 80% of those test coverage that you need out of the box and has a nice UI MCP so you can build those tests in conjunction with your organization. So you can then tune them and tweak them and add them and it's open source, runs on your infrastructure. Has a very reasonable enterprise price. And so lastly, we've really been enjoying our MCP server that we put on top of it. It's complete.

00:32:09 What the MCP interface exposes

Chris Bergh: You can do almost everything you can in the product. You can profile, create tests, run tests, edit tests. And then there's a REST API to programmatically put that as part of your a part of your pipelines or your CI pipelines. So, we've got some blogs and and talk about it. And, so that's exciting for us. We just I think and you know, for one of the things I did, recently was I downloaded within like five minutes, I downloaded some New York taxi data. I profiled the data. I ran our test engine on it. It gave some tests, but then I asked Claude to recommend some tests and it recommended some tests saying,"Well, no New York taxi can can drive a,000 miles on a trip or no price could be over $10,000." And it created some great rules. Based on that context. And so, we've got a blog on that, that that I think is really cool and just just makes this very fast to be able to say, okay, because the the biggest problem here is you can code really fast, but you need to have pair that coding with a test engine to make sure that that all works.

00:33:18 Fast to install, and over to the demo

Chris Bergh: And so, it's fast to install, fast to use. And I'm going to stop there. I finished a bit early, but see if there's any questions I can answer. All right, I think Chip from our team was just going to do a quick demo to talk about some of these things. Chip, are you ready?

Chip Bloche: Yeah, I would I I saw that Darcy has a hand up. Is that is that a

Chris Bergh: Yeah. Yeah. Attendee, do you have another another question or is that just from

Chip Bloche: Question?

Chris Bergh: Before?

Attendee: No, no. That's from before. Sorry.

Chris Bergh: Okay. Yeah, I did that for you. All right, Chip,

Chip Bloche: Great.

Chris Bergh: You're on.

Chip Bloche: Okay. So, let me share my screen because I I wanted to, give you just a brief demo of what the, MCP looks like with Claude code. We're, as Chris said, we're really excited about it. Second here.

00:34:35 Demo: Claude Code on the TestGen demo database

Chip Bloche: So this is a this is a a basic Claude Code screen. We're attached to the demo database which comes by default with the TestGen installation. A and this is a fresh a fresh conversation. And I just cleared it and a and I the I want to kind of show you how a very simple walkthrough might look like if you're if you're inter want to interrogate TestGen for information about data quality. One of the really interesting things about using TestGen in con TestGen MCP in conjunction with an agent is that it it lets TestGen act as kind of a ground truth deterministic source for data quality information. So there's of course there's a lot of stuff you can do just with a plane agent. You can in you can write queries against a database. You can surface information. Everyone has been experimenting with this and and there's some tremendous capabilities there. But I I I think what TestGen does in this context is to allow you to improve your results.

00:35:53 Deterministic results instead of fresh inference

Chip Bloche: You're not asking the agent to do fresh inference. Each time it's looking at data quality issues. You're reducing the potential for hallucinations. You're making you're rely relying more on deterministic results that you know are real and and it this this can help you really in a wide variety of areas. So, so starting out let's say I have TestGen here and I've done some u testing and I can ask a question here. So, oops. So what are the most important data quality issues to deal with? Well, one one challenge that people have dealing with any kind of automated data quality system, TestGen included, is the you know the the sheer amount of signal that you get. You can get a lot of issues. You can be inundated with with test fails and you sort of start to tune out. The the Aentic interface gives you the opportunity to use the information about about prioritization in TestGen to help you decide which are the most important issues to address.

00:37:39 Using the issue model rather than raw inference

Chip Bloche: And that's what this is doing. It's it's actually looking at the data quality issues, but it's also using what we call the impact dimensions to help help to figure out which are the most important issues to deal with. You and here we go. So there there are quite a few things here, right? It's it's pointing out you know this data is a little old. So so you can see that the dates are a little are a little old, but it's it's pointing out what it thinks is the biggest set of issues. It looks like it may have identified a bad customer data load. And notice this is really interesting, right? And notice that it's taking individual issues that TestGen has identified and it's bundling them. It's it it it's suggesting potential reasons why some of these issues are related and might be together part of a larger problem. And I think that's really interesting, right? That that that again is the combination of working with TestGen's ground truth data collection on quality issues and using the agents capabilities for inference really gives you some cool results.

00:39:14 Demo: a duplicated customer ID

Chip Bloche: So let's say you know we have an issue here with bad consumer data. There's a customer ID duplicated value that's a big issue. There's there also potential issues in one of the fact tables. You can dig down a little bit deeper here and say, let's see here. So, let's see. Let's see what the issue is with duplicate IDs. And and and I the interesting thing about this is when you do a demo, you never know exactly what you're going to get because this is probabilistic. So So it'll be it's always an adventure, but you get the idea. Oh, wait a second here. Let me just fix this. Let's try this again.

Chris Bergh: Chip, I like your red

Chip Bloche: I have so many different Claude Code sessions up at once.

Chris Bergh: Background.

Chip Bloche: I give each of them a different color. Okay, so here we go. So, it's pointing out there's one duplicated ID that's in 26 rows that accounts for all of the duplicates.

00:41:49 The query that isolates the issue

Chip Bloche: It actually gives you the query to run in order to identify just what the issue is. It's this it's variant of same query that was the original test query and it it points out that these 26 rows are entirely different people. So this is really interesting, right? This is this is this is giving you more information than the test alone gives you on on the issue. Is able to do some additional investigation and and can help you follow up on the problem. And it's it's it's describing multiple related issues with the load and really gives you someplace to go to do further troubleshooting. Let me approach this from another angle too because I was just playing with this. This is kind of interesting, right? Let's let's say you are working on an analytic query and you're you're kind of approaching this from the opposite side here. Let's let's say you're working on a a query and you want to say okay I want to run this query against my live data.

00:43:05 Asking what could go wrong with a query

Chip Bloche: What could go wrong? You know are there any data quality problems that might relate to this query? So I did a simple query here that is a grouping query, right? Looking at sales sales revenue by region and product type and counting the number of customers. Oh, you know what I didn't do? I didn't hold on. Let me I realized I didn't ask it what to do here. Evaluate this query. Will I run into any Here we go. So let's see what it comes up with here. This is this is really a different kind of use case. But it t it takes advantage of the idea of having a a ground truth source of information about the quality of your data. So that even if you let's say you're dealing with data quality issues that you're not able to address. There might be issues that you know there are lots of reasons you can't address issues there. Your weight.

00:44:59 Upstream fixes and downstream dependencies

Chip Bloche: Waiting for upstream fixes or you you are you have downstream dependencies that you don't want to change. You want people to be aware that these issues exist so that their work downstream of you is not affected by the issues and they can take them into account. This is an example of a way to do that using using the TestGen data that is collected across a a a big window on your data set. You're not just you're not just with testing you're not just looking at a few critical data elements you're actually spreading your net very wide and this is a way that you can make use of that. So here you go right. So, so we have multiple issues that this particular query is going to

Chris Bergh: And Chip, I should add one of the things. So,

Chip Bloche: Hit.

Chris Bergh: So why does this work? Well, TestGen profiles all your tables, right? So it's got not just the the description of the tables, but it's also got semantic content and a whole bunch of profiling information.

00:46:10 Pushing context to TestGen and Claude

Chris Bergh: So it can use that. We're sort of pushing context to TestGen or to Claude to be able for it to be able to make sense of this and and and do it. And that's actually one of the great things here is like if you didn't have that context, Claude would have a lot harder time making these kind of judgments.

Chip Bloche: That's right. It's it's context and it's deterministically developed context which is really important in in with all of the confusion that can come when an agent is reinventing the wheel. So here's an interesting one, right? This this lists a bunch of issues that you can deal with. Maybe you can deal you can deal with them in a downstream query. But this one strikes me as interesting. They're different six different spellings and this came from a hygiene issue based on profiling of of product type categories. So you can say, you can you can ask for SQL. That could correct this issue in the source data.

00:47:46 Building tools to patch issues, and close

Chip Bloche: So, not not only are you identifying problems, but you're building tools in order to patch data quality issues. And this might be something that you do to temporarily. It might be something that you do in a for a view or a or a process table. It might be something a downstream user does when creating a u a final data set for analysis. So here we go. Right. So it gives you a set of queries that will pick one variant and update everything to the one variant and you can work with it. And and it also makes a very good point, right? This only fixes the data, not the cause. So, I the really interesting how the combination of the of the expertise of the of the agent and the builtin information provided by test can kind of work together to to make your troubleshooting more effective. That's and that's my demo.

Chris Bergh: All right. Well, thank thank you, Chip, and thanks everyone for taking the time. And we'll send out the transcript and the slides and the recording here today or tomorrow. And thanks again for taking the time to join us on our webinar. Bye everybody.