The Four Points in Your Medallion Architecture Where Data Testing Really Matters

00:00:00 Welcome, and what a medallion architecture is

Chris Bergh: Hello everyone. My name is Chris Bergh. I'm head of DataKitchen and I'm here to talk about where to test in a medallion data architecture. So some housekeeping before we start. I will share the slides and the recording and the transcription this week probably today or tomorrow. You'll get an email and if you have questions put them in the chat window and I'll answer them at the end of the session and we've targeted kind of 50 minutes for our presentation with questions at at the end. And so hopefully that will that should work and okay good. So let me let me start the slideshow and keep going. So what what are we here to talk about today? So there's a data architecture pattern called medallion that has gained popularity if not the default architecture in many large organizations. So we'll talk about that and then we're going to talk a little bit about where do you test data in a medallion architecture and we're going to identify four points.

00:06:35 Bronze, silver, gold, and sometimes platinum

Chris Bergh: Then we're going to talk about well you've what is coverage? How do you make sure you have enough testing and then some ideas on where to start and then a conclusion. So the medallion data architecture in generally it's a layered architecture. So what it means is following left to right, you take source data and you land it in what's called a bronze layer, which is in some ways a copy of exactly what is in the source data. So a table-to-table copy. From there you put it in a silver layer and and that tends to have some cleanup on it. So maybe you're mastering data, maybe you're patching some of the source data. And then from there you're putting it in a gold layer which is kind of fit for consumption. That may be aggregate tables, that may be facts and dimensions. However you want to polish your data to make it look good for consumption. And so sometimes after gold there's platinum. Sometimes they're called L1, L2, L3.

00:07:38 Dev, QA, prod, and 840 comments on why engineers don’t test

Chris Bergh: And it's just very popular in the in the lakehouse. And Databricks popularized it but this is a popular way to doing it doing doing it. And so another way to look at the Medallion Lakehouse is to look at it a little bit more holistically. Right at the very top here we have production where it goes from bronze silver but going from the bottom up we have development QA and production. And so this is where your data engineers are building a data transformation. They're adding a new data set and it has to go through a a process to get from dev to prod. And so that is one way to go about doing medallion. And so, just as an aside, we did actually use Claude Code to go scan the internet and get over 860 840 comments from data engineers about why they're not testing. And so, what was really interesting, right, and and we wrote a blog about this, which is at the end, but why aren't people testing?

00:08:43 Systems built on hope: no time, no support, no visibility

Chris Bergh: And not even a medallion. It's a lot of people are they build data and analytics systems on hope. They they don't they trust their data providers. They trust everything will work. And it was interesting that the a lot of the the topics were around I don't have support. I don't have time. Errors aren't visible. You know, you know, I everything works until I get yelled at. Things are changing too much. The tools aren't good. And you know it's just the industry is immature. And so I think this is also part of our goal in doing this is is to make it clear on exactly how you should do things and and testing data in a medallion architecture for the sole purpose of delivering insight to your customers that's trusted and error-free which has the added effect of making you have a lot more time to do cool stuff. So what should be tested in a medallion architecture? So I'm going to have a metaphor on this in four parts.

00:09:41 Four places to test, and the pets metaphor

Chris Bergh: Data quality, data observability, data production, and deployment. And I'm going to use some metaphors here around something called a PET and cattle and data and observability and the assembly line and the code factory and production and deployment. And so we'll talk a bunch about that. So the idea here is that when you're speaking of in the upper left here, data quality, you're trying to get data fit for purpose. You're really focused on the source data itself, improving it. And that's a hard thing because lots of organizations have lots of data and different owners. And one way to think of it is they're kind of pets. You want to improve them, love them, get people to improve them. And we'll talk about how we do that. And then the second is opposite of pets. It's cattle. You've got a lot of data coming into your data and analytic system from different places and different sources. And you want to sort of spot the sick cattle very quickly when you're looking at that herd of a thousand cat cattle.

00:10:44 Cattle, the assembly line, and the code factory

Chris Bergh: And that's really about anomalies. And then the fourth part here in the in the lower left data production. Well, you have an assembly line, right? You've taken that data from source. You've ingested it in somewhere right in a bronze place. Now you've got that production process. And the idea here is you want to stop if anything looks weird. And this is the case that you know data something could be wrong with the code acting upon the data or you only find problems when you've joined two or three data sets together and then the problems emerge. And we're going to talk about stopping the line tripwires and and use the assembly line metaphor. And then lastly is deployment. And that's where you know our from a system standpoint the way that we deliver value to our customers is not just data. We have a bunch of code acting upon that data in different tools. So how do you change that code and not have problems? How do you update a the name of a column?

00:11:40 One null check doing four different jobs

Chris Bergh: How do you add some new transformations and have it all work? So this is what we're going to talk about and use this way to these are the four points that we're going to talk through. And so the common thread here is that there are things called data quality tests which check data. So they're automatic. They are scriptable not manual. They are code objects and and so a simple test like I'm going to check for nulls it has a very different purpose right in very different cases. And this perhaps is another reason why it's hard to test. Like I have a I have a test that says check for nulls. Well, that could be a quality metric. You know, this column shouldn't be null. That's a problem with our data quality. Now, maybe some of the columns are nullable. But or or not, and you've got a column that's was full and it's suddenly null, right? And so or 50% of it is suddenly null.

00:12:32 The same test at ingest, in production, and in development

Chris Bergh: That's an ingestion. Maybe something's strange. There's an anomalous behavior. Something's changed over time. And in production, maybe you've got a fact table and suddenly one column's null in the fact table. You've integrated three data sets. Maybe you should stop production before that fact table gets to your customers. And then finally, everything's working and you're I've changed some code and suddenly a column's null that wasn't null before. And that probably means you've screwed up something with your your code in development. And so the same test really has four different places and four different ways it could be used. And so I think that's really interesting and why testing is so important, right? Because all of these end end up with problems in front of your customer. So let let's take this and sort of lay it out on the architecture that we have. And so the circle on the left is, if you remember this diagram, there's source and bronze and silver and gold.

00:13:30 Mapping the four test points onto the architecture

Chris Bergh: On the left here, we're talking about source data quality testing. And here we're checking data in source systems or its reflection in bronze systems. And then ingest data observability is really I'm checking to see if my data is arriving on time or something weird's happened. And then in production, I'm checking as data goes through bronze and silver and gold. And then in development I'm testing data dur development. I'm but I'm doing something different here. I'm I'm testing it with think of it in production I've got today's data with yesterday's code. In development I've got today's code with perhaps yesterday's data test data and a lot of meta like what's a metaphor here? So, we've we're talking sort of cattle and pets, but I also think there's some P's here that we want to talk about first with data quality. It's really about persuading people to make changes. Most people who deal with data don't own the source system. They have to talk to someone else to persuade them to change a field to improve it.

00:14:36 Four Ps: persuade, poll, protect production, protect the push

Chris Bergh: And the activity of working in the in ingest polling for problems is really about doing that polling for anomalies. And then the third one is protecting production with tripwires. And then the fourth P is protecting the production push. So I love my alliteration so sorry about that. And then as a way to do this, we're going to talk a little bit about our open source TestGen tool and how that happens with profiling and dashboards which are a great way to persuade people monitors which are about sort of polling and then protecting production with tripwires and we'll give an example with Databricks. So let's go through all these and I'm just going to check here to see if there's any questions so far or any comments. I don't see any. And again, feel free to put comments in and I'll hopefully check them every once in a while. So the so data quality. So it it's a pet and and why do I use the term pet?

00:15:32 Point 1: source data quality, table by table

Chris Bergh: And in DevOps, there's this metaphor called pets and cattle. And so a pet is something that in a technological sense, you kind of have to know how it works. You kind of tweak it a little bit. You make sure it's right. In the DevOps world, that's opposed to a cattle where everything's sort of scripted and runs automatically. And so, like, if you've ever sort of set up your machine, for some reason, you got to click this button, that button, and then you forget about it, and when you do it again, you don't remember, that's a pet. And so, I think that's metaphor applies to data quality. Because we're trying to improve the source systems and find data that's fit for purpose and it really is a sort of table by table, column by column time. You can't persuade people to change thousands of things. You've got to get influence to make that happen. And so we've talked quite a bit about this in previous webinars about how to make this happen.

00:16:29 Scoring a chosen set of fields to earn influence

Chris Bergh: And so the first thing is to get a score on your data and to pick not every field but a certain set of fields to be able to do this. And we're a believer that all those scores are for the purpose of influencing someone to make a change, someone who owns the system. And so this influence game, work small and iterate. Be very precise on which ones of your fields that you use to to make to make it go. And the the challenge here is we're really focusing on sources, right? It's not everything. And here's the case. You could have perfect data sources, perfectly fit for purpose, but your reports could be wrong. Your transformations could be wrong. And so that's it's great, right? You're you're really trying to make data not only good for analytics, but good for everyone, but you're not really focused on what happens downstream from it. You know, the data could be perfect, but it couldn't show up, right? You have a freshness problem.

00:17:40 Why a dashboard has to trace back to a test

Chris Bergh: So going to the going to the next slide, the way that you get influence on people to make changes we believe is to have a data quality dashboards and we think very specific data dashboards are important. Maybe some of them are focused on critical data elements. Maybe some of them are can are traceable to business goals. Maybe some are data source focused. This data source. Maybe some are by the user. Here's a data scientist. Here's the eight fields that are really important to them. And all these different dashboards have their place. And so we're a great believer in one size doesn't fit all sort of a multi-dashboard strategy that's essential to address different stakeholder needs. But we do believe in this sort of data quality dashboard test equivalence principle and that is a dashboard must be traceable to a data test. So a dashboard and a score is meaningless unless you can say people exactly what they should do. And a score without a clear fix this P path path is just kind of noise, right?

00:18:48 Every test result is a workflow ticket in disguise

Chris Bergh: And so we think that test results should drive the calculations of dashboards and every test result is really sort of a workflow ticket in disguise. Right here, go do this. Have you done it? And so we've even seen some people dispense with dashboards altogether and just deal with workflow tickets and test results. That's fine. You know you could have just test results without workflow tickets. But the idea here is that sort of theoretical abstract data quality dashboards don't don't work. They've got to be driven on very specific desk test results. And so you know we have a tool that does that. Our open source tool can calculate these test results as based in test generation and automatic open source. And so what's the summary? So your source data are pets. It's about persuading people to make it happen. Make it fit for purpose. You know you do that by building data quality dashboards. But those data quality dashboards must be based in in reality and in tests the but it's not everything right there's limits errors in downstream tools you're completely blind to right the you know in this case you could be blind to even the bronze layer but you're certainly bl blind to the silver layer and the gold layer and what's beyond it.

00:20:11 Point 2: data landing in bronze, and who moves it

Chris Bergh: The benefit here is you're just kind of talking about data. So you're just checking and connecting to data sources. You could check and do data quality just in your bronze database or you could point it right at your source databases. It it sort of depends. So its benefit is it's it's pretty simple. And so and we've built some technology that can make it very simple for you to do. So let's go on to the next one here. So we're not talking about source data anymore. We're talking about source data landing in your bronze database. And so you have systems that push data into and transfer data into your bronze database. Maybe you have Fivetran, maybe you use Informatica, maybe you've got custom scripts. But they're sort of moving data from one place to the other and landing it in your bronze zone. And so that's sort of a transportation problem, right? And and a lot of times the people who are in charge of the bronze data, it's not six tables, it's 600 or 6,000 tables, and they don't have time to improve the quality of it.

00:21:14 Spotting the sick cow in a herd of 6,000 tables

Chris Bergh: What they're really trying to do is sort of find that cow that's has a lame leg or is looking sickly or find the cow that isn't there anymore. And so, it's really about polling the bronze database to find problems and then look for variance. You're kind of monitoring the whole herd herd. You're not trying to find out if this meat is going to be better tasting than another if this cow tastes better than another cow. You just want to know the cow's there and it's generally looks pretty good. And so what that means is you're continually monitoring the bronze database for things like is the data fresh? Did it drop in volume? Was there some schema or change or was there some specific metric or data quality test that you want that drifted on you? And so what that means is you're really doing anomaly detection, time series anomaly detection. You're looking at these values over time and trying to find out if there's problems. And what that means also is that you're have an early indication of something going wrong downstream.

00:22:20 What ingest monitoring catches, and what it misses

Chris Bergh: Right? The the challenge here is that again this doesn't tell you whether silver's right or gold's right. It just pulls your database and tells you that bronze is good. So data errors could get into production, right? Because you're not stopping anything. You're just notifying that hey the data is late. Well, your ETL process runs at 1 PM every day. But this is great. It's a good way to catch errors. And so, that sort of gap in test coverage downstream is important, but it's think of it as a passive way to look at it. Not an active way to stop your production. And, but it depends on, you know, your job. Some people's jobs are really data ingest engineers. They're looking at thousands of tables. Can be very useful and certainly it's better than being completely unmonitored. And so what are these monitor types? Well, freshness like did am I getting the data updated correctly?

00:23:13 Freshness, volume, schema, and custom metrics

Chris Bergh: And what that means is trying to learn the update frequency on a table-by-table basis and then make predictions on whether it happens. So is this table updated every Tuesday? Well then if it's Wednesday and we haven't gotten it, probably something's wrong. And then volume like am I getting more or less records? And then schema literally did the schema change? And then lastly is is metrics. So this could be case of sales volume. Did sales volume or sales volume in a region go? And these are all early indicators that are something's wrong in your data. And so we released several months ago this a tool that actually does that for you. It pulls your data finds anomalies and and drills into it. And so here it looks for freshness changes, volume changes, schema changes, metrics changes, learns your data, sets this up very very quickly, and also allows you to build custom metrics based on things like if you look on the lower right here, the average discount.

00:24:15 Catching problems on arrival rather than at the source

Chris Bergh: And so why is this important? Again because you're the finding problems in front of your customer is bad, right? One way to do it is solve at the source data. Another way to do it is catch it right when it arrives. And so you can go beyond this and do a whole bunch of data quality testing. That's certainly fine on arrival, but just looking at these like cattle and looking at freshness, volume, schema, and various metrics is a great very quick, very easy way to start. And so, again, just to summarize, we're talking about cattle. You're talking about polling for perversity. You know, you're finding errors at the point of ingestion, not anywhere else, not not in silver or or gold. And this really doesn't stop errors getting into production, but it does it it stops a class of errors. And you know, it allows things to go into production that probably shouldn't, but again, in the land of the blind, having this running is so much better than just hoping that things work.

00:25:16 Point 3: production, and what a tripwire is

Chris Bergh: And so it really just allows you to connect to the data and find if there's if there's problems. So now we've talked a lot about source data systems. We've talked about bronze. Now, we're going to actually get into the process of of filling out the the silver and gold, the production process. And so, here the goal here is how do you stop things getting to your customer, right? And and the answer there is tripwires. You want to be able to stop things before they get to production. And so what that means is being able to put quality control steps in your process kind of within the transformation logic. And so ideally and so what what this means is is you can add these steps kind of from bronze testing silver testing gold to make sure the data is correct and there's no customer visible errors. And so what's happened here is you want to add a thing called a tripwire. And a tripwire is a stop in a process that your orchestrator is doing because most of these processes are done through a tool like Airflow or Dagster or any other ETL tools that orchestrate a series of steps that happen on time.

00:26:37 Demo: tests inside a Databricks bronze-to-silver job

Chris Bergh: And some of those steps are SQL, some of them are not. But what you're looking here is really trying to look at the data journey across all those steps. And you can extend it not just to looking at the data, the bronze and the silver, you can look at the tools acting upon the data. So let's give an example here of of medallion. And so we've if if I look at this diagram, we've got the bronze here in Databricks. And if we go in and look at it, we've got some tests we've got the bronze to silver transformation. And here I can just kind of show you in in Databricks. So what we've done is gone in and we've got a a process here in Databricks that does this transformation and you can look at it and say okay here's the transformation it's a really simple case it's just like loading one table but if I go back to the run and look at the pipeline I can see tests that have done and the way this works is this process is has some Python code that's calling out to our TestGen application running the tests and then getting the results back.

00:27:46 Pushing test results into the data catalog

Chris Bergh: And so that happens here at various points in the process. And so if I go look at this, I can see I've run this test. I can see the timeline and then I can see the results. And so all of this sort of test silver and test gold actually gets pushed up into the data catalog. So if I go into my database and I look at what's happening in gold, I can see the summary of the table. And it gives me the time the tests ran. It gives me a little interesting integration, but we're also pushing the test results from our test engine into into there. And so, oh, I got to start I got to start this up again. It went to sleep. But the idea is that the the the test data can actually be integrated into your data catalog. And so we think that that's a good way for you to kind of stop the line if something's wrong and may be able to be remediated before it gets done.

00:28:47 Necessary and sufficient: check the data and the tools

Chris Bergh: So kind of going back to our slides here. Going slideshow. So here's the errors. This is what you saw. Now to how do you make sure production is perfect? Well you've got to check the data in bronze and silver and gold as part of production have tripwires. So we've shown that but you're you have tools acting upon data. So for instance sometimes your airflow gets stuck for three hours and you don't know why. Or sometimes your reporting tool the refresh didn't happen. So the necessary and sufficient condition to have no errors in front of your customer is to check the data and monitor the tools acting upon the data. And so you know we think that how you do that so if I you could put is putting tests and monitors as part of the data production process. And there's two options to check the tools. One is that you could use kind of a testware toolware orchestrator. So you could plug in the monitoring as part of your tools.

00:29:57 Several orchestrators, and a fire alarm panel rather than a blueprint

Chris Bergh: And so there are a number of orchestrators including ours that do that. And so another way to do it is sometimes organizations will have multiple orchestrators. They'll have one process that loads data. They'll have another ETL for bronze, another ETL for gold, and another script that calls on schedule to load, you know, Power BI. And there you need something to look over all that all those steps and and we have an open source what we call a data journey observability tool to monitor all that. But you know the idea here is that in data production you need to test every table and every zone. You need to monitor every tool and be able to look at that. And so we're not a huge believer in that data lineage helps you in this guard. Can sort of describe it. But we think the metaphor here is that you need to have a fire alarm control panel, not not a blueprint. And you need to be able to look and and somewhere in your data production process, you need to look and say, "Here's the problem." and then be able to go fix it and maybe consult lineage to to diagnose it

00:31:03 Point 4: the code factory, and stopping regressions

Chris Bergh: More. And the problem here is that to really do this right, you've got to put tripwires in and then you've got to be able to either look at your tools together in a tool aware orchestrator or have a data journey observability tool. And so that takes a little bit more work. Not impossible, but a little bit more work. So, we've gone through three parts so far. I'm getting my source data. I'm loading that into bronze. So, I've got my cattle and my pets. And then I'm taking that source data and I'm putting it through bronze, silver, gold with all the tools. That's my manufacturing line. Now, we've got another manufacturing line to overuse a metaphor. We've got the code factory. And this is where I'm not really about changing data. It's about proving that my code acting upon data is still working. Specifically that it doesn't cause regressions that it both the new code I wrote works and the code previously.

00:32:06 Data tests doing double duty in development

Chris Bergh: Now most organizations kind of don't do much on this. They hope it works. They, you know, somebody eyes it up. Maybe if their engineer's good, they'll write a unit test. But that really does very little to stop regressions. And so what are we talking about here? So in development on the bottom, QA and production, you should test data during that process and the testing of that data. Those tests are really double duty. They're because you're using test data that's pulled from production. You know, maybe a filtered one or maybe an exact copy, a zero copy clone, and you're running that during development. You're actually using data tests to find out if there's problems in your code. And what that means is your data tests then are are serving this the this duty of checking that let's say I've changed the name of a schema in silver. I, you know, change the name of a column, right? There's a misspelling, I change the name of a column.

00:33:09 End-to-end regression testing across bronze, silver, and gold

Chris Bergh: Really simple, right? And I do that. Well, there's some process that knows about that column name in gold. So, that process would break, right? And then in gold, if there's some tool that actually looks at that column and loads it up into a report, that report breaks. So, the idea here is you're trying to spot these regressions in development. And you want to run your process with the new code change in a development on a on a on a developer box and then in a QA environment with multiple developers. And so regressions are bad. And to do that you need sort of end-to-end testing across the bronze and silver and gold layers just like you're running in production except in production you're running with active data. In QA development, you're running with test data. And that is something that you need to figure out how to do. And so, I can't emphasize enough here. Full regression testing is incredibly important. And why does that matter?

00:34:11 Why regression coverage matters as the team grows

Chris Bergh: Well, it matters because it makes it so that you can change your system faster with less risk. And so, you can have smaller releases, you can have more people releasing. And that becomes incredibly important as your team grows. If you're one person and you, you know, keep your data engineering pipelines to yourself, but when you hire two or three people, and especially when you start running, AI agents on it, having good regression test coverage is incredibly important. And having enough tests to do that. And also, and that's part of also having your deployment process worked out. So, you know, this is I think sort of the the highest impact but also the highest complexity because you want to have good test data. You want to have a good development environment that's u from a DevOps perspective like your production environment. You've got to be able to move things from one environment to the other. So, you've got to have a CI and CD process. And then you've got to have this regression testing across all parts of your medallion architecture.

00:35:12 A bug fixed in development costs a hundred times less

Chris Bergh: But if you fix a bug here, it's like a hundred times less expensive than fixing in production. And so we do that. Our tools work with your CI and CD please. But kind of repeating what I've said, this is super high impact, but it's more complicated. Our viewpoint is unit tests and manual business reviews, they're not going to cut it. You're going to get screwed. It's just not enough. And so you need this automated. You need to be able to press a button and hire a 22-year-old, have them change the name of that column in a database table and then have a big red light saying, "Whoops, you you broke something downstream." that's a really important part. And in some ways, making it easy to bring 22 year olds in is and and do work and find regressions is a is a great marker of a high performing data engineering team. And like I said, stop encoding regressions, it's hundred times more expensive once it gets in the live and you're having to go apologize to people and rebuild your system, etc. So, let me go through these again just so everyone understands.

00:36:20 The four points recapped, with each one’s limitation

Chris Bergh: So, we've got these four places, right? Data quality, the metaphor is pets, you're trying to persuade at the source, you're trying to test data at the point of origin, and the key limitation is well, you're just trying to improve source data quality. Doesn't really help your medallion data observability. You're trying to test and pull for all these cattle just to make sure that they they're arriving on time. It's really at bronze and the problem is you know there with any anomaly detection algorithm there can be irrelevant pings and if you find the problem it's probably too late. It could already problems could sneak into production. With data production you've got this assembly line. You're doing tests in and tripwires in various steps along the journey that data goes through bronze, silver, gold, and you're trying to pro production. You've got transformation steps, but you do have to do some tool integration. It's it's a little bit harder. And then lastly, we talked about the factory, right? I'm I'm trying to check my code to make sure it's great.

00:37:20 How much testing is enough?

Chris Bergh: I want to do this as part of the CI and CD pipeline. And it does require test data and other things like I said environment migration CI/CD get there's a lot of things that are are good here so now the question is these are great these are great four things Chris it sounds good two parts sort of how much and and how do I start and so that's what I'm going to talk about next and the how much is is really about test coverage and cost and so we're I'm going to first talk about test coverage. Now, the problem with partial coverage is it's better than nothing, right? If you have all your data and you can't judge its quality, you can't judge if it's arriving on time, it's going through silver and bronze with no tests, yeah, it's you got a lot of problems. You're just expecting to get yelled at. And so trying to do these having or testing one table out of a hundred or a few columns, it's better than nothing.

00:38:24 Cover every table and every tool

Chris Bergh: But you keep getting problems. You know, you have you know, it's sort of the the depth versus breadth gap, runtime versus dev time, prevention versus detection. These are the tradeoffs that you have to go through as you walk through coverage on your tables. And so the idea here is like how do I know I have enough, right? I've got so that like my customers don't yell at me or I reduce the frequency that my customers yell at me. So the idea is you just want to cover all your tables in bronze, silver, and gold. And then you want to cover all your tools. I mean, it's that's pretty simple, right? You want to make sure you've got tests running on every table and and you're monitoring whether your tools are running on time or they have problems. It's very very common sense. I think let's start off with how to think about this. So for every level in your medallion data architecture, every table should have tests and every column in every table should have tests and if you have significant business metrics i. E. The way your customers are looking at sales or production volume or sales by product or sales by or those metrics should have tests on them.

00:39:37 Three tests per table, two per column

Chris Bergh: And then every tool that you use that sort of touches data, your ingest tool, your ETL tool, your orchestrator, your BI tool, they should all be checked for errors and timing. And so what does that look like? Well from our standpoint every table should have at least two tests if not three. You should be monitoring schema, you should be monitoring volume and you should be monitoring freshness. So maybe it's three tests per table. And then every column should have at least two tests. And every significant business business metric

Attendee: Chris, we lost your audio. No, no audio yet. It It shows you as mute.

Chris Bergh: Oh, I forgot about this here. Here, let me see if that works.

Attendee: Yeah. Yeah, you're good.

Chris Bergh: Can you hear me?

Attendee: Yeah.

Chris Bergh: Okay,

Gil Benghiat: Yeah.

Chris Bergh: Sorry about that.

Gil Benghiat: Yeah. Thanks,

Attendee: No worries.

Gil Benghiat: Attendee.

Chris Bergh: I Sorry, I was trying this new microphone.

00:41:07 An example stack: StreamSets, Airflow, and dbt

Chris Bergh: I forgot it it taps out. All right. Thanks. Thanks for bringing that up. Let me go back to slideshow. So we're talking about coverage. So that's a lot, right? So let's look at this in a medallion architecture. So let's say in a medallion I've got sort of StreamSets doing data loading. I've got airflow moving from L2 to L3 with dbt. I've got airflow from

Attendee: I think we lost your slides as well,

Chris Bergh: L3.

Attendee: Chris. I don't know if it's just me,

Attendee: Mhm. Yep. Yeah.

Attendee: But I lost your slides.

Attendee: I don't see a slid either.

Chris Bergh: Okay. Thank you. Sorry about that.

Attendee: Yep. You're back.

Chris Bergh: Good. Okay. Sorry. My microphone didn't work. And so you can hear me and you can see my slides. Great. So this is just an example data medallion.

00:41:59 Counting it up: about 5,000 tests

Chris Bergh: And this is where like medallion's kind of broken into chunks, right? You've got the loading to bronze. You've got the bronze to silver, the silver to gold. They're kind of separate processes. And so you've kind of got three tools, you got sort of five jobs, you've got three database levels to cover. And so how do you get all this coverage? Well, if you've got a 100 tables and 10 columns, that means you've got sort of 2200 tests in L1, L2, and then in L3, bronze, silver, gold, right? You got maybe you have only 10 tables in in gold or L3. So and then you've got your tools like you want to monitor three tools and then you've got six jobs detect for timing. That that's a lot of checking, right? And and that's sort of like 5,000 data tests. That's just sounds crazy, right? So in my experience you know we I have our teams part of our business is doing data engineering.

00:42:58 Why engineers have no time, and what auto-generation changes

Chris Bergh: And for the last 12 years, writing 5,000 data tests as part of what people do has been part of the data engineer's job. Making sure you have adequate test coverage because you know, up to five or seven years ago, I think testing is part of you know, part of the data engineering's work. But like how do you do that, right? That's a lot of time, right? And so some, if you trust what Reddit said, data engineers just don't have time because their bosses are saying that they just got to, you know, get stuff out and they don't really care about the quality. So that's one of the reasons we built our open source tool, right? It autogenerates all these tests. So instead of grinding it away or doing it piecemeal which is not bad we generate 80% of the tests that you need automatically and then give you a nice UI to do that. And so instead of spending maybe 156 days of of if you trust our our math here you can kind of do that in minutes and and get test coverage quickly.

00:43:59 Usage-based pricing punishes you for testing

Chris Bergh: 80% of it not not 100%. And then another part too is you know we've done some studies of looking at data observability vendors and data quality vendors. They have a lot of usage based pricing where they're kind of looking at credits or tables. And then as the number of tables goes up your cost goes up. And so it's like the more you test the more you pay which makes absolutely no sense to us because we think you should be testing in development and in production. You should be testing your source data. So testing, testing, testing all the time. Reducing errors gives you more fun time to do work. And so the tax on that just doesn't make any sense. And so you know, setting yourself up for a $10,000 starting fee and then, you know, a $300,000 yearly bill is is crazy. And it's it's this is sort of what observability companies do both in IT and not. They're trying to get you to make this happen.

00:44:56 Flat pricing, and deploying open source at every layer

Chris Bergh: So we don't think data testing should be a quality or luxury good. And so our open source is completely Apache 2, $0, forever. Has only one limit. You can run as many tests as you want, use as many tables as you want, but it only is for one user and and one database. And so our pricing is very flat rate, not by use, but by users and connections. Again, unlimited table, unlimited volumes, unlimited testing. So it's simple and predictable. And so we're trying to get a small team of a half a dozen data engineers and you know maybe they've with with a couple of databases you know 10 $20,000 a year we think a month of a data engineer salary is is the right price. And so one way to deploy open source is to get coverage is since all of our features are 100% of our testing features are standard unlimited tables unlimited volumes you just have to deploy TestGen at every level of your database a separate instance and the benefit of course since we're capital is trying to make money is that having a single version of TestGen allows you to look at all of that in one place and then another part is having a single user interface for all of them to go

00:46:03 Where to start: monitor bronze in an hour

Chris Bergh: To. So lastly sort of where to start. So if we go back to that survey that we did 849 comments on why data engineers test the the quick takeaway is like I don't have time. I'm going to get yelled at and I don't have any money to spend on it. And so, I think there's more in there and I encourage you to go off and read it, but like how do you get quick results with a little effort and no cost. So, I'm going to show you that this is in fact exactly why we built the tool. And we use it internally and we get great value out of it and our our customers are using it as well. So, the first thing is just focus on bronze like forget everything else, forget development. Just start with the bronze database. And so all you need is a connection to that database. TestGen's a readonly connection and then you know first is like I just wanted to monitor the arrival frequency volume schema a couple of metrics.

00:47:08 The $1, $10, $100 pyramid, and shifting left

Chris Bergh: That's it. Just start there. You know you're getting tables in. Just make sure they're they're on time. This this can this will literally take you one hour with no cost to monitor a thousand as many tables as you want. The second is well maybe you want to start looking at source data quality build a quality dashboard. And you know we've got again that's one connection no cost building a dashboard really simple in our tool. It and so that's and those two great are great place to start. Doing more is is is cool right? But here's the thing about why we believe testing is so important, right? Because if you this sort of $1,10 $100 pyramid every time you prevent something with an automated data quality test, it saves the remediation cost and saves the embarrassing failure cost. So if you shift left, find things and shift down, find problems earlier in the production process of data and find problems earlier in the development process. And you need to do that.

00:48:04 Environments with almost no errors

Chris Bergh: If you test every table and every tool, you're going to have just so many less errors. In fact, our experience is you have you run environments where there are almost no errors. Which is just frees you up from so much of the burden and embarrassment in data engineering. And so, the conclusion here is really like stop being reactive. You know, the tools are out there for you for you to do this. If you fix these symptoms downstream you know don't have testing be an afterthought it's sort of a core discipline for every data engineer in medallion architecture and not and like I said we've got our open source tools work on all parts of these parts of the process and the great part of it is we believe that we can generate 80% of the tests that you need 100% of the monitors a data quality dashboard just super quickly like in a half an hour, an hour of your work and and encourage you to to to try it.

00:49:01 Closing, and the first audience question

Chris Bergh: Just go download it, point at your database, try it today. We've got detailed documents. In fact, we just updated our documents one and we've got if you like it, you're using it production, you get tired of jumping from one IP address to another, then get the enterprise version. It's it's reasonably priced. And so that's the end. And so I'm going to stop and ask questions and then when the questions are over I'm going to talk about we've got two more webinars that we're going to do in a little bit. But just any more questions. All right. I don't see any questions. Yeah, Attendee, you have a question.

Attendee: I do. Hi. Can you hear me?

Chris Bergh: I can.

Attendee: All right. Awesome.

Chris Bergh: Yeah.

Attendee: Thank you so much for this presentation. I came in a little late, so I might have missed this if you if you spoke about it, but you know in the examples there's a dbt and Databricks.

00:50:06 Q: does this work on an AWS-native stack?

Attendee: If you're using like a different tech stack like you know at my job we're like AWS native and for orchestrators you know we're still trying to push forward like something like a step function which is not as robust as like you know the Apache products like Airflow.

Chris Bergh: Sure.

Attendee: I'm curious like does this tool you know work well with like the AD AWS like native configuration or

Chris Bergh: Yeah, absolutely. Yeah. In fact, like most of our our work is AWS native and I think the the real idea here is like where are you going to test, right? Where's the first place? And so in a medallion like you all you really need and what database do you use?

Attendee: Right now we're using we don't have like a traditional data lakehouse architecture. So we have like lambdas that you know put we essentially use S3 buckets as like our staging areas bronze, silver and gold and then ultimately all this gets fed into like a Postgres database for consumption through like a website.

00:51:09 Testing S3-to-S3 hops with Lambda and step functions

Chris Bergh: Okay. Okay. So there there's two ways to think about this, right? As you if your data is kind of have lambdas, it's going from a set of S3 buckets to another S3 set of S3 buckets. You need to in order to stop errors getting into your data your data you need to check things you need to check it with automated tests and so one way to do it is just to write them yourselves right go in and have some step functions that after the data lands you have some Python code that goes in and is called to make sure the data is there that like you know that's a really simple way to do it another way to do it is perhaps with our tool we need a engine that actually can execute queries. So there's like Redshift Spectrum where it can call through and and check against those S3 buckets for for errors. That's one possibility. And you know as your data goes m maybe your data architecture is instead of being persistent databases like you know Redshift or Snowflake it's it's S3 buckets and that's a common pattern right going from S3 bucket to S3 bucket.

00:52:16 Just start testing, and write the results where you’ll look

Chris Bergh: You you want to make sure as you go that you have some way to stop if there's errors, right? And even if you're using step functions, having some tripwire or some notification. Even a simple way is just write errors to a table and then as you're going or write errors to a bucket and be able to look at that bucket. So the the I'm a big believer in like just start doing testing. Like the simplest thing to do is just write some tests, you know, find things where you've had the most problems and write some tests manually. Have them write the results to some S3 bucket and look at the results of the S3 bucket once a day or once an hour. I mean, just start really really broadly and make sure that you start having start taking hope out of your system and start

Attendee: Great. Thank you so much.

Chris Bergh: Testing

Attendee: Appreciate it, Christopher.

Chris Bergh: All right. Any other questions?

00:53:11 Q: what is the application team’s role in data testing?

Chris Bergh: Happy to talk. All right. So, let me just share the conclusion here. So, we we have like I said before, we have another part of our business in addition to producing software tools. We actually do data engineering on behalf of customers with those tools and so we've been doing a lot with with Claude Code like everyone does and and using oh actually Attendee has got Attendee you have a question let let me before I go into this Attendee feel free to ask your question either talk or just put it in the chat

Attendee: Okay, I'll talk.

Chris Bergh: Window

Attendee: Yeah, my name is Attendee. Thanks for the presentation. I'm really glad I joined this webinar. So, I, had the engineering team, that builds the apps, that generate the source data. Data engineers from other from another team data team started listing some tables that have problematic data with complete with SQL queries and the impact of that is that important business reports and dashboards become inaccurate.

00:54:24 Treating the data your app emits as an interface

Attendee: So my question is what's our what's my team's role in data testing? It is it in the data quality process or yeah can you explain a little bit more?

Chris Bergh: Yeah. So you're in charge of a a team that has some kind of transactional application that produces data and you have a team who uses that data to deliver insight to other people and they've given you they found problems in the data that you're producing.

Attendee: Correct.

Chris Bergh: And so honestly most organizations you're you're ahead of like 80% of the people even actually caring and going to a webinar on it. Most people who run transactional applications don't even think about their data teams. They're like, "we changed the schema. The data is wrong. Not my problem. It's, you know, it's good enough for me." so it's you, it's your problem to fix it. So that's one way to do it. You just throw it over the wall to them. I think it's better that you have you think of your data as an interface to your software.

00:55:27 Regression tests on the schema your consumers rely on

Chris Bergh: And so if you change the schema of it or if and how in software development, if you build like a REST interface to your application, you're going to have a bunch of tests that run against it, right? You're going to want to know it works. You're going to want to know if it changes. And so that those SQL that they gave you might be a foundation of regression testing of your data or your schema to make sure that in your software development process as you're changing things that interface to the data is as good as you can make it. Now the ownership of these things around data quality and and this question of who owns it is is a really interesting question right because the data might not be wrong it just may need to be improved and so that may be sometimes a third person that comes in a data quality person who will try to advocate for change to a project to improve a table that your tool generates because that could be useful for some other purpose in the organization.

00:56:29 Who owns improving the data

Chris Bergh: And so that's where I think as a software development leader, you have an interface and data to your to your partners and data engineering, make sure that you don't break the work that they've done. And if and then if you want to improve that data even further because that data could not only be used by analytics, but it could be used by your application, you need to start thinking about how to improve that data. And that goes to a more system systemic thing about perhaps risk reporting or other internal reportings or company goals to measure the quality of your data and improve it. So I hope that answered your question.

Attendee: Yes yes yes. Thank you. Thank you very much.

Chris Bergh: All right, so we got four minutes left. Like I said, we we we've got two webinars coming up. One is about the context i. E trying to get people who use your data to use LL or when pe some people who are using your data may be trying to query the data and analyze it using large language models like ChatGPT or you know Genie or Cortex these tools that are embedded into the platforms and so how does that how do how does that work and so we've been doing some work with our customers in general there's data and then there's the thing next to the data which is context or

00:57:47 Coming up: context engineering, and Claude Code

Chris Bergh: Metadata. Those things together actually help make it tenable. And so we've written a blog and are going to give a webinar on our our experience in this idea of context engineering and that's in two weeks. And then the next one is we're after that we're talking about everyone's been using Claude Code. Claude Code's been become a staple in our company this year. And to do data engineering. So how do you actually use Claude Code in this and and part of it is is we really believe in both cases data quality testing is really important because data trust is important that's here the DT part of giving insight and then data testing is really important and we lay it out right here. So both of these things are important and there's other considerations on how to make your AI tools. So we have two webinars coming up in two weeks and another two weeks to talk about it.

Attendee: Christopher, I added a link for you on metadata.

Chris Bergh: Like I said,

Attendee: I'm on your team with metadata as a driver of operations and to reduce the need for resources and improve quality.

Chris Bergh: Oh,

Attendee: We've got a book coming out May 1st on Kindle

Chris Bergh: Great. All right. Well, yeah, thanks for sharing that.

Attendee: Put yeah I'd be interested to hear your thoughts on that.

Chris Bergh: We'll take a look.

Attendee: And if you want to get your name out there, I'd welcome an endorsement.

Chris Bergh: Okay. All right. Great. Well, let me check out the book, Attendee. Thank you. Thanks everyone. We promised an hour and now we have an hour. The the slides and recording and everything will come out in a little bit. So, I appreciate it and give TestGen a try. Talk to you later. Bye.