DATAVERSITY Demo Day: Data Quality with TestGen and Observability

00:00:00

Hello and welcome. My name is Mark Horseman, data evangelist with Data University. We would like to thank you for attending Data University Demo Day, data quality. We're very excited to have you join us for this event set up to give you an overview of the tools and services available for your enterprise data management programs.

Just a couple of points to get us started. Due to the large number of people that attend these sessions, you'll be muted during the webinar for any questions. We will be collecting 'em by the q and a section. If you would like to chat with us or chat with each other, we certainly encourage you to do so.

And just to note, the Zoom chat defaults to send to just the panelists, but you may absolutely switch that to network with everyone. To open the q and a or the chat panel. You'll find the icons for those features in the bottom middle of your screen to answer the most commonly asked question. As always, we will send a follow up email to all registrants within a couple of business days containing links to the slides.

And yes, we are recording and we'll send a link to the recording of the session as well as any additional information requested throughout the webinar. And welcome to our second session with DataKitchen and Gill Bent. Gill is a co-founder of DataKitchen, a software and services company on a mission to help data workers improve their lives through the adoption of DataOps.

Gil is the co-author of the DataOps Cookbook, the DataOps Manifesto, the DataOps Wikipedia article, and speaker on DataOps at many industry conferences. He has coached companies on how to get started with DataOps and how to best use DataKitchen software. He has a Bachelor of Science in Applied Mathematics and Biology from Brown University. A master of science in computer and science from Stanford University has worked at at and t Bell Laboratories, sase Phase Forward and Leapfrog Rx, and has been working with data for most of his career.

And with that, I will pass it over to Gil for his presentation and demo. Take it away, Gil. Very good. Thank you, mark. Yeah, thanks for that, uh, introduction. Hey folks, I'm coming to you from a, a cold, sunny day in, in Boston, Massachusetts, and today I'm gonna talk about TestGen and observability. But first, uh, why, why TestGen and observability?

Well, um, all of us are in data and analytics know that it's hard. There's, there's multiple people have done surveys and fine pot projects fail. There's too many errors, people are stressed out. Another problem, things are fragmented. Uh, there's lots of different tools, uh, that there's teams that are in silos and there's really data all over the place.

And I think we've all found these issues here of poor data quality customers, finding errors, data errors. I really hate it when that happens, too much to do. And, and a tool chain that just barely runs. And another reason we kind of get ourselves into this is we focus on building, build, build, build and not really operating.

So how do we fix this situation? Um, well, at DataKitchen, we feel the answer is DataOps. Uh, we're not alone in this feeling. Uh, Gartner has said that DataOps teams that, uh, adopt DataOps, uh, can see 10 times the productivity. And what DataOps does, it helps you increase quality while decreasing errors and also increase cycle time.

Once we've seen this, uh, transition in companies that adopt DataOps, where the time they spend on waste and errors and operational tasks are reduced to leave time for the good stuff, which is delivering features and data sets to customers, and also improving internal operations. Now at DataKitchen, we have a lot of resources on DataOps, but I'll, you know, for this talk I'll focus on really, uh, two ways to get started.

Uh, one is, is we think you should focus on the production system. DataOps is very comprehensive. It encompasses both production and getting features into production. Your CI CD pipelines, uh, to get started, just focus on production. And there, step one is improve data quality. You know, improving data quality is a gift you can give to your future self.

And second, get a handle on production errors.

And at DataKitchen, we have two tools to help you with this. They're both open source, they're free to download, free to use for the individual, and they are DataOps, data quality TestGen, which we just call TestGen. And the other is DataOps observability, which I'll refer to as observability. So with that introduction, I'd like to show you the tools.

Uh, the first is TestGen. And where we will start here is with this data quality dashboard. You can see that there's scores, um, and what the scores let you do is use it as a lever

00:05:00

to influence people. It's, it's one thing to know that there's data quality issues, but as a data group, we should come with data. And also we find that people respond to getting a letter grade. It's like, Hey, hey, supplier, uh, you're, you're only at a B. We, we need you to be at an A.

It helps an organization to be, as we like to think about it, data driven and not denial driven. So you can see we have total scores. Um, we also have a mechanism to let you really focus in on the data that matters. So we're able to give a score for, uh, critical data elements.

And we also have ways to, this is all configurable. You can look at your scores based on quality dimensions or your own definitions. And here we have, um, two groups of data. One we're calling MDM and one we're calling sales. And you could see the MDM team's quality is higher at a glance than the, than the sales data that's there.

And if you'd like to improve these scores, you could drill down into the details and you can see the contribution. You can see that the, um, you know, for each table, how they're impacting the total score. And then to really get something done, you could go down to the individual column and see how those are impacting the score and therefore improve it.

So how did we get to this point? I'd like to walk you through the product from the beginning. You know, it starts with a database connection. We support, you know, the popular, uh, SQL databases, Redshift, Snowflake and so forth. Um, this is your regular connection. There's an idea of a table group. So once you connect to a database, you can zero in a bit further.

You can have inclusion criteria for tables based on name exclusion or just, or just make a, a list of tables just so you can really focus in. Once you connect to tables and say, which, uh, a database and specify tables and say what you're interested in, you run profiling. Um, and here's the output of some of those profiling runs on this test dataset.

You can see it's very quick. And once you get profiling, you automatically have two benefits delivered right away. The first is a filled out, uh, data catalog. We'll look at that in a moment. But this, since this is, uh, about quality, let's look at these quality issues or, or data hygiene issues. So we can see here, um, that these were found automatically with, with really no configuration.

Uh, there's a list of them. The tool does its best at bank at ranking them by definite, likely, or possible. So you know where to get started. And let's just, uh, look at one of those here. Let's look at, um, one of my favorite ones here, which is similar values matched when standardized.

So once you, um, there's a explanation of what's available here, we can say, okay, what's going on here? And just really pop up the data to see what's going on. And if we zoom in, we see that in this, in this field, uh, there's 1, 2, 3, 4, 5, 6 different ways that e-bike is specified.

You may say, Hey, this is just, uh, uh, you know, data. Does this hap or does this happen to you? Well, um, we ran this on some public data from the city of Boston, uh, the building permit data, and we'll zoom in on a little bit of here. It seems that the city of Boston itself is having a little trouble spelling or being consistent with how they spell Boston.

We could see, uh, 1, 2, 3, 4, 5 different ways Boston's specified and maybe two more if you include Chinatown. So this is a very real error and, um, causes downstream data issues.

Okay. Um, that's the source data. We, you're able to look and take a, um, a look at the profiling information that's available. Once you, once you do that, you can see what the other values are there, frequency as well. And then if you really want to influence something, someone, you could create this issue report.

And let me just download that here. It's a, it's a PDF report that you could include. And let's take a look at that. So it is one thing to find issues, but how do you get someone to, to fix them? Well, this, that's what this issue reports about. It, it explains what issue there is.

00:10:00

It shows you the schema table and column. It gives you some sample data, has a suggested action. And then to really be useful, it, it gives a query that will help them, uh, see what's in there and clean things up.

And with TestGen here, you can, um, you can multiple select, and you could send them all of this at once.

So that's one thing you get for free. Uh, a number of hygiene issues. Um, it lets you focus in you, you also get a data catalog. You can see as you'd expect, tables, columns, you could see profiling information. So if we look at, um, uh, eBike products,

resize of it, and we could see with the different, uh, things here, we could look at, for example, uh, frame size. And again, for the profiling, we could see there's small, medium, large na and missing, no, actually, I know that this is flagged as a hygiene issue as well. And then if it's a numeric type, it really will tell you everything you wanna know about it.

Uh, min max average standard deviation, and, um, the 25th and 75th percentile all in one quick thing there. Um, what you're, what also is interesting is that you're able to auto this system in order to help you focus will automatically flag, uh, what it thinks, um, are critical data elements. Um, and you, you can also, uh, yourself, um, override that.

Is it a critical data element, um, or not. Um, and then also you can, um, group your and, and further categorize your software. So here, the whole table, um, of Debi is from the MDM, which we saw before. And the sales table here is, is mark sales. And that's how you can get these custom categories in your scorecard, uh, to drive change.

All right? So that's what you get just by running profiling. You get a, a list of hygiene issues. You get better understanding of your data through this data catalog. But then, um, this is called TestGen because it is able to automatically generate tests. So based on the profiling information, um, you could hit the button, generate tests.

I've done that already. It just takes a few seconds to do. And here is an entire list of tests that, uh, can be run. You can, um, you can click on these, you can edit the test to fix them, and you can export them and you can copy and move them between different test suites.

Uh, once you run a test suite, you have, uh, test runs. And these are, these are very quick to, uh, run most of this on this sample data set. Uh, just took, took a few seconds. If we look at some examples here again, we see how many passed, how many failed, how many warnings we got, you can export the whole table.

Um, and let's look at one of them. Let's look at a test that failed on tax id. We see it's a pattern match. Uh, we had established a baseline based on the profiling. Uh, we set a threshold at zero, and it somehow, something in the threshold went from zero to one. So what I like to do is just use, um, look at source data to see what's going on.

And we could see that this is a, a tax id, um, based on the US social security number of three digits, dash two digits, dash four digits. And we could see that, that there's a, a data issue here. So again, we can, uh, create an issue report, uh, to send it to the folks who are, uh, in, in charge of this, uh, data.

Let's download it. And let's take a peek of it in the browser here.

And again, this is a, so there's, uh, issue reports for hygiene issues and issue reports for test failures. And we could see, uh, a history. So you get an idea of when the, uh, issue was introduced. It gives you the sample data of what was there, and it gives you the query so you know when it's fixed, um, that it could be, uh, all cleaned up.

All right, so, so this is what, uh, TestGen gives you,

00:15:00

you know, out of the box profiles, finds issues, gives you a data catalog, has profiling results, generates tests. You can run tests over time, but also it has a way for you to add your own tests. Uh, here's a test suite with three test definitions. And what you can do is, um, you know, put in your own SQL and the, and the rule is if the res seql, if the SQL returns no rows, it's, it's a pass.

So here we can see select where occupation equals junk. That doesn't exist. It passes. Uh, here's a test where the occupation equals nurse and it fail and it fails because there are rows like that. That's just the definition of test. And finally, there's a project dashboard

that shows you, um, you know, that helps you keep, um, more information, um, understood. He, here we have one connection, one table group, just a few runs. You can imagine many connections, many table groups. Uh, this, this will give you a, a quick view of, of what's going on. So, so in summary, you know, here's a tool that will automatically find things, run things.

It's very customizable. And by using scores, by using these reports, you can affect change in your organization by sending those to suppliers, sending them to system issues, and that will help your data quality program. So that's part one. That's how to improve data quality. Now, what about the rest of production? What about the way your, you need to run and look at different processes?

Well, we have another tool for that, uh, that's observability. And this gives you a process view and a, a process lineage of your, of what's going on in your data operations. It, it, you know, each box represent here is a data is a data pipeline. Uh, we can see in this one, it starts in Azure Data Factory.

There's a Python program that runs. There's, uh, Databricks and then finally Power BI. We, uh, get information from the system two ways. One of them is that we have, uh, agents that can be dropped around your system, and they will detect what's happening and send events into this. Another is we have a API.

So you can instrument your code, and between those two, you can see what's happening in the system. So let's take a look and see, um, what's going on in this, the single pane of glass in this mission control for your data operations.

So as it runs, you get a view of the status. You can see that this, uh, had a, a warning. These two ran correctly. And, hey, wait a minute. Uh, this segmentation didn't run before. Uh, the Power BI dashboard was refreshed. It, it let you know that this journey was out of sequence. You can see in this timeline, view is going on.

You know, customers that have especially a, a new manager of, of data operations, find this very easy to, to instrument their code, see what's going on, or drop in some agents. And then you get a, a bird's eye view of everything that's running and their status, good or not. Uh, you can see in this batch run tabs, it also accepts tests.

So in a single place, you can see if tests are working or not working. You could drill down, see the results. Uh, you can filter, really zoom in and see which tests are not working. And finally, we have a list of all the events, uh, that this system received. And if there's some type of error, uh, you can then click back into the source system and see what's going on there and get some more detail.

So this is, um, this is a demo. We're clicking into an image of Databricks, but in, in a live system, it would be a link right to the, uh, the pipeline that ran, so you could go and get some more details.

So, um, all this information is at your fingertip. People, uh, don't want to just watch this and not, um, not be alerted. And that's why in, in here we have a whole host of rules for different types of alerts. So for example, when a run has failed for any comp,

00:20:00

any component in the journey, you can get an email. We support two types of alerting. One of them is an email, another is a web hook. Uh, you can see if something starts late. You can see based on metrics, is is something being exceeded, is something out of sequence. And then this is, this is actually one of my personal favorites.

Did something not run when it was supposed to? I don't know if you've been, um, if, if that type of error has has caught you, but you think everything's fine, there's no errors, no alerts, but you didn't get the failure because it didn't run. And, you know, through expectations and observability, uh, you can look at that.

So to summarize, uh,

for DataKitchen and to improve your, your data operations to establish a quality program, we think that there's, uh, two simple steps, uh, that you can use Our open source tools. One is to improve data quality, start with TestGen, um, second to get a handle on operations, use data observability, and then you'll have more time to either work or, or relax a little bit.

So just a pointer when you get the slides, the, there's some, uh, more information here. Uh, you know, we advise, you start with TestGen. It's, it's very easy. You can get started in, you know, under, in under 10 minutes, you could start to see some, um, hygiene issues and, and start to affect change in your organization.

Uh, you could read our DataOps manifesto. We have a free book we call the DataOps Cookbook. We actually have, uh, two books that we've written. And then if you're interested in, uh, DataOps, you could read materials in, uh, the, the cookbook or different white papers or, and there's also a free, uh, DataOps certification that we offer.

So that's, uh, that's what I have for my presentation. And Mark, do we have, um, any questions? We do have a couple of questions here. Um, we'll start with the, the first one on my list, does the profiling extend to referential integrity and aggregation types of checks? So the, the profiling, um, there are other types of checks that the, um, the, the testing does as far as referential integrity.

It looks across tables, so that will, um, but most of the profiling is single column. You know, if you want to do different referential checks, that would be a, um, you know, that could be done with custom SQL as well.

So when it does, uh, profiling on a single column, like what kinds of profiling checks does it, does it run, does it do like some statistical analysis like min max range mode, median, that sort of thing? Exactly. You know, it depends on the, uh, data type here. So let me, um, yeah, so, so for example, if it's a, um,

you know, if if it's, if it's,

yeah, so if it, if it's constants, you know, it, it will give you, um, it, it'll look for patterns, it'll look for pattern breaks. And if it's numeric, yes, it does, it does more of a statistical analysis of MinMax median and and so forth.

Um, and just a follow up question by somebody, does this work with Azure sql, SQL databases? It does, it does. Nice. Yeah. On our website, um, you know, it, we, we have a complete list, but yes, that's, that's one of them. So what, what it doesn't work on is, is, is files, you know, sitting in, in S3, for example, but in any SQL database or, uh, or, you know, we would advise, just load it into your favorite SQL database and run, run your data checks right there.

Okay. Um, we do have a question that popped up, um, about critical data elements. Is it possible to add custom complex business rules for critical data elements? Oh, absolutely. Absolutely. So there's, there's, uh, I think there's, there's two parts, uh, to that answer. The first is, you know, the system will, will discover and suggest what critical data elements are.

You can override those. And then for your own business rules, which are arbitrarily complex, um, we support SQL to write those rules. So there is a, there's a place to put, there's a, a test called the custom SQL test, and you could make your rules as arbitrarily complex or sophisticated as you'd like. Um, in, in your experience, Gil, when the tool

00:25:00

finds a critical data element and, uh, and that's put forward to the business, how does that discussion work? Is it accepted in or is there a conversation? Like what's the, the typical use case there? Yeah, it's, it's, you know, what's really best is, is a conversation because what, what is a critical data element?

You, you know, what, what the tool is. You know, actually, let me back up. One, one element of of DataOps is agile. So we believe, and we, we've seen this work really well, is to iterate you and you have to start somewhere. And that somewhere is, is what the tool can suggest. But if, you know, there are many organizations that have thousands and thousands and thousands of tables.

So if you were to take this tool, run it on thousands and thousands of tables, it's really overwhelming. So you wanna focus your effort on what really counts and what does count. It's, it's really what does count to the, what counts to the business. So, um, you, you could, you could skip the critical data element discovery, uh, mark them, um, based on a conversation of the business, and that works really well.

So it's kind of a, it's kind of a what works best in your organization? Do you wanna have a a blank interview to start the process or do you wanna say, Hey, look at this, please edit. Neat. Um, so we do have another question from, uh, from an attendee here. Uh, they say they're trying to draw a connection between TestGen and observ observability.

Can you map the data pipeline in the journey and observe profiling errors? Yeah, so profiling errors, um, turn up as hygiene issues and those, um, so really what the really the way the whole that, that's a great question. So I'll, I'll back up. So step one is profiling and hygiene issues. Um, and that really what that does is that creates a baseline that are, is then used to generate tests.

So a work, so a workflow is, is do is run profiling, get hygiene issues, review those, um, and, and then, uh, generate your tests. Um, and, and again, you could put in as much work as you want to into saying, do I, do I need to re um, uh, get rid of all the hygiene issues right away, yes or no?

Um, so TestGen is really focused around, you know, hygiene issues and, and, and, and data quality. Uh, observability is, is about two things. It's about, uh, process health and data health and seeing all of that in one spot. So what we automatically send to observability from TestGen are the, are just the test results and then also the process results of did my, did my data pipeline succeed or fail?

We do have a, a question that popped up in chat. Does the op side monitor occurrences of exceptions that are out of expected occurrence range? For example, could it use the statistical process control thresholds such as two or three standard deviations over normal, kinda like leany kind of stuff? Yes, yes. There's, um, you know, some of those tests are automatic and some of those could be, uh, written as a custom test.

Cool. Um, what is, why is the job never ran, counted as an error? We got as a question in chat, Why is, well, if, if a job is, um, that, yeah, thanks for that clarification. The j if the job never ran, um, you know, just imagine you have a, a important process that runs once an hour or an impor, you know, or the important dashboard run that happens on Fridays or, or something that's running on a schedule.

That's, that's something that would be a process that doesn't run. It's like, oh, we were supposed to pro, or, or you could say even on the ingest side, oh, we're supposed to get a certain data delivery, um, at a certain time. And that doesn't happen. That's a process that does not run. So it's really, uh, those are for processes that happen on some type of schedule.

I mean, count is not an error. I, yeah, the, they were following up their question here. Yeah, Sure. Um, so, uh, why would it count as not an error as well, like on the, on the flip side? Well, it's, it's, um, if, if, if you set an expectation, uh, just to simplify, if something's supposed to run every day at noon and it doesn't, that's an error.

Yeah, yeah. That's my experience too. And it, yeah. And if it's, um, yeah, I think it's,

00:30:00

I think it, and we're able to detect that and, and report and alert on it through, through the observability product.

So as a, as a data quality practitioner, um, when we're implementing these sorts of things, like what is the, what is that use case, um, for TestGen? So like, I can, I can kind of see it all come together, but like, how would a how would a data quality person use TestGen? Right? It, it, so, you know, I think that a number of different roles can use TestGen.

And one of 'em is as a data quality person, you know, someone, uh, doing data governance or someone whose job it is to assess data sets, um, you know, they would start with, with the data in a SQL database, and then they would go through those, uh, those steps of, here, let me, let me just back up a little.

I have a, do you have a picture? So this, this is, this is what the, uh, the workflow would look like. Uh, profile the tables look for the hygiene issues, then generate tests. Uh, there's a step in here of, with these gotchas, go back to the, uh, the supplier, um, you know, and then as data comes, rerun those tests again, generate scores, look for improvement and, and repeat.

So what would a program look like? You know, it, it depends on your goals and a lot of data quality, uh, professionals, uh, kind of have two tasks. One is to, to characterize, Hey, is this data good? So, uh, you know, some of our customers have just run hygiene tests and, uh, given the reports back to the suppliers and say, Hey, here's our findings.

Um, and then, and then the other job is to actually improve data quality. So how can you know someone, you know, a, a data quality engineer or, or a data governance person actually improve the data when, when they don't own the data. And that's to, to come with their own data, quality data, go to the supplier and say, Hey, I've noticed these issues.

They're having impact on my customers or the business and, uh, can you please fix them? So that's, that's how the use of TestGen would fit into, um, a data quality program for, for those two roles. I, I will mention that, um, this can also be used by a data engineer as a, as a framework to run tests.

So as data goes from, um, pipeline to pipeline, uh, one can, um, you know, uh, TestGen also has a command line that can be called externally and can, and it can execute tests at the right moment to make sure data flowing through the pipeline is high quality. And that's another, uh, DataOps practice. It's an interesting call out to me, Gil, because a place I worked at recently, I'd often speak with data engineers and they're like, can we use this data quality stuff to, to help us with our ETL deployment and testing?

So, um, yeah, that's, uh, that's neat how you kind of have that on here. Um, Yeah. And, and if, if I could add, if I could add Yeah, go ahead. In, in working with data engineers, you know, it's, um, you know, part of our service, we, we, we've had a best practice of when you, when you're work as a data engineer for an entire data pipeline, you know, put tests at the beginning to make sure your data quality is good, put tests in the middle to make sure your assumptions are not being violated, and push tests at the end to make sure your, your final product is what you expect.

And we have found that it's, people have trouble writing tests. So what we did is we invented this tool to write the test for you, uh, to have a starting point, and then you could iterate and really be proactive. 'cause as a data engineer, you wanna find problems before your customers do, and let, let them know, know about it.

That's what TestGen will do. And that, that's one of its uses. And the other, as a data quality, uh, person or data governance person, you wanna be able to influence the people who own the data to improve the data. And we, we think do that with your, you know, like Deming said, and God we trust, but everyone else must bring data.

You know, TestGen will help you bring data to the table about data quality For sure. Uh, this slide has, I think, peaked the interest of chat because, uh, we've got a, what does open source mean? What does, what does the open source based TestGen mean to the user? Does that mean they, uh, they're freely available to the, to the user?

So what, what, what does open source mean in this context?

00:35:00

So, in, in, in one word, yes. Uh, it, the open source tools are free, uh, to the user. So we, we have two versions of our tools, what we call open source and, um, enterprise, the open source tool is focused on the individual. It, it has all the features needed for an individual to be successful.

Um, it is free, free to download, free to use. It's, it's under the Apache two. Oh. Um, so you, you could even take the, you could even look at the source code and modify it if you'd like to, but most people, uh, uh, take our prebuilt version, install it, and, and off they go.

Uh, you know, if you find it useful and want to start sharing, what the enterprise version brings is multi-user, multi-project, and multi connection. And, and it's at a and a very modest price point.

We do have an another, uh, question in, in the q and a here, does the scoring happen for each rule based on the filtered population, or is it done for all data in a full table or a data set? So when you've got those wonderful percentage, um, uh, ring charts are, are, are, is that for a individual column or a whole table?

You know, it's, they're, they're really rollups. So if the, the, the way it works is, um,

if, actually, let me, let me just drill down to this view here, is that every, every column, you know, here, here we're looking at one column, um, and, and total amount. And we can see that it has it, its own score and, and its score is, is lower because there's a few, few issues as we go down, you know, some, some columns have no issues found and their score is a hundred.

So if as you roll them up, um, you, and that's where critical data elements come in. Um, you know, if you're really focused on, let's say the three metrics that matter to the CEO, that, and you define a, c, d, E being that then only those three columns that participate in the CDE will get their roll up and contribute to the score.

Uh, you could say, okay, show me all tape, show me the whole system and everything I've profiled, then it will roll up to that. So really it's, um, we're, we're really slicing and dicing the scores, uh, and rolling them up into what, what makes sense for the user.

Nice. And, and I think our, our previous conversation has peaked the interest of chat a little bit. Mm-hmm. Um, when we were talking about, uh, uh, data engineers, uh, using this to write tests and manage, uh, ETLs, um, at, at what point would those tests be run? Like are, are they run at control points in the pipeline, kinda like data contracts and, uh, the, the question or reference is, uh, Chad Sanderson, uh, in this, and I, I think I'm familiar with that a little bit, uh, where it's, it's talking about, yeah, I have a obligation to, uh, promote good data into the reporting environment or get good data out or, or have a downstream obligation.

Yes. Yes. So, um, that, that's a great question. So where, where are the, where are these, uh, where are these run run and what do you do with them? So they, they should be run, you know, at, in, in general at, at three points in the pipeline. You could run them after every process, whatever makes sense.

But, you know, is the, but the big questions are, is the source data good enough to continue or is there a, a very bad error where you should, you should have some kind of action, like not use the latest update, use the previous update. Um, and, and really, you know, you, you know, we, we think of, of data pipelines as a, as a, as lean manufacturing.

And, and what does Toyota do when there's a, a very bad error is they allow engineers to stop the line, fix the quality issue, and then continue. So that's, that's really what you'd want to build into your production pipelines is, is use the severity, um, of these, uh, test failures and, and you know, we have, we have warnings and failures and actually use the failures judiciously and actually stop the line and make a fix.

So, um, you know, it, it's really where do you put them in your pipeline? It depends on your pipeline, but I say, you know, at the beginning, in the middle, at the end, kind of three places and, um, and use it to stop production if it's a very, a bad error. 'cause it's better to stop production and be late than to give bad data to your customers.

There's a lean word for that I, that I almost remember. This feels like a, like a trivia thing. Um, is there a capability, uh,

00:40:00

for different roll-ups for a data set? So if, if you're looking at something from a sales perspective, but, uh, or an accounting perspective, like when do the, when the users interact with this, uh, what are, what are the controls there? So when the users interact, you know, you, you, you get to say what the user, uh, interacts with.

So you can, you know, this supports read only users and you could make your own scores here you can say, what am I interested in? You know, I, I'm interested in just CDE scores. Um, um, you could filter, you know, there's different ways to filter. You know, some of these are automatic by table group, but some of these are what you define.

So you're able to make different scorecards for your different constituencies. Something for you to look at and share and, and, and customize it that way.

Awesome. Well, we, we don't have any other questions that I see in chat. Um, do you have any final parting thoughts, Gil? So I think, um, you know, I think data quality is hard to do. I think, um, it's, it's hard to write tests. So we, we've automated that it's hard to influence people.

Um, so we, we provide data for that and, and really I would like, um, I would like people to, if they can, you know, download it, try it. You just need a read only database connection to get started. And, uh, many data engineers and many people who work with data have that, uh, give it a try and, and let us know.

We'd love to hear some feedback. And if you have further questions, um, uh, I know in the follow up, please include my, my email is gill, GI l@DataKitchen.io. Let me know if you'd like to talk about anything further. Wonderful. Well, thank you so much time for this presentation and, uh, everybody feel free to stay tuned while we get the next speaker set up and the next presentation launched, and we'll begin the next one right at the top of the hour. Thanks again, Gil.

All right. And I'd like to thank everybody for the questions. They were really, uh, excellent. And thanks for hosting Mark.