Automated Test Generation: Why Data Teams Need It

00:00:00

Hello everyone. Um, my name is Chris Bergh. I'm here with Chip Bloche, and we're gonna be your webinar host today. Um, so hopefully everyone can hear me. Chip, can you hear me? So hopefully everyone can, I can. Oh, so and so, um, I, let me walk through the, what we're gonna do today. So I am your host, Chris Bergh, c e o, and head chef of DataKitchen. And, and Chip, do you wanna say hello?

Sure. I'm Chip Bloche, uh, uh, director of Data Engineering for DataKitchen. All right. And so, um, just some housekeeping before we go on. Um, you know, the slides and the recording, we'll send an email to y'all with, um, uh, when we're done. Uh, when we're done here, if you have any questions, uh, just put 'em in the chat window. Um, and we'll I'll try to monitor it during it, but, uh, we may not get there, but, um, we'll try to answer it at the end. So feel free to, to ask questions.

So hands outs and recording will be at the end. So what are we gonna talk about today? Um, we're gonna talk about a new product, sort of, uh, that DataKitchen has called DataOps TestGen. Um, and we're gonna try to put it in the context of, um, what, what we call a data journey and the failures in the data journey and the five pillars of the data journey and why DataOps TestGen fits as one of those pillars. And then we're gonna introduce the new product. Uh, we'll give a customer example cuz this product's been actually in use for, um, well over a year. Um, and, uh, chip will do a quick demo and then we'll answer some questions or we'll answer it throughout. So going to the next slide. So, uh, we've, we've talked about sort of why DataKitchen exists, or what's the problem with data and analytic teams.

And I think a lot of times is that, that their projects fail or that when they get something in production, no one trusts the data, or no one trusts the teams, they have too many errors. Um, and, and we did a survey recently where teams are just stressed and unhappy. So if you look at the data on what data and analytic teams are doing, it's, it's a very sort of unhappy state. Projects are failing. Things are, uh, people don't trust the data and teams are stressed out. And so for us, um, the way that we've looked at the world is sort of opposite of a lot of people.

We, we don't think the problem is that your database isn't too fast or your e t l tool isn't easy enough to use. Uh, we think that there's a system that you need to build around your data and analytics estate, and the purpose of that system is to drive these three values, sort of lowering errors in production, increasing the cycle time at which you can get new things into production with low risk. Um, and then just trying to automate various parts of your, um, of, of the work that your, your data and analytic team does. And, you know, I think there's a lot of reasons why we've written a lot about it, but I think just one of the reasons is Japan, there is a lot of data tools out there, um, and Snowflake and, and, uh, Databricks are having their conference, and of course there's some new data tools out there.

And so every day it's this sort of Cambrian explosion of data tools across everything, um, databases, data sources, ETL tools, visual visualization tools, governance tools, data science tools, AI tools. And so, um, you know, the way data thi the way we think about it at DataKitchen is we, we want you to focus first on the data journey.

So f focus first on lowering the number of errors in production. And for us errors, we use the term errors in importantly because we think it's not just data quality, it's not how good your raw data is. That can be very important, but it's also all the other steps where it's raw data to integrated data to being used in a tool. And all those things need to be put together.

And so for us, we've kind of boiled it down to something called a data journey. And, and I'm gonna talk a little bit about five, the five pillars of the data journey. But the main problem is, you know, errors are in a lot of places, right? They're not just in your data. Um, and you know, we're gonna talk about testing your data today a lot, but, um, in my experience, you know, data problems are frequent and often, but they're also, you have report problems or you have tool problems or code problems. And so, um, you know, reducing errors along that journey is, is very important for data teams. And if you do it, it actually gives you time to do more things.

And so that path from data source to customer value, we call a data journey. And I'm just gonna quickly go through these five pillars mainly as context, um, for what we talk about. So, so in this diagram, the red parts here are pretty typical. There's a bunch of data sources. Uh, maybe there's a load process in here. Uh, maybe there's a transform process, maybe there's a predict process, maybe there's a report process, there's your data, your infrastructure. Um, and kind of functionally where these ideas go is, is we think the five pillars of data journey involve data at rest.

And we're gonna talk a lot about that today with TestGen, but also talks about data in use and also talks about looking across all the

00:05:00

steps that data takes and looking down all those steps sort of down the stack because problems aren't just in data, they could, uh, occasionally be in your server. Um, and then setting expectations on all this. So, so you know that it's right before your customer sees it. And so the, the, the first one is looking, uh, the first pillar of a data journey is kind of looking across your steps, um, and seeing if things are wrong and order.

And sometimes you have tools that are a collection, they're unordered or tables that are unordered, but a lot of times the data journeys have an order to them, right? You have to do your data work before you do your model work, before you do your visit work. And just making sure that that order happens, that SLAs are met, that timing's met, that the, that um, there aren't delays.

And, and really getting the process lineage and the process reliability, right, is the first pillar. And then the, the second pillar is, well, when things go wrong, they kind of go wrong in different places. A lot of times it's data, but it could be, as I said, your server or your infrastructure or bad code or your network. Um, it could be cpu, it could be disc space.

And sort of monitoring those things in a very it way is also important to do it. Um, and that's just is your tech stack working? Um, and then the, the one that we're gonna talk a lot about today is like, is your data right? Um, you know, uh, and you know, is its schema, right? Is it fresh?

Is it volume? Um, is it right from a business perspective? Like have your sales gone up or gone down? Is the percentage that they've gone up or gone down? And that's really about what a lot of people call data quality. Um, for us we have a little more expanded cause it's not just about the raw data, it's also about, about how you put the data together.

And so we're gonna talk about a tool to help, um, automate the testing of data at rest or the data quality validation testing of data at rest. And that's the purpose of, of our, our webinar. And you know, the other part is that you have data that's not in a database, but it's in used, it's actually in a predictive model or in a visualization.

And how do you test that? Because those are software tools that are driven by code that could have errors. Um, and so how do you make sure the user experience is right with those tools? How do you make sure they trust the data even though the data in your database could be perfect? Um, somebody in a model or a visualization could have muck with it, it could be wrong and your customer could find it wrong.

And then sort of lastly, sort of the expectation layer, like you've got, you know, your data journey should run with a certain cpu. Your data, raw data and integrated data should be perfect. Your model should be have this level of prediction, um, your, uh, uh, and how all these things have to happen in this order.

And then how do you collect all those expectations and, and tell people and notify. And that's really about alerting and and sharing that information. Um, and so, uh, you know, we've got three tools that focus on all the pillars of the data journey. We've got this new one right in the middle called DataOps TestGen, which I'm gonna get to in a second. And DataOps automation, which focuses on data and use and our DataOps observability tool. Um, so, uh, what we're gonna talk about for the rest is this, you know, how do you test data at rest in a, in a database, uh, as an example.

And so how do you, how do you do that? And so, well, why, I guess the real question goes to sort of why, like why do you, like, why do we build a new product and, and what's the problem that, that it solves? Um, and so we, um, you know, we, we've um, been doing data testing and talking about DataOps for seven to 10 years now.

And so why, it's also, why now? So what are the challenges with testing? I I guess a lot of people have databases that are kind of black holes, they're dastardly dark databases, they don't really know what's in 'em, they don't understand it. They have a lot of tables, it's sort of a black box to them. And then, um, they have, data teams have a lot to do, right? Their customers are very demanding.

Things are breaking, so teams are stressed out. And then obviously there are embarrassing production errors, um, that your customers find and they find them before you. Um, and then data teams don't, aren't able to kind of have the business knowledge. You, you may have a thousand tables. How are you supposed to know all about the thousand tables?

So if you're gonna test data, how can you test data without really connecting the business and, and the levels in data and the syntax and symantecs and pragmatics of data to put it together. So how do you actually, if you don't have time or the knowledge to create data validation tests? And so we purposely built these two things to solve those problems.

And really it boils down to two things. One is that you're so overworked that you can't really trust the data or understand it, right? And then second is you don't have the sort of domain knowledge or time to fix it. And so these two big problems I think are very important in what we've tried to set up to solve. Because fixing the ability for you to have to trust your data, um, is uh, really part of the data journey. It's not the whole journey, it's not everything, but it actually is important.

Cuz data's combinatorial complex, there's a lot of, uh, uh,

00:10:00

and there there's a lot of trust but verify that needs to happen data. And so, um, you know, I think helping data engineers understand the data, helping data engineers create tests against the data, and then helping them, uh, make sure that those tests are with what the business needs and getting other people in the organization like data stewards involved, are sort of the design points for it. And, um, I, I'm just gonna have a personal aside here. Uh, and so, um, I'm gonna talk about my own personal history here.

And this is gonna have to be funny. So like, why do we build it now? Um, so I've been CEO of the company now for, I don't know, nine or 10 years. And, and we've been, I've honestly been doing data testing since 2016, and I've always sort of written my own data tests and um, you know, I'm a coder guy, so, but like, why did I, why was I sort of blind to those problems? And, um, and Chip was the one who actually helped me help us as a company figure that out. Well, I reflected back on my experience in college, and when I was in college, I had a girlfriend and she was sort of, uh, very musical, very artistic, and very sweet. Um, and one day she couldn't get into her dorm room and her doorknob was stuck.

And I, you know, uh, she was frustrated. She'd tried for 10 minutes, she couldn't get it in. So she finally found me. I went up and of course, me being the engineer who had taken everything apart that they could when I was a kid, I went and did left, right, boom. And the door opened and it just made sense to me. And of course, being the sensitive guy, I was at whatever, at, at 20, I told her, how can you not know how this door, this door not works? Almost laughing at her.

And of course she was mad at me for, for a week. Um, and then even playing forward, uh, when we started the company in 2014, I was a data engineer for a pharmaceutical company. I did, it was a small company out about 15 data sets. And I did the weekly data engineering kind of using our as a way to use our, our first, our our first tool. Um, and, you know, for us I was like, oh, I'm just gonna do tests cuz I'm a software engineer.

Software engineers write tests. Um, and then, you know, I think, uh, chip has really, um, kind of been the genesis of this. And, and he sort of has been building it in conjunction with our, a couple of customers over the, over a bunch of years. And, and finally very gently said to me, Chris, uh, this has really got value.

We should make this into a product. So this whole slide is, thank you Chip. Um, what do you think about that Chip? I, I like it. I'm gonna remember this slide. Well, it's the hardest thing about, uh, about it. Sometimes your own blindness can lead you astray. And so, um, I, I do think the real problem is it's really hard, um, to write data tests.

And that's what we're trying to take away. We're trying to automate it so you, it, it, and so you can, uh, do more of the stuff that you like. And, and so what is our tool, right? So it's a tool that profiles data automatically generates data tests, runs those data tests on data, um, and then shares the results pretty simply.

So kind of breaking it down into buckets, and I'm gonna go through each one of these. It's 28 simple, fast, automatically generated data quality checks. And I'm gonna use the term data quality data validation data tests, tests kind of interchangeably here cuz the, there's not really a common word. Um, and it does those because it has 51 characteristics, uh, that are done on profiling every, uh, table and column in the data.

And it also has 11 sort of best practice business context, fill in the blank data tests. And so, um, you know, chip and, and our team has been doing data tests for so long, we kind of see the pattern. And so there's a lot of cases where you can have a data test where you can sort of fill in the blank and say, okay, if you, you put in these parameters, you have a data test, and then after profiling, we have something called 13 bad data detector tests.

So if you profile your data, you can start to understand what went wrong and you know, your data is your data, but, uh, you can actually start to look and possibly, uh, patch it or go back to your providers. So 28 automatically generated te data tests. 11, sort of fill in the blank tests, um, 51 profiling characteristics and 13 bad data TE tests.

And so let's go through those. So, um, what does it mean to automatically create a data test? Well, um, the way I think of it is, is data has both syntax, it's and semantics meaning. And so looking at the syntax of the data is done through profiling, right? Um, looking at is a value present and a list of values.

So you profile the data and find a column has three values in, and then next time you run this, you get a fourth value. That may be an error or may not be an error. Um, you should know. Um, and then there's another case. Did you get, uh, a, a date in the data that's incorrect? And, uh, and so this idea,

00:15:00

or did the, is the schema of the data incorrect or did it not? We get, you get data not in the right format. So chip's gonna talk a lot about all the different checks that we have to, that are automatically generated, but it's, the idea is can you build this kind of quick scan of your data to find any problems, and then also be able to change it quickly so that you don't have too many false positives.

So think of it as like an initial screening or looking at every table and getting a test on every table. Um, and, um, I think that's the, the first bucket. And so you're trying to make sure that your data is, uh, when you get a refresh that it's correct. And then a another test is that, you know, these are sort of business rules and like, how do you know as a data engineer what the business is?

You have a model of sales and marketing and, and manufacturing in your head. Very few data engineers do that. But like looking at, um, looking at trying to build these kinds of rules then to see if the, you know, the total sales per region is the same week to week or there's been a big increase.

And the idea here is that your business customer looks at the world through a specific lens and they've memorized values and you're trying to find variations and, and, uh, problems before they do. And the idea behind this is that it's fill in the blank, right? You as a data engineer could put initial values, but then you can ship it off to your data stewards or business experts and they can add it in. And so another sort of theme in this is can you have the, the testing environment shared between you and, um, your data stewards or your business customers can, uh, you know, the, you, you maybe as a data engineer be responsible for running the tests, but can you get more people help you create and manage them, uh, as data changes? And then data profiling is very common.

Like there's a lot of data profiling tools out there, right? And think of it as an x-ray of your data and looking at date characteristics, min max volumes, mirror counts, percentiles unique values. And so we have 51 of those that we use. And, and primarily it's, it's to help you understand the data, but it's also as the basis to build those 28 tests. Um, and then the 13 bad data detector test, this is kind of looking at your data when you initially profile it and say, this data is just wrong. Like, um, you know, you've got some null columns, but you've also got some columns that are spaces that, that doesn't really make any sense. Um, and, uh, or you've got leading spaces in data or you've got invalid zip code formats, things where the data just looks wrong and maybe you control it, maybe you don't. Maybe you can give feedback to your customer.

Maybe you have to take, take the data as is. That's not your case. Um, but like, you know, whether it's feedback to your data provider, whether you can patch the data, whether you just have to live with it as is, um, it's really good to have this kind of, um, initial screening to help you improve the, the data once you profiled it.

And so, um, these tools themselves actually fit, um, the, the DataOps TestGen fits within our, our product suite. So we have, um, for instance, our observability on the left that collects things on the run status and the logs and the schedules and the metrics and the test results. Um, and we're just such a big believer in data testing that we, we, you know, we've got a tool to help you generate tests.

We've gotta help you an automation tool that helps you write tests, uh, yourself, and then we have an open API that you can put in your existing tests or use some other tool. Um, and all these things are important because, uh, testing, data validation, qualifying things and finding out problems before your customer sees it is really the, is really essential for your team to move forward with DataOps.

Um, and so it's been in development for two years. It's in use at several customers. We had the first commercial release this month. Um, the results are visible and DataOps observatory, it, it's a SQL base tool. So if, um, it doesn't actually run on Python, it actually runs on different databases and it's a, it's a, um, table driven SQL tool. So the, the profile results, the configuration are all in tables that you can, uh, uh, edit and change.

And so right now the configuration is done by the C L I, but the, a UI is coming later in the end of the year. So think of it, um, as a tool that where the results are visible in DataOps, observ observability, I'll show you it, it runs against databases. And, uh, because most data is in databases, um, and it's configuration is done by a C L I and a UI is, is upcoming.

And so let me talk about a customer example. And so, um, there's a, a pharma company that does this really amazing sort of, uh, cancer cell therapy, uh, um, drug where they take some of your blood, they look at it, and then they actually figure out from genetic tests,

00:20:00

what exact therapy you need, and then they give it back. It's just, it's incredible. Um, and that data stream is not just used for analysis, actually used for like monitoring reporting. And, and they have effectively have 70 different data sources. And, uh, the people who use the data want data that they can trust, but they want the data as soon as possible because like, they're also using right to monitor the, the production process, right?

If someone's blood got late, they want to know. Um, and so the, the point here is that DataOps TestGen has been, and for a year, year and a half, actually doing the testing against this data, all these different data sets. Um, and what that has means is that the customers almost never find any errors.

And it's not the only place that it does test, it does the data at REST test. There's also other places where, um, data and use test is done. Um, and it was done with a very small team. And so, um, normal big pharma companies to say, build this whole system with reporting, um, done in a year or three and a half people, that would be crazy.

I've seen teams that have literally 10 times that size and then do less. And so by automating, by following DataOps principles, by helping the team get time by generating tests for them and, and, and configuring this for TEM and also having the data stewards be involved, or the data the business owners be involved in configuring these tests, enable them to work really, really fast. Um, and so at another way to look at it is, um, you know, there's just a bunch of different data sets that go in and, and DataOps TestGen test at every step of the process.

And so kind of going to the bottom gear from, if you look at these columns, there's sort of data is posted in s3, it goes through a file, Damon a file processor, it goes through ingestion and data prep. Um, it goes through, um, some way to actually prepare the data, and then it's actually put out to, um, a Tableau dashboard.

And at every step of the way, um, checking, getting alerts and doing automated testing is helped by DataOps testing. And so there's another reason that's a very powerful way to make, um, data teams productive. And the productivity comes from, you know, writing tests. That's hard. And so it's a, it's a tool to help you write tests to who it's a tool for you to help, uh, configure those tasks with your business customer and data steward. And it's, uh, and, um, you know, we're pretty excited about it. And so, chip, how did I do? What do you think so far? Is this, uh, anything that I should add? Uh, Sounds good to me. I, I think that, you know, the, the challenge here for data engineers, uh, is, is that we're the first line of defense for process issues, but we're, we're often the, uh, last in line when it comes to knowing about the contents of data and, and, uh, that's the, the, uh, one of the challenges that we've been trying to address.

Yeah. And that, that's really tough. Like first in line for problems last in line to understand data that that's just like, that doesn't make sense, right? And that's part of the stress that teams have. Um, and so trying to understand the full context at which data is used is hard. And so, um, you can have this sort of layered approach, right? Where these, um, automatically generated the task can be your first or first line of defense, and then your second line of defense can be these sort of configurable, fill in the blank business rules, right? That, that's a second line.

And then like even your third line of defense could be tests that are very specific to your type of company or very specific to your business unit, your company or the domain you work in. And so, testing, um, in drug discovery has very different needs than in, um, manufacturing or than in financial services. And so this, you know, thinking of it in three buckets, tests that can be written automatically based on syntax, there's sort of fill in the blank tests that can be based on semantics.

And then there's sort of pragmatics like tests that are very specific. Those are probably always gonna be custom. And all three of those types of tests are needed in different places and at different times, depending. But, uh, giving you time to actually do that at both for data at rest and data and is, is what we're trying to do to help here. And so I, I'm gonna talk about CHIP'S demo and, and chip's gonna show you it.

So he's gonna actually fire up the command line and, and show you it, and I'll show you the results and, uh, and DataOps observability. So let's talk about kind of before you're producing data. So you get some data, and what CHIP'S gonna do is profile the data. And when that profile runs, he's gonna get a bunch of test results that identify bad data. And again, you could fix it, you could live with it, um, you could patch it, and then he is gonna press something that auto generates the data tests. And, um, you know, before data production, we also have another class of tests,

00:25:00

this fill in the blank tests. You could configure them. I don't think he's gonna do that in the demo. And then during data production, the tests execute during production. And so you may have your E T L or orchestrator or Crown job, it may be, um, it may be Airflow, it may be, um, any number of tools that, that actually execute jobs.

And then you're running these auto generated data tests, and then you're actually, uh, perhaps running these business real tasks. And then the test results are stored in a database, and then they're actually can be sent off the DataOps observability or, or, or anywhere. Um, and so kind of, uh, that's the first two steps before data production, during data production.

But then there's kind of that case of after data production and, you know, it's, it's sort of like, it is sort of tweak and improve, right? And so, uh, one case in improvement is kind of following the bottom here. You get new data, so your data in your database is changed. So you repro profile data and refresh the auto-generated tasks based on the profiling. And this, um, has to happen at some frequency. It doesn't have to happen every, every, you know, it depends on the frequency of your data, how often it changes.

Maybe it's every day, maybe it's every week for you. Um, and then there's sort of tweaking the configuration of these tasks and he'll show you a YAML file, and there's a table representation too, and that's sort of data engineer refinement. But then there's like, how do you get other people who know more about the business in, well, the exporting the test definitions, right?

And sending them to a data steward or a business experts. Now, we don't have a UI for that, and we're gonna have a UI by the end of the year, but that's also part of the value here, as, as, and we've done this without a UI by sharing spreadsheets and sharing files. And that's actually worked just as pretty well. Um, maybe not every data steward would like it, but, but it works. Um, you know, it, it is workable. And then the data steward tweaks, oh, this test should alarm at 70 instead of 40.

And you take those improved test definitions and you put 'em in TestGen and they run, they run again. So that's what he is gonna walk through, um, the idea of before data production, during data production, and kind of after data production. And so Chip, I'm gonna, um, stop and share the screen. I'm gonna make you the, uh, presenter and change presenter to chip.

All right, there you go. Okay, great. So let's see here.

How's that is my, is my, uh, terminal window red terminal window visible. It's beautiful. It's, it's a nice sort of, um, isn't it, isn't it Harvard red? That's your color? Is that it? You can see, you can see why we want a ui, but, but, but, uh, let's, let's, uh, that, that's in the works, but, but, uh, let's, let's take a look at what we can do here with, uh, with the, uh, C L I I think we've been able to do, uh, quite a bit. Um, one thing that, that I really, um, appreciate about, uh, having this as a tool is, is that we're able to share, you know, we're talking about sharing, uh, information with data stewards, but what we're also doing is creating a feedback loop on a regular basis with both, uh, uh, data stewards potentially and subject matter experts in the business so that we're able to, uh, share, uh, responsibility and accountability for data quality rather than, uh, you know, uh, play a game of, uh, of, uh, of hot potato. It be, it becomes something that, that is, uh, everyone's responsibility together.

So with that in mind, let's kind of move forward here. So, so, um, first of all, I'm just gonna grab an identifier

and what we're looking here is what we call, uh, a list of, oops.

It is what we call a list of table groups. And a table group is basically a specification for a set of tables in one particular database. Uh, TestGen can work with, uh, multiple database connections. And, and, and that's a real advantage because what we're able to do is to provide, uh, a kind of a consolidated hub for, for distributed data sets.

00:30:00

Uh, uh, in typical complex environments, you have different clusters and databases in different places. You may have data marks, you have self-serve environments. And what we're able to do is to consolidate, uh, a set of, uh, sets of rules and standards that can be applied to databases in multiple locations. So in this case, it's very simple. We have a little, uh, demo database on a, on a, uh, Redshift cluster that consists of a, a couple of tables, uh, but the, the tables are still big enough that it's a little bit awkward.

So these are the tables we use for testing. We're using, uh, uh, one table has 46,000 rows, another table has 8,000 rows, and it's, uh, somewhat of an hundred powered cluster. But, but, um, we, uh, are are able to just simply, uh, use some, uh, rules to specify which cables we're going to go after, and then we can run our profiling. I, I, I think in the interest of time, I'm not gonna run the profiling here.

Let me show you the profiling that we already have. It takes about five minutes to run profiling and to run testing. So, so I'm gonna show you what we already have for those things, but let, let me, let me show you here.

So I'm gonna,

something's pointed to the display. So these are diff multiple profiling runs, uh, that we have run in, uh, two tables, 184 columns. Now we can look at some details on profiling for each of these. We can, we can spit out a whole bunch of details, but on screen, we'll only look at a few.

So this, you, you can see here, this is providing some basic information. We, we have a whole bunch of, uh, different elements that we use, some of which are, are, uh, more useful for direct review and some of which are more useful as inputs into the test generation process.

Go. Hey, chip, could you just walk through a row here and sort of tell us, tell us what it, what's, what's in each column and sort of what it means roughly? Sure. I th this is basically just, uh, the, uh, each, each, uh, column in a table with the column mean. We have something which we call general type, which basically is alphanumeric, ga or boon.

We have the specific type as described in the database, and we have what we call a functional data type, which our system derives based on, uh, multiple sets of information where we're, we're trying to identify functionally what purpose the data type serves. And that allows us to, uh, that provides input into our choice of which tests to apply to that, uh, to that column.

All right, great. Thanks, Jim.

So now I'm gonna show you the results of the profile screening process, which is basically that initial bad data check. It's, it's that process to look for oddities of, uh, inputs that may be addressed, uh, as part of a data hygiene process. It's not necessarily something that you'd apply as a regular test in production, but, but it looks for potential issues that, uh, you might wanna review as, as, uh, uh, data hygiene concerns.

And so here's some examples in our test data of, of some common tests, right? So we have cases where, uh,

00:35:00

There may be, uh, a small number of blank values, and maybe this is, uh, a, uh, a column where there shouldn't be any blank values. Uh, mo uh, similar to that, mostly one value where, where, where, uh, uh, looking for cases where, you know, why, why are there a very small number of divergence, divergent values?

Um, there are columns where no column value is present at all, which could indicate a problem. Um, there are cases here where there, uh, multiple, uh, depictions of blank values where you might have some null values and some empty string values, and some of what we call field values, which, which would be something like X X X or uh, dash, dash, or even the word missing, or na, you know, quite often in data sets, uh, um, codes, let's say that come from Excel, like, uh, na result can make it make their way into data. When, when, uh, uh, we don't wanna depict that as a, uh, as a value that's actually present.

Um, and then we look, uh, among the other things we look for, and this is just some of the tests that happened to have been triggered by this dataset, this test dataset, but we also look for these pattern inconsistencies in data where, where we, uh, boil down, uh, data sets into, uh, common patterns.

This, this works really well to deal with things like, uh, uh, uh, data coming in from multiple sources like, uh, uh, phone numbers, for instance, that may come in, uh, formatted in different ways and from different sources. And, and you can catch those, uh, those formatting differences. Now, some of these may be nothing at all. They, they may not be an issue at all, but this just alerts you to kinds of common issues, uh, that you can run into. And then, and then, uh, you can make that decision about what is, uh, you know, what could indicate a problem. Other examples of things that we have, for example, I think Chris mentioned leading spaces, that's a common thing, or importing data that somehow ends up with quotes, uh, in it.

This will, this will catch that as well.

Then we get to the tests that are performed or that are, that are generated in production. And, um,

what we can do is take that, uh, let's see here. Let's, let's generate some tests here based on our profile.

Um,

So we're gonna use that, that, uh, table group,

and we're going to, uh, create those tasks in a, uh, what we call a test suite, which we're calling demo tasks for this purpose.

So we've, we've, um, just generated a bunch of tests that, that, uh, was pretty easy, and now we can, uh, list them.

So you profile the test, you profile the data, and now you've generated, you know, 28 different test types.

So wow. So there's 781 tests that you've generated, 781 tests on the two tables, 184 columns. So we've, what what we can do is, you know, in, in practice we, we run profiling on a weekly basis, and we periodically are able to refresh our tests. What this does is to, is to refresh tests if they already exist, uh, with the latest values.

You also have the ability to, uh, to lock some tests from being refreshed. Let's say that you have tighter standards than are present, or different standards that, than the ones that are automatically generated. You can set your own standards as part of your test refinement process and tweaking process, and you can lock them so that they don't automatically get refreshed.

00:40:00

But let's take a look at some of those tests here.

So that's not really all that helpful. That's great, because there, it's so wide. I don't, if I widen this, I don't know what people will see. It's still pretty big, but there's a lot of information about the test. I think it might be better. Let, let me show you the, uh, the, um,

yamo view instead

That, that's just sort of like a printing of what's in the database, right? And so this is the way that puts it as like, in a YAML file, it's easier to understand This is, this is easier to read. And, and, uh, now the, the, the other view actually, anytime we, uh, pull data in the cli, it's also creating a CSV file.

So we're able to pull that into Excel and, and, uh, share it as an Excel spreadsheet. We, we also share them this way, it's a little bit more documented, uh, because different tests have different parameters and different sets of parameters, and this will show only the parameters that are associated with a given test.

So, so, um, as an example, uh,

there's a, for, for a particular column, uh, the address column, there's a certain percentage of records that are unique. And we're, and we're testing for a shift in the percentage of unique records versus baseline data. So as you would imagine, these would be, uh, highly unique in a, in a address column. And if for some reason they suddenly became less unique, uh, there could be an indication of a, of a data issue.

Yeah. And so what's good is this, you know, both the CSV format and or the yml format could be shared with your, um, business customers to be configured. Um, the other part too is if you have separate development environments versus production environments, and some organizations are very strict about the separation, um, you can put this YAML file and get, and then that you can take it out of GI and then use that to run it in, in development. So you can almost like have an air gaped environment, which some organizations have between, um, between the development environment and production.

And you can build the test and build the test configuration, um, but then see if it works, but then in production actually executed

That. That's right. And, and, and you, you can also have different, uh, versions of tests that you can, uh, generate into different test suites. So you can run a, a newer version experimentally and see how that goes. You can have a version that you retain that's in production, so that, uh, if you need, uh, approval for changing standards or, or, or changing your changing expectations, then you can continue to run your original test suite as you, uh, um, test and gain approval for your new test suite. So there, there are all kinds of ways to do it.

So now I'm going to go back to the other screen,

and

I'm not gonna run test results. It takes about five minutes in this, um, in this environment. But let me show you some test results that have been run Previously.

Put it this way.

So these are some previous test runs, and what this shows you is for each test run, uh, the number of cables, the number of results, the, um,

00:45:00

Tests that have failed and, and whether or not the tests had been sent to observability. So in this case, most, uh, uh, previously, uh, uh, this was actually, uh, last night that I ran this, in this case previously, uh, all the tests passed. That's really the way it should be when you first run tests based on profiling, because if it's the profiling that generates your tests, and then you're running tests on exactly the same data, you shouldn't get any errors. Um, so I introduced one, I I, uh, uh, changed a, um, uh, uh, data point, and I introduced an error on purpose. And we, and, and let's take a look at how that error looks.

This, all this information for the test suite and the test time and the run is all stored in a database. And, you know, it can be shoved to observability, it could also be put to a governance tool. So for instance, I know tools like atland and Elation have ways to augment their information about, you know, the data catalog and what do the rows and columns mean.

And so we could push this to, to them as well. And so by having it persisted in a database, by having it configurable, uh, by having it understandable, by having it tweak tweakable by, maybe you have different versions of test config for development or not, um, it gives you a lot of flexibility to, um, handle the change that's inherent, uh, that chip talked about the change in data or the change in requirements.

And, and we're also in the process of building our own, uh, uh, dashboards and visualizations as, as part of our upcoming ui. So I think that's something that should be helpful as well.

So, looking at the test, we had a column called party status. Uh, there were only, I think, uh, two or three different possible values at the time of profiling. Our test generation made the assumption that those values would be the only values that are present going forward. So we have a list of value match test, and then I changed the value, so it failed. And, um, this, this, uh, um, pro provides that information and it, it, it, you know, you can imagine in a case like this, it might be that it's totally legitimate, that, that a new value is introduced and it really isn't a data error at all.

I wanna point out that even if that's the case, this is still information that's important to share with downstream data consumers because, uh, they may be building, uh, uh, uh, deliverables. I've seen, you know, deliverables such as, uh, dashboards and, and models that depend on an assumption of a certain number of, uh, potential values. And now for whatever reason, that that assumption has to be, uh, reexamined.

So it's not just a matter of finding errors that are production failures, it's a matter of, uh, communicating with people up and down the data journey. You're, you're, in some cases you may want to go back and contact upstream data providers and say, something's incorrect here. In some cases, it's valid and you wanna contact downstream data consumers to say that there's been a change and the, um, the, uh, tools here and the reporting capability provide, uh, a way to do that or to track those changes.

So it depends on sort of your architecture pattern, right? Like, let's say you're getting 10 data sources. Well, it's obviously important to check those data sources when they arrive, um, to make sure that there's any problems, um, uh, where, um, row counts are less than you have, they're syntax errors, et cetera. Um, and you, cuz you want that sort of fresh, the fresh most perfect data that you use during production. And, and that's where you're sort of assembling the data and doing all the other pieces in the reports and dashboards and models. That's another use case.

It could be on arrival, it could be in production. Some architectures have two or three database layers. Layer one, layer two, layer three sort of testing, uh, between those is is also part of it. Um, and then, uh, as Chip said at the beginning, testing across databases, maybe you have four or five databases where you wanna profile and understand, uh, and check, and maybe that's like an offline activity that, that you wanna help. Um, just understand your, the, the data system that you're in.

00:50:00

One area too, where this can be very useful is, is looking at, uh, legacy pipelines and, and legacy tools where you're, you're trying to provide an additional measure of, uh, oversight into the results of those tools. You may have to live with them for a while, but a but a common, um, tool like this allows you to, uh, work with them and make sure that, that the end product of those tools, uh, is correct regardless of what their processes are.

So now let's, let's export the, uh, information to our, uh, observability tool.

And this basically exports everything that has not been exported so far for this, uh, test suite.

And we're done. And, and, uh, Chris, do you wanna show how things look? Or, or I can show however you prefer. Oh yeah, why don't you show, yeah, if you, if you could show it, um, that would be great. I've got it.

So this, this is, uh, uh, our observability tool. Let me refresh. What happened is, by doing that export, we created a new instance, uh, of our, of what we call our demo journey. Each set of tests comprises one instance. Uh, and you can see multiple sets of tests on the same data are available here. Let's, let's dig into it

and we can see overall the status 781 tests, one failed,

and we can click in directly on the tests here. And this shows by default, the one that failed that we saw here before. Now this, this is actually our, um, general, uh, test, uh, visualization that, that is, uh, usable for input from multiple testing, uh, products. One thing, you know, as part of our ui, we're working on a, a more detailed view of our tasks for, um, for our own product. But the, but, but this is what this looks like.

You can, you can show all your tasks if you want,

and, and you can see all of our 781 tests.

That's great. Yeah, that's, that's good. We've only got five minutes left, so, um, let me, um, let, let me take over and I'll, uh, I'll just close it out. So,

So, um, you know, I think we've got, um, we, we've, uh, got some actually really good interesting things to read about TestGen. Um, you know, we got a white paper, uh, a product overview. You can actually look at the documentation on our website, um, to learn more about it. Um, and of course how the test and product fits in with the data journey.

And observability are also really, uh, I think really quite interesting. There's a document on the five pillars that the data journey manifesto. So Phil, uh, we'll send, again, we'll send these slides out and we'll send the recording. Um, and to kind of, uh, end with, uh, with a commercial. Um, you know, our goal is, is really that, um, you have more time to do what you like because less customers are yelling at you and you're feeling less squeezed from your data providers.

And so to do that, we believe that you need to observe the entire end-to-end data estate. Um, and to do that, you actually need to generate data tests because that's time consuming. And so why we've built that product is to save you time. Now, there's a debate that ship, and I have, we'll, test Jen Con do a hundred percent of your tests.

And I don't think we have a debate. There's always gonna be some custom tests. Um, you know, I think there may be more. I think it's really dependent, you know, maybe TestGen A and it's, um, fill in the blank test and automated test can get 60, 70% of your data testing needs. Now Chip sometimes says more 90%. Um, but you know, I think the, the point is most people that I talk to have have two.

00:55:00

They maybe have a row count, um, or maybe they're just seeing if, uh, this, uh, and they're actually not testing their data at all. And that risk of hoping your data right is, is not a great strategy. And so if at all, you can have tests and, and, uh, that's why we built observability so open.

So if you've already got existing tests, you can fit it in. And then sometimes you need to have tests that are built specifically for APIs and pythons and languages. They're not data tests, they're more tool tests. And we've got a way to do that. And so I wanna thank everyone, uh, today. Um, we're just about out of time. Um, and we, I see there's one or two questions, but we'll try to follow up by email. Um, and thank you everyone for, uh, for, uh, we're excited about this new product.

It's available now. Um, and uh, thank you very much. Bye. Thanks everybody.