On-Demand Webinar · 55 min
How to Do a Data Quality Assessment
A data quality assessment that produces change rather than a document: start with one consumer's urgent need, pick five columns, profile, generate tests, score, hand over specific fixes. Includes a ten-minute demo doing the whole loop in open source TestGen.
What you'll learn 6 points
- Data owners generally think their data is fine, because it is fit for their own purpose. The example on the call: marketing wants an upsell campaign, but 50% of records have a good email, 50% have a good street address, and only 20% have both, so the campaign cannot run.
- A data quality leader has influence rather than authority, which makes an assessment an act of advocacy rather than analysis. Chris's line is that data quality leadership is sales, not science, and the thing worth measuring is whether anything actually changed.
- The method is a loop: start with one consumer's urgent need, pick three to five critical elements, profile the data, draft checks, get alignment on why and when, score, hand over specific actions, monitor, then take the next tranche.
- The demo runs the whole loop on one table in about ten minutes: connect, define a table group, profile against 51 characteristics, get 11 hygiene issues, tag five columns for marketing, generate 76 tests, and run them.
- Adding one custom test for null yearly income moved the dashboard score from 99 to 94 and produced a PDF issue report listing the 5,397 affected records — the score and the fix are the same object.
- TestGen reads rather than copies. It makes a read-only SQL connection to your database, and only profiling results, test configuration, and test results land in its own Postgres database, all running in Docker on your own machine.
Prefer to read it? The written version is in Webinar: A New, More Effective Approach To Data Quality Assessments.
Slides
Transcript
Show chapters and dialogue 12 chapters · 8,964 words
- 0:00 Why data owners and data consumers disagree about quality
- 4:59 The tragedy of the commons, and why influence beats authority
- 9:59 The method: start with one consumer’s urgent need
- 14:58 What the open source tool does: profile, screen, generate, score
- 19:57 Issue reports, iteration, and why waterfall assessments go stale
- 24:57 Demo: a marketing table group, and profiling one table
- 29:56 Tagging five elements and generating 76 tests
- 34:56 A custom null test drops the score from 99 to 94
- 39:55 Two open source tools, the price, and the first questions
- 44:54 Q: PII, and which databases are supported
- 49:54 Q: governance versus observability, and unwilling data owners
- 54:53 Q: how accuracy is defined, and closing
00:00:00 Why data owners and data consumers disagree about quality
Chris Bergh: All right. Hello My name is Chris Bergh. I'm CEO of DataKitchen and I'll be your host for this exciting webinar today. We are talking about a topic that I think many of you have done. Many of you have heard about, but actually we have a interesting new twist that's enabled by our open-source software that I'm super happy to talk to you about. And in terms of logistics of the meeting, what I will do is share the recording. I will share the slides and share a transcript in an email to you soon after the meeting. And if you feel free to add questions in the right hand corner there's a box and you can put your questions in there. I will try to answer them at the end. Although I may answer them during as well. So what's our agenda today? So we're going to talk about data quality assessments, but what I'm going to kind of set the background of the world that requires a data quality assessment. And then we're going to talk about a new way to do data quality assessments that's enabled by our open-source, but it's a very iterative, focused, influence-based way to do it. I'm going to give you a demo of how to do a data quality assessment in our open source software. And then we're just going to have a quick conclusion. So, it should take again 45 minutes to an hour. And thank you for joining. So, what's the today? I guess my goal is to show that you can as an individual do a data quality assessment and affect change in your organization. And our key themes are really how to drive that change. And how to influence others to make that change and how to focus on specifically on the needs of the data consumers, how to start small, how to iterate quickly and how to use our source free technology as the enabler to make these things happen as an individual. And we have some anti themes today. So we're not going to talk about how to assess data without action. We're not going to talk about what's the best algorithm to measure data quality. We're not going to focus a lot or at all on data quality categories. We're not going to focus on large scopes or slow waterfall delivery or manual expensive projects. And so what's the background? So people who own the data where the data comes from in general and not always but in general find the data quality in their systems good enough for their particular needs. Data is kind of fit enough for what they're doing with it and honestly that they sometimes have little interest in fixing data for other people's needs. So as an example let's say someone owns the digital channel and they care deeply about email addresses but they don't really need streets. The Salesforce knocks on doors, so they actually really care about street addresses, but they only rarely send emails. And then the person who owns the ERP system cares about the right corporate address. And if we look at data, consumers, whether they're in marketing or sales or finance, they want data that's fit for their own purpose, which is often kind of beyond the parochial needs of the data owners. And often they find data is not fit enough for their needs. And they don't understand why the data quality is so poor. So let's have a simple example. Marketing wants to do an upsell campaign to customers at high potential company Seems like a reasonable thing. But they find that 50% have good emails, 50% have good street addresses, and only 20% have both. And only 75% of the customer accounts are aligned to a corporate account. So they're like, "This data is crap.
00:04:59 The tragedy of the commons, and why influence beats authority
Chris Bergh: I can't do this campaign and can you help me data quality person or data team and so this is kind of why data quality has always remained a significant challenge and it's not getting better right according to a dbt survey two-thirds to three-quarters of people don't trust their data and it results in potentially billions in losses and so one way to frame it is that it's from an ecology term called the tragedy of the commons. And this is where you have a piece of land that's not owned by anyone and maybe you have three or four or five families grazing on it and then the grass gets all overgrazed because no one owns it, no one cares. And data is kind of similar to that, right? One really owns data quality in the sense that we're talking about it fit for purpose for multiple people or maybe everybody owns it and it's kind of a shared responsibility and people focus on short-term gains over long-term integrity and kind of things happen and then they get going, right? Once you've overgrazed your land, erod then there's gullies and it even gets worse. And so a lot of times data teams, people who do data engineering or science or analytics or governance, they ingest and integrate and make data useful and data quality problems are often show up to them, Because they're trying to make it useful for lots of people. But they could happen in the raw data, but also they have their own quality problems. They have the production line quality. Maybe they've gotten perfect data but somehow their transformation or their visualization is wrong and who owns quality is also quite interesting right but sometimes there's an assigned role of a quality and governance team and I imagine some of you today have that title some people are data engineers some people are on the business side but it's broadly owned and today I'm going to kind of use the term data quality person or data quality leader but if that isn't your role. I mean in general the person with whatever title they are who's interested in making data quality improved in their organization and so for us we kind of see this not as a way to overcome the tragedy of the commons and the fact that data owners don't care. It really becomes an influence challenge because data quality leaders don't have power. They just have influence. They can't make anyone do things. And so how do you get someone to take action? You could for instance read Dale Carnegie. And data quality in my mind it's about changing and improving data. We're talking about assessments which is a measurement activity. But really the proof in the pudding is that something has changed. I'm not going to talk a lot about the world's perfect assessment. Because getting people to do things, getting people to improve data is really the coin that you should be measuring yourself on. And lastly, data quality leadership is sales, not science. And so it's about influencing people to take action. It's about convincing data consumers and the data team to work to make things better. An influence or a selling, it's a skill that can be learned. I'm an engineer who started my career at NASA. Lots of people can learn to get true to themselves and how to improve things and all of us believe that data quality is an issue. And we can use that as a way coupled with sort of a fast agile way of work and try to avoid at the end of the day becoming a data nags. It's really about selling and selling means convincing and convincing means getting stuff done and kind of nagging is sort of anti- sales in my mind and I don't use sales in a negative term here. I think, my wife's been a teacher for 35 years. I think she's probably one of the best salesmen I've ever seen, right? Because she convinces a bunch of middle schoolers to learn. And so, now I'm going to go on to the next part.
00:09:59 The method: start with one consumer’s urgent need
Chris Bergh: And again, if you've got questions, I'm going to talk about this way to do a data quality, assessment. I'm going to give a brief overview of how our open- source tool does it, and then we'll, jump into a demo. So, this process, and there's some words here, But the word at the top, right, start with a specific data consumer's urgent need. So, start with that marketing. I've got to do this campaign. Or we've got this compliance goal in our bank that we need to go and so actually start with that and so Attendee actually talked about advocacy as a word instead of sales and I think that's a better way to say it perhaps but really you've got to start you're advocating for a customer's needs and Again with multiple people in this organization some of whom think the data is great some of whom think the data is crappy and some of whom are want to fix it you need to be able to connect those and if you start with a data customer and select a small number of data elements to improve the most critical ones and here and this is important right start small and iterate and identify the technologists or data people or data owners who can effectively change those elements and gain access to that data and learn the data by profiling it and develop quickly draft checks that identify specific data quality challenge Automate it, Make sure that you can be able to say, "Okay, here's how this data is wrong and here's how you can get at it. Here's the query that goes with it." And then kind of gain alignment between your customers, your owners, your technologists on the need to improve and how you can do it. And so I think that that's an important point in influencing and something I've learned in my career working in industry is a lot of times we talk about what we're going to do, but getting aligned on why we're doing it and sort of when we're doing it, that alignment is incredibly important. And it doesn't have to be a three-hour meeting, and once you have that alignment, you've got to sort of what does success look like? And that's really what the assessment and scoring that we're going to show and that you've seen that we're going to talk about. And it's really building a quality score on those specific data elements. And then create specific detailed actions that need to take place. So share with these people do these things. Here's the exact issue that you need to fix. And then monitor progress. So as the data changes, as time goes on, you don't snap your fingers and data quality improve because everyone has day jobs, watch and measure your data quality improving over time and then kind of iterate until complete. And then the last cycle here is kind of start with another tranche and go back up to the top. And so here we have a very small cycle iterative focus customer focused way to make changes. And the benefit of this also is instead of having a broad brush big bang we're going to fix a thousand data elements including fax numbers it allows an individual to get successful and I think also the social proof and I've talked to a number of data quality people who are frustrated and upset that they're nagging people and no one's listening. And so getting alignment, getting actions done is a way to improve your own social proof that you're effective. And so what I'm going to do now is kind of go through each one of these steps and kind of talk a little bit more about what they are. And then I'm going to sort of talk a little bit about how our open-source software supports them. So the first step kind of start small. And I kind of think of these data elements as kind of a data product or a set of a product that's going to be used by your customer. And if you focus on the value that those elements have and really don't pick too many, just maybe start with three or five. And iteration speed is key. You'll have more time to do more with success. So this is an important part of the agile idea. By doing less, you gain the ability to do more. And if you work on things that are critical, you're going to gain leverage. And find the right people to influence. Don't have a meeting with 30 Get a few people that you can influence to make this happen. And this sort of product thinking, focusing on value, starting small, iterate influence is really, I think, a key way to do this. And this has been hard, These are good words, right? Everyone says, but what I'm really excited about is that we have an open source tool that allows you today to be able to do these things. And so part of that is learning the data and testing the data. And you need to gain access to the data and to develop draft data quality tests that identify specific data quality challenges and get a feel for the data.
00:14:58 What the open source tool does: profile, screen, generate, score
Chris Bergh: You can learn more and profiling as part of it, getting access, seeing understanding the data, so then you can communicate the challenges to people. And so, what I'm going to do is kind of talk about our open source tool here just as a way that supports this activity. And I'm going to give you a demo in a little bit, but as a takeaway, I included some slides. So, we have a tool. It's fully open source. It's full feature. It has a backend. It executes what we talk about in databases. So we support six or seven databases. And it has a very reasonable enterprise version but what we're trying to focus one user. So you can use it, you can download it, you can go at it as a single user. You have all the functionality and it really does these five tasks. It connects to your database. It profiles it. It helps you screen data sets and automatically creates what we call hygiene checks. This looks weird, maybe you should check it out. And then it actually does what we call generative data quality. So it has sort of an engine. I hesitate to call it AI, but I guess broad it has a series of decision trees that look at your data and the profiling results and recommends a whole fleet of data quality tests. And then as the data is refreshed or as you hit a button. And then lastly, it does data quality scoring. And so I'll walk through this in the demo, but as a leave behind, you kind of connect to data. And so we've got a way to connect and a connection string. And then you set up something that we're going to call a table group. And a table group is a way to say here's the set of tables in a database that I'm interested in. And then from that, you actually hit a button and profile the data. And then we've got a bunch of profiling characteristics. In fact, we got 51. And then once the profiling is done, you can see the results and you can kind of dig into the data. We have a data catalog where you can kind of go through the tables, the columns, and see all the stuff and I'll show you that in a bit. So, the next part of success is sort of getting everyone on the same page. And so getting that alignment between the owners and the technologists, maybe the data engineers, the data customers, the data quality teams and kind of starting I think asking for help saying look this is a problem. Can you help me? And saying look our data owner has got these five fields they need to improve because of this business need. This aligns to our corporate goal. Can we get this done? Can you help me do this? And I think I've learned that asking for help is often a great way to I've often been surprised at how often if people are willing to help. And then build an assessment dashboard. And we're going to show you how to do that today in the tool, but I think assessment is based on actual problems and saying I've got this column. It has 30% problems. And that's part of the assessment. And we to do that. You want to be able to do that quickly, easily, and not spend weeks or months assembling this together and do it on your key data and Keep it based on data quality tests on specific fixable actions and kind of I'm personally against the sort of quasi scientific language. I don't think that it all comes down to specific data elements. You can group them however you want. You could use the DAMA data quality categories. Fine. You could group them by critical data elements. You could group them by they all begin with a letter G. It's really about business and ROI and action and then tracking improvement over time. And I link a blog post of all the different types of data quality dashboards that you can do. And so part of our tool and what makes it cool and I think distinctive is once it profiles the data, it builds a whole bunch of data quality tests. So there's 32 automatically generated data quality tests. You just press a button. And they're there and they actually go into building this data quality dashboard. Plus it's got a nice UI to build a custom test. And I can show you that. And our goal here is to get you kind of 70% of the way there in data quality tests. No one has time to do this. It takes so much time. We have years and decades of experience with data. And so we've built this tool that it's a generative data quality. It generates them for you. And then we also can build a data quality dashboard. And so I'll show you how to do that. And then lastly, how does the scoring process work? To make scoring actionable, we start with data hygiene issues and then data quality tests and then data quality tests. All three of these components and how you disposition those, it's not relevant, it's a warning, it's a serious error. All of these things go into the calculation of the score. And so again, these are actions that you You can turn tests on and off.
00:19:57 Issue reports, iteration, and why waterfall assessments go stale
Chris Bergh: You can reconfigure tests. You can regenerate data quality tests based on new data. And all of them make the scoring process very dynamic and very easy for you to manage and control. This is a UI based tool, not a code tool. And so making things happen is really important, being able to go in and make a change and trying to say here data engineer here's what you have to Make it easy, make it simple, make it motivating. And we'll show you this idea of an issue report. And we also have an observability tool that can also put it in Jira or Teams or other tools that can also be a way to send this. But an issue report is a great way for an individual to who's is kind of working on their own and then sort of monitor and iterate. I've built a dashboard. I'm learning things and you're kind of constantly refining. Maybe you can tweak the tests. Maybe you disposition the test. Maybe you learn more and you can sort of watch the chart of data quality scores over time and see them improve. And so we're going to talk about that, We're going to show how to build a custom test. We're going to show about how to disposition. We're going to show these marketing scores changing over time and we'll show all that in the product. And then lastly, I think is do it all over again. The fundamental part is sort of start with a small number of data elements, attach a tool like TestGen to it, develop a series of data quality tests, get a dashboard tweak and influence. And it's much easier to start on real data that you can look at and feel. And it really helps you maximize your influence with customer focus. And if you don't have it, you can tap an expertise by kind of saying, hey, look at what this tool says. Is that right? And so for our way of thinking is that we're a big believer and the focus of our company always have been on this idea of DataOps, which brings together ideas that come from different industries, agile from software or lean in manufacturing or DevOps and software and bring these ideas together and I think these really apply to a data quality assessment right you get something working right away generate 80% of the rules automatically kind of before things are working and then you start measuring and evaluating kind of live you don't wait to do sort of a big long analysis And then you start establishing standards by using the tool. And that cycle actually maximizes your learning because you're doing real data. You're giving real feedback and you're iterating and improving. And so I guess my belief is that you don't want to wait months to deliver your value as a data quality leader. You don't want to go through this sort of waterfall process of analyzing and planning and designing and building and test and deploying. It's a very slow process, often takes months and oftentimes by the time you get it done. The actual assessment that you've delivered might be out of date. And so this kind of start smaller and deliver assess smaller bits over time actually enables you to get more done. And here's a simple cartoon to talk about it. Right? So let's say in this traditional on the top data quality assessment that takes months, you've gone through analyze and plan and design and build and you've delivered some learning and value at the end, In an assessment and maybe you've got set one and data set two here, but by doing it in this iterative way, you're getting learning and value continuously. And you have a chunk data set set three, data set four, data set five. You end up having more that you've delivered because you're working in an iterative way. And this I think is the key takeaway from this. So the last thing is we're giving a methodology that's very focused. Start with a customer need work on some data elements. Recommend specific data quality develop a dashboard, monitor and cycle through and the whole point is that it avoids kind of perform overly broad and ineffective data assessments. This narrow focused action. However, there is another way to do this that there is also a road for a broad data survey that we're not going to talk too much about today that TestGen enables because we do connect to hundreds or hundreds of tables and you can start to sort of go in and screen hundreds of data elements. So you could do another type of assessment that's this data. I'm looking at a 100 tables and each one has 10 columns. So I've got a thousand columns and I'm using it to kind of find troublemaking data elements that could possibly affect your outcomes. And so what this means is you could also provide value.
00:24:57 Demo: a marketing table group, and profiling one table
Chris Bergh: You could say, "Look, you want these five fields, but really this other field actually is really interesting and have you looked at this field to help what you're doing with your business goals." So, you can be more proactive by doing this broad assessment. And of course, again, just we're focused on the narrow way of doing it, but this looking broadly and bringing it to narrow is also another way to work. And so, let's see. I'm going to take a break here and see if there's any questions. And so, okay, I So, now I'm going to switch and actually do a demo. So, to do that, I'm going to go into our product here. And kind of what we're going to do, I'm going to share this tab. So, what we're going to do is we're going to do this entire process here in about 10 minutes. And so, that's what I'm extremely excited about this product. This is the product. You can use it today. And so like I said, it's fully open source. And so here's the project dashboard. It's got a bunch of components in. We've got a help center on one side. We've got a Slack community that you can join. Our team is very helpful when you have questions. We've got a whole training portal and a certification that you can take. And this gives you an overview. And the important concept that we're going to talk about is a table group, which is a group of tables. In a particular database. And here I've got four databases with some sample data in we also have a quality dashboard. And here's an example of the quality dashboard. So I'm going to show you how to build one, what goes into it. And so to start, I'm going to look at our data catalog and look at this adventure work data. And so there's a lot of tables here. And there's just a lot of interesting information that we gather from profiling. You can even go in and preview the data and see what it looks like. And so I'm going to show more about this, but what we're going to do is take a slice of this adventure works data. So I'm going to get We have an existing connection to the data that's in a Postgres database. And so we have some table groups for it. And this is all running on locally on my Mac. And so, I've got a table group and I'm going to set up a table group. And so this means what it means is I'm going to actually go in and I'm going to call it marketing because we're going to our marketing customer. The schema that it's in is public. And I'm going to only be interested in one table and that's our customer dimension table. And that's all. And I hit add. So I've added this table group and then I'm just going to go in and edit it and just make sure it worked. So I test it. Okay, I've got the dim customer. This is my table group consisting of just one table. So great. Now I've made a connection. I've got one table. What's my next thing to do? I'm going to go in and profile that. So I'm going to run profiling. And so what this does is take these 51 built-in characteristics and profiles the data and populates the data catalog. And so if I go in and I can see my profiling runs. Here's a list of all of them. And so I did one for marketing. It took three seconds. It's completed. It discovered 11 hygiene issues. And I'm going to find those up by looking at the catalog. So here I've got the marketing table group. And I look at dim customer. It gives me a overview of the entire table. So it's got 29 columns, 18 18,484 rows, a bunch of data points. I can preview the data again looking at it. It's got things like last name, it's got an email address. It's got some of the yearly incomes are null. That's interesting. I've got some tags on each one of the tables and it's identified automatically, some things that are potential PII.
00:29:56 Tagging five elements and generating 76 tests
Chris Bergh: It's also identified what we call hygiene issues. And these are things that don't look good. And so there may be cases where if you standardize the values, they're similar. And we can talk more about this, but this is actually really interesting in data quality. So we have about two dozen of these hygiene issues and PII characteristics that we automatically surface on your data. And so what I'm going to do is say, okay, I want to develop a data quality dashboard. I know I've got five elements. So, I'm interested in saying these are the five elements for my data quality dashboard. So, I'm going to pick last name. I'm going to pick email address, and I'm going to pick yearly income. And I'm going to set all those with a bit of metadata. And I'm going to call it the stakeholder group is marketing because I'm doing this dashboard for a marketing campaign. I'm going to hit save. So, now I've said things like yearly income is tagged with marketing. So I've tagged my five elements that I'm interested in developing a data quality dashboard. And so some of them first name are here. I can go in and say, what are the first names? I can kind of peruse the data and understand a little bit more about it. But one of the real benefits of TestGen is that it generates tests for you. It's a generative data quality application. So we have to build tests that run against that data. And to do that, we have to do something called a test suite. And so I'm going to add a test suite here. I'm going to call it marketing test suite. It's got a description. All that stuff is there. I'm going to add it. And so now I've got a test suite and it's called the marketing test suite. I don't have any tests. I'm just going to hit this button on the right called generate tests. And quickly it goes. And again, it's one table. It's 1,800 rows, but it generates automatically 76 tests. And so that's what's cool about this. I've got 76 tests out of the box that help me assess data quality that I can use as part of my process of creating an assessment. And then I can take these tests and this test suite. And kind of going back, I can actually run them against the data. I'm just going to hit run tests. And so again, this is data that's in a database. It's executing a bunch of SQL queries against it. And when it finishes, it creates what's called a test run. And so here I can see that 75 tests pass, zero have warning, one failed. And I can kind of drill into this to look at each one of these. And I can see that all these tests have passed. And that's interesting, And so I've got these tests. They've run against the data. I have one that fail. Let's go look at that. That's interesting. So, I've got some non-standard email addresses. So, I ran the test once. What is this? Let me look at the source data here. Okay. I've got some but email addresses can't have these looks like South American. They've got some accent marks on time. Those aren't valid email addresses. And so, This may not be a problem. And so, what you can do here is disposition these. You can say confirm this is I'm going to confirm this issue as relevant or I turn it off. Or I can deactivate it. And here's as a data quality person, you can start to manage how these tests impact your assessment process. And so I've built these tests. I've run these tests. I want to go ahead and do the last thing and build a And so I go to this quality dashboard. And I can see that there's one here. We've automatically built one called marketing, but I'm going to go in and kind of view the details on that. Here's our marketing dashboard, and I'm going to edit this a bit. And so, I'm We really aren't looking by critical data elements. I'm not a big fan of quality dimensions. So, I'm just going to get rid of the category. So, the table group is marketing and the stakeholder group here is also marketing. And so I can see here in the columns that these are the only five columns I'm building my score on. Email address, last name, yearly income, and we can see the impact on the score, the total score. And I'm going to save the changes. So now I have a score dashboard that actually gives me what I want. And so I can go back and look at my quality dashboard. I can see the marketing things and I can see the score results. So I can go look at the details and I can go in and see the score trend. It's always been one. I can see the columns. And so time may go on, you get more refreshes, the tests run again. And you can see your score change. But one way to show this is to actually say, I remember looking at my data and yearly income had a bunch of null characteristics in that doesn't seem great. So, I want to add a test specifically for that. So, I'm going to go to my test suite. And I've got my marketing tests and I've got a bunch of test definitions here and I'm going to add one. So, I'm going to add a specific custom test.
00:34:56 A custom null test drops the score from 99 to 94
Chris Bergh: And so this one is a test that defines a column value. And so this test is really about yearly income. That's the subject of this test. And I want to know if anyone there's at least one error where income is null. And let's see, that's a little chunk of a SQL statement. So I'm going to validate it, make sure, okay, that Yearly income is null. So I've created this custom test that just checks to see if it's yearly incomes null. And so it's added to my corpus of tests. So you can see it here. We have some other yearly income tests that look at they're really anomaly detection tests. So, these are looking for if I've profiled my data and a bunch are missing and I get more data that has even more missing, it'll tell me. And that's a really interesting task, but we're not interested in that for our score. We're interested in aggregate how many of these have a null in. And so, to do that, I have to go back to my test suite. And I want to run my tests again. So, I'm going to click the button and I've got my marketing test suite. I'm going to hit run tests. And so I've run the tests again. And this is performed in the background. And I can go to my test runs. And I can see here that the tests have run twice. And look at there's interesting this errors changed. So, if I go look at this I look at the test run of this and I click into it and I can see that this new test I added yearly income is failing and it fails. There are 5,397 missing ones. If I go click at the source data, it's smart enough here to says, okay, it pulled up all the records where if I scroll over here, the yearly income's Great. So, it works. And I can take this and generate an issue report. And it's a PDF issue report. It pops up here. And it's just like in our UI. We can see it and it comes up and it pops up in a separate window. And so I can recalculate the scores. And let's go look and see what happened to our quality dashboard. And so we can see up in the upper right here that it dropped that in the score details. I can see the score dropped from 99 to 94. And if I go look at the columns, I can see the big income now is the yearly income has dropped. I can go into the issue. I can drill in. And so either this is the challenge in dashboards, right? You've got the data can change and your view of what is good or bad can change. And so this is the agility here. It's all under your control. It's all we're trying to make it so a single person can have a lot of success building dashboards, communicating issue reports, drilling into the issues and finding out what the problems are. And so I'm going to go back here to our actual slides. And I'm going to go through the conclusion now or and just before I go to the conclusion I'm just going to talk a little bit about the product and then finish up. And so we sort of follow this process right profile data screen for data hygiene gotchas generate data quality tests and then you go through the source code changes or not you execute the tests you generate the scores you ine you share the issues. This is the general process flow that we're talking about all at your fingertips. We have a very simple technical architecture. It all runs against your databases. And then we have two open source tools. One is our TestGen tool. It's very focused on data quality. But there's another part of quality is the process quality which is maybe my data is perfect but moving the data building facts and dimensions building reports building models something went along in the production line and this is where we have another tool that integrates very well with observability called our DataOps observability and this is the sort of because at the end of the day your customer looks at a report and it's wrong and they're going
00:39:55 Two open source tools, the price, and the first questions
Chris Bergh: To say the data is wrong and it may be that the source data is wrong or it may be that you did something to the data or maybe the report is misconfigured, maybe the data is right, maybe the way you've integrated the data is incorrect, maybe the model's wrong, maybe the report's wrong. Being able to pinpoint that and get a good answer is a great way to improve your operational efficiency. And so we have two open source tools to do that. So what's our goal? So, as a company, I don't know, we spent about $6 million developing these open source tools. We're a profitable company. We've been in business for about 10 years. Our goal is to make DataOps happen. And so, our goal here is to give a single person with limited time a tool to affect meaningful data quality change. And so, we've done market research. We've talked to a bunch of data quality people. A level of frustration of feeling data nags is intolerable. And we think this idea of working in an agile DataOps way is the way to gain that power back. Why? Because it's no cost. It runs on your laptop. It has unlimited database tables. It has all the features, UIs, single users. It does generative data quality test creation. And if you want to have more than one database connection, it's a pretty reasonable price. $100 per user and per database connection per month. And so we think this gives you data quality influence superpowers. So that's our goal with this and we're very excited. And so in a conclusion, you really have two changes how do I deal with all this data at scale and how do I influence change? And we're recommending that you get going quickly. You focus on a specific customer, you get specific data elements, you measure, you improve, and you start with using open source that gives you because there's no other tool like what we've built that has this combination of features and I don't know whether you have to pay for it or not. I think it's a great way for you to start. And lastly, we've just got a whole bunch of other tools to help you about with DataOps. We've got TestGen. We've got our observability product. We have a certification on data observability. We've written two books on how to do agile in data and analytics. How to actually do DataOps. We have free certifications. So I'm going to Go back to this slide and I'm going to look at and see if there's any comments or questions I can answer. So we've got some questions. At 12:30 I have a question for the end. Do you have a recommendation for example interviews to understand key use cases or data elements from data customers? Are there ways to do this quantitatively? Let me answer that. So what are important data elements focus on? And so one way to do it is identify data elements that are critical data elements. And so we have a feature in our product that surfaces what potential critical data elements are just like potential PII. So that's one way just start with critical data elements and use that as the first thing. I'm a big believer if you're going to focus on your customer, talking about what their needs are and trying to focus on what's going to give them a win. So, talking to customers and starting and then translating that into what data they're going to need, I think is a great skill for people to develop. And I don't think software can do that or ChatGPT. I think talking to people, finding out what they need, translating that and here's these data elements that you need is and so the next question is where to find test databases to try TestGen. The good thing is the install comes with a test database. So you can use that and I can share after some test data that I've used. There's plenty of sites that have test data out there and we actually have in our docs. So, if I go and bring this tab up, just going to share this tab instead. We've got some quick starts that you may go to. And we have a quick start here of TestGen and it goes through a bunch of different databases, but this has a step walkthrough on how to do it. So, including getting some the Adventure Works data that I talk here and go to download it and sort of walk through with screens exactly how to build hygiene issues, data catalog, kind of goes through what I did today. And that's great. And so this is a great place for you to start.
00:44:54 Q: PII, and which databases are supported
Chris Bergh: So going back to the questions how do I address the concern of data privacy with these clients? That's a good question. I think one of the things is that we try to surface what we think are PII what are potentially so our tool is not a data security tool. It doesn't protect or encrypt or anonymize data. So that's sort of beyond the scope. How you treat these PII data is kind of up to you and not in the scope of TestGen. Is it possible to custom quality checks for a large language model? If what are the metrics used? I'm not sure I understand that question. Data that goes into an LLM in order to do things like what's called RAG or actually setting a whole bunch of data into the context window of the LLM. It's great that has good data quality and we can help that. The tool itself doesn't have a connection to any of the popular LLMs, but that's a topic that we should talk about and I think there's ways that we can help, but I'd have to learn more. And Attendee says, "As a solo shop, I appreciate the investment." Yeah. And that's the thing I think this is everybody in data quality is a solo shop to a certain extent, not just consultants. Everyone's got to get their own personal leverage to make stuff happen. And when you've got to talk to someone to build something, you've got to go through this long waterfall process, you end up not getting anything done. And so here's a case where you can get something done really in a half hour, and then start making changes. And I think that's really what it's about. And so, right now, an attendee asked a question, is TestGen only for relational databases? And that's correct. It there are seven or eight types of databases that it supports. It does not work on databases that are sort of hierarchical like MongoDB it doesn't work on images so that is a limitation we need a query engine and so we support that so there's another question what are the rules for data quality what is the criteria that the rule is checking I think that's really a great question So no one knows what the rules for data quality are. There isn't a standard data quality diagnostic and as a result, most people aren't actively checking data quality. That maybe they're sort of handwaving and looking at things. They're not actively making concrete active data quality tests. So we've tried to fill that void by actually building a whole bunch of data quality checks that are automatically generated. And so you can actually see those types of tests that are in the product. And so what are the rules in terms of the industry standard? I think that's a sort of beyond the scope. But we've tried to fill the void by giving you dozens and dozens of existing data quality rules that you can use today and a capability for you to build themselves on the hope that you get 70% of the way there with a few clicks and then you can focus your time on the things that are unique to your domain because there is no tool that's going to be able to do data quality for a pharma manufacturing company, for a financial service company, for a geospatial company, for an internet company. These are very different and domain dependent but there are some things that are invariant across data sets that we think we've captured. And An attendee has a question on the opportunity to contribute. Absolutely. This is all on the source itself. We've got the link to do it, but you can also go in and it's all on GitHub. So you can go in and see this. This is our TestGen. It starts with our installer, but you can go in and look at TestGen and download from there if you want. Does TestGen store the source data or just metadata? So, that's a good question. We don't store any of your data. And so, I'm going to go back and share this tab instead and kind of go to the architecture one more time. And so, there is a database here that stores the data. We make a SQL connection to it that does reads. So, we don't copy the data. All the data is resident. We do store the data about the profiling, the test configuration, the test results in a Postgres database. This application itself all runs in Docker containers on prem. So you go in and install it. You can access it through the UI or the CLI. So it runs in Docker Compose. And our installer is we've done a lot of work to make it pretty easy for you to try it out on your laptop. You do need it's a Windows professional because it's got to have Docker or I have it running on my Mac and so keep going on the questions. Could you please share your views on the differences between data governance and data observability? I guess to me data governance and I'm data governance is a lot of things, right?
00:49:54 Q: governance versus observability, and unwilling data owners
Chris Bergh: It could be the role, it could be the activity, it could be the broad category that govern that Gartner does, but it's the process of maintaining a data catalog, defining the state of what is and sharing it. That's why we have and we're not trying to be a data governance tool. There's plenty out there about tracking lineage and catalogs and what isn't. And data observability is also a really good thing. We define data observability as a process of looking at the entire production system that includes data and so we've got a few more questions Gil brought them on. Should a minimum requirement for data owners be their willingness and ability to focus on meeting the needs of all the downstream customers in their domain? So that's a really good question. How do you get data owners who are willing to change? You might not have to because it depends on where you change. So, you may ultimately want to change the source system, but they may not have time. So, a lot of cases data quality people are putting it into the analytic system. So, they're changing it not in the actual core databases, but they're changing it in the sort of let's see, I'm trying to find the slide here. Let me show you this one that there's sometimes fixing it in the source systems and in that case picking people who are willing and have time to make changes is a good thing to do but sometimes you actually just fix the data quality in the data warehouse or in the landing zone of the data warehouse that's also also an option. So keep going on the questions. Can you please speak to the role of the business and IT process improvement in order to prevent future data errors? Where does that fit in the approach that you describe? I don't think it does, honestly, because I think ultimately you need to have an interface into the source systems that blocks people from putting in data that's incorrect. And so that's sort of beyond the scope of what our tool does. Potentially it's an area of improvement in the future. But I think that's a really sort of general problem. How do you stop data quality errors in the first place? Not fix them afterwards. And those do involve establishing standards. And I think one of the things that test can provide is we create standards automatically. We allow you to edit them quickly. We allow you to turn them on and off quickly. And so by setting those standards, at least in one place you have them. Perhaps those ideal solution is to have those standards spread broadly and are sort of queried by the user interface every time someone checks something every time someone fills out a form field on a website. That would be great. But that's sort of beyond the scope of what our tool does. Do you have use cases to understand use cases or data elements from the customers? I think be beyond the tool is if you find data elements that have real value to a customer and that customer is willing to push and not be passive about it. That's the great case and I'm sort of against boiling the ocean. I think starting off with five good elements for one interested customer is great and get something done. Get something in the bank, Make a small deposit of value. Maybe it's only $5 or five data elements. You don't have to in put $500 in the bank. Just start small, but then keep adding. And I think So, lastly, a question about how is accuracy defined and actually measured. So, that's a great question. In fact, there's all the sort of, data categories here. So I'm going to share this tab and so we do have these quality dimensions. So you can look at the data unique completeness. The way that we define that is by the pieces of the test. We tagged a test saying This test is a uniqueness test. And that's how we define it. We take the idea of accuracy.
00:54:53 Q: how accuracy is defined, and closing
Chris Bergh: In the semantic idea of accuracy does this field actually represent what's happening in the real world? I don't think a software tool can do that, but we can actually help you if you have ways to express that You can input that into the software as a custom test. Lastly, so I think we're about to run time. So what I'm going to do is the recording is going to go out, the transcription is going to go out, the slides are going to go up kind of probably by the end of today or tomorrow. So, thank you all for joining. We're really excited about this. Please give it a try. We will support you. Get onto the Slack. And if you've run into any issues, our team is eager for you to help. We're very excited that this is something new in the marketplace that's going to be really help. Data quality people make a difference. So, thank you very much. Have a great day.
Machine-generated transcript, lightly edited: filler words removed, product and speaker names corrected, and audience members anonymised. Chapter times are scaled from the meeting clock onto the recording, which is shorter than the meeting. Speaker attribution is as captured on the call.
Questions from this session
How do you run a data quality assessment that changes the data?
A way of doing data quality assessments that ends in changed data rather than a report. Chris Bergh sets up why data owners and data consumers disagree about quality, frames the work as influence rather than measurement, lays out an iterative method built around one consumer's urgent need, then demonstrates the whole loop in DataKitchen's open source TestGen — connect, profile, generate tests, score, issue report — before taking about fifteen minutes of audience questions.
How do I work out which data elements to assess, and can it be done quantitatively?
Two routes, and Chris recommends both. The quantitative one is to let the tool surface candidates: it flags potential critical data elements the same way it flags potential PII, so you can start there. The other is to talk to the customer about what would count as a win and translate that into the columns they need. He was direct that no software and no chatbot does the second part for you, and that translating a business need into data elements is a skill worth developing.
How do I address data privacy concerns with clients?
TestGen flags what it thinks is PII during profiling, but it is not a data security tool: it does not protect, encrypt, or anonymise anything. How you handle the PII it surfaces is your call and outside the scope of the product.
Is TestGen only for relational databases?
Yes. It supports seven or eight relational database types and needs a query engine to work, so it does not run against hierarchical stores such as MongoDB, and it does not work on images.
Does TestGen store the source data, or just metadata?
Just metadata. It makes a SQL connection that reads, and the data stays where it is. What it stores in its own Postgres database is the profiling results, the test configuration, and the test results. The application runs in Docker containers on your own hardware, reachable through the UI or the CLI.
How is accuracy defined and measured?
Through the tests, not through a taxonomy. Each test is tagged with a dimension — uniqueness, completeness, and so on — and the dimension score is the sum of those tests. Accuracy in the semantic sense, whether a field truly represents what happened in the real world, is not something Chris thinks a software tool can determine; if you have a way to express it, you encode it as a custom test.
Where to go next
- Install open-source TestGen Apache 2.0, runs in your own database. Docker Compose to a first quality score in about 15 minutes.
- Every on-demand webinar The full recording library.