Managing Stress in Data Engineering: Data Quality and Testing Techniques for Data Observability

00:00:00

Welcome everyone. My name is Chris Bergh CEO and head chef of DataKitchen and thank you for spending the time to come to our webinar today. I am joined by Chip Bloche. Welcome chip and thank you. He's going to be my partner in crime today chip. Do you want to give a little introduction to yourself?

Thanks, Chris. I am a data engineer a long time data engineer and I I am director of data engineering it DataKitchen and I've basically been spending my career working in the trenches dealing with errors of one kind or another and I just have a question for you. Why is it what that whenever the subject is error? You'll always ask me to join you in the presentation. Should I be concerned about that? You should check? Yeah, you know, you're you're my error guy.

No, I think you know well, I think errors are a real problem and I guess you know my background is I I wasn't like a long time. Data engineer, but but I managed data engineer teams, and I was a data engineer for years and sort of problems were the sort of Bane of my existence. I just didn't like things going wrong. And so

You know, I think. A lot of the stress that people have in data engineering is sourced in errors. And you know, we're not going to talk about meditation techniques or good any presence to take what we're not your therapist. So we're not going to help you with that type of stress. But what we are going to do today is kind of walk through some ideas on how to actually fix it permanently.

And so to do that, I want to talk about a few things in terms of housekeeping. So we're going to actually share the slides in the recording. So you'll get an email and everyone will get an email. We'll put up on our website too and put your questions in the chat window. Well answer the questions at the end of the session and we'll sort of periodically monitor it throughout throughout today, and we've targeted 50 minutes for our presentation, but we may go long so, but we'll try to stick to the hour.

And so so today's webinar is really about testing sort of the why and the what and the when and the where and the how and the who have testing and then, you know in the middle chips going to actually talk. The biggest session today is really about how to do testing. And so today this isn't as much a product discussion or a product future discussion. This is really a best practice and idea discussion.

And so like the sign says we're not gonna help you meditate. We're gonna help you solve it. And so why do you want to solve it? Well, I guess basically your house is on fire. And so there's a lot of things that you should do to help your house environment mainly put it out. But also you should help make sure that you have smoke alarms in every room and if it's the if you remember remember change your smoke alarm batteries at the New Year's in your house if you have a house So why is your data house on fire? And so there's four data points here one is that most data and analytic projects fail? According to Gardener according to a survey with Eckerson most data analytic pipelines just have way too many errors, and we've always heard that models never get into production. And and last year we did a A survey with data.world of 700 data engineers and that's where we got this this number. That's 78% of them were so stressed and the number one source of their stress was errors and problems and so it's great to see a therapist, but let's get out the root cause of that in our presentation today.

And so I think the main source of that is that you can't answer very basic questions and I've got a list of them here and the most important question I think is is the thing that you're doing is the product you're giving out the dashboard the Rapport the data set the model is it correct? And is it refreshed with the latest latest data another way to say that it doesn't have errors.

And do you know that ahead of time and if you don't that's a problem and that's a source of a lot of stress. So hope is not your enemy our hope is not your friend in this hoping that things are right likewise. There's just a lot of things about the day-to-day production process that we can't answer in any way that that takes time like well resources did each file arrive is my source data good did each job run in order how many jobs are in yesterday versus today likewise in development? We can't answer the

00:05:00

change if I change something like a sequel a model report. Will it break anything in production? So these are very and the reason I've highlighted those two sections is I think they're actually very important error rates and cycle time. I think lead to productivity and that's kind of the the why you should care part of this presentation is that testing drives productivity and results and so if you can get your production errors Of any Source production errors from having bad data in the beginning something happening during the process of building your dashboard or model. If you can get those to be very low you end up with a lot of time that you can do great things likewise, if you can deploy quickly, you can iterate faster and therefore learn more accurately what your customer wants in both of those have drive. If you work on those they drive incredible productivity benefits five times in Gartner's latest report even 10 times productivity. And why is that? Well, it's just waste right you're it. If you have errors in production, you're wasting time finding and fixing them.

If you're putting errors in production you're wasting time and if you're taking forever months to actually figure out what your customer wants. You're not accurately doing good work and you're wasting time. So really productivity is a is it's counter is is removing waste. And so let's start at the top here. What is testing? What do we mean by this term? Well, so that this testing may not be the right term but it's the one that we're going to use and we mean checking that things are right and so for us there's a lot of terms that we've heard about the for instance data quality and a group called daima, which is a great group has got the dimensions of data quality. And so for us the idea of Errors is greater than data quality because errors could be because you have poor data quality in the beginning, right, you know garbage and garbage out, but you could have perfect data, but you still could have errors Something could go wrong in the processing a server go down a server could have take too much time. Some code acting upon the data doesn't work a dashboard doesn't work and so or in all these things contribute to error rates and so for us errors are the thing that we want tests to solve and so for us we're going to use the term testing as the way to do that.

And in this we're going to say tests and talk about tests checking or Q. Seeing or monitoring these two processes. One is we talk about the value pipeline, which is the journey that data takes from source to value and all the different tools that it goes through in steps in. The second is the Innovation pipeline, which is how you deploy things to production and both of those need to be tested you want to test in production. And and why because you don't want to learn about data report quality model issues from your customers you want to know if things are right before they see them.

And then second is you don't want to bake break production when you make a change to production. And so both of those things are cured with tests and there's other things that we've talked about to do it in terms of orchestration and Automation and environment management. But for us today, we're just going to focus on tests. And you know, we're using this very broad term and some people honestly use tests in different ways. But when we mean tests, we actually mean something very specific. We mean automated tests tests that actually happen automatically without human intervention. And so it's not just manual checks. In fact, we're going to talk a lot about why Manuel checks aren't good.

And so for instance, when should we do testing? Well, first of all test everywhere in production and we're going to talk our last webinar. So that's something called the data Journey which is all the steps that data takes from when it lands to when it gets to your customer. And some people call those quality control check some people call those anomaly detections, you know, the unfortunately the market really hasn't settled on terms for those. I I happen to just like tests and so tests in production, we're going to talk about those but we're also going to talk about tests and development and that happens based on, you know test data but we're gonna happen when it's in development or a QA process and people call those other terms. Sometimes these tests are called impact tests to regression tests or unit tests or functional tests. They take a language from software engineering and so for us when we use the term tests in this they mean tests that run into development and test that run in production.

Um, and so our belief is that really tests have this dual nature and so in production your data varies as you load data things change and

00:10:00

so but but your code that's acting upon data is fixed, but that's inverse in development. You're you're data is fixed IE. It's development data, but your code varies and so across both these tests can be useful. You can have a lot of tests that one in production should also run and and development and really when you're testing any data and analytics system that's composed of tools that runs on servers you're really testing data. And so at the end of the day tests are data tests, whether you want to call them that way but they have different purposes. So there's a dual nature of tests.

And so the fourth category sort of where to do testing. And so, you know here at DataKitchen we've been using this concept called a data journey and it's and what we mean by it is is you imagine where data comes in from anywhere in your organization and outside of your organization and all the path that it takes to get somewhere and then it ends up in your customers hand and maybe that's a dashboard or maybe that's back into an internal system or a report or an extract but you know, the the the path that it takes from source to Insight value and so how do you actually check that right? Well, first of all, you know, the Market's huge there's 80 billion dollars in spend every year and all the tools and everyone loves their tools to do ETL or El T. You're data science or data engineering and there's 50 plus Tools in each category and a lot of companies have more than one tool. Right? And and so how do you actually make sure that as it goes through all those tools? It's right.

And then a lot of companies have just data Journeys everywhere. They don't have one pipeline or one journey in production. They have lots of them and some of them are batch and some of them are streaming and some of them are are manual. and so

and so

If we go to the next slide. You know, what we think is that every part of the process every tool acting upon the data needs a test. And so for instance if when you get your data from a server from a source, is it good is it right? The second is the things that are acting upon the data your ETL or elt process your model your Visa. Are they correct? Because all many of those all of those tools are code govern processes and code could be an XML configuration file adjacent file. It could be python but really they are code encased in a tool that's acting upon data and you want to know if that's right and then at the end just want to know if you're Outlets are consistent and and you know, I think one of the reasons I was not surprised that 78% of Data Engineers want a therapist is that in my experience? I just you know had the morning dread. I didn't like finding think problems when I went into work in the morning and then having having to fix them or run around to to go back and then of course, there's expectations, right you expect everything to be perfect. But you're also expected to run at a time. And so there's a set of expectations and tests that happen against a data Journey.

And then there's just lots of different ways and we're going to talk a lot about tests and ways to check and people, you know, I've been over the pastor of nine years we've been involved in the company. I think a lot more people are putting automated checks in and some of them are simple. Some of them are unit tests unit checks, but I think it's becoming much more common now for your average daily data engineer to put test somewhere in the process and this also has to do with a term called the definition of done and what does done mean of your work and for us we have a very expansive definition doesn't mean it works on your machine done means that it works in production done means that has tests that tell that you write on means it's in source code done means that it's delivering value. And so I'm gonna hand it over to chip now. And what we're gonna do is he's gonna go in and talk a lot about how to do testing. So let me make chip the presenter here.

You said we're not going to do therapy Chris to does that mean I can't blame my parents for my parents. You can actually yeah such a chip. What why did your parents make you a data engineer? You can blame my parents too if that helps you. um, let's see do I have

You are the presenter now. Let's see.

That's the one.

How's that looking?

00:15:00

It looks great chip. All right, good. We're got this, right? So, let's see here. How did you testing? Well these this first slide is something that is is a Repetition from what Christian already said and maybe we could have done a little bit better QC on our slide deck. I apologize for that. But it but it's important to reiterate, you know, always test automatically no if I made that slide, are you saying that there's an error in my slide? No, not at all.

Always test automatically not manually and and you know, this is the it's the same reason that you do your processes automatically right humans make mistakes and and the purpose of a good testing plan is to put the your people in the best position to succeed and and we say automatically for example, you know, this is something I've seen, you know spreadsheets cut and paste did you data verification?

That's not automated. Other you know, we I I've seen that I've opened up someone's spreadsheet a partner spreadsheet that they had provided to show that the process was working correctly and the spreadsheet had circular reference errors. So the problem with manual verification is you have to keep verifying your verification and and an automated process works much more effectively. Yeah, and also like if you build a big checklist for someone to do manually they inevitably get it wrong. And so I remember in 2006 or seven. I had a an engineer with this third like five page checklist and he just sometimes would forget it. He was a fun guy but like man checklists are boring. Um, and it's boring work and I guess you know, it's this Echo what chip said do it automatically, you know, don't we're not religious. Maybe do it me and you really wants her twice to get a feel for it then automate it.

Right and and always test every step of the data journey. I mean first of all this helps you to verify that all your steps were successful, but the second piece of this is that by putting tests in place at every step. You can much more quickly get to the root cause of an error.

Okay. So so what are potential target areas for testing? Well, we've talked about development versus production. I like to break testing out into production testing out into production process testing and production data testing the development as we've said that's you know, your your goal is to qualify a version of your pipeline for release. The problem is that the data is always going to surprise you there are going to be changes to the data. They're going to be and normally is that you're not going to be able to control or or Predict and that's why production testing is so vital and this is hopefully something that everyone knows by heart probably in this in this group. We've all we've all been through that. We've all been bitten.

um production process testing Here we're talking about process steps. We're talking about the validities of the inputs to your process and any given step and and what the what the outputs look like. It can be process control where you're checking individual data points against Aggregates. And and the goal here is to just make sure that a process worked right you're looking to alert people if there's a failure or if they're invalid results production data testing is a little bit different. It's it's really it really allows you to focus on more the unknown unknowns. It's the the ongoing testing of source data and it helps you identify those unpredictable.

differences

So so what are some looking closer into the difference between processed QC or testing and

00:20:00

data QC process testing is has to be synchronous with your with your process. You want to stop the process if if it something fails and be able to go look into it.

it's it because it's part of your process you often have to be sensitive to Performance not always sometimes performance is less of an issue, but but often it is

You'll you know you even even sometimes if it's performance is not an issue. It might become an issue tomorrow. Right? No one's going to come to you and say we you know, we we love your data. Could you please refresh this less frequently, we've had situations where we've had weekly builds and and just recently a situation where people have come to us since and said, this is great. Could you make it daily we made it daily and then they said this is great. Could you make it hourly so we had to make it hourly. So so performance is is always an issue when you're doing testing even even if you feel like you've got some measure of time as we've said it it process QC Shadows application steps. It tracks, uh in line transactions and updates and it allows you to really to catch issues that you already know to look for right you're testing for specific air conditions that you're anticipating could happen.

Data QC on the other hand, it can be synchronous but it can also be asynchronous. It can also be testing that you perform outside of the process itself and that gives you the freedom to be, you know, a little more comprehensive you can do some additional testing that you might not want to. a burden the process With as opposed to shadowing the application steps it Shadows the data architecture and you can analyze aggregate results you work backwards to a cause and helps you to catch the unexpected and I'll get into the kinds of data testing that we do but in general we divided up into three kinds of testing we call it qualification screening where you're validating assumptions about your data business rule tests where you're comparing your data to predefined rules and consistency tests where you're comparing your data to Baseline Baseline data set.

So what does effective process QC look like in production? Well?

You'll have real-time tests. At checkpoints where where you verify steps in the process flow as we said one thing. I always say is don't be afraid to test the obvious. if you think that something, you know a should should be and and you know it you think it's just a there's no way it could be wrong. Well, you know, you're you what's gonna happen down. The line is people's assumptions may change. Someone may make changes to the logic another person might get involved or another group of people might get involved in in maintaining code and there could be significant changes to basic things and sometimes it's those tests of obvious.

Conditions that can really get you to to identify a problem.

So why do you really want to test the obvious? Why just the obvious? Be because sometimes as I said, right because it sometimes the the somebody can go in for example and and make some change any and it will fundamentally change obvious data there, you know, so something you think should be a give in, you know turns out to not be a given and that could be an indication of a larger problem.

Thank you.

So, you know, we're talking about testing every step. This is a typical production process at DataKitchen where you know, what can go wrong in this kind of process. We have we have data inputs that come from

Let's say SFTP or a data Lake other sources. We

00:25:00

we have file Watchers and automations that were freely data into a staging location. We do ingestion into the data warehouse. And we do analytical processing and then we fresh refresh external deliverables. You know, what are we testing for? Well, lots of things can go wrong. Right and this is just an example of all the different areas where you potentially can have problems and that kinds of testing that you want to implement. So, you know, you know, we've seen everything from formatting problems sudden changes and Upstream data to you know changes in our own processes that cost errors.

to areas where we have to verify external deliverables like Tableau reports and and the beauty is that all of this can be Automated and you can confirm one set of data versus another. Now what what can process QC Miss? Well, it's it's basically anything you're not explicitly looking for right? So you can have aggregate Trends or shifts in in data versus prior iterations of the data. You can have scheduled cadences we've seen that where where there may be unexpected delays and data from one source, if you're doing data integration and your data from one source doesn't match the data from another.

There can be you know in event driven systems or transactional systems similar kinds of issues where where multi you know, separate updates can affect data consistency and you can also have air conditions that that really require some slower testing. That's where we get into Data testing.

So what's effective data testing in a production environment? Well, we start out with preparation. First of all where we like to do some extensive data profiling. We identify Baseline characteristics of data and we do qualification screening to evaluate Source data for kinds of typical standard problems, then in production. We not only do we like to do automated testing but we really put an emphasis on parameterized testing. That is we have standard types of tests, you know, which which will you know at this point we have maybe 25 different standard types of tests, and we're able to run them by creating a parameters in a table.

And this means that the testing itself doesn't have to get reprogrammed, right the idea is that we know that that the test engine is going to work in a consistent way as long as the parameters are correct. We this allows us to do business rule testing by manually filling in the parameters. We we compile sets of business rules that we can test against the data. It allows us to do consistency testing by by Automatically deriving tests based on prior data.

And in addition we save some room for custom testing because there's some kinds of tests that that you just can't do by a limited number of predefined tasks. Then as as follow-up, the testing is not just performing these tests, but it's also reviewing the results and refining them. The goal is to highlight what's actually actionable information to separate the signal from the noise and to make changes to your criteria as the data legitimately changes.

Also by splitting things up this way. We're able to split work up among people with different responsibilities and different levels of expertise. Maybe a data Steward versus a data engineer versus an analyst. Yeah, because maybe somebody doesn't know the business right but they can profile the data and look at kind of the syntax of the data and build rules, but maybe someone who actually knows the details of the business more and can work on a different part.

Is that the idea exactly?

So what does qualification screening look like the the goal is to identify data characteristics based on the actual data based on profiling. So

00:30:00

so you you periodically review The Source data, we have our own. More extensive profiling tool that we built specifically for this purpose and we do things like validation that that dated dictionary is correct. We try and identify a requirements for data hygiene that we may have missed by manual review. And also this inherently creates documentation that can then be shared and and gone over by by different people and in both the on the business side and on the technical side, so as an example, you know confirming the data typing she might see numbers in Alpha columns.

A really common issue we face is blank value representation. Where where? You will have data coming in maybe from different sources or maybe even from the same Source where they'll be different designators for blank values. We might want the designator to be no, but we might see in some cases nulls we might see I I've seen everything from the word missing in all caps to TBD to be determined the word null as an alpha.

I mean we've we've kind of seen it all so so part of our screening looks for different ways of designating blank values. This can have a big impact on Downstream analysis, right if someone is trying to exclude no values or identify missing data, they may be looking for for looking for in one way and and it could be coded in in lots of different ways.

and we also you know have other kinds of tests as well. The business rule testing now, here's here's where we really try and leverage domain expertise. It's really this is the testing based on criteria that we get from the business or based on our own knowledge of what the data should be. And in this case too, right the rules become the documentation. We are ideally feeling in parameterized tests.

To to then verify the rules and the good thing about this is these are targeted kinds of tests, right? So these are these are are bound to be more signal. We know that this is that if there's a failure in business rule tests that that really is an issue and typically examples, you know, validating lists of values that are provided to us or enforcing any relationships between tables looking for duplicates that kind of thing.

Consistency testing is more open-ended, right? We have Baseline data. We have characteristics that we've gotten through our profiling and that allows us to automatically infer rules based on that prior data. Now, the rules may not always be correct because you're just looking at your past data. But it's a starting point and very often the rules are you know, we're our goal is to start out with inferred rules that are maybe 80% correct. And and then we continue to test and refine those rules. The idea here again is simple tests in some ways testing the obvious and we're deploying this net that kind of acts like a burglar alarm and sometimes if you see suddenly blank values in a column because when they were no blank values before for example, or or an increase in the in the percentage of blank values that might indicate that there's been a more radical shift maybe to similarly typed character columns were switched in your Source data, so you could have very complex causes to simple problems.

yeah, and this side that I can't emphasize how much this these tests have gotten me out of it because You know, I've given an analysis to a business customer and they look at it and like within a half a second they go that's wrong. And you know, I mean I took a lot of calculus in college. I thought it was smart and how did this you know guy who drank a lot of beer know this and I think Your business customers know their business and they they have metrics that they keep track of and sometimes dozens in their head. So if you're doing a sales and marketing thing they know their top customers or their top regions of their top products.

And a lot of times it may not be.

00:35:00

It may be small things that got it wrong small Excel files that group data or some minor code change that could have a big effect and so by by doing this kind of consistency testing sort of one example is historical balancing taking the top 50 metrics and comparing that, you know, you're you're top products your top prescribers your top regions your top customers and just doing a comparison from last time it is that kind of burglar alarm or that alert and another example, I often get is the Hubble telescope I join NASA in 1990 and like the Hubble telescope was blurry and everyone was embarrassed and and why was that because they just didn't look through the Hubble telescope when it was on the ground to make sure it was right and so just don't don't be sure that everything works trust and verify everything and even look at it Through The Eyes Of Your Business customer and I've found too and we're gonna talk about this later when you get things wrong.

Not another consistency test, you know, look at that's a great way to check the whole entirety of the system to make sure that it's right.

And and what you know going along with that one thing about these consistency tests is you really do curate these tests along the way right? You have to keep nurturing them because because valid data does change criteria and the more you review the results of tests and de-emphasize the ones that are less important or turn off the ones that are that are false positives the more you do that the better your test results become in the future and and the more Effective people are in finding what's truly actionable information? So it's so it's not just a one-time deal where you set up tests it. It's ideally an ongoing interactive process between defining your criteria and conducting the test and refining your criteria. Yeah. How do you optimize that signal to noise ratio? Like not having too many false positives?

You I think you you keep testing and it can happen pretty quickly. You're you start off with probably more tests that you need than you need and and you you know, you start to see how new versions of the data respond and you really learn over time kind of the the pulse of your data. There's the shape of your data and in our tools what we do is, you know, if you can make it as easy as possible for users who are evaluating test results to a mark and disable a test or or change test criteria on the Fly.

Then it becomes a much easier process to to shape that testing it parameterize testing helps that because criteria is is not built into code in that case. It's just built into a tables of tests and you can edit that criteria directly in the tables and you know it can Reflect that the next time you run the test.

So here's an example of consistency testing. This is an example that comes from real life. We have a column that came in data. It's coupled by and who knows what that means but the data dictionary type because a data Engineers are lazy was varcar 255 and the documentation for that column. It was documentation. There isn't always but there was documentation that said tracks completion status.

Well profiling told us some some valuable information about that column. I was turned out that that every entry was a validate. It was 80% filled in we had a minimum date that was fairly recent and a maximum date. That was very recent. And present in the data was at least one date per calendar week starting with the earliest d.

So what what we try and do is to identify a functional data type as opposed to Data dictionary data type or a documented data type. How does that data point function? In real data for gates, for example, we might have historical dates like birth dates where new dates come in out of

00:40:00

order. There's no chronological expectation. We might have a transactional game where the new data that comes in is the more recent data. We might have a schedule day which includes, you know events that have been like like Dockers appointments in the past or in the future or shipping dates, and we might have a date that's just in the future just you know, like a best before date or expiration date.

In this case, we would categorize that again. It's it's an automated process as a transactional game and that would mean that there are certain tests. That would be likely to be appropriate for that column. So we would say okay, the minimum gain is constant. It wouldn't change new data the new Estates that come in are going to be more recent.

We wouldn't expect shift in the percent missing for those dates the fraction of values that are that are missing and we would continue to see dates present every week. We might have in some we have other tests for dates present every month. If it's less frequently or even Gates present every day for for really frequent transactions, but these are some basic tests that we can just derive automatically based on the profiling data and bring us much further along than the information that was initially provided to us.

Some other kinds of examples, you know other kinds of testing, you know, uniqueness lists of values Alpha and decimal truncation. That decimal truncation listening sample where we're we created a test after finding out that you know, we started receiving a data point that had previously been reported as decimals as as integers only and that was a significant change. We had to go back and deal with that. Nobody caught that now we can automatically apply that test as part of our data testing.

and and other kinds of standard testing based on functional data type Finally, we have our custom testing and and this is the hard stuff right? This is the stuff that might be vertical domain, you know industry specific tests, but things that are not easy just to apply using our parameterized engine so so might be kind of weird things like, you know, these are these are some examples we had a case where you know, a customer never had shipments on Fridays because it took two days for their shipments to get to the air customers and so they didn't want to have shipments that would arrive over the weekend that would be a an example of something that would be maybe a single test in that we would typically do and we would we have Provisions for SQL testing just directly within our platform.

The idea here is to Target our programming to the only to the hard stuff, you know, we don't want to reinvent the wheel and

You know, they're possible we want to use pre-existing tasks, but it's not always possible. The problem with this kind of testing is like anything customized right? It's it's always harder to maintain and you you need to test your test you to verify that you're tests are working. There's just more work involved. So even in these cases we look for reuse. We look for opportunities for reuse and to it and to keep better to program in improving our test engine than to program individual tests where possible and and one example of this was an aggregate test that was always different. We kept applying we found ourselves comparing aggregations.

Values with group buyers kind of historical balances of one data set versus another based on, you know might be a you know, some of sales for particular categories you want to make sure that they were consistent or that they never decreased. And people were doing that manually where we build that into to a parameter driven test and we think it's a better test and the test itself is more reliable and more consistent and it's also a healthy question for you. What percent of all the tests do you think are custom Our our goal is less and less so so I would say what we try and shoot for is is maybe 10% custom test. It doesn't mean that there is less.

00:45:00

That you know that there's no work involved. There's no there's no human work involved in an automated test. But the human work is a little bit different. It's about refining the tests that exist and and make you know, making your test better rather than just programming tests based on a predefined set of values. Yeah. Yeah. And so this is It's a trip and I have a difference of opinion here. Like I come from a software background and sophomore Engineers always right there on tasks. There's there's very little tests that you can have a machine do and well, I think what chip is saying is that a big chunk of your tests. You can have automation help you make it easier and faster.

And and so that's what our team is is actually been working on here is to try to try to work through that but I think it's very hard to cross different Industries to have every test cover like, you know, we do with customers who do drug Discovery who do Financial Services who do Manufacturing.

You retail those are very very different domains and and there are things about the data that you can you know profile based data, but there are it's just very different types of tests. So I think you know no matter what your opinion is. You're always going to need some custom testing and that's part of your job.

That's that's definitely true. Right you're always going to need it. But you're you're goal is to diminish it if you can if you can accomplish something through. A pre-existing parameter driven test that's the better way to go you would you agree with that Chris? Yeah, actually right because the parameters themselves and we're going to talk about this in a little bit. I cannot can be edited by someone else right and and over time you can take that burden and get it off you and you know build the test framework, but the tweaking could be done by data stewards by business analysts, you know by by at because what you want to do is build leverage in the work that you do and allow it to be handed off. So I think parameter-driven everything is is the best and and tasks are just one one of the cases of where parameters really matter.

Sounds good.

So talking again about kind of the follow-up process. To this right there. There are lots of different ways. You can do follow up. It's it's important you you're working on refining your parameters. You can also do statistical process control. That you can't do necessarily with individual.

builds you can use Aggregates to do things like like look for drift and shift and identify, you know, a compare sets of outliers identify potential outliers things that you might not want to stop a build for or you might not have information within an individual build, but it would be through aggregate analysis that you can you can realize that there's a problem and follow up on it.

And and you can also measure your progress, right? This is really important transparency of how you're doing. If you even if you're measure is not perfect and this is an example of a of a measure just in just a little formula that you could use to you know, if you have if you have a particular test type that detects 10 issues and you know that 60% of those issues are real problem, but 40% or not you can you can

Make a guess at the number of real issues. You have compared to the total number of columns and rows and you can get a percentage accuracy function as a score now. That's it's not perfect because you clearly have some data points that are more important than others. You may have critical data elements, you could there you could run something like this against just the critical data elements. But if any way that you do it, it's a consistent score that you can use to compare, you know, one day against the next and that can actually bring you some value.

Um, what what we've done I've talked about what we've done and we're in the process of developing an automated testing solution, we built it for ourselves and and the goal is to Create a sustainable framework for for the

00:50:00

processing data QC that we've been talking about. What am I talking about? When I say sustainable framework? Well, it's something that people are gonna are really going to get benefit from and and continue to tend in feed because you can't have a static testing process. It has to be an ongoing thing where you're where you're continuing to review your results and and make changes as you're as your data changes as your process has changed. So it's got to be easy. It's got to be self-documenting it we want to be extensible. So you're not constantly having to hard code when you're dealing with changes and it has to lay out a plan of action. What do you do if things go wrong? Can you you we're not just reporting. We're actually acting and there might be certain kinds of problems that are less or more serious and you want to flag.

tests as being you know showstoppers versus just warnings and you want to alert be appropriate people. So that's that's a critical part of this whole thing if it's easy to go through those the those steps of dealing with errors, then you're testing system helps you even more

So yeah and and chips taking the lead on it and one of the things that we believe as a profitable company is that we have part of our company that's always done data work for customers and it's actually really good because it puts reality into the software building process. And so we've we've demonstrated this beta product at a couple of customers they're using it and now we're actually going to release it as a product this year probably around Q2 or Q3. And so this sort of automated testing a module we're very excited about mainly because I've gotten this question I think, you know running a company I've learned to be wrong and one of the things I was wrong about is is not Everyone likes the right tests and and having a solution where we can accelerate how people actually test their data and make it easy for them to do it and and be able to provide a framework to support them I think is going to be a big help.

and and I would add that the framework is not just For testing it's it's for what happens after the testing is done. And that's why we want to you know, our goal is to connect up our testing system with our observability platform because it's you can have the greatest testing scheme in the world. But if all your tests just go into some logs somewhere and they're not being monitored and your team's performance isn't being measured and and the people are not getting feedback about Source data, if if those things aren't happening if the appropriate alerts don't go to the appropriate people. It's really not that useful.

So that so so that's my piece Chris. Oh, all right. You want to make me the actually let's see. I guess I can would you make me the presenter? Yeah, let me do it.

All right.

So let's just we've got we're going to bit long. So hang hang with us. We've got maybe five or ten more minutes to slides and then actually I've been looking we got some great questions to answer. So let's talk about people and so who creates tests and who cares if tests fail So the first thing is, let's look at who creates tests.

so

You know, I think everyone can create tests right data Engineers data scientists bimless data stewards. Operations and they do different things right data Engineers are about putting together data sets. Well, they can test the raw data and test the process data scientists should have checks for their models how predictive accuracy they are. Bi in English should be able to if they're doing some data prep.

They should help test that data prep and data stewards who actually know the data could configure parameters of the tests. And finally the operation people should actually test if things are running on time. And so everyone can create tasks but that doesn't say you can't have Specialists. Right? We've done webinars about DataOps Engineers. There's been QA automation testers. That doesn't mean that those roles don't

00:55:00

have to exist. In fact, they're better in some ways to have someone who kind of sits across the process and helps because a lot of times it takes a very different part of your brain to create something and then the pull it off into trying to find cases where it breaks and automate those and sometimes that is you know, there are more Engineers who are very used to and I think I'm one of those we like to create the sunny day case but like don't want to go down every quarter where things could go wrong and having specialist rails like DataOps engineers and QA automation testers can really help.

And so what happens when things go wrong? Well, I think you need to tell people right? And so the first line of defense is always your operations team, right who's ever running it and maybe you're data team is a build a build it and run a team which is great. But if it's not then you've got a routing problem, right who should it go to because if there's a problem with the data, it should go to the data Engineers or the model or the dashboard. And so and even you should tell your customers if things are going to be late or wrong and so really it's becomes partially a routing problem of who wants to know the errors because a lot of our Data Systems have Um timing, you know, they're expected as chipset every day every hour every minute when things are late or waiting people get upset. And and of course, we don't want to have oftentimes the people who get gets upset as a has a Veer a theme in front of their title, and that cannot be fun.

And then when you find errors in a process in production, make sure that you find them early. Um, in fact some rules of thumb is is often when the data lands to start your process check. It don't wait until you know, if you've got a bill that happens every day and the data arrives in the morning and that builds in the in the late evening. We'll check your data right away. What and why because it gives you time to fix things before your your customers patch the data find out if it's a real problem Etc and so find errors early and often and set alerts early in the process and try to de-risk what's happening. So again, this is this is where the hope is your enemy strategy hoping that things were hoping that the data is right, you know, I love hope everyone loves loves hope but hope is your enemy in in data production and hope is your enemy and dating analytics development.

Likewise with data things are not just wrong. I mean, sometimes they're just wrong, right the data didn't show up that's wrong. Well don't you know don't run it or backfill it. Um, and sometimes data is just weird. It's sort of wrong like well, we usually got a million rows. We got 800,000. Is that wrong? Sometimes you don't know sometimes you have to actually talk to your stewards or your business customers to find out if something is is strange but no about the warnings. And so I think having multiple not just having it's good or bad having the thumbs up which is important but these gray areas is important and and different groups have done different things with gray areas.

You know, some people have said okay just investigate the warnings other people say turn all the warnings into a parameterize the warning so they actually fail and oftentimes the warnings can be useful for sort of a heads up to your business customer. So think of failure modes as as a gradient not as just a binary.

We're Chris about that sometimes data can be right and it can still need to be investigated. So you may have a change in data. You might have a three categories and in a particular column three codes and then suddenly a fourth coat starts to appear. And I've seen that break Downstream processes and analyzes, even though it's totally valid Upstream. The word hasn't gotten to the analysts and there might be a you know, statistical processes that are that break because of it. Yeah, and you know what people love you to tell you when you know things before they do, you know, basically, I think the mental model of our customers is they think of us as plumbers. It should just work and you know, if the plumbing is gonna break you can talk to them ahead of time and saying look we're something's weird here, you know, the plumbing is going to break and they thank you for it.

And so in general data analytic teams customers of us think that we do magic and want us everything in data analytics to work like a faucet and and they're very happy if they we give them a warning saying the water looks funny. And so I think all those things are really true and and I think that's part of the partnership that can be established. And and one of the reasons that we want to do that is to get out of the way of like the embarrassment that team has because they haven't caught anything right because a lot of times the psychological context is

01:00:00

your customer thinks that you're an idiot because you're not catching the obvious things that they can catch in two seconds. They're not doing those kind of consistency testing. They're not trying to investigate that gray area. And so a lot of again, the stress can be alleviated by doing the work right doing the work of testing and automating and that's our argument that we've been making for years to the the data industry. And I think hopefully we've we've got that to you we've we and I think more people are believing it.

And and so the second to last thing is how do we categorize tests? And actually, I hope that we would discover a Dewey Decimal System for tasks or some way to categorize all tests and Chip and I failed and there's just a lot of category confusion at a talk about tests, right and and You know, I think there's a kind of starting in the lower left hand. They're sort of the demo dimensions of data quality which are accuracy consistency validity completeness. And you know, I think of those as like about a sort of a kind of the the balance sheet of it sort of data at rest. And what is characteristics are and then there's other tasks that are kind of going up from the the bottom right? There's categories that come from software development like in a development process unit tests functional test and end desk performance tasks and and people categorize their tests with that and they say, oh I wrote some unit tests.

And then there's there's terms that we have used in DataKitchen location balance statistical process control just sort of variation over time and then the nomenclature that tip it chip has been talking about today. And another way going on. The left hand side here is kind of categorizing by what you know and how you can create tests. Right?

So if you know one thing should run you can write tests on that if you know data structure and syntax if you don't know much about the business, but you understand data. Well, you can you can correct trying to create some tasks but there's other things that if you actually know the business you can help.

And so I think and lastly there's just thinking of it as locations, right? You could do it on the raw data or on the integrated data or in the tool modified data. And so we failed utterly to give you a good nomenclature. And so I think the industry is kind of utterly confused. And so, you know, I think automate tasks and you know the kind of sort of wisdom that we've Given in this sort of test everywhere test often test and development test in production. And you know, you can apply your own language this and maybe someone smarter than us. What would develop the good the condominator here, but we fail so we're sorry for tests that are gonna get me fired disciplined or that No One's Gonna know about yeah. Yeah and just write some tests. It doesn't matter what they're called right some tests so you don't look bad and you can deploy faster.

And that how can I organize my team? So um You know, we talked about everyone doing testing but we didn't talk about I think the most important thing is how much time you should spend on it. And so, um, you know, spend five to ten or fifteen percent of your time on testing and and why that seems like an insane amount of time but for some people and if you actually look at software development, um as a comparison and you know, a lot of teams spend a quarter of their time on what are called DevOps which includes testing and Automation and a whole bunch a number of other things, but you should work on your process and another psychology of data analytic teams is we're all very busy and we're constantly pulling out of our next cast to do and it's hard to talk to your customer and say well look I need to put in some automated tests. So this won't happen again and so by doing that you're actually enabling yourself to go faster. So advocating time to develop automating testing making sure that it happens I think is important.

Just as kind of sum up kind of our key points on testing is is everybody should write tests, but there can be Specialists. When should the test be written. I've been kind of before referring it after it's you know, there's different philosophies on test driven development and I I think a lot of times in my experience we've Put data together. We with very minimal tasks exposed to the customer to get feedback then iterated both on what the data is like and then on the tests that are developed and and then even going you could have systems that run very well for a year with zero errors and then suddenly some condition happens that you that you didn't expect and you need to add another task. So tests should always be part of your discussion with your customers and then I should ship said parameterizing tests are important mainly because you can adjust these tests sort of after their live and 10% of your work should be testing manual tests

01:05:00

are bad. Maybe that's not that's a judgmental word manual tests are good at first, but they should be automated. And the demon dimensions of data quality are really I think more about raw data than actually about this sort of process-oriented testing. And I've heard a couple of people say this in development unit tests on fake data. Um, I've got testing well, I think unit tests and fake data are not enough for rent progressions. They're helpful and and in software unit tests are meant to be a quick run through before you check in your software and that's not bad, but they're not the end all be all.

And then um, you know, I think the other point is sort of love your errors here. We talked a lot about errors and so love them. And so when things go wrong get excited because then you can add a task that that make sure it never happens again. And so what's our conclusion? So our first conclusion is that Thing in automation drives less errors increases the velocity at which you can deploy and makes your life just better right you can do more work. It's more fun. You don't feel like such a schmuck from your data customers and and Life's good and so doing this work is really important. And of course, we've got some products to help right? We're a software company. So we have an automation product that can Orchestra help with orchestration that can help with deployment and environment management. And then we've got an observability product that can help with watching the end and journey and then of course testing throughout and chips talked about our new our new module to do automatic test generation that's coming up this year. And you know, we've got a great architecture that can run on Prem or in the cloud and so we've, you know, would love to have you come and look at our product.

and so lastly, we've also done a lot of work on educating so we've got Two books that we've written. We also have a training course on the ideas of DataOps and then we've got a whole bunch of stuff that we've written about our new observability product and testing. And so I want to thank everyone for going along today and I just want to take for the remaining chip. I want to go through some of the questions here that I saw and maybe we can answer a few of them and whether is a lot of questions, so, okay.

And so let's talk about this one. There's a question here chip about data contracts data contracts seem to be all the rage, but are they not just tests that exists at the interface of teams. How do you view data contracts? Do you have any thoughts on that chip? I would say that they can be very valuable. It's in some ways things are all the rage can sometimes be the similar things in another, you know, another terminology.

Um, I I one thing that

we've found about proactive attempts to do this as has been that they just don't keep up with with reality and it doesn't

protect you from having to perform tests as part of your processes and after the Fact you you still you will see changes that come no matter what is in the data contract and and you will have to adapt to them and you'll be responsible for dealing with them. Yeah, that's a great question. We just actually wrote A Blog on it and I think you know, I think tests are the terms and conditions of your data contract.

To be sure right if you look at I've Had The Misfortune of looking at a lot of contracts and like contracts have just terms that legally use term term term you got to do this and when things go wrong this this has happened and things go wrong. And so and so they're called terms and conditions and I think the terms and conditions of a contract are tests. And so to me testing and data contracts are the our I don't know how you can have a data contract without tests. It just becomes another Useless document that you get written once and never updated and so to me active the more you're coding the more you're automating the more you're testing the better you're off.

and so and so here's another question. What do you guys think about AI best AI best-based testing like what anomalo is doing the chip? You want to answer that? I also have an answer to I I think it's fantastic. I I think anything else you can bring to bear to to automatically derive. Test results is good. I think we're learning about how to apply these tools and and you know, by all means it's something to experiment with.

Yeah, I guess you know our our test engine has a bit of AI I think specifically there

01:10:00

are domains. We're really good time series. Testing is important. Like I think there's a lot of real-time trading on Wall Street where you're looking at sort of Trends over time of a particular stock or a particular investment vehicle and sort of really good anomaly time series anomaly-based detection like an envelope does or custom there's other customers time series databases. There's tools that can do it and I think you know in my experience That's not needed in most cases for most people are doing data analytics but in specialized domains it does and then I guess from my standpoint having sort of studied AI.

A while ago and a fan of the technology. I just be cautious. I mean, we you know, there's a bunch of hype on AI and that we've seen and and you know, I think a lot of people sell AI as magic and then it just becomes you either become it's parameter tweaking or it's wrong and so I think I'm more of an automation fan try to automate first unless you're domain Demands a very sophisticated sort of Time series solution.

Here's a question. So about data profiling how can you handle sort of privacy gdpr auditing requirements with data profiling? And what that means is like how can you profile data when perhaps you're not supposed to be looking at?

Well, so how can you handle that? So what what we would do in cases like that is to Shield that data from our profiling tools. So we might profile a view instead of a table. We would the idea would be to have you know, only profile what was accessible. To you know, what should be accessible to the profiling.

Okay. Yeah, I think you know it's hard depends on who has access to your development data. Right? And so if you're gonna have test data, um, perhaps you've cleaned it and stripped it of of gdpr some development teams have access to it some don't and so you could either it dependent. I think it depends on the access of your development team to your your gdpr level data because you can always sort of clean of that data and still do profiling.

um And so another question you have to we have two consumers who have two different universes of freshness in use cases. So one set of testing covers the first consumer who paid for the quality effort, the second consumer has a larger University runs their own test manually.

And so have you have you ever observed that or like there's a universe of data with some tests? And then there's a bigger units person data. That doesn't have any test. I think that's a way to paraphrase that. Have you seen that chip? Um, so

I I don't know exactly. I don't I can't tell exactly from the question. Just what the different use cases would be of the two different data sets. I mean, I would say that we always you know, we're striving to have more accurate data period and and

so it were we always want to go. Above and beyond and do as much as we can to make sure that the data is accurate or not, you know within the time constraints that we have. But there might be testing for example with some of our consistency testing that we do asynchronously or a different points. Where where we're Not necessarily applying it to the data as as it gets released, but we're we're monitoring it over time.

Yeah, I think.

You know for my standpoint like I think of data as a Continuum, right and if you have a small data set or a medium data set or a large data set you may want to have a small data set that you use in development. Right? And and again if you're seeing tasks as something that you can use in development and production, that's a good reason the parameterize your tests. So have those per have a set of parameters for small have a parameters for medium who paid for it in a set of parameters for large and try to you know, keep the tests invariant of data size and enduring of environment production or development and use parameters to turn things on and off or adjust to the size. And I think that's a good technique to help you manage that sort of that sort of challenge.

and

so here's a question. My team is doing a lot of testing yet. When when delivering some information is usually the wrong the first time I get it delivered.

01:15:00

So we'll do a comment on that chip. So is that when when delivering information to the team? It's usually wrong. I'm not sure.

It's probably both right you get the information wrong the first time and then when you give it to your customer, you get it wrong. Yeah, if it's if it's wrong upstream and it's gonna be run Downstream, that's well. I mean that you know that that's our goal for a lot of this testing right? We're looking to test we test Source data and we test process data.

The our qualification testing is is a series of standard tests that that reviews basic common things that goes that can go wrong with Source data, but by by making our testing process a separate service What we're able to do is to plug in Suites of tests at different points in the process so we can be testing data as early in the process as we want and and much like some of the things that Chris was talking about earlier Chris made the point that that you you want to give yourself time to follow up on a problem before maybe that data is is putting to Downstream use right if we if we receive data in the morning.

We want to ingest it in the morning and do as much as we can even if that data including testing even if that data is not going to be used until later in the day because we can have the time if there are problems to go back to the source and follow up and and try and deal with errors before we need it. So the more time that we have the better off we are.

Yeah. Yeah and I think on the other side like giving giving error for air giving data with errors to your customer like that, you know, a lot of people say no I can't um, it's imperfect I got to be perfect or they're gonna get down my throat. And so I I you know, I think there are some organizations like that that they're you know, there's sort of Shame and blame organizations.

I think the challenge with that is is it's oftentimes those customers also want to have new insights and new data very quickly. And so trying to go from a shame and blame organization to an organization where you can say, look here's the new data. I've done a few days of work to integrate it. It's probably not it's it's correct as I can get in a few days. What do you think is this sort of directionally correct on what you want and that term of directionally, correct? And it's not perfect but it's directionally, correct. It's helpful with customers who are tend to shame you because oftentimes if you do that you learn more about what Customer wants and then you can further improve both the quality of and the type of work that you're doing the data you're integrating at how you're integrating and adding more tasks. And also the same thing too is saying some customers are very day driven and they say gotta get it done and then, you know, you build up technical debt and and talking them about how to actually put in more Automation and testing is the benefit is as a benefit. And so this is becomes part of the discussion and there's a question here about sort of testing and Agility and projects and you know to me agility and projects is is comes down to like what we talked to pull before before what's definition have done but also comes down to the discussion between you and your customers and you know, you're not only building features for them. You're building a system around those data sets and and data features in order that you can work quickly with low errors, and so reducing techn That refactoring adding more tests adding more automation should always be in the discussion with your team because it enables leverage and sometimes you've got to work on the car, but oftentimes you've got to work on the factory.

And and that can enable you to produce more cars. And so I think these discussions are harder and organizations where shame and blame chronominates, but but they can I've it takes time but it can it can work and that trust between your customers and your team. It's a beautiful thing when it happens because you're able to move very quickly get feedback in iterate and I've seen and as a result when you have that I've seen very very small teams.

Five people do the work of literally 50 people and other organizations when they've been able to work in this quick and iterative way and it gotten more work done and it is a better quality and has less errors which is which is amazing and that's really why one of the reasons why we started the company and why this sort of agile approach automated approach testing approach to data analytics is very important.

And so I think we've we've got most of the questions here 50. You see any ones that I missed in the list here. I I don't have the questions. You know the questions. Oh, okay.

01:20:00

I think there's one or two questions here that are that I think are a little bit more.

Okay, here's one from about business rule testing this the last one I'll answer or we'll answer about business rule testing. That is so challenging and most times business rules. Don't match with logic. How could you explain it to a marketing guy? Yeah, that that's that can absolutely be true. And and we're you know again we're dealing, you know, part of part of the testing process is Reflecting back to our users what the data really is, you know, so something me not, you know you the real data may not match business rule criteria and the you know, sometimes we're just the deliverers of the bad news.

We have to let them know and and that you know, we do build trust that way. I think the goal is for people to go back and some the solution sometimes is is Upstream of us. There are other kinds of solutions, but that's good. We're doing our job in that case and and if we're you know, the what what good testing does and transparency is to build trust by making these things apparent to everyone and we're all working together to make the best data that we can Yeah. Yeah, and I think what's should be saying a lot of the role of our data and analytic teams is to in the a DataOps in general is to take things that are hidden and bring them to light and a lot of times, you know the worst thing and I've had those experience early in my career. You've got data that's in your system that's been wrong for months and no one actually noticed it and just writing an email to A Thousand several thousand people saying, yep that date has been wrong for the last you know, that's a painful email that I've had to write in my career. And so I think you know, the the challenge here is is being able to invest the time in your system to make sure that it doesn't happen and have that discussion and talk about the risk and time trade-offs and I think in good functioning teams, you know, when you get crappy data from your customers, you know, you're going to either have to patch it.

Use last week's data or go back to them saying hey, you know you nut head it the data is broke, but in all cases, you know, if you do it in the right in the right way, they're happy. I often seen data providers be very happy saying. Oh, wow, you found out that there's a problem. You know, I have other customers who are getting this you just really help me out and and I've had cases where the data providers saying.

Hey, can I run your rules myself to make sure that they're right. Think everyone is everyone benefits when we know about problems, and we don't hope that things are right, and I think that's on that note. I think we're gonna end and again, I hope you found it valuable. There's a number of questions about the recording and the slides.

We will send an email with both the slides and they're recording to everyone here. And again, we want a bit long, but but thank you for spending your time. And I hope you all found this valuable and Chip. Thank you so much. Thank you, Chris. All right. Bye everybody. Have a great rest of your day.

Thanks everybody.