How DataKitchen DataOps Automation Delivers End-to-End Data Observability

00:00:00

Welcome, everyone. Thanks for joining us today. My name's Beth Pfefferle, I'm the VP of marketing at DataKitchen, and I'll be the host today for the webinar. So today we have Chris Bergh, who's the founder, head chef, and CEO at DataKitchen, and he'll be talking about how to achieve data observability with the DataKitchen DataOps platform.

So for those of you who are new to our webinars and don't know Chris, he's a leader of the DataOps movement. He has more than 25 years of research, software engineering, data analytics, and executive management experience. He's been a COO, CTO, VP, and director of engineering, and he's also the co-author of "The DataOps Cookbook" and "The DataOps Manifesto." So before I hand it over to Chris, just a few quick housekeeping items.

If you have questions, please enter them in the question box on the webinar control panel during the course of the webinar, and we'll collect all of those and answer them at the end during the Q&A session. Also, the webinar is being recorded, so we will send the link to the recording to all attendees at the conclusion of the webinar.

So, with that, I'll hand it over to Chris. Okay. Thanks, Beth. So I'm going to turn off my webcam, and so hopefully you can see my screen. So this topic of data observability is very near and dear to my heart, and it's really about loving your errors. And so when we use this term data observability, sort of what is it that we mean? Well, think of it kind of like a crystal ball.

It's sort of how to observe errors in your data and analytic systems before they happen, or more specifically, before they get out in public. And the two things-- We're going to have four parts of the agenda, and the first part is sort of what prodded me to do this and my co-founders to do this, and what are the principles that we've learned along the way, because we've been doing it for a while. And what does our product do and what kind of proof does that... And so for the next 45 minutes, we're going to talk about it.

So the first one is a bit fun, and I'm going to talk about why it's sort of near and dear to my heart. And so, my career sort of spans 30 years. In the first 15, I wrote a lot of software for companies like MIT and NASA and Microsoft and startups, and did a bunch of roles from individual contributor to running teams. And then in 2005, I got the bright idea that I go do this data analytics stuff, and I thought, "Oh, it'd be kind of easy." My kids were small, no problem. And so I had teams of data engineers and data scientists and people doing database working for me, and my life really sucked.

And so here's some stories from that life of 2005 to 2008, and maybe you can relate. So, the first story is my kids were in grade school, and I go to sit in soccer games, and then my BlackBerry at the time, I think, would beep off and go, or I'd get an email and go, "Uh-oh, something broke in this week's build.

I got to go figure out how to fix it." And I'd explain to my wife and sort of walk out with between my tail between my legs, "Yeah, honey, I know it's the second time I'm leaving the soccer game. I know I'm supposed to watch the kids with you, but I got to go." And so I don't know if any of you can relate to that, having things breaking and walking away.

A second story is also near and dear to my heart is, I worked for this company that did analytics for healthcare, and we had a lot of sales reps on, and one team had over 5,000 people. It's one of the largest pharma companies in the world. And we were integrating data and batch-producing analytics, and at this time, they really wanted batch-templated Excel exports. And you know what? We did them all.

They were all delivered right, but guess what? The data was wrong. And so I had to explain to the person who was paying us to do this that, yep, it was wrong, and yep, it's our fault, and yep, we don't know why it happened. And so it's such a headache to have those calls and have the guy who sort of very nicely threatened to kick us out and...

So that's just not fun, right? And especially salespeople, because they're so extroverted, right? And he got calls, I got calls, it was just not a good week. The third case is, I worked with a bunch of data engineers, right? And so I'd sit down and talk with them and he'd say, "Yeah, I really wanted to get this thing in for my customer.

So I made this last-minute change in the build. I didn't know if it really worked. And you know what? It's okay, because it's been live for a week and nobody's called me." And so he had no idea if it was right, but he was happy because no one called him. Here's a third case. And this is from, I would actually drive into work, and I had a BlackBerry at the time, so I probably wouldn't check my email.

Or even on the long drive, I'd just get this pit in my stomach like, "Oh, no, something's going to go wrong. I'm going to get a call. My day's going to start off crappy." And I'd sort of scan for that email saying, "The data is wrong again, and you suck." And I'd just sort of brace myself. And it's

00:05:00

not fun to have that situation, honestly, because I would have to go off and find the problem, get people together, make some phone calls, talk to people, try to smooth it over, and it's not a fun way to work. And we had hundreds of data providers, right, of different places. Some of them were from the company, some of them were syndicated, some of them were small files, big files. And we had a situation where the data provider kind of changed the format of their data feed, right?

They ended up transposing some columns, changed the meaning, changed some of the semantics, and guess what? We didn't notice. And it was like for three months, we didn't notice. And so all those dashboards, all the reports that we produced were wrong in some respects. And so that's not great. Again, you have to make phone calls and try to not get your whole team fired.

And then, another case. So, as I said, I took this when my kids were small, so I was 42. And on my birthday, I had one of our ETL engineers come in and talk to me, and he was just really upset because he felt like he's a really smart guy, really worked hard, just felt like he couldn't do anything right, and things were breaking left and right. And so Finally, after talking to my wife for a bunch of years, she said, "I just can't take this anymore. Either just quit the stupid job or fix the problem." And, I've been married now for almost 30 years, and you know how your spouse can be sometimes, she'll tell you the absolute truth.

And so the story here is, how did I fix the problem, or how did we fix the problem, my co-founders and I? And so the first thing is that we've been actually fixing this problem, we're in our third generation of solving this problem. We built two previous systems, one called Pup and one called Boom!

in G Pharma, which are terrible names, honestly, but, we're engineers. And then in 2014, we really started DataKitchen kind of intent on solving this data error problem, this observability problem, but as well as other problems about the cycle time, collaboration, and process measurement, and all that we've put into the idea of DataOps. And so what did we learn, and what sort of things did we pour into our product? And I'm going to go through that in a bunch of slides.

And so here's kind of the summary. So, I think with pride, I can say since 2018, working for me and our founding team have delivered hundreds of millions of data sets and dashboards and models with almost no errors. And that's kind of incredible, right? Very quiet. Not to say we haven't had errors, but almost no, very rarely.

And I guess the first thing is just avoid manual tests. Like number one. And then number two, and we're going to talk about this, is you run a factory and that factory takes data, integrates it, produces it, models it, and produces things in the form of data sets and dashboards and other tools. And have a factory that produces Toyota Corollas and not AMC Pacers. And this guy who did factory work, W.

Edwards Deming, was right, don't blame people. It's often the process that the people work in, and you as a leader own that process. And errors, and we're going to talk about what that means, are just a huge time sink. And then, don't wear the sort of hair shirt as hard work and overtime and suffering like a badge of honor. It's actually not a badge of honor, and yeah, we all have to sometimes work nights and weekends when it happens, but we're having a bunch of people who are burnt out.

And the main idea, I'm going to say this over and over again, is make sure the data, the transformations of the data, the artifacts are created from the data are all right before your customer sees them. It's not about data quality, it's about low rate of errors. It's really about an end-to-end system. Make sure you don't put a bag on the side, and 90% of your tests should run in production and in development.

So let me have some slides here to talk about these ideas. There's a bunch here. So the first is it's not about data quality, it's about errors. And so if you look at your error rates, right, data quality is a component. And so think of data quality as like your balance sheet and your error rates as your cash flow. And a balance sheet in financial is like a point-in-time measurement, whereas cash flow looks at it.

And it's not that data quality is not important. Having poor data quality, sort of garbage in, is garbage out, but other things could happen. You could be late. Some code or configuration that you've checked into the system worked. Your server could have a problem. And so all these things are kind of in the idea of error rates, and so focus on error rates and not data quality. And that's sort of why we talk about data observability as a component. It's a set of technical practices and cultural norms and architecture patterns that enable low error rates.

Just like DataOps has that, plus more.

So one of the realizations I made is that errors in all this form just take up too much time,

00:10:00

for a lot of reasons, right? Because there's a lot of complexity in your organizations, in people, in your tool chains, in your data. There's a lot of collaborations when things go wrong, right? You have your best people, you have meetings, you've got to figure out where it is. There's finger-pointing. People get tired. If you've got a good team, they're going to sort of work really hard to fix it, and it ends up taking just too much time.

And I think that's one of the challenges here, is we're spending too much time with the hair shirt of I can fix errors very fast instead of focusing on the good things, which are getting features done. And so by focusing on errors, you can actually reduce the amount of time your team spends on errors, which actually gives them more time to do more cool stuff.

So the second part here is you run a factory, and so you want a factory that produces Toyotas and not AMC Pacers. And so think of it in this way, that data comes in on one side, and we call this the value pipeline. And there's a set of workstations in that factory. Maybe they're accessing code, or maybe you're doing some transformation, or maybe you're doing some applying a data science model to it.

Maybe you're visualizing it, maybe you're updating, maybe you have a data catalog. But all of these things are adding parts to it. They're putting the data together, and each one of these workstations, my goodness, there's like 50 tools in each category, for databases and servers, for data pipelines and data science and data visualization and data catalog. And you could span this out.

There's subcomponents of this. There's two types of tools to do data prep. There's self-service data prep, and there's ELT and ETL. There's lots of ways you could do this, but think of it as a factory, right? And in running a factory, you want to make sure that everything is working as intended. So test and monitor every step of the pipeline because, like in a factory, the earlier you find the problem, the less rework you're going to have to do. And that's a lot of what the problem of error is, is you've got to rework. So I found if I accessed the data and the data was wrong and I found that right away, you may have a chance to go back to your supplier and fix it before you have to ship it to your customers.

If you wait until it actually gets to the customer's hand, you have all that rework time that goes into it. So the earlier you find out the errors, the better you are. But also because you're manufacturing, you want to check all the time because errors could not just be from the source data, they could be from something that happened during the processing of the data.

And so you want to check, is your data right? Is your business logics all right? Are your outputs still consistent? And then if something's wrong, get alerts, because you want to know about it sooner rather than later. And in production, time matters, because we're all under time deadlines, and sometimes you have the option of patching the data or using the last data, or talking to the supplier or delaying your release, but getting alerts. And so think of you run a factory, test every step of production. The workstations on the factory are the $100 million market of analytic tools, and you want to be able to test across that and not have to replace all your tools.

And so another idea from the factory world is statistical process control. And that is, over time, the production work that you do is actually an interesting source of data. And so for instance, there's a thing called a Shewhart chart, where it looks at the upper and lower bounds. And so if every week you get a million rows, and the next week you get another million rows, the next week you get another million rows, and then the next week you get 50 rows, yeah, something might be wrong.

Something might not be wrong, but you should know about it before your customers do. And look for trend breaks. And so looking at that statistical history in data size, shape, pattern, summarizations, I think is really an important concept because, again, you're trying to know about things before your customers do, and you provide value to your customers by finding this information out.

And maybe it's wrong and you can't go on, or maybe it's just weird and you've done your customer a favor, whether it's a business dashboard or an end customer, because you've noticed it sooner and can release note it or at least give them a heads up.

And so the other part is testing and monitoring is kind of not a bag on the side. And there's this derogatory engineer term saying, "Ah, it's just a bag on the side of the thing we built," meaning it's not really part of the system. And I think that's true, and as you go through your production process, your pipeline process of building things, and for us, when we think of the factory, we don't just think of the data factory that does ELT, produces schema. We also think of models and visualizations. And everything needs to be tested because at every chance along the way, something could be wrong.

And so think of it this way, think of an analytic system, and you're going to decide where to put the if then else clause that groups, for instance, let's say, your major customers into three buckets. And so those three buckets, that if then else clause, it could go

00:15:00

in the actual part that accesses the data. It could be an attribute of a dimension of your fact table. It could be put in because you've derived it from data in a data science model, or you could configure it in your visualization tool as a calculated field. All those are, from a computer science standpoint, if then else's. All those are code that should be tested, and in some ways, your data may flow in and break that if then else.

And so you want to know, so that means you need should test everywhere, because all the tools in our data and analytic toolbox are in some ways code development tools, and they can build the equivalent of if then else's and complex logic. So that's why you need to test it. And so don't put it as a bag on the side.

Integrate it as part of your system. And then find it early and send alerts. And I think alerts are really interesting because sometimes they could be a ticket, sometimes they could be a Slack message, sometimes they could be an email. And so you want to be able to get them to the right person.

You want them to be at the right time. So alerting's really important. Also providing information in the alert where the problem went wrong. And so it's not like, "Hey, it's broke. Can you go dig into it and find it?" I think that idea of catching the error where it happened and giving them, "Okay, it stopped.

Something is wrong at this point," and that helps you a whole lot, and I'll give you a customer example about why knowing exactly where the problem is matters.

And then another way of thinking it is checking internal consistency. And so building data and analytics systems is kind of like following the bouncing ball. Data comes in one part. You access it, you transform it, you model it, you visualize it, you report it. And so you could see data come in and it's on an SFTP server, it ends up in a bucket store in your cloud.

It ends up being in a sort of a raw area in your database. It ends up being in a processed area in your database, and sometimes those are three. It ends up being then cached in an extract for your report server. And so you've got that same data set kind of in now what? Four places.

And so how do you know things haven't been lossy along the way? That that million rows that you pulled off the SFTP server ends up in a million rows or equivalent in the Tableau report. And so you need to check the consistency of it. Now, is this an error that happens all the time?

No, but it certainly has happened to me, and it's one of the toughest errors to find. And so being able to do this test that we call a location balance test, I think is really important to be able to get at it. And then another type of test that we support is kind of this idea of a historical balance test. And that's the case where you want to be able to go in and check the data with the lens of your customer.

And many times, or sometimes, I've put analytics in front of a customer and just found it was wrong. And that's not a great feeling that you do some work and they look at it in two seconds, they find it's wrong. And a lot of times there's very small data files that actually make things wrong.

Like here's a case of a product grouping. And you can see here that the product grouping in production sort of have G1 and G2, each of them has three. But actually the new one, G1, has got four in and G2's got two in. So something changed. It's a very small file. Maybe that's an Excel file or maybe you've got a good hierarchical data management system, but that's a small file that changed, and it has big effects that are very public. And so one way to get this is to look at top products, top regions, top customers, the top end, and just compare between what you last showed to your customer and what you currently showed to your customer, and just look for big variations.

And sort of look at the world through their lens. And because business people Have 80/20 rules. 20% of the customers deliver 80% of the value. Know who those are, monitor those and check against them. And again, the whole idea is you're trying to build tests that reflect the business view, or the domain view of your customer if they are in business and they happen to be in drug discovery.

And then lastly, and this is something that we support, and it's also my belief is that I don't particularly want to have different people use different tools. Let me rephrase that. Different people love their tools, and I've gotten in discussions of R versus Python or SQL versus doing it in a visual UI. And so, I like to write SQL.

Some people like to write Python, some people like to use their favorite GUI tool. And if you're going to interact with data, use those tools. And so build the automated test in the tool of your choice. And I think that's a really important trend here because a lot of times it's hard to get your data engineers or data scientists to learn another tool.

And if they can do their data work in the tool they're used to doing the data work

00:20:00

and fix it in some framework like we're showing here, I think that's a good compromise and also helps adopt the idea of testing. And then lastly, we actually have a bunch of reports and sort of looking at errors in your system from a historical trends or longitudinal trends, and then breaking it out by data providers.

And a lot of times you may have dozens of data providers and there's one or two who you kind of intuitively think are noisy, and they cause a lot of problems, but you don't have any facts to back it up. So just track how many errors they've produced. Track how much time it's cost you, and here's a report called a tornado report that shows that, and it gives you leverage to talk to them.

Because you run a factory, right? And you're dependent upon suppliers. Your suppliers are the data providers for you, and maybe it's your internal ERP system, or maybe it's some third-party API. But they could give you data that's wrong, and you want to know about it and be able to actually apply some pressure to them saying, "Hey, this problem caused me three days of rework by my team. It caused me to be four hours late." And that affects people when you've got data.

And I think data people are affected by data. So get some data about your data production because you can use it as leverage to help those who provide you data debt to improve.

And then, the last concept that we're going to talk about, and we're not going to spend too much time on it, but tests themselves have a dual nature. And so the tests that you run in production are also really useful in development. And you could call them different names, right? You could say in development they're more regression testing, functional testing, end-to-end testing, unit tests, smoke tests.

They're kind of names that software developers have created to categorize their tests. But I think 80% plus of the tests you run in production are useful in development. And in one case is the data is variable, but your code or configuration is fixed, that's production. But in development, your data's variable, but your code is fixed.

And the thing is that the tests and parameterization of the test can actually help you, and be able to then prove that the system works. Because a lot of what we're saying here is observability is about testing. And there's some ways that we can help automate the creation of tests, but in a lot of ways, it's work that your team's going to have to do.

And it's worth it, because it's the gift that you give to your future self, as my co-founder says. And the second part in testing and development is the data that you use in production is also useful in development, like your production data, your test history, some of the artifacts. Because in our software, we have an idea of a kitchen.

And in that kitchen, we've parameterized it such that you can take and use in a development kitchen, for instance, saved input production data or referenced input production data, and that saved test history and that saved artifacts. And all these things are useful for you to test what's happening in development. So I can see, well, how do I do a historical balance test where I save off some of the production data and I balance it against it? And maybe that is a ref, maybe it's not a save, maybe it's actually a reference to the production database, or maybe it is a save.

It depends on how you implement it. And so what you're trying to do is make sure that you have a consistency between your production tests and your development tests.

And so lastly, there's a bunch of other capabilities that I'll talk about in a demo. And I think that a couple one is being able to bootstrap, have some ability to kind of start tests from profiling data. I think that's a good help. Being able to route your alerts to the correct party, that's a good help.

And a lot of people talk about data lineage and where the data comes from. I'm actually more interested in the things that act upon the data, the processes that act upon the data. And keep a database of those, and that's what we do in our product, the code, the test results, the timing history, process steps, et cetera. And don't think of testing limited to data or the artifact. It's actually timing.

If your process takes too long and you're not going to make your SLA, well, that's an error. And then find problems fast, but also make it easy for people to get at where the problem is and diagnose it. And be able to figure out how to either patch it or fix it. And then think of coverage in testing.

In software development, there's a thing called code coverage that tells you how good your tests are. And that doesn't exist yet in data and analytics. And we've got a version of it in our software that allows you to kind of look at the extent of the testing across your full pipeline and see if it's good. And then a lot of our work In data and analytics now is broken between on-prem and cloud or multi-cloud.

And you need to be able to test in both places, and I'll give an example of that. And then finally, one principle that we believe in is that trying to keep your code and pipeline and test code together is actually a good best practice, because if you're going to rev your code, you should change your tests.

And so whether you believe in test-first development or like I

00:25:00

do, sort of test next to development, the process of creating something is creating work and adding tests at the same time. And that way, not to say that your data engineers or data scientists should spend all their time testing, but I think having some good tests that can both run in development and production is a good work for them to do.

And so I'm just going to switch and just show our product for a few minutes, just to kind of highlight some of these ideas that we poured into. And then I'm going to give a few examples and then finish up. So let me just show our product here. And so, this is DataKitchen. We're a software company, so we have this thing called a recipe, and this one happens to run in Azure.

And so it takes data, and the team who did this happens to like doing ELT in Azure, happens to like doing Jupyter Notebooks and Tableau, and it's a data catalog that happens to be a wiki. And so all these things are going on, right? And this can run from an event or in production.

And so what our software does, it doesn't actually do the work, right? You've got tools that do it. Maybe you like Informatica and not ELT, or maybe you like SAS and not Jupyter Notebook. But you're going to get data from an SFTP site and pour it through all the steps in data and analytics. So how do you know it's right?

Well, you actually test it or monitor it in production, and that's what these numbers mean. We go in and check to see if things are right, and we look at the data or the things that are happening from the data. So let's go and just take a first one. I'm getting some data from SFTP.

And so in this case, we have in our system a way to actually grab some data. And so we've got some tests that we're running. And so for instance, we're looking at is the SFTP line count greater than 1,500? Did we actually get data off the SFTP server? And is that size right? And is the hash of it right? The MD5 hash of the data we get versus another.

Again, did we actually transfer the data? Now, you could argue that networks are reliable, you don't have to worry about this, but again, I've seen errors just on accessing data. Now there's another case. So we're putting it in, it gets into a bucket store, and then it actually gets into the database. And so we've got a bunch of tests that run against the database.

And so here we have embedded in our system kind of a DBT-like ELT tool, and you can see the parameterized SQL and SQL. Now you don't have to do SQL. I like SQL. But then we've got a bunch of tests based on each one of these steps. So it's like compute the percentage of invalid orders.

Here's a way to compute that, and here's the test that actually shows up on that. So the percent of invalid orders, the percent is less than 2%. And so that's a test. Or the percent of invalid zip codes, or make sure that the number of raw total orders is always increasing. And so this is the case where you don't want to have the number of orders decrease for some reason, and that's if you're getting data on a periodic basis, you want to make sure that there's always monotonically increasing orders. And here's where we look at the variable history.

And so that's another example of a test. And then, so this is sort of access tests. The middle are sort of data and schema tests. Here's a case of a location balance test where we're looking at the differences between the fact table and the stage table in the database to make sure that they're the same size.

And then there's tests that actually have to do with the model itself. And so here, I'm looking at the predictive model. This happens to be an IPython notebook that runs. And here's some tests that are against it. And so here I've got some tests looking at the root mean square error that comes out of it. And so these tests all run with the recipe. So we create this thing called an order run, and they all run with it. And so if I look at it, I can actually look at my history, and I can search for my statuses and say, "I want to find the ones that have warnings and failures," and I'm going to apply.

And so here's the case of that SFTP minimum file size. You get these little yellow triangles. That means everything still runs. It's just a notification, and it looks a little weird, and maybe that's okay, maybe it's a problem. And so with data, it's not just right and wrong, although sometimes it is. There's this gray area where things are weird and maybe you should check it out, and that's what a warning is, and that's what all these yellow are.

They send an alert or a Slack message telling you. And so we're looking not only at the data, but we're also looking at the model, and here's where we're looking at the root mean square error model and seeing if it's right, and suddenly this test didn't work. And here's a case where it failed, and you can see that send a fail alert.

And you can also look at timings to see if the timing is right for that build. And so all of these things are cases where things have gone wrong.

00:30:00

And so if I go look at a specific case where it went wrong, and again, the idea is love your errors. So the fact that you found an error in production is actually a good thing. So we found that the model wasn't predicting right, and here we can see that the test failed, and we can see that that number didn't work.

And so I think the idea, again, is being able to sort of on top of your system, not replace what it has, but be able to work with it to be able to run these tests and be able to see all the tests across all the system, and then be able to actually keep track of the history of the tests, and even the logs of what happened. And so you can kind of see the error warnings, or you can go off and see everything that's related and all the logs. And so this is just maybe the simple thing, but when your operations team has to find a problem and they got to go, "Okay, where were my warnings?" Or "Where are my errors?" I can find it easily And then they can go off and export it and send it to someone saying, "I got this." And being able then to-- sometimes you want to be able to run this again and resume it.

Other times, you want to look at the details of the order run, and there's a whole bunch of metadata that we create, including all the source code that acted upon the system, the test results, and the timing results. So if you get sued, you can go back two years and say, "Here's exactly what we did.

Here's the code, here's the test, here's the results." It's a small thing, but actually that metadata is helpful in more regulated environments. So,

let me go back to my webinar, and let me summarize this in a second here. And so something happened to my arrangement of my slides. So, okay. Well, I'm just going to talk through it. So, unfortunately, the layout of my slides got broke. So,

I guess the principles that we have here are automatically test in production on top of your entire tool chain, send alerts when there's problems, keep track of history, and make it easy to create tests. And so that's our principles. And then we talked about a whole bunch of different test types and statistical process control, location balance, historical balance, and basic business profiling tests. And also their equivalent of how those tests map into what software developers call regression functional system tests.

And the whole point here is by focusing on lowering errors in production, you end up with more time, and you end up with less stress for your team. And I think that's actually a really good thing. So let me go on to some examples here, in my final few minutes. So, as you know, we're a company that's been really talking about DataOps, and one of the ideas of DataOps is being able to deploy quickly from dev into production. And what makes things be able to deploy quickly? Well, there's the automation of deployment, the CI and CD, but there's also the development of tests.

And the second part that we really focus on has been our subject today with data observability, low error rates. And so a lot of people come to us, and it's interesting that we found different people come to us for different reasons. And sometimes it's around, "I want to be able to deploy quicker." Sometimes it's around, "I want to get some sandboxes to help my self-service customers, so I want to improve collaboration." And sometimes it's low error rates, and it's kind of 30, 30, 30% or 33% each.

But if I had to pick one, they're all basically equal. Lowering error rates is a big deal for some companies because, as I said, once you get

sick of that embarrassment, you want to fix it. And so here's a case. And so what we do is try to work with customers and say, "Okay, here are the business goals you want, error rates, cycle time, collaboration, and what kind of features do you need?" And if you're focused on errors, you kind of focus on this production monitoring and observability at the bottom.

And so here's a company that did that first, and I'm going to talk about an example. And so they get data streaming from their vehicles. They have different teams managing different parts of the process. And as a result, they had so many data errors, they had the lack of customer data trust. And so here's the production environment. It's cool, right?

They had streaming, they had NiFi and Kafka. They had sort of a fast batch with an enterprise service bus. They had Informatica, they had predictive models, all taking data from vehicles, from internal ERP systems, internal systems. It all went into a big Oracle data warehouse on-prem. And then they had Tableau and Notebooks working off, and they had their business customers on the right, and they were also starting to move stuff to Redshift.

And so the problem was, if you look at this from an organization standpoint, there's IT teams work on the data sources, getting data in. There's different streaming teams for each type.

00:35:00

There's IT teams, there's an ETL team, there's an EDW team, a BI team, a data science team, and a cloud team. And so you've got these different groups, right? And sometimes they all-- in this case, they all sort of work for each other. I think the BI team did not. But what happens? And who are you going to call from "Ghostbusters" when something goes wrong?

Did somebody misconfigure Tableau? Did somebody muck up the warehouse? Did the access from it go? Is something actually wrong in the vehicles? And so this causes a lot of problems, right? Your VP or CEO and their favorite report gets something wrong, their favorite dashboard, and suddenly everyone's running around chasing, working nights and weekends. People are finger-pointing at each other.

"It's your fault." "No, it's my fault." And so I just don't think that that's a great situation. I think that's actually a first-order thing that you should solve, because this ends up eating time. And it also slows you down from doing more good things. And the way you solve this is by kind of overlaying a system on top of it, and that talks to all those tools and actually runs those tests against it.

And as we said, it fires alerts based on the types of tests that we talked about, business tests, historical balance tests, location balance tests, profile-based tests, and tells you if something's wrong as soon as possible. And so those alerts actually are a big deal because then you can know that something's wrong. And I just think in the data world, things are going to go wrong because data provider, data changes, and you're playing this sort of two-sided game where you're trying to work the data to see if it's right, see if it's predictive, but you're also trying to build an application like a software engineer to make sure that people are adapting. You got both sides going on.

And finger-pointing doesn't help you in that process. And so one way to get beyond finger-pointing is sort of establishing an end-to-end system across it, and that lowers errors. I've never found a customer who said Where I've said to them, "Yep, we had an error. Yep, we caught it before. And yep, it's our fault. We're going to fix it next time." And they're like, "Fine, at least you're going to fix it next time." So let's go on to the next one. And here's a global pharma company.

And they had two different teams, one in New Jersey running a Spark cluster and one in Azure. And one of the interesting problems is that they work together, so the kind of drug discovery data they did in New Jersey is also used by the cloud team who's also involved in drug discovery in California. So they had a problem in that the source of data from one team and the schema and the data in it was not working. And so they are drug discovery, so they have a pretty free-form way of working on their data.

And so one way to do this is that the people could actually go in and change the schema in the database in New Jersey and change the schema in the cloud. And they trusted their data engineers. They had sort of a high-trust environment. And I'm agreeing with that, but how do you monitor when that happens?

So one way of testing is actually to just test over and over again to check if see something's wrong and not only check during the actual production process, but check when the production process is done, if your schema is rewrite and not read, people are going to actually be able to write it and check and see if it's right.

And in addition to all the other principles I talked about, the kind of active monitoring, sometimes when you have skilled people who have access to changing production databases, I think is an important task. And then finally, I think it's actually really easy to start with DataKitchen. So let's say I'm an individual data engineer, I'm working, let's say, in a cloud, and I've got a data system running.

This stuff sound great, Chris, I want to start working. Well, there's two sort of decisions here. So you want some tests quickly. You've got something working. So you have your favorite ETL and ELT tool. Let's say you've got Airflow or Talend or Informatica or one of the 50 companies, and it's all working, right?

And you've got Azure Data Factory or Amazon Glue. But it's running sort of open-ended. You hope that the data works when it comes in, you hope it's right, and you rely on your customers to find errors. And so you've listened to this webinar, you think it's a good idea. How could I do this? Well, the first thing is, it's kind of one, two, three. You just call DataKitchen, and we can actually run the tests and execute the tests for you by our REST API.

It's sort of non-blocking, can be run from anywhere in your process, and we'll send you alerts and notifications and have a control panel, just as you saw before. And so it's a very simple way for you to get comfort that things are working. And I think this is a really great step because at least some tests running in production are better than none.

And being non-blocking tests makes it that you

00:40:00

know that some other new tool's not going to mess things up. Now, there's a case where you do want to be blocking, where you do want to stop a process, and that's this idea of what do you do when there's a big error and you just don't want to proceed? In Japanese manufacturing, they call that an Andon Cord, says, "Stop the assembly line." And it's that idea that you can empower workers to be able to stop and focus changes. And so there are cases when the data or the processing is so bad you should stop.

And you also want to know, in other cases, where you've got multiple data processes happening in different locations, that you want sort of a single pane of glass that oversees it. And another way of using our system is where we could call your ETL process and do the work and say, "Here, go run this." And maybe that happens to be in Amazon, happens to be in Azure, happens to be on-prem or all three, and happens to be multiple tools. And we can call, and then you get this idea of DataKitchen can run the tasks, call the tasks, send the alerts, and stop if you need it, and gives you sort of an Andon Cord.

And so it's a way to kind of what we call meta orchestrate all your tools. And I think both are actually fairly easy to do. And I think both have benefits of working. And so the conclusion is-- And here's a theory that I want to test on you, and this is an idea, and we've been practicing it with some of our managed service customers, but the idea of continuous production.

And so if you're in the cloud, disk and CPU are relatively cheap. So having variations of production, it becomes actually really easy to do, especially if you build your pipelines in a more functional and idempotent way, meaning that you sort of take data and you run it through, you get the answer, you take data, you run it through again, you get the same answer.

And so with disk cheap and with the fact that most people have 1,000 terabytes, it's relatively small data, throwing those datasets around a cloud is not a big deal, so you can actually have multiple versions of production. You could have one that has 1,000 people on and then one that has one person on with just a little variation to see if it's right.

And maybe you've got another one just to the right. Or you could get into champion challenger A/B tests, where you could use IP load balancing to go between the two or some way to switch back and forth. And so what our software provides and is sort of a way to do infinite variations of what happens.

And if you look at it, isn't just development a variation of production? And so in some ways, this idea of continuous production, where you're always working and building on variations of production, I think is a very powerful idea. And it's enabled by the fact that you can do this in the cloud. And having lots of variations and managing those variations, I think is a good thing. And also having multiple versions of production may seem crazy, but I found it very powerful that you could actually have something that runs in production with 1,000 users.

All those tests ran, it's perfect. And then you have a parallel version of it, and you just tweaked a table. And you ran all those 1,000 tests, and maybe you add one new for that new thing that you added. And you've got that one or two customers on it. And the benefit of that is you're getting feedback quickly.

And at the end of the day, we just want analytic teams to run with low errors, be able to deliver value to the customers quickly, and iterate and improve. And sometimes iterations can come from having lots of variations of production. And so testing and data observability leads to, I think, in the cloud, leads to this idea of continuous in production.

And so, what do you say to your boss? How does this matter from an ROI perspective? Well, we've done some surveys with Eckerson, and actually, a lot of people have a lot of errors, unfortunately. And the goal of data observability is to fix this, is to take that graphic equalizer and push it up.

But we're a DataOps company. There's other problems that go with this, as I talked about, cycle time and collaboration and process management. And you can start just fixing one, or you could push up one of those sliders and push it down, just focused on observability, or you could push on all of them. And so I think that's the real idea here, is it is possible to reduce your errors.

It's not something-- You don't have to sort of suffer and walk into the cafeteria, and you don't have to live with all those embarrassing situations I had for many years. You can learn from that and implement some ways to work. And also, you can do more than that. You can not just flow errors, you can actually deliver faster, and you can actually get rid of the Hatfields and McCoys in your organization.

And so, that's what we're talking about with DataOps and data observability. And as you've seen, we've got a software product, so if you're interested in talking with us, I think we've been doing this for quite a while. We've got a lot of experience and expertise to help you. And so that's it, Beth. All right. Awesome. Thanks, Chris. So we have some time for questions.

00:45:00

So if you have any questions, just enter them in the question box and we'll get through as many as we can in the next few minutes. So here's one to start, Chris. Is there a template to measure the maturity of DataOps of an organization and then produce a guide path for maturity? Yeah, we do. Actually, we have one.

We were asked by a bunch of customers, so we have a maturity model, and we have a template, and we actually look at sort of a bunch of characteristics of-- One of the columns is kind of data errors, data testing, data observability. And so in there, we have a way to do that. And also, Beth, that's going to be available off of our website, so you can do it yourself. Is that correct?

Yes, it will be available soon.

Yeah, and so you can do it yourself. We've had time where we've just done it with people in a talking discussion over a phone call, and we've got a white paper that talks about it. So there's a lot of ways you can judge your sort of maturity on DataOps, and of course, your maturity on data observability.

Great. Thanks, Chris. Okay, so next, have you all worked on asset data, and have you helped implement data cleansing around creating uniform asset data models?

An asset, they mean like financial assets? Is that what they mean? As a term- Not sure.

Please. Beth, I'm not sure I understand the question. Yes, I'm not sure. Let me see if this is the turbines, heavy machinery, and turbo- Oh, those kind of assets. Yes. Like heavy asset data. Ah, okay. Like actual physical assets. Oh, cool. Not just that financial stuff.

There's two answers to that. So, I look at it from my engineer's head and say, well, data's data, and the data could come from a machine or could come from a stock ticker, and it ends up being data. There are some issues with asset data in manufacturing, where there's some processing that has to happen closer to the machine.

But yeah, I think the idea of integrating data, and especially machine data can be very noisy, and how to find anomalies in that data and applying sort of filtering logic to that data. I remember when I worked in air traffic control, we used to get data from sensors from planes, and we applied a Kalman filter to the data to kind of smooth it out.

I think that's an important aspect to be able to look at, keep the raw data, keep the smooth data, and in an automated way, try to look for some kind of anomalies in the data. And it depends on your domain. Sometimes you want to smooth the anomalies, sometimes you want to highlight the anomalies.

And so that's where we can kind of help work with you and decide what to do with that anomalous data. But our software, in either case, can work and help. It's a framework to help test anomalies or a framework to help you run algorithms to smooth the anomalies.

Great, thanks. Okay, so next is how do you deal with the concern that running data quality queries on your data is a burden to the systems?

Well, I'm going to be spunky here. I just think that's a freaking excuse. So you're worried about CPU cycles when you can log into Amazon and configure one more virtual CPU, and take care of any burden, and you're relying on your customers for QA. So, let me take the spunk out of that. Yeah, there are some cases where having hard deadlines and people have databases that have limited CPU cycles, and what we're saying is that, yes, you should do these tests. And we've run into customers like that.

They have an old Oracle Data Warehouse. They're counting CPU cycles. They don't want to miss their cycle and sort of the thought of running a bunch of tests on top of it freaks people out. And so, in that case, I think oftentimes it's better to know before your customer sees it, or sometimes there's an exception.

Better to know very soon after your customer receives it. And so you can also put the tests and do exactly the opposite of what I said, have them as a bag on the side, take some exports, run them on the extract, run them in another database. You don't have to run them on top of the system.

You can run them on the data as it gets before the database and run it afterwards, and that way you don't- That way you don't run into CPU cycle or disk cycle limitations on old style on-prem hardware. And so that's a case that we've run into and a case that we've helped solve. And so yeah, and you miss the sort of finding alerts in the middle of the process, but sometimes you can sort of say, "Okay, halfway through the process, I'm going to do a quick export of a couple of key tables and run tests against that," and that can

00:50:00

give you at least part of the way there. Okay, great. So next question, this is a good one. What kind of pre-work needs to be done before calling DataKitchen to establish enterprise-wide DataOps?

Enterprise-wide DataOps. Well, that's a big question, right? Because I think the first thing that we do is try to find where we can add unique sense of value quickly. And oftentimes, but not always, it's, I want to lower my errors in production. Sometimes it's cycle time of I want to deploy things faster. And so I think it's really about applying the spirit of Agile to the deployment of DataOps and finding areas where you can deliver value to your customer quickly, get it in the bank, and then iterate and improve. And so that tends to be actually, in some ways, a contingent discussion based on what the individual problems and individual data sets.

And so to me, DataOps is something that should be delivered incrementally and have incremental value. And once you start to get the ball rolling, we found a lot of people have expressed interest in this because nobody really wants to have those problems that I had at the beginning. And there's another, you look at some of the other values like deploying slowly, there's a whole other set of embarrassing stories.

And so, I think that those things are worth the change and worth thinking about in a considered way. And we, with our partners, can help you do that.

Great. Okay, we have one more question unless someone chimes in in the next minute or so. Feel free, if you have a question, make sure you enter it. But the last one I have here, are there any additional resources on the specific types of testing we should be doing with SQL pipelines?

Yeah, I think in our book, so both on our blog and our book, we've got a bunch of examples of the kind of tests that we do, that we've done in SQL and there's some code examples, there's some principle examples to show. And so one of the things that we hope to do is continue to have some of our chefs write blogs that sort of go through the examples of what's a good SQL test. And in a longer, different version of this presentation, there's lots of ways to write good SQL, and just one key tip to take away from this is the SQL ACCEPT statement's really good. Just be able to write a query and compare it to another query and see if the results are the same.

That's one very simple way that you can check one table against another. And so yeah, check out our company blog and check out the DataOps blog for some more tips. All right. Well, that is all the questions that we have, so I just want to say thanks, Chris. That was great, useful information as always.

Thanks to all the attendees for joining us today. We really appreciate that you take time out of your day and share it with us. If you do think of any questions later on, don't hesitate to reach out to Chris or myself, and we're happy to help. And just a reminder that we will be sending out the recording of this webinar as well as the slides to you in the next 24 hours or so. So be on the lookout for that in your email.

So that's it. Thanks again.

Thank you. Have a great day.