Tame DataOps System Complexity with a DataOps Platform (Demo)

00:00:00

Good afternoon and good morning, and even good evening to some of you. Thanks for joining us today. My name's Beth Befferly, I'm the VP of marketing at DataKitchen, and I'll be the host for the webinar today. Our topic is how to tame DataOps system complexity with a DataOps platform. We have Chris Bergh here, who's the founder, head chef, and CEO of DataKitchen, and he'll do a live demo of the DataKitchen DataOps platform for you today.

In the first part of the webinar, he'll go through the demo, and then we'll have some time at the end for Q&A. So if you have any questions, please just enter them in the question box on the webinar control panel, and we'll collect those and go through as many as we can at the end.

Also, just to note that the webinar is being recorded, and we'll send a link to all the attendees at the conclusion of the webinar. So I think that's all the housekeeping, so I will just hand it over to you, Chris. Oh, thank you, Beth. Thank you for that introduction, and everyone, welcome. So the topic is taming complexity, and so complexity is a really interesting word in data and analytics because a lot of people focus on the complexity of the data itself, whether it's big or small or fat or thin or structured or unstructured, and I think they miss an important part of the complexity of the processes that act on data. And so that's really where the focus of DataKitchen is in DataOps. And I'm going to begin this demo with kind of talking about those processes and using them as a setup to how we tame complexity. And so the most important thing to start off and just kind of frame DataOps is this phrase of, what you do is much less important than how you do it.

And so I'm going to just kind of talk about that in our slides here. And so, I spent a bunch of years writing software and managing software teams, and then about 2005, I started to manage data and analytic teams, and it was actually really hard. And the first thing I thought was, "Oh, I just have poor people, and so I'm going to fire some people, and I get better people, and it'll all be wonderful." And you know what? That didn't work. And I started to read why, and there's this guy, Dr.

Deming, who was sort of about total quality management and the Toyota production system, and he said 94% of the problems are actually the problems of the process, the system, not the individual or not the specific tool. And I sort of took that to heart, and Elon Musk says, "Think about the machine that makes the machine." And in a lot of cases, we talk about data and analytics.

We talk about a lot of really cool stuff, like what model that you're using, what algorithm is it, how are you transforming data, how are you visualizing data, how you govern data. But we talk less about how you develop, how you deploy, how you monitor, how quickly you can iterate, how you collaborate, and then how you measure those processes. And that's really almost what we're going to focus on today because our belief is that the process and the people and the operations, if you get those right, you're able to super quickly change the tools and the technology and the data.

And so in some ways, the main problem that we see is not data silos, it's people and process and tool silos. And so what does that mean? And we're going to sort of talk through these four themes today. One is that the process is really focused on trying to have low errors, where by observing and testing and monitoring data to see that it's right, being able to deploy quickly, the low cycle time, being able to collaborate between teams, and being able to measure your progress.

And so unfortunately, a lot of teams suffer from these poor processes, and their cycle time is low. Think of it takes months to deploy. They are constantly going into the office and getting a nastygram from their customer saying the data's wrong or the data looks weird, or their teams are fighting against each other.

And from our case, we've talked and sort of defined what DataOps is, which is really this less about data focus, and it's weird that it should be called DataOps because it's really data process ops, but it's a set of technical practices and cultural norms that really focus on those things, cycle times, error rates, collaboration, and measurement.

And so today, we're going to have this diagram on the right we're going to use to talk about what our product does. And the first circle is about the journey that data takes from value, and the second circle is about deployment, and the third circle at the bottom are about collaboration. So the first part of our demonstration is we're going to talk about that production process. So imagine you've got a bunch of tools indicated by those gears and light bulbs in the diagram, and data's sort of coming from sources on its journey to deliver value to your customer, and lists or artifacts or charts and graphs.

And so one of the biggest problems in that is, how do you run a system like that that's complicated, that has a lot of tools, has a

00:05:00

lot of data, and also how do you get it so it reduces a lot of errors? And so when we wrote the product, we looked at a bunch of different tools, and our product is named DataKitchen, so hopefully, if you're anywhere near mealtime, we're going to overuse food metaphors, so apologies ahead of time for our use of this term called a recipe. And that's what these steps are.

And what happens is, imagine this is a journey, or in our demo, it's a journey where data comes in, it's transformed, forecast models apply to it, it's visualized, and then there's a data catalog and data security. And there's a chain of tools in here. This happens to be an ELT process. This happens to be a Jupyter Notebook. This happens to be Tableau.

This happens to be a wiki-based data catalog. But we don't particularly care how you do your data work, right? You have your favorite tools. Maybe you like ETL, Informatica versus ELT. Maybe you like SAS better than Jupyter Notebooks. Maybe you like Cognos and not Tableau workbooks. But this chain of running these tools is your production process.

And so how do you make sure that this production process runs correctly and your customers don't find out that there's any problems? Well, one of the things that we want to do is to make sure that-

Make sure that you test the system while it's running, and that's what these numbers mean. So imagine as the data's flowing through either batch or streaming, and it's going through this process of getting into a bucket store in Amazon, being transformed through ELT, that we're actually grabbing little bits of data out of the process and checking to see if it's right.

And it could be right because it's the right size, it could be the right shape, it could be the right semantics, the right counts, or it even looks different from the last time that we saw it. And then it's not just the data, it's actually the artifacts that are created from data. So is the forecast model giving an accurate prediction?

Is the Tableau workbook actually showing what you do? And the system runs, and when we run it, either from an event or from a scheduler, we create this thing called an order in our system. And an order is a way to collect up the history of all the times this recipe has run. And so what that history is actually enables you to do some interesting things.

And so here, these graphs are ways that we've grabbed bits of data or artifacts created from the data over time, and there's different values on the side. And so if I go look at this, and I'm going to change the number I display here. And I can see that I want to look at this, for instance, and say, "Here's the number of the percent of invalid orders." So this is a check saying the number of orders in this case is the data that's flowing in.

And it's below two, and then all of a sudden it breaks a bar and something's wrong. So every time this runs, we're checking that the data's right, and then it suddenly broke a bound. And what this triangle means is we actually give an alert. So we'll send an email or a Slack message, create a Jira ticket, and tell you that something looks weird.

And that could be, sometimes you actually want to stop the whole production line and sort of give it an andon cord. In that case, we'll actually send an email or stop the production line entirely. And in some cases, it's not just the data that matters. It's a case of, is the data science model predicting correctly?

So in this case, we've got a category. The data in this demo is from kind of a store sample data, and one of the things the store sells is furniture. And so it's actually building a forecast model for furniture sales. And in that forecast model, we're looking at the root mean square error. And so most of the time, it's actually in line to make sure that's right, because it's not just is the data right, it's the things that are created from the data are right.

And you can look at this over time, and actually, it's not just the data or the things that are created from the data, it's the timing. Is the system running in the time that you think it is? Because all these things are cases that I have found where customers complain, and they end up calling you on Friday afternoon or someone in your team, and you're sitting there late Friday night trying to figure out where the problem is.

And so for us, we think the most important thing in making this happen is to build these tests that run against the system. And if I go in and look at a test in our system, and again, this is one type of way that people interact with the data is to do ELT, and there's other ways, and we're going to show you different kinds in this demo.

But in this case, people are actually-- I'm a SQL guy, so when I wrote this demo, I did SQL. And so here, actually, we're writing some SQL that goes against it, and we're just making sure that it's right, and we write a test against it. And so this test says, "Does the fact table equal the stage table?" And so in a lot of systems, data systems, data bounces from an FTP site to a staging area to a raw or in a database, to a processed area, to a report.

00:10:00

And how do you know you haven't lost anything? Okay. Great. And so we described how we did that, and we talked about a thing called a recipe, and we showed all the steps in it and all the tools that it talks to. And then we were talking about what happens in one of these circles.

It's called a node in the recipe, in the different tools like Tableau or a data catalog. And then we said when that runs, it creates something called an order run, and that order run is evidence that that process happened. And it actually keeps track of all, whether it ran, and it keeps track of all the tests that were applied to this.

And so again, these tests or monitors look at the model, they look at the data, they even look at the Tableau workbook to see that it's right. And that's an important part because I think from, at least my experience, one of the most important things that you can do as in an analytic team is try to focus on lowering your error rates and embarrassment. And to do that, you need to automatically test everything that you're creating in production, kind of on top of the tool chain, and send alerts and notifications as soon as something's wrong, and then keep track of history, and then actually make it easy to create tasks by allowing people to use their favorite tools. So if you've got a bunch of Python programmers or SQL programmers, let them use that to create the task.

If you've got a bunch of Informatica programmers, let them use that. And so, in other webinars, we talked a lot about different ways to test data. Now, there's tools out there that are called data profiling tools, where you can take a raw file and profile it, and it can tell you, well, for instance, this column's got three values in it, and we can create that as a test.

And then we talked a little bit about the statistical process control test, which look at trends in data over time. And we showed you an example of a location balance test. And one of my favorite tests is that test that actually looks at your data system through the eye of your customer. And I don't know if you've had this problem, but a lot of cases, I've worked with someone doing visualization, someone doing data, perhaps a model.

We work for a few weeks. We put something in front of a business user, all proud, and then we show it to them and say, "Isn't this great?" And they look at it within two seconds, they say, "The data's wrong." And how do they know that? Well, in a lot of cases, your business customer kind of memorizes their top regions, their top products, their top customers, and they're looking at the values and they go, "This looks wrong." And so we think that you should have a test that looks at it longitudinally, do something called a historical balance test ... what your customers have seen now versus what they saw previously.

And so at the end of the day, why do you want to do this? Well, having less errors actually means more time for innovation and more customer data trust, and not a small thing, less stress and less embarrassment. And let me talk about an example of a customer, and here's an American transportation company, and they had a really interesting data architecture that came in from their vehicle. Some of it was streamed in in two ways with NiFi and Kafka. Some of it sort of went in in fast batch in an enterprise service bus. They also had Informatica.

They also had some models that ran in real time. It went into an Oracle data warehouse, and then they had notebooks and Tableau sitting on top of it and a cloud initiative. And so when something would happen, a VP and a business customer or someone in a region, they would say, "The data's weird," and they had no idea where it was.

Was it the vehicle stopped getting data? Did something happen in these ingestion routines? Did something happen in the warehouse? Was it the visualization? And sort of finger-pointing and pain ensued. And so how do you actually know that? How can you locate the problem quickly, and how can you make sure that the problem never happens again?

And so, what they did was actually take that recipe that you saw and sort of overlay it on top of that complex tool chain and sort of grab bits of data and do two things. One is sort of meta-orchestrate all the tools, and then two, write those tests and monitors against them. And if something looked funny, send alerts, emails, Slack messages, JIRA tickets before your customers see them. And now, doing this isn't going to resolve every data issue because data is data and it changes, but it's going to give you a heads-up if something's wrong. And then second, when something does go wrong and you don't have a check ahead of time, it's going to enable you to say to your customer, "I'm sorry something was wrong.

We know the reason, and we're going to put a check in to make sure that it doesn't happen again." And so I think that's a very important point. And so that's really talking-- And I think the actual sort of business benefit, the sort of why you care is, I've been doing data and analytics for a while, and we spend a lot of times in meetings, a lot of times talking, and that focus, and a lot of time just fixing problems.

And that's time taken away from actually doing cool new stuff. And so I think one of the benefits of DataOps is by focusing on lower errors, you actually get more time to do what you actually want to do. So that's one process. You've got production systems with a bunch of tools, focus on lowering errors, and therefore, get more customer data trust.

Now, the second process I want to talk about is actually kind of independent of that, and that's the deployment process. That's how do you create something new or change something that already exists and then deploy it

00:15:00

into production? And so in this case, we're going to give you a quick scenario where there's a VP of marketing and they have a data engineer and a data analyst, and it's running in Amazon. And the data engineer happens to use Pentaho. But like we said, it could be Informatica or Talend or however you want to do it, SQL or Airflow, or it could be Power BI or Qlik. And so what we really want to help you do is say, "When there's a new customer, I want a customer segmentation." And to do that, sometimes you actually need to have a new person involved.

So in this case, there's a new person, a data scientist. And sometimes to make a change, you actually have to change something small. But in this case, you actually, the data scientist adds a model, the data engineer stores it in a column, and the data analyst visualizes it. So everyone's got to make a change.

And so what we're going to show you is how that change is made. And then, more importantly, we're going to show you something called a kitchen, which is the place to work, and then from that, how we deploy. So let me go back into the product, and I'm going to switch my customer here and log into a different customer to show you this demo.

And so I'm going to log in. And so I'm going to go into this kitchen called Production. This represents what's happening in production, and we've got this recipe called Agile Analytic Ops, and this is a really simple one I'm going to show you. It's just got five steps, and all it does is actually just call some Pentaho ETL process and then publish a Tableau workbook and then test the Tableau workbook. So what does it actually mean to interact with an ETL tool? And so maybe you've got Azure Data Factory or maybe you've got Airflow, but every one of your tools has got some kind of file that is the result of creating the work. And in

Pentaho, that is actually these XML files or these files called KTR files. And so what we do with those, we actually run them in the Pentaho engine, and then we test them just like we tested SQL before. And here's a simple test. It's get test data is greater than or equal to 1,000. And so the way that we do that is we actually wrap the

Pentaho engine in a Docker container, and in that Docker container, we are able to execute that Pentaho work and then, of course, test it. And so when this runs, and I can hit Run, or I can schedule, hit Run this variation, I create an order run. And so just like before, I have this order run, which is the evidence of what happened, and each one of these steps executed correctly in each one of these tests. And so here's the result of that Pentaho test run.

So I can see that test actually executed and passed. But what's also interesting is I'm running tests against Tableau, and I can see all these Tableau tests pass. And so why would I bother to test Tableau? Well, it makes nice bar charts and line charts, and we're actually not testing those It turns out that when you're working in Tableau, you've actually got a little bit of a development environment that's set up. In this development environment, you're able to build things like calculated fields or if/then/else statements.

And from my standpoint, that's actually code in a software engineer code that you should test. So what we're doing here is we're actually testing Tableau. So we're digging into the Tableau workbook, we're pulling data out of it, and we're testing it to make sure it's right. And all of this is, again, why bother?

Well, in production, you could have some variant dataset, and that third clause of the if/then/else statement that you've built into Tableau could be wrong, and your boss doesn't know that if/then/else was in Tableau and not in an access script or not in an attribute of a dimension. They just know it's wrong, and you've got to, again, find out that there's a problem before production.

So let's go back and actually do what the VP of marketing wants. Let's go in and actually make that change. So to do that, I actually have to create a kitchen. So if I go to my list of kitchens, this is what's in production. I'm going to create a child kitchen. And so doing development in data science and analytics is interesting because you've got a lot of pieces you have to put together to do the work. The first part is you actually need something.

You need an area to develop that's not production. So you need to have a copy of the tools, a copy of the software, the servers, and the test data. So you need a tool chain environment to work in. And so sometimes that test data is a copy of production, sometimes it's a filtered, smaller subset of production, but you need test data.

And all these things have to come together. And you also need people. Maybe you're just working by yourself, or you're working with a team, and you also want to be able to do what my boss at NASA told me many years ago, "Build a little, test a little, and learn a lot." And so we scope the orders to the kitchen, but we also allow you a lot of control over how you run the recipes.

00:20:00

And this one that I showed you with five or six circles in is kind of a toy recipe. Most of our customers have graphs that contain hundreds of nodes, and of course, you don't want to run hundreds of things every time. You want to focus in on a small part and change that and then be able to see if it works. And finally, we think that all the work that you do, and this is an important part of the next thing that I'm going to talk about in the demos is, is we put it all in Git, and that enables you to do something called branching and merging so that you can have a version of the code and your coworker can have a version of the code, and you can work independently, but then be able to merge it back together.

And so to do that, I've got to walk through this wizard. I need to name this kitchen. I'm going to call it Dev Sprint. And I'm going to hit Next, and I just want to have warnings and failure emails go to me. These are the alerts I'm interested in. And then I'm only interested in changing just this one recipe.

And then, beyond the scope of this demo, there's a role in our software for people who decide what does an environment need. What kind of servers do you run on? Do you run on Azure or do you run on GCP? Are you on-prem? What kind of databases, what kind of data science tools, what kind of biz tools?

And so in this demo, our DataOps engineer gave me two choices, creating a Redshift cluster or creating a SQL Server database. And here I've got two options. One is I could create a Redshift cluster from scratch, start it up, or another case is I could create a schema, or I could just create a place in an existing database. And I'm going to do that because that takes just a few seconds, whereas creating a Redshift cluster can sometimes take a minute or two. And so I need to name that schema, so I'm going to call it Dev Sprint.

That is that place in the database. But also I'm creating a place where my data analysts work. So Tableau Online has a different scoping mechanism. So there's two servers that are running. One is a database, and I'm creating a place in that database called Dev Sprint. And another server is running to do the reports, and I'm creating a place in that database, and I'm giving them names.

And so that's why I'm putting in the term Dev Sprint. And so I hit Next. And then a last part of our system is that, since we are dealing with servers and logins, you don't want to have all that information stored in plain text. You want to store it in an encrypted vault and call it by reference. And so we have a vault integration.

In fact, you can have multiple vaults because in some organizations, they don't want to have the production logins and passwords known by the development team. But you still want to be able to easily have your code run in development as well as in production. So I'm going to hit Next, and then I'm going to hit Create kitchen. So what this kitchen wizard does is goes off and does all these steps and builds the kitchen, creates the Git branch, creates the workspace for me to do my development. And so what I end up with is in my kitchen list, I can actually see this Dev Sprint kitchen.

Now, kitchens have security applied to them. So some people have access, some people don't. Some people can change different things in the kitchen, but I'm just going to go into this Dev Sprint kitchen now by clicking on it. And I can see in the upper left-hand side, I'm in the Dev Sprint kitchen. I've only got one recipe now. So I have my own hardware, my own software, I have my own test data, and I'm on my own Git branch. So I can change it and be very comfortable.

And I don't have to worry about whether I bump into anyone else or something breaks in production or my notifications go to somewhere else. It's a way to create a virtual sandbox. And in some organizations, just the process of getting sandboxes can take six weeks. And that's just too slow for the speed of business.

So we don't have a lot of time in this demo, especially since my network went out, but I want to show you actually how people make changes. And so in this scenario, we've got a data scientist using their favorite tool, a Jupyter Notebook. So they have to go in and actually do their work in their Jupyter Notebook, and then they've got to add it into the recipe. So I've done it ahead of time.

And what the data scientist or the data engineer would do, they'd hit Add node, and we've got different node types, some of which actually do talk to databases, some of which talk to different tools, some of which actually have a And we're going to talk about this in a little bit, a reusable component model called an ingredient.

There's a way that you can reuse existing nodes as templates. And they added that in, and what happened is, if I drill into that, we can see that the data scientists did their favorite work now. Here it's an IPython Notebook. And you can see that they did their cluster here, the K-means cluster. And that's fantastic, but how does that fit with the rest of the world? And more specifically, how do you know that it's correct?

Well, I wrote a very simple test against it that checks the output of this Jupyter Notebook and sees if it's correct, and looks at the integer to see if it's right. And so now, if I go back, and I go back to my variation, because I'm in my kitchen, because everything's set, because I've got tests running against it, I can just hit Run Variation,

00:25:00

and I can just run this. And one of the ways to think about this is that we're building, taking in our kitchen, we've got a copy of all the recipes, all the tools, and all the code that runs that tools. And we're saying, "Hey, go run this and see if it's right." And so we're going to create an order that executes on that.

But we're also going to apply a set of variables against it to make sure that it's correct. And so some of them may be the email address that we're going to use, or some of them may be the hostname of the Redshift database. And you notice that this is actually a vault reference. So we're going to pull that out of a database at runtime and then even filter it out of the order run history, because sometimes Amazon actually puts secrets into their error messages.

And so we apply these variables and compile the recipe to execute it. And so we can see this order run as I click into it. And we can see each step going, and we can see the status as it goes by. And so one of the most important kind of concepts here is how do you know-- is about blinders.

And a lot of people who work in data analytics kind of live with blinders. I've made my change. Maybe I've done simple unit tests, and I'm done. My work is finished, and I don't have to think about anything else. I can go on and take a vacation or go away for the weekend. But that's not true. What happened here is our data scientist put something in the middle of an existing process.

They put these two segments in, these two nodes in, and how do they know that they haven't broken anything downstream? And so, I think that's a really important thing. You need to know when you've changed something if you've created a regression. And so what we're trying to do here by putting the whole system in a recipe is take those blinders off, and so that you can see not only that this new thing that you developed, this Jupyter Notebook works, but you're also testing to see if what existed before still works. And here we're actually rerunning those same tests that we ran in production, those Tableau tests in development.

Their purpose is different. They're not actually checking the data, they're checking that there isn't any regressions in code. And so testing and meta-orchestration are important both in development kitchens and production kitchens. In production kitchens, your data varies, but your code that's acting upon it is fixed. And in development, your data's fixed, but your code varies.

And so thinking of these tests working in both ways actually helps you check for regressions. And this is an incredibly important point because so many data teams, they don't really test. They don't really know if things work, and then they find out that there's problems in production, or they find out that there's some change that they wanted to make and it didn't end up working. And so if we go back and look at this recipe and kind of drill a little bit sort of deeper into each step, we can see that there's the Pentaho work, the Tableau work, and the Tableau notebook test. And let me just, because I built this ahead of time, let me make just a small change today, and I'm going to make a change that says, "Today," in capitals. And since I've built this already, I'm just going to hit Update, and I'm just actually making a Git commit and a change.

And so the question comes then, since I've made my change, I've proved that it works, and now I want to get that work from development up into production. And there's kind of two ways that people have arranged their technical infrastructure to do data and analytics. In some cases, it's stateful. That is, you've got to take the code and move it from one environment to the other and patch the environment and then make it work.

And I think that's a fine way to do it, and we can handle that. The way that we've written this demo is a different way. It's sort of more of a functional architecture. And it's something called idempotent, which means we can run it again, and it'll automatically update. And so the benefit of that approach being functional and idempotent means that you can move it from one environment to the other, and then the code, when it runs in that new environment, will work. And so we'll be able to make the change. So I'm going to do that here.

I'm just going to merge it up into the kitchen, and then I can actually see that change that I made. It's a really simple change in the description. And then I'm going to hit Merge. And I'm going to actually delete the child kitchen because I don't need it anymore, and I'm going to do that merge up into production.

And so I go to the production kitchen, and I can actually see that change, and I can see the history. So I can actually see that history that I made in Git, that simple, dumb change that I made. And so it's available, and when the next time it runs, it's going to work. And so I guess the question is back, why should you care? And so I think the first case is really a lot of organizations are spending a lot of time trying to get something from the mind of their data scientists or data engineers into production. And that latency of deployment is

00:30:00

weeks or months in organizations, and that's just too slow for the speed of business. It should be hours or minutes. And when they do it, it has a high error rate. Things don't work because they've got their blinders on. And what that means is you've got a data and analytics team that's both unproductive and unhappy. And so, I think that's one of the benefits of DataOps.

And so let me go on to the third part of our demo. So there's different ways that teams organize. So sometimes you've got a data engineer and a data scientist kind of working together next to each other. And so how do they actually work? And so it actually it's very easy in our system because if I go back to the kitchens and I say the kitchens themselves are environments, but they're also Git branches. And so if I look at this, I see Priya and Tom are both working on features that are actually going to go back up into Snowflake when it gets into production. And so each one of them are working in their own kitchen, and then they merge their work up into kind of an integration kitchen, and then that gets back up into production.

And so but the fact that we have all the tools and all the code that acts on the tools in one place allows us to merge that code together so that teams who are working together, maybe on the same team or the same boss can make it work. And so, that's one sort of, I think, very simple way that we allow sort of teams to collaborate, people working on the same project. There's gets to be another case that's actually, I think, a bit harder and not handled very well today in most companies, and that there's this idea of self-service analytics where you have multiple teams all around your company using tools like Alteryx or Looker or Power BI, to be able to do work. And then, but they depend on data that kind of a home office team or a central IT team may create.

And in this case, there's a Boston team that builds a SQL Server database, and then the data analyst builds the visualizations, and the VP of marketing sees the results. And so the VP of marketing doesn't know about that there's two teams who have two separate bosses working in two different locations. They just know something's wrong.

And so, how do you make sure that if the data team in the home office changes something, that the self-service teams know about it? And the current answer is, let's have more meetings. And I just don't think that's acceptable. Not that I don't like meetings, I just think there's a technical way that they do that. And the technical challenge here is that you need to have two teams be able to own parts of their own recipes, but they need to be coordinated so that each team can work independently, so that if I change something in the home office, I can see if I've broken all the changes. And I've talked to companies who spend millions and millions of dollars of building a data lake and a data warehouse, and they have thousands of reports, and they have no idea if they work, they have no idea if they're updated, and they have no idea if they change anything in the database, they think the world's going to break, and they just wait for their customers to find the errors. And I just don't find that an acceptable situation. And so the way that works in our product is I'm going to go into a different kitchen here. I'm going to call this home office kitchen.

And so there's a home office recipe. And so this is the one that actually does both. And so we've got this feature in our software that we talked about called an ingredient. And an ingredient is a way to encapsulate a recipe that is perhaps owned by a different team or owned by your own team and run. And so this home office ingredient, if I double click on it, I can actually see that this one does something interesting.

It takes some data, creates a star schema using SSIS, which is Microsoft's older data integration technology, runs some tests, and then segments some customers using Python. And so this is a very simple one that the home office owns and runs, and of course, like all of our systems, they test. And so, but we go back to it and say, "Well, if I made a change, how can I tell if that change works?" Well, there's an ingredient that's owned by the local office, and in this case, they're actually running an Alteryx workflow, adding some data in, and then actually publishing a Tableau data source and publishing a Tableau report. And so both of these are owned and run independently, but they're coordinated by a recipe that happens above them.

And so this idea of an ingredient that encapsulates recipes allows local control, but a central view, I think, is essential to solving this kind of collaboration problem in companies. And so, we've talked through kind of three processes in DataOps. One is the production, where you want to run with low errors. One is deployment, being able to get ideas into production fast.

And then one is collaboration and two types of that, inter and intra-team. So the last thing that we want to talk about is really measurement. And so, you'd think that people who run or work in data and analytic teams would be interested in data. But a lot of organizations aren't very interested in the process data, the

00:35:00

process analytics. Like how fast are you deploying, how many errors you have, how productive is your team? You would think these are important metrics. And so these kind of metrics are what we get out of our software. And so we've got a part of our process analytics module kind of focuses on two areas.

One is sort of production metrics, looking at data sources, the history of the running, the timing results, the test results, model metrics to make sure that as the machine is running day in and day out, where some organizations have not two pipelines in production, but hundreds or thousands of pipelines in production, that everything's working.

And that system throws off data that's worthwhile. And then the second one is I want to show you is really about productivity metrics, and how do you know that your team's working well? So if I go into the report section of my product and I look at this report called the project dashboard. And so this project dashboard is kind of talking about what a specific team does. And one of the most important things that a team does starting in the lower left is are they delivering on time? Have they met their SLA?

And here we're tracking, for a week in June, are they late? And then are they on time? So that's good and sort of well, is what they're giving correct? And so here we're looking at these red bars over time. How many major errors did they have? People called up, things were wrong, things were late, and we can see that going down, but we also see the inverse, that their number of automated tasks is going up and monitors is going up.

And that's a good thing, right? Because that enables them to actually make sure that they have no problems. And another benefit of the testing is actually the productivity the team has, the number of deploys that they're making from a feature branch to development, from dev to production, and we're tracking that. And then even looking at kind of more qualitative characteristics. What is the number of tests they have?

What's the ratio of the tests to each node? And so, if you look at it from a manager standpoint, you want to make sure your team is productive, you want to make sure that they're satisfying their customer needs and make sure that they're not creating problems. And so this is a way to look at, analytically, the work of your team.

And also it can help drive DataOps adoption, because partly to enable change, to enable that transformation, you've got to actually keep track of data and convince people that this idea of DataOps, that you can iterate quickly with low errors in a collaborative way, is actually possible. And so, just to kind of end up and kind of to finish up the demo. So one way to look at these processes that happen in data and analytics is to think of them as a graphic equalizer. And most organizations kind of have their cycle time to deploy is measured in weeks or months.

The amount of errors they have is just unacceptable. And they may not even know because they're sort of hoping things are right. They spend a lot of time in meetings, they produce a lot of documentation, but they're just frustrated. And they have no ability to measure their processes. And the result of that, actually, is that they have very high costs in productivity. And there's a huge failure rate in data and analytic projects. Gartner estimates upwards of 50%.

And we've invested a lot over the last years in machine learning and data science and big data and streaming data, and we've got a lot of cool two, three, four-letter acronyms that are out there, but we're still having a lot of cost and productivity, and the move to cloud is actually making it more complicated. And you actually end up talking to business customers.

You find out that, and like NewVantage Partner did a report recently that showed the lack of success and the unhappiness that your business partners have with how your data and analytic team's doing. And so I don't think that has to be the case. I think best-in-class organizations are realizing that a new database doesn't really matter, and chasing the latest algorithm, while nice, doesn't really move the needle. You need to actually take this graphic equalizer and push it up, focus on, and that it is possible to go fast and not break things and automate the working relationship for less meetings with your team.

And best-in-class organizations do all these things well, and the end business result is that you can actually do more work. Your team is more productive, and that your customer satisfaction goes way up because they can ask you questions and you can give them follow-up answers, and they can trust the data. And so that's what we've seen from our customers with DataOps and why we built the product.

And we've been talking about this for years, and in addition to our product, we've also got a lot of good information about what DataOps is and DataKitchen itself. So that's the end of my demo, and again, I apologize for my network glitch. Beth, is there any questions that came up? Yes. So we'll take as many questions as we can in the next 10 minutes.

So if anyone has any, just send them in the control panel and we'll go through those. So, our first one is, what

00:40:00

would be the approach if existing pipelines used Airflow for data orchestration? Would it be to replace them with DataKitchen and build an overarching orchestration, or keep them and build DataKitchen around those? Yeah, I think it could be either/or. Yeah, but I'm a Python programmer, so I actually like Airflow, and so I tend to think that there's a lot of jobs that people do.

And some of those jobs, the data job is nice to be done by Airflow. Maybe you want to write Python programs and do your work in the Python process space. Other people like to use SQL, other people like to use Airflow, but that's, again, only part of the picture. You've still got predictive models, you still got data visualization, you still got data governance, data security that are all applied, and all those things should be deployed as a unit. And so, that sort of meta orchestration, orchestrating all the tools that do the work with data, whether it's Airflow or data science or data visualization, is what DataKitchen specializes in.

And then laying it with tests. So, if you're doing Airflow, there is a DAG in Airflow and there is a DAG in DataKitchen. We actually think of ourselves less as a DAG, but more as a DAG of DAGs. And that working with all these tools is what enables you to do DataOps.

Great. Thanks, Chris. So we actually have two questions about BigQuery. So is BigQuery or Databricks a lower layer for DataKitchen, and are we compatible with BigQuery? Yeah. So we have an integration with both those tools. And so, BigQuery is a fast database, kind of like Snowflake or Redshift, where you may store your data and transform your data, so we actually can work with BigQuery, and we have a direct connection to it.

And so it is one of those tools that we meta orchestrate. Likewise, Databricks is an interesting thing because it's a database itself, but it also has a great set of tools to do data science and to do what they call MLOps, which is a way to track experiments, and we're compatible with all those.

And so I think the idea here is you may do your work on Google Cloud, right? And it may be on BigQuery, and then the data may be exported into Databricks to run your data science model. And that may end up in Looker because Google bought Looker. And so you've got all these tools that you're working with in addition to a bunch of subsidiary tools. ... that you have to do.

And so do you want to do the work to actually integrate that tool chain to build DataOps? And so we think that that is something that is a value add that we can bring, and we can actually kind of be the superstructure around all those tools in Google Cloud to enable you to do DataOps.

Okay, great. Thanks, Chris. Now, how do you think about how granular a node could be? For example, you could have an ETL node, but that could contain dozens of transformation nodes. Is it better to have granular tests or high-level tests?

I think I'm more of a fan of having the nodes be of a smaller grain and trying to find out the problem sooner rather than saying, "Okay, we've got an Informatica set of jobs, and this node calls 30 jobs," and you don't find out if there's any problems until all the 30 jobs are done.

I think the sooner you find out that there's problems in the data in production, the better. And so testing before is fine. So let's say you've got an existing set of Informatica jobs and 30 of them that run. So one way to adapt DataKitchen, and this is perfectly fine, run some tests before those 30, run some tests afterwards. That's better than no tests, and a lot of organizations have no tests or monitors in production.

I think it's better if you sort of break the grain down and say, okay, maybe you break those 30 up into six nodes, and maybe two of them are in parallel, and test in each one of those. And that way, you find out if the problem sooner. And maybe those 30 Informatica jobs take four hours to run. You don't want to wait four hours for it to work. If it takes four seconds to run, then, well, put the tests on the outside. So the rule of thumb here is think of what you're doing as an assembly line.

Find out sooner rather than later where the problems are. And if you need to sort of disentangle the work that you do into multiple nodes, that may be a better thing to do. And if you're in a hurry, put all of them in one node and some tests in front and tests in the back, and both are work.

But getting the grain down, I think, is an important sort of operational metric that can help you both in development and, of course, in production. Okay. Here's another question on testing. It's actually from Steve Emerick, who I think knows you from the past, Chris. Hi, Steve. He says hi. So given the theoretical existence of literally millions of possible data

00:45:00

quality tests in your methodology, how do you distinguish between data quality problem discovery for problems that exist in the data pipeline versus regression test creation? How do you support the discovery process? Yeah, I think Steve, that's a good question because as Steve knows, I come from a software development background, and software's got the same problem. It's sort of combinatorially complex.

How do you test a three-tier web application in a user interface? Because people could go in lots of different places, and you can't possibly test every specific combination because it's too much. And so you end up having to build scenarios and functional tests and end-to-end tests that reflect all the scenarios. And software engineers have gotten good at sort of writing those tests, and knowledge of the application and the use case matters.

And so I think that's very similar in data and analytics. Plus, there's another spin on it, right? Because you're not building just an application, you're also integrating data, and the data may not actually support what you want it to do. So you've got this double-sided problem. And so I think to me, the answer is that iterations sort of trump all. Being able to get something out in front of your customer in a week with one test, and then the second week adding 10 tests, and the second week adding 30 tests, I think is actually a great way to go about it and incrementally building the tests.

And we don't have any magic that automatically creates your tests for you, but what we found over the years is there's patterns to tests that we've written about, the statistical process and location balance tests, and they also sort of represent your domain and your understanding of the data. So I think testing's important. I think testing is an activity that your data engineers and data scientists should be involved in, in addition to your test automation.

And doing manual tests and sort of putting them in a test system and then forgetting about them is probably the worst possible world because you run your tests in development and check for regression and put them up on a shelf and never use them again. So overall, automated tests that run in production, automated tests that run in development, the combinatorial complexity of creating tests is something that is part of the work that you need to do as a data engineer or data scientist.

Great. Can you give us an example of a client using a system that consumes data, but you couldn't integrate within a kitchen? If so, what were the reasons, and what kind of process would you suggest in those cases? Yeah. In general, we kind of don't have a lot of problems integrating to tools because we're not a data tool. We talk to the tools that act on data. And so whether your data happens to be in flat files or happens to be in a Mongo database or a rows and columns database, we're talking to the tools that talk to the data.

And so, in general, the way that we talk to those tools is we kind of say, "Here, imagine the tool as a black box." And we say, "There's something that configures or code that runs that tool." And maybe it's an ETL engine, or maybe it's a C compiler, or maybe it's the Java or Python virtual machine, but we're saying, "Go run this." And then we're monitoring it, pulling log files out, and getting test data back out of that. And so we treat these systems kind of as boxes.

And so we haven't really run one system where, since we're not actually touching the data, where the data goes in. And so perhaps there is one that we'll run into What we have found is that there's some tools that actually don't support branch and merging very well. Like for instance, your Tableau workbook is kind of a big hunk of XML, and they've even put a hash on it now that makes it used to be able to edit the XML, and now you can't.

And so in that case, merging is more like kill and fill than actual merge. Which you could do with, for instance, SQL code or other code. And so some tools are better

supporting the idea of having Git and branching and merging versus others.

Okay. Well, we have two minutes left and one last question here, so that's good timing. Do you have experience with trunk-based development with DataOps and a lack of a dev test environment?

Yes. Yeah, I think anyone who's done data work has known that there's the quick patch to production, right? And

a lot of times you have your weekly release, and you find out that there's a problem during production. Maybe you have a five or six-hour build process, and halfway through, you get an email, and you're like, "Oh, man, this data file, it didn't show up. Or it showed up and it's just wrong, and I want to patch in last week's version of this data file in order to get the job done." And maybe you get the clearance from your business sponsor to do that. And so that is sort of making edits in production. And one of the benefits of this approach is you've got something running in production, you can quickly create a development environment, make the change, and deploy it back into production.

00:50:00

Now, some cases, you may not want to run all the tasks to be able to do that. Maybe you just want to make the change and that's fine. But the important part is that patch that you may do, you may want to keep it around for the next week, or you may want to remove it for the next week.

So patching running systems as well as a more formal development and process where you've got a separate environment is something that of course we handle. And is it recommended? Well, that's a different question, right? In general, one of the things that's hard when you're doing data work is to avoid hope and heroism. And the kind of heroism of editing production shouldn't be done very often, but there are cases where it's okay to do.

And that's sort of your judgment call. Likewise, if you're going to hope that something works and not know it, there's probably cases. But in general, you should know things that work, and you shouldn't be a hero 98% of the time. And those 2% where you want to have hope and heroism, think hard about it.

Well, I think that's a great concluding statement for the webinar today. So we're out of time. Thanks so much for giving us that demo today. I hope everyone found it really helpful, Chris, and thanks to all the attendees for joining us. If you have a question and we didn't get to it or you think of something later, please feel free to reach out to Chris or I, and we'll make sure that we get those questions answered.

We'll be sending out a recording of this webinar and the slides probably within the next 24 hours or so. So be on the lookout for that in your email. So I think that's it for today. I hope everyone has a great afternoon. Thank you very much. Bye.