How to Build a Successful Cloud DataOps Program

00:00:00

And even good evening to some of you. Thanks for joining us today. My name's Beth Befferly. I'm the VP of marketing at DataKitchen, and I'll be the host. Our topic today is how to build a successful cloud DataOps program. And many of you may already know our speaker. Many of you have attended a lot of our other webinars, but we have Chris Bergh with us today.

He's the founder and head chef at DataKitchen. He's a leader of the DataOps movement. He has more than 25 years of research, software engineering, data analytics, and executive management experience. At various points in his career, he's been a COO, CTO, VP, and director of engineering. He's also the co-author of "The DataOps Cookbook" and "The DataOps Manifesto," and a regular speaker on DataOps at many conferences.

So before I hand it over to Chris, just a few housekeeping items. This webinar is being recorded. We'll send out an email with the recording and the slides to all participants shortly after the webinar, so be on the lookout for that within the next 24 hours or so. Also, you're all on mute, so we'll use the last 15 minutes of the webinar to answer questions.

If you have a question during the course of the webinar, just enter it into the question box on the control panel and we'll collect all these and answer them during the Q&A session at the end. So, that covers all the housekeeping. So without further ado, you can take it away, Chris. Hi, everybody. So, as Beth said, I'm Chris Bergh.

Thank you for attending, and here's what I look like. And now I'm going to turn that off because it's more enjoyable to look at the slides. So we're going to talk about something called cloud DataOps, or how to do DataOps in the cloud. And we're going to have sort of four areas that we're going to focus on today.

And the first is kind of just talking about what it means to work in the cloud. And I'm going to use a term called tool chains, which are all the different tools that the cloud providers provide, and in fact, beyond the cloud vendors, and how you stitch together systems that build and deliver data and analytics for your customers.

And it's very interesting. It's also gotten very complicated. And then we're going to talk a little bit more conceptually about sort of why taking all those tools that they have, the DevOps and workflow tools, and sort of stitching them all together doesn't quite work and will fail at doing DataOps. And we're going to talk about an example of that.

And then we're going to talk through the idea of a DataOps superstructure as a solution to that. And then I'm going to go through one example. And so, as everyone knows, in the past sort of five, seven years, all the big three data providers, or the big three cloud providers, AWS, which has got the biggest share, and then Azure and GCP, have a set of tools that you can use to take data and put it in a database for analysis.

And first, it usually goes into a bucket store. Maybe it streams into a bucket store, then it gets into a database, and a lot of those databases are tuned for fast queries and columnar representations. There are tools to actually transform data, tools to visualize data, tools to run models on the data, and they're all really interesting.

And so, for instance, just if I want to put data in an analytic database, right, you've got

Azure Synapse in Azure, you've got Redshift, you've got Google's BigQuery, you've got different bucket stores. And if I want to transform data, you've got things like Data Prep or Glue, and other tools that you want to do. And so, in essence, what I think the cloud providers have built is that you can, in essence, build your whole data and analytics stack on their platform.

And now, people may plug in different tools. So for instance, Snowflake's got quite popular in different cloud providers. But there's a complex set of tools that you need to work with in order to actually do any sort of data and analytics in the cloud. And so, some examples from Azure are like, if you sort of go and Google the web, like I did, and say, "Well, what's an Azure data pattern?" Well, you look and people ingest it, they get it into Azure Data Factory.

They get it into Azure Data Lake Storage. Sometimes it gets into Databricks for training. Sometimes it's in something called Cosmos DB, which is a fast read for a web application. It does Synapse, which does algorithms on top of it. And there's different ways that people have put these patterns together. Sometimes it's Power BI, sometimes it's not.

Another example is here's one where it goes in and it's ingested in a blob storage. There's Data Factory, there's Analysis Service, there's Power BI. And here's an example of someone how put together all the pieces in a presentation where they tried to apply sort of agile principles to do DevOps, and Azure and other ones have a full suite of DevOps

00:05:00

tools. And so if you look at it in more of an abstract way and start thinking, "Okay, I've got to put together a data and analytics system." And on the left-hand column here are kind of the components that you would use. So for instance, there's an analytic database or a blob store, a streaming tool, an ETL, ELT tool, a data science tool, a BI tool, a data governance tool. And so let's say you have your company or you prefer the AWS stack.

Well, if I'm going to build everything in AWS, well, I'm going to use Redshift as a database. I'm going to put the files before it gets into Redshift into S3. I'm going to stream with Kinesis. I'm going to do my ELT with Glue. I'm going to use SageMaker to do data science. I'm going to use QuickSight to do BI, and then there's a Glue Data Catalog.

And okay, there's another set of tools in Azure or another set of tools in GCP, and they have different names and different tools, and sometimes they exist or not. And then if you go kind of more traditionally, so for instance, a partner of ours, IBM, it's got Cloud Pak for Data, and they have exactly the same set of tools that can be put into a- Into a Kubernetes cluster. And traditionally, you find in bigger companies, you find the fifth column is sort of on-prem.

You find Teradata as a database, maybe Kafka and Informatica and SAS and Tableau and Alation. And then there's actually even an open-source version of all this, right? You could grab a free version of a database. You could use Kafka and Airflow and Jupyter Notebooks, do your data science, Superset, Apache Atlas for your data governance tool.

So these tool chains exist, and then I'm not even going into the entire sort of free market version of this. And I think everyone knows that if I want to go in and look at a different data integration tool, there's all sorts of interesting things out there. There's tools like DBT and existing-- There's a whole bunch of different tools to do ETL, like Pentaho, as well as data science. And then there's the versions of these tools that split up because some of these tools are more technical, like data science and Databricks, and then there's tools that are less technical. So for instance, you could use to do data prep or data science, you could use

some of these self-service data prep tools like Alteryx. And so there's just a great variety of tools that you have to put together in order to do DataOps. And then there's also some supporting tools that you need to do. So for instance, if you're a believer in doing DataOps, you need to store your work as code, and that goes in Git.

And then there's something called the Secret Store, where you should store all your logins so they're in an encrypted store. If you want to run queries across the clusters of machines, well, there's different types of K8s, Kubernetes. There's a different Kubernetes cluster. There's EKS and AZS and GKE and OpenShift and just vanilla Kubernetes.

And then a single sign-on provider. Each directory has a way to manage sign-on across each one of these machines. And so I think there's just a bunch of supporting tools that you want to be able to put together. And so if you think about it, if I'm going to try to start doing DataOps, I've got to kind of pick my stack, right? If I'm going to do analytics, I'm on Azure or GCP, I've got to decide what my analytic toolchain is, and then I've got to put together all these tools.

And that sort of makes it, I think, hard because you're assembling all these components, and you're doing this giant system integration task. And that may be fine for the consultants and teams who want to put it together. But if you're a believer in DataOps, and we didn't talk too much about what DataOps is in this, but really the process of iterating and deploying data and analytics in a more agile manner, you've got a lot of challenges, and here I kind of outlined some. And I think the simple term is it's just a lot of assembly required. You've got this powerful collection of data tools, and there's no system that makes all those data tools together.

It's about integrating all these tools into a whole, and it's a bit overwhelming to design a solution in this world that works and follows the DataOps principles that we've laid out in other discussions, mainly just because of the complexity and stitching together a solution. And what we're suggesting here and what we're going to talk about next time is sort of why you need a superstructure that lies over all these tools to make it easier and how it's very difficult, and in fact, almost a software development task to put all these things, all these pieces together.

And so we're going to use the term a DataOps superstructure as a way to talk about it. And so that's the first part is, yeah, I guess the cloud is complicated. And in fact, data analytics has kind of always been complicated, right? There's always been different databases and tools, even 15, 20 years ago.

And now the question is, if you want to go fast, if you want to not break things, and you want to follow sort of the DataOps principles we've laid out in the manifesto

00:10:00

in the book, well, why can't I just use DevOps tools and workflow tools, and why do those fail? And so why can't I just use what's there? Why do I need more? And so what we're going to go through and talk and really is about why there's sort of six principles in this next part of the presentation.

And so the first one is you need an end-to-end, something called a meta-orchestrated production-type pipeline. And so we're going to use that term meta-orchestration quite a bit, which may sound weird to some people, but we'll explain. And the next thing is really it isn't about DevOps CI and CD. We're going to invent a really awful acronym called DataOps CSMOIDM, and why CI and CD just doesn't capture the complexity of what you need to do in DataOps. And we're going to talk about the lack of an environment pipeline, the complex team and data center coordination, a common system and vocabulary, and process measurement, and about how to help your teams change to DataOps. So I'm going to walk through these sort of six comparisons about why kind of, given the toolchain of data and analytic tools that exist in Azure or exist in GCP, why sort of putting them together to sort of build your own DataOps solution is very, very difficult and some of the things that you need to think about.

And so the first is, as we've talked quite a bit and we had a webinar on it last year, is the process of doing data and analytics is really an assembly line process. And if you look at all those tools I laid out in the last one, whether it's a database or a data pipeline or a data science tool or a data visualization tool, the metaphor that we talk about in DataOps is that it's an assembly line.

And as data goes through that system, whether it's big or small, or fast or slow, or structured or unstructured, you're kind of assembling data in all those components. And so you may think, "Wow, those are a bunch of tools. I can just use a tool like Control-M or some other workflow tool to stitch that all together." And so that may be true, but I think one of the things is that you really want to have kind of-- and think of it as an intelligent pipeline or a test-informed end-to-end pipeline that orchestrates all these pieces. And in some ways, we I have used the term meta-orchestration because within each, a data pipeline, or a data science pipeline, or even data visualization, steps are happening. Sub-steps are happening.

You need to coordinate the steps and the sub-steps together, and you need to do that in an intelligent way. And you need to do that, that integrates production testing and monitoring and alerting. And also, you just need a set of connectors. You got to connect all these tools, and that in and of itself is a pain. Integrating with your favorite data science tool, your favorite ELT tool, your viz tool to be able to do this, this sort of meta-orchestration, or sometimes we called it a DAG of DAGs, a directed acyclic graph of directed acyclic graphs.

And so, the idea here is the assembly line's complicated because you've got a lot of tools, and if you're going to try to run it like a factory, you need to integrate to all those tools and make it work. And while workflow tools like Airflow or Control-M have similarities in concept, they're just missing several features, the sort of meta-orchestration or connectors or the sort of intelligent test-informed way of running.

And so, the next one is CI and CD versus C-SMOIDTM. And so I'm laughing because, first of all, it's a terrible acronym. But of course, I made it up. So, in DevOps, in software, there's this idea of continuous integration and deployment. And integration is the part where if I'm working on a big software project, I make a change, and then I'm going to integrate it with everyone else's change.

And then I want to know quickly if I've made that change, if it's affected anyone else. And then the deployment means if when I make that change and the integration happens, can it get into production automatically? And so continuous integration is different than continuous deployment in the world of DevOps. And those things are true. The role of integration testing is not often done in a very robust way in data and analytics, but it has a role. Not as complicated a role as in software, but it does have a role.

And I think it's a little bit bigger in terms of when you think about DataOps. So the first is that this idea of a sandbox or a development environment is a lot more complicated in data and analytics. So you need kind of sort of continuous self-service sandbox. And then the part that we've just talked about is you need meta-orchestration.

You need to continuously and intelligently orchestrate across all those pieces with that sort of test informed. And then you need to do like you do in development, you need to do continuous

00:15:00

integration and deployment. But then once in development and once in production, you need to do continuous testing and monitoring. So I took all those Cs and then the S-M-O-I-D-T-M and made one acronym, and I know it's terrible, but I think it tries to express that there's just more going on than deployment here in the DataOps processing. And so the missing really capability that these tools don't have is this ability to orchestrate and monitor and test these pipelines, the development and the production pipelines. And if you think about CI and CD tools like Jenkins and Azure, they don't cover the whole process, and they're really focused on software development tools, on not data and analytic tool chains.

And this idea of the duality of tasks and integrating, because really the core idea here is you need to prove that your end-to-end pipeline works in development before you get it into production. And the core idea behind that is you need to orchestrate all the tools. You need to have the relevant branch of the source code that you work on. You need to have a test data, and of course, you need to have tests. And all those things have to run together.

And so that makes it very hard to do. In fact, you have to build a whole bunch of stuff in Jenkins and Azure to make that happen. And then you can sort of see a link to a webinar where we talked about this idea in more detail. And that kind of gets to the third principle, is that putting together an environment, a sandbox, a place to work is actually quite difficult in data and analytics because of all those pieces that you have to put together.

Because then you've got all the tools themselves, and so whether it's an ETL tool or a data science tool, and you want to have a development version and the right libraries, et cetera. And then you've got the source code that's running in each one of those tools. And then you've got the test data against it.

And so all that people of software and configurations and data and hardware and access controls all have to be worked together in order for it to do it. And those are all different systems and have different capabilities. And so if you really want to give someone the ability to actually develop something on their own and create a sandbox or a development environment or a DevBox, and the cloud is actually-- What's great about the cloud is its ability to do things that are done ephemerally.

You can spin up environments and shut down environments. And it's been built on a different principle than

most installed software, in that everything has an API, and therefore you can use that API to configure it, and you can kind of do what's called infrastructure as code. You can actually code up your infrastructure, and then you can code up-- What we're saying is you code up your infrastructure and call it, you code up your test data, you code up the parameters that call it, you code up the source code and pull all those things together, and that creates your environment.

And these ephemeral environments that you can kind of create and dispose of are actually really exciting because not only that you can do them in development, you can actually run parallel versions of production environments. And that actually makes a really exciting way to make your customer successful because the whole idea of-- One of the important ideas of DataOps is that you're delivering value to your customer quickly, and that value in the eyes of your customer is really the key success criteria.

And so how do I get something to my customer if it's a bit risky? Well, maybe I just code up a different version of my environment, and instead of having 100 users on it, I give a version of this environment for a week to one of my end customers to see if it makes sense to them in a development way, and then they can look at it and give you feedback.

And so the ability to copy and paste your entire analytic platform saying, "I've got this running, and I want to copy and paste another version of it, and give it to one person with a couple of new tweaks, and copy and paste a different version of it with a few other tweaks." It's really important.

It helps the team understand what customers want because you're giving them versions of what's in production. And that sort of infrastructure has really been enabled by the cloud in a way that you could do before with virtualization, but it was seldom done. And so, one of the things that you think about is that there's DevOps infrastructure tools like Puppet and Chef and Ansible, and those are great. But what they don't do is, they're sort of developer-level scripting tools, and they don't do things like test data management. It's not done where people in a self-service way can go do it.

They don't integrate all the version control and the tool chains. And so this idea of having a self-service analytic development environment that could even go off to your BI users or your data science users who may not be interested in working with the infrastructure, I think is a very important idea. And we've got a whole webinar that talks about

00:20:00

that. And so the third thing, in addition to the differences between deployment, the investments in production, and the environment pipelines. And so last year, we've got three hour-long presentations that go into details on each one of these pipelines.

So if I go to the fourth part and I say, well, one of the interesting things is that in data and analytics, there's lots of complexity. And most of the people that we talk to are in some way, shape, or form transitioning to the cloud. So they've got some stuff on-prem, and they've got some stuff in the cloud, and maybe they have two clouds, or maybe they want to be cloud agnostic, but they're in some transition. And in some ways, the teams that work may be different.

There may be an on-prem team, there may be a cloud team, maybe they're working, maybe they're the same, maybe not, and the tool chains are different, right? So they may use Informatica on-prem, but Azure Data Factory in the cloud. And so how do you get this-- when you're in this mode of trying to transition and be able to work half a foot in one world and half a foot in the other, your analytic users don't care that half the processing was done on-prem and half the processing was done on the cloud. And you don't want to spend a year rebuilding everything in the cloud from scratch because you know that if you spend a year with no customer input, either your budget's going to get cut or you're going to end up with a big pile of complexity that you can't handle.

And so, what we think is that this DataOps capability to sit across all these systems, to be kind of the Switzerland or the hub and the spoke of all this world. And you've got all these disparate tools on-prem and in cloud, and you're doing analytics in both places, but you want to be able to make sure that your customers know that they're right.

And so I think that's a really important capability that could sort of extend beyond the cloud, because transitioning to the cloud in some ways is harder. And in my experience, recreating or rebuilding things that already exist are often hard in data analytics because of the hidden business logic that's not often well-documented, and you have to end up recreating when you move it from one platform to the other.

And that may be obviated or may be removed by the need when you lift and shift things, but it's still a complex approach and you're still running in two places. And that may also look like another case where this sort of complex and team coordination happens, is that there may be two teams where you have a home office and a self-service team, where a lot of organizations have decided that their centralized data team is really a data enablement platform or a data lake platform, a data lake house where there's groups of people using tools like Alteryx and Tableau and Looker and Trifacta being very close to the customer, doing a lot of good data and analytics work.

But the stuff that's at the enterprise level or the centralized level, those two teams have to work together. And so none of the cloud platforms really have this ability to orchestrate across teams and that comes from the ability to create components and having in this diagram, there's sort of a component that wraps up what the home office does and a component that lacks what the local office does, and being able to call those independently with sort of local control, but centralized guidance. And so I think these things also go to play with sort of sandboxes for self-service, and I think a lot of organizations have ended up kind of throwing up their hands or maybe hiding their head in the sand saying, "Okay, this is our way. We've got a data enablement team.

And yeah, I know there's a lot of work happening by the people out in the different branches or different lines of business and somehow it's all going to work." And then they end up with thousands. Then they end up with sort of building a set of complexity that it's very hard to get out of.

And so, you want to think about how to coordinate this work because your self-service team is still just doing-- They have self-service tools, but they're still creating code or configuration and complexity almost at the same degree of complexity that your centralized team are.

And so, the fifth case is sort of a common system and vocabulary. Now, if you think about all the tools that you do and with data science and data engineering, you've got five, seven tools, 10 tools that you're working on. And then if you actually want to apply sort of these ideas of DevOps and infrastructure as code, you've got another half dozen of tools.

And so software engineers like that complexity, right? They like writing scripts and they think about, they get excited, "I've learned a new tool, and I can put a new tool on my resume." Most people who do data and analytics are not interested, particularly in all the tools that it work. They maybe have one tool that their expertise on, but they are really interested in

00:25:00

getting value from data, not in the infrastructure and the toolage to run it. And so I just fundamentally believe that software engineers and data scientists and data engineers, maybe we all sort of took the same courses at undergraduates, but just have a different view on life. And I think you need a simple, kind of a user experience that spans this whole process, that stitches together all these multiple tools.

And if you're trying to put together sort of seven to 10 tools and DevOps and version controls, it's very quite hard. And if you look at even applying the ideas of ops on top of it, you've got DevOps and DataOps, and the market's talking about ModelOps and AnalyticOps. We're recreating the problem again, on top of all the different tools that you use to data, where some would argue, and some of the analysts argue that you need lots of ops systems and that ModelOps is different than DataOps, is different than Self-Service Ops and DataSecOps.

And really, I think our perspective at DataKitchen is that by having a sort of, system level view across all these roles, data governance, data engineering, data science, and a common platform that does all the operation functions on top of all your tools is the way to go. And getting that sort of system level view and management process, is very difficult. And, in fact, most teams, they don't even try.

They end up saying, okay, if they are a believer in the DataOps principles, they're just going to focus on data engineers alone, and they're going to forget everyone else. And so, our belief is that what is needed for DataOps is kind of a more straightforward set of abstractions and a user experience kind of to deal with that complexity.

You want the machine to handle that complexity and not your users, because at the end of the day, you want the concepts that we've talked about to be self-service, to be intelligent, and especially for users who are increasingly just want to focus on business value. And so, you've got a lot of people, a lot of data and analytics work done in the lines of business where they're using self-service tools and they don't want to learn the details of AWS. And in fact, most data engineers don't want to learn the details of AWS.

And if you want people to work in an agile way, in an iterative way, getting a single tool that manages that complexity we think is essential. And it can also drive just the adoption of DataOps principles. And so the last part is, we've talked at a number of times about the work that you need to do to help your organization change from being kind of a waterfall or, sort of a hope and heroism-based organization and the transformation that needs to take place and attitudes. We've talked about the ideas.

We've talked about how to transform your organization, and we've given webinars and blog posts, et cetera. And so one of the things that I've noticed is that, people who've been in data and analytics have a very central and core disbelief that can you actually make changes to a running system quickly and not have problems? And for people who've been in the industry for a while, that belief is pretty ingrained and they say, "No, if I got something working, I'm not going to touch it." Right?

And, the fact that what DataOps is saying is you've got something working, get it done fast, and then keep touching it, keep changing it, always changing it, because you're always trying to get something in front of your business customer to see if it's right or not. And so that goes against the grain of a lot of people.

And one of the ways that you can convince people, is with demonstration projects. But another way is, data people are informed by data. And so if you start measuring the processes that you're using and start saying, "Look, we have been able to make these changes to our production system quickly, and look, our number of errors are going down and look, our number of tests are going up.

Look, our models, we can tell when our models aren't predictive." We can actually understand as the data's flowing in, the source data, the aggregated and integrated systems that are being built off the data that they're actually ready and we can get metrics on that. And I think those metrics are, number one, can help you as a leader run your team, in a better way.

But number two, help the people who are sort of from Missouri on this idea of DataOps, be able to say, "This is working. Look, I can see it in a chart. I can see it in a dashboard." And so this process measurement, I think is key to change and as a data and analytic person, running your organization in a data and analytic way, I think is an important thing to think about.

And so we've talked a lot about sort of why sort of DevOps and workflow tools sort of fail at DataOps. We talked about the complexities of toolage in the cloud and in fact, outside the cloud. And so,

00:30:00

I hope that these set of arguments have taken hold and so what would you need to solve this? And so we've been talking about this term of a superstructure and as a way to sort of stitch together all these components. And I think, the challenges are that if you look at how people have tried to work and build and work in an iterative way, they just end up having running into these same sets of problems, right?

Number one is we find that they just have a lot of errors. The data's wrong, they're late, people are fearing going into work, or they just to make the whole system work, there's manual steps and people are taking time. Sometimes there's several staff just involved in manually executing these steps to make things work.

And those are very expensive resources just to sort of follow a flowchart and click buttons and check those things right. And then we talked about the ability of the importance of environments, the importance of coordination. And really the idea here is if you can put new ideas into production, a new dashboard, a new data set, a new model, and get feedback from your customers quickly, you end up actually doing more work with less effort. And that's the sort of magic of agility, right? Because you are putting smaller things in front of your customer, you're learning more, and then you're also learning more about what you don't have to do. And if one of the magics of working in DataOps is that you get a project from your customer, you think that you're going to have to do 10 things, but by the time you've gotten the fourth thing, you realize that, oh, I only have to do five things.

And those other things I thought I needed to do, well, they're not important. And so that's where you end up saving time. And so this is the set of constraints that we see for people who are trying to do DataOps in the cloud, or attempting to do DataOps cloud, or thinking about it. And so from our perspective, and one of the reasons we built the product is this idea of a superstructure across all these tools, and that have been provided by the cloud and for a fact have traditionally been provided on-prem. And our software has something called a recipe, which it does the sort of meta-orchestration across all the tools.

It does automated testing in production and development across on-prem and in cloud. It does sandbox management. It has the sort of common vocabulary around kitchens and recipes that makes it simpler for people to understand the complexity of the workflows that they're dealing with. And then finally, it's sort of got built-in measurements and history to be able to make this happen. And so we've thought pretty deeply, given our long experience in working in data analytics, both in the cloud and on-prem, with teams across who do every role in analytics. And if you're going to do DataOps, you're going to end up having to build something like this superstructure, right? And so the argument that we're making is why build it? Why not rent it? Or why not buy it?

Because that superstructure is going to get you faster to be able to solve the real challenge, which is how do you actually do DataOps in production? How do you actually get your team to work in an iterative way? And then how do you really deliver value to your customers? Because that's the real end game for everyone.

And so we've talked a lot about, in different webinars, about what our product does. And so it does this orchestration and testing, environment management, kind of deployment and collaboration and measurement. And one way is to kind of look at this from an example standpoint. And so here's an example of a system on Azure. And so let's say you're getting data from Salesforce and maybe some SQL database and maybe some on-prem database, and you're getting it into either streaming or in batch into Azure Data Lake Storage. And you're using Data Factory to kind of integrate the data. Maybe you're pushing it into Databricks or Synapse.

Maybe you're putting it into Cosmos for Fast Query or Power BI. And there's all these steps and tools that you have to use, because remember, there could be problems that happen every way. And so our view is that you need the superstructure from DataKitchen across all these steps, and that by having this superstructure, you're going to be able to deliver changes to Data Factory, to ADSL, to Cosmos quicker with less errors, and therefore you're going to be able to get more done by investing in this type of automation.

And here's another way to think about it, and this is from an architectural standpoint. And so a lot of times when I see-- I'm a technical guy, and I see diagrams like this, that here's our data architecture. I think it's also important to talk about how you architect for change. And so if I look at the architecture of the system and I want to make a change to it, I want to take a piece of it, and how do I take a change to each one of those?

00:35:00

Well, I've got to create a branch, I've got to create an environment, I've got to deploy it. And so you've got another set of tools to do that. And then there's a whole set of enforcements of what it means to have a good test, what it means to deploy. And so the temptation is that these pieces that are sort of missing from the equation, the automated testing, the meta-orchestration that we built on purpose, because we had to, because we were running our own data and analytics shop and just trying to adapt the software tools was frustrating, if not impossible.

And so we had to build it. And if you look at this same picture from a software architecture standpoint, you can see all the tools in the production environment, but also those tools are in development and test environment. And so there are in Azure, you've got to put together all these pieces, their directory, the secrets, their version control, their deployment tools, all their tools, and you've got to, in essence, build all this on your own. And so that's our argument here, is that there's a bunch of things that you need to do if you're going to pull all these pieces together and kind of looking at it from more of a feature and function standpoint.

So there's a bit of an eye chart that I'm going to walk through. So the first is this idea of meta-orchestration or an end-to-end pipeline, and it's a DAG of DAGs, not just a DAG. And so you may have a single tool that does things. Maybe you've got ADF or Airflow or Glue or Control-M.

But it doesn't do meta-orchestration. It doesn't have built-in connectors to your tool chain. It doesn't do integrated and testing and sort of test-informed orchestration. And then we talked about the crazy acronym, the CI SMOIDAM. And there's a lot of CIC tools like CodeDeploy or Azure DevOps or Jenkins that do the CI and CD part, but they don't do the other parts that are necessary to actually do DataOps.

And then we talked about this idea of having tests across all the tools and integrating that and on-demand infrastructure. And then this whole point of collaboration, and that really is, I think, a very important part because of the fact that data and analytics is no longer a central function. It really is distributed across the organization.

You've got to handle that distribution of function as a central tenet of your architecture, or else you're just sort of- putting it off of some future state. And to do that, you need a single pane of glass or a common abstraction. And then in order to make that change, you need measurement. And so a lot of these things just aren't there in any cloud vendor. And so, from our standpoint, that's why we spend years building the tools.

We were sort of lazy software engineers, and yeah, we could've stitched this all together, but you need to do these pieces in order to actually accomplish DataOps. And so let me talk through just one example before I finish. And so here's an example of really a big global top 10 health company, and they've had a complex data landscape.

There's tens of millions of dollars of investment, huge team, and they can't get the parts to work together. And so as an example, they had a team in Texas that does data and analytics, a large Hadoop Spark cluster, kind of a best-of-breed tool chain, all put together and sort of really high-value drug development data.

And they need to work with a team that's in Azure, that has an Azure data chain, and plus some proprietary tools. And that they have wanted to mix that sort of high-value drug development data with their own research data sets. And so they tried to put this together, and they just found when things were going wrong. And so, like an example is if you were taking data in the Hadoop world and putting it into the Azure world, what happens if someone changed the schema?

How would you know about it? What happens if someone decided not to fill the table, or the meaning of the table changed? And just the simple integration where you've got really smart people doing really great work was not working for them. And they were solving it in this sort of, "Let's have more meetings, and let's have more emails and documentation" way.

And I'm a technical person, those are fine, but is there a way that you can solve that without meetings and documentation? Can you build the system to solve it, and not to have meetings to solve it? And so the aspects of that and to solve this, if you can solve these kind of coordination problems, it actually means that the teams can work better together and get more done.

And it stops the Hatfields and McCoys, stops the lacks of data trust. And you can see their tool, Spark and Hive and Kafka and String Sets and Postgres and ADSL and Connectica and Hive and HDFS and Python. They've got a complicated tool chain. I mean, drug discovery is a complicated business. And so, the first case is how do you actually just ensure this consistency, right? And how do you detect drift in schema?

How do you help make sure that the data is actually transferred right, it gets in the right place? And,

how do you

00:40:00

find out if someone, since it is a development environment, goes in and out, some just changes the production schema? How do we know that it's still right, and get an alert if it's not? And so this idea of meta-orchestration, testing and monitoring, alerting, and having an abstraction that goes across both teams, I think is a really helpful way to get at it. And then the second part is, well, things are going to change, right? How do you deploy things and then undeploy if they don't work? And then how do you keep history?

And they actually used Azure DevOps as CI and CD, which we integrated to. And so, to be able to use the idea of if you run a recipe, if it's successful, merge the code up, and if it's not, exit with error, and be able to actually work in the greater process that they had on using Azure DevOps. And I think these things can help roll back, version, and do that sort of si seulement that I talked about.

And so this is a case of a really big company, really smart people doing really great things, where if you don't design for collaboration, you don't design for DataOps, you end up with this seemingly simple problem, but not actually solved. And so I think that's a challenge a lot of organizations have. And I think that's the inversion idea that happens in DataOps, is you start thinking about these problems as first-class things to solve, and don't build a system that has this built in.

Don't build conflicts. Don't spend all your time on one part of the system only to have a conflict later on. And think in a more systemic way about how you organize all your teams, about how you organize your deployment. And so, just a summary before I take some questions. So, I think it really comes down to avoiding failure.

And so, we've seen some teams with people say, "I've got some DevOps tools. I got some workflow tools. I'm just going to build it." And our recommendation is to stop, is to not do it, because you're going to end up spending years of development. And as the guy who wrote the first version of DataKitchen's product, yeah, it's a lot of work to build this stuff.

And a lot of times we've seen companies sort of try to do this and then fail. And I think it's good that they're trying. At least they're working on it a bit. And as an example, there's a pattern I've seen of I do a couple of unit tests, and then I put my unit tests into Jenkins, and then I've got my glue code and Airflow, and then I deploy to production.

And so, that doesn't really achieve the goals. Unit tests are good, but of course, unit tests are not enough. You need to test the whole system. And that sort of takes the theory that, okay, I've got tests and I've got automated deployment, and therefore I'm doing DataOps. It doesn't actually achieve it. And so it's sort of like Potemkin agile or wagile.

You're sort of doing part of the job, but not all of it, and it doesn't actually achieve the goal, and it doesn't actually let you deploy quickly. It doesn't let you deploy in an automatic way. It doesn't let you see problems before. And so, our recommendation is don't apply the sort of patina of software developer-focused DevOp tools on your more complex data science and analytic projects. Sort of work from first principles, and say, "Okay, if I'm in development, I've got to test everything thoroughly. And if I run in production, I want to know if there's problems. I've got to handle this coordination and measurement problem upfront." And so that's it in terms of our cloud DataOps. If you want to learn more about DataKitchen, our products, more about our ideas, we wrote a manifesto and a book. And if you come from a DevOps background, we've got an excerpt from Gene Kim's latest sort of DevOps novelization book called "The Unicorn Project" that you can download.

And so that's it in terms of my presentation. And so now I'm going to turn it over to Beth and see if she's found any questions. Yes, thanks, Chris. Yeah, so we have about 15 minutes for questions, so if you have any, please just enter them in here in the question box, and we'll try to get through as many as we can.

So a first question for you, Chris, is going back to the beginning of your presentation when you talked about meta-orchestration. Why is Airflow not enough? It has a DAG and it has connectors.

Yeah. Well, I think in some ways if you want to build-- It has connectors to data sets. It doesn't have connectors to data tools. And so the orchestration is, in some ways, a lot of people use Airflow as their Python-based way to do data preparation. Some people use

Airflow as kind of an ELT tool, but it's often used as like, I want to do sort of Pythonic way to do data integration. And so the data itself is loaded into the Python process space

00:45:00

and transformed. And so, I think that's a great way. If you want to do your data work in Airflow, in Python, fantastic. If you want to use Airflow to do ELT, fantastic. But both of those are different patterns to integrate with data, and that's only part of the process. And so oftentimes, companies will have a little bit of their data work in Airflow.

Maybe they'll have some ELT at the end, maybe they'll have some Airflow. Those are all different tools that you use to do data work. And so you need something on top of those tools to be able to do it. And in fact, on top of the data science tools and the viz tools and the governance tools. And so that's where this idea of meta-orchestration comes from.

And yeah,

Airflow is a DAG, and I'm a Python guy, so I actually really like Airflow, and actually some of its competitors now, like Dagster and Prefect, and actually there's a whole bunch of sort of DAG runners out there. And so just if you can run a DAG, it's not enough. It's really about managing the DAG of DAGs and managing integrating testing and having the integrations to all those tools.

And so, as a software developer, I could put my hat on and say, yeah, if you've got a DAG, you could go build it. A DAG is five lines of Python. But there's a lot of stuff that you actually do to actually have to make this idea of meta-orchestration and DataOps work, and that Airflow does not have.

Okay, great. Thanks, Chris. The next question is: how do you integrate data governance and data dictionary tools? Oh, that's a great question. So, I think for us, let's start off with the idea of a data dictionary or data catalog. And so a data catalog could be just the description of... It could be a lot of things, but let's take its simplest case as a description of the tables in the database. And it talks about what are some relevant values, maybe there's some text description of that.

And so,

there are lots of data catalog tools out there. There's open source ones, there's closed source ones. But they have a representation of that table kind of in a website that you can go look at and edit and find. And so let's say I'm going to actually change that table in a database. I'm going to add a column to it, and I'm going to change the meaning of an existing column. And so that change, that additional column that could be done as changing what's called the DDL, the schema of the table, and you're doing difference in how you load the table.

And then any visualization or model that uses that has to do it. And of course, the last change that you need is you need to change the description of that table in your data catalog. And so you've got this set of changes that seem pretty simple. Okay, I've got my alter table command, I've got my add column command.

I've just changed two things in one table. But you've got this coupling to things that are downstream, models, visualizations, and the data catalog. And our point is that coupling shouldn't be manual. Data governance should not be a manual activity. It's sort of data governance as code. And that if I'm going to make that change, I should have that change as a coded unit that I can deploy all at once in my new table.

My changes to my visualization and my model and my data catalog, and I should test those in development before it gets to production. You shouldn't hope it works. A lot of times, data governance activities lag weeks or months behind what happens because, oh, I've made a change, and then the data governance people come along and they're feeling like they're sort of abused and that the data team like doesn't, "Oh, you changed the table?

I just had learned about it." And they're always sort of following behind. And I just don't think that's the right. I think if we see governance as code and updating the data catalog as part of it. And the other part that goes with it is sort of the lineage of where the data came from, and did that change?

And the process lineage, were tests run on it? Which tools act upon it? And those things that a DevOps, DataOps tools like DataKitchen can provide. And so, it's a simultaneously doing-- We sort of see data governance as code and DataGovOps as the principle and stop seeing data governance as a thing that happens at the end, but as another code-driven process or configuration-driven process in your organization.

Okay, that's awesome. Does the DataKitchen tool distinguish between real-time and trend analytic processes?

I guess for us, we think that whether it's your batch cycle is a day or a minute or a second, whether it's actually streaming data or some form of batch cycle, that's sort of your design decision, right? And how you build your data and analytics system. And so, likewise, your design decision of what data is incorporated, whether it's big or small or structured or unstructured.

00:50:00

The principles of DataOps apply to any type of data system like that. And so it doesn't matter if it's streaming or not, it doesn't matter if it's batch or not or what your batch frequency is. You still need to follow the sort of seven steps to DataOps that we've talked about for years, about source code and testing and deployment.

And with streaming data, you've got different streaming tools and patterns that apply. And there are different design patterns on how people do sort of time-based coupling of their system versus event-based coupling, where you want to work in a different way if you're having an event-driven system versus a batch-driven system. But the principles of DataOps apply, and our tool works in both cases.

Okay. How do you achieve end-to-end visibility across all data pipelines? Well, for us, it's having one thing. What the end-to-end visibility comes from, what software does, right, is having a common framework that all the tools plug into, and having a database of record that stores that framework, and then being able to track those changes as code.

And so we apply the same principles that we've learned to the end-to-end meta orchestration, is that the meta orchestration itself is code, and the meta orchestration itself is stored in Git and versioned and deployable. And so that's one of the key pieces, is that if you've got the end-to-end system in all its complexity put into one abstraction, and we call it a recipe, and you could call it whatever you want, a pipeline, a DAGs, however you want to do it, that becomes the common plane at which people can work.

And you could have your

dev team, you could have your data science team, and your engineering, your data science team, your data engineering team, all working on that shared common framework or abstraction that we call a recipe. And so it comes down to that, is saying, okay, that's what software does. It builds a common way of looking at things that is sort of, in some ways, helps people deal with their

limitations. And back when I started my career, I did a project for five years at NASA that worked in air traffic control automation, and we had a way to kind of see the big picture of all aircraft coming into an airport all at once. And that's too hard for one person to see, but a computer can see all those things together, and that's what's good for computers.

But it wasn't smarter than the air traffic controllers. It gave some advice, and sometimes the air traffic controllers' local knowledge had to override what the computer thought. But this idea that computers are good at kind of seeing systems of things together and having a system, having a place where you can put all your tools together, that we've talked about, all these tool chains, and one abstraction actually makes it easier to accomplish these goals, and it becomes the sort of necessary step moving forward.

And that's why we had to... We looked at tools like Airflow when we built it. We looked at workflow tools, and they just didn't accomplish what we need, so we had to build our own core abstraction called a recipe that coordinates all these pieces.

Great. Can you do DataOps with Subversion? Does it have to be Git? There's different types of... My co-founders and I had a long discussion, and yes, you can do it with Subversion. It's version control, and so it's not distributed version control, but it is version control. And so as long as you're keeping track of versions, I think it's better. And there's an argument that says centralized version control with locking is a simpler way to do it. In fact, that's how I learned to do version control.

But there's another argument that says Git sort of branches and pull requests is a better way to do version control. And I think if you look at the popularity of Git versus Subversion, Git's more popular, but there's some companies that still stick to very centralized version control. And so tools like Subversion can still work, and you can do DataOps. I think it's just important to get the work that you're doing in data and analytics in version control, right?

And so whether it's Subversion or Git, it doesn't really matter. There's differences on how you branch and merge and resolve and lock things, but using version control is the most important thing, and whether it's Git or not. Although my co-founder, Eric Estabrooks, would disagree because he loves Git. I like Git. I don't love Git.

Great. So is your DataOps platform a licensed solution, or is it professional services to build out the infrastructure on Jenkins, Git, et cetera? No, it's licensed software. And there is some integration to the tools that you have, especially if there's some special tools. But we sell licensed software, software as a service that you can use.

And a great follow-up question is, how can you save money on tools by using

00:55:00

DataKitchen?

Well, I think it comes to two things. One is, I think, kind of at a core level, your tools aren't going to make you better. There's a book, before Lance Armstrong went down, we found out he was a cheater, he wrote the book, it's called "It's Not About the Bike." And I think that applies to data and analytics. It's not about your tool.

Having a faster database isn't going to help you do more data and analytics. Having a better UI and visualization to do segmentation or data science isn't going to help you build a better model. Maybe it'll help a little, but the main thing is if you can start thinking about the systems in which you work And investing in the factory, investing in the machine that makes the machine.

Because the problems are often these processes and systemic problems, not just the tool. And in fact, if you can build it so you can take a tool out easily and put in another one where you're not dependent, and a lot of tool vendors now want you to do everything in their tool, right? Put your data into your tool, do your data transformations, your visualizations, everything, and that sort of lock-in is hard. And I think my experience in working in data and analytics over the last 15 years is people love their tools, but then loving that tool is often fickle, and if they find a new one, they're going to want to use it.

And so, I think it's the sort of right tool for the job, and if you see analytics as a workbench and we go down and look at my workbench, I've got a lot of specialized tools for special things. And I think that's the right philosophy instead of having one tool to try and do everything. And so, there are people who use Leatherman to do all their work, which is a kind of a hand tool. I just think having a good toolbox is good and being able to pull things in and out is even better.

Great. And somewhat related to that, this question is: I love that I won't have to build the platform if I buy DataKitchen, but I'm worried about lock-in. How can I migrate off of DataKitchen in the future or take over a portion of the platform?

Well, that's a great question. And so, for us, one of the things of thinking of the work as code is that you own the code. And so, the work that the recipes that you write, the tests that you write are all sort of your intellectual property. And so, I think that's a good thing. And so by having that, it's not in a proprietary format, or you can sort of take it and it's in Git, you can look at it in text. And so if you find DataKitchen doesn't work and you want to have the market for

tools is growing. There's DataOps competitors that do model ops or DataOps or who do kind of focus more on observability and testing. And so by having that in code, if you decide DataKitchen is bad, which I'm sure you won't, but in case you do, you could write a converter to take that code and put it to the code of the tool of your choice.

And so, we certainly don't believe in lock-in. We believe the power of expressing your work as some abstraction code configuration that you can work with, and then having you own it, and then that way you don't have lock-in. Okay, great. Well, we're at the top of the hour. We have a few more questions, but I think we'll get to one more.

If we didn't get to your question, we'll definitely follow up with you directly. I think this one's a good closing question, though. For a data engineering team, what would be the first DataOps step to implement to make the most impact within the scope of that team? Oh, write tests. Write some automated tests that run in production that check your data to see if it's right, that check it over time to make sure the data is, if you're building and integrating data, make sure the tables are right.

Just write tests, and have them run production and have them-- write a bunch of tests and at the end, have a table that says all the tests pass, and then just go in and check that table to make sure all the tests pass when you've done your data engineering work. That's the very simplest thing you should do, and everyone should do that because don't trust your data providers, don't trust that the data's actually processed. Try to avoid having your customers find problems in your data, and every time they find a problem, put a test in. And then every few weeks, look back and say, "Okay, what are the problems we've had?

Let's see if we can put a test in." And just that one thing alone will make your life much better. So automated testing in production with a low-shame environment where you reflect on what happens and look back and see if you can put more tests in, and try to avoid that dread of going into work in the morning saying, "Hey, did this actually work or am I going to get an email?" The shame of walking to the cafeteria when everyone knows that you or your teams pushed out a bunch of data that's wrong, try to avoid that, and automated testing is the way to go.

All right. Well, we are out of time.

01:00:00

Thank you, Chris, so much for that excellent presentation. Thanks to everyone for taking the time to join us today. As I mentioned, if we didn't get to your question, we'll follow up with you directly. We'll also be sending out the recording and the slides within the next 24 hours, so please be on the lookout for that in your email.

And if you have any additional questions, please don't hesitate to reach out to Chris or to myself directly. And we're happy to help you with anything that you need. So, have a great afternoon and evening. Thank you. Thank you. Bye.