Orchestrate Your Development Pipelines for Fast and Fearless Deployment

00:00:00

Good afternoon, everyone, or good morning to those of you on the West Coast. Thanks for joining us today. My name's Beth Befferly, I'm the VP of marketing at DataKitchen, and I will be the host for the webinar today. We're very excited to continue our webinar series on orchestrating the three pipelines of DataOps. Today's webinar is the second in the series, and we'll cover how to orchestrate your development pipelines for fast and fearless deployment.

Before we get started, a few housekeeping items. This webinar is being recorded. We'll email a recording to all participants, so please be on the lookout for that in your email in the next day or so. We'll also send you a link to the slides in that same email. Also, you're all on mute. We'll use the last 15 minutes of the webinar to answer questions.

So please enter your questions as the webinar goes along in the question box on the webinar control panel. We'll collect all those questions and answer them at the end during the Q&A session. Finally, I'd just like to introduce our speaker today. Chris Bergh is the founder, CEO, and head chef at DataKitchen. He's a leader of the DataOps movement.

He has more than 25 years of research, software engineering, data analytics, and executive management experience. At various points in his career, he's been a COO, CTO, VP, and director of engineering. Through these experiences, Chris realized there had to be a better way to quickly deliver innovative analytics without errors, which led to the founding of DataKitchen. He's also the co-author of "The DataOps Cookbook" and "The DataOps Manifesto" and a regular speaker on DataOps at many industry conferences.

So with that, I will hand it over to Chris. Oh, thanks, Beth. Hey, is my screen being shared? Just so- I don't see your slides. Okay. Hmm.

How's that? Do you see them now? Yep, now I see them. Fantastic. Except you're not in presenter mode. Okay, there we go. There you go.

All right, good. So hopefully you can see my screen. So, yeah, thanks everyone for attending. This is actually number two in a three-part series that talks about the three pipelines of DataOps, and this one's actually about deployment, or sometimes we call it the innovation pipeline. And so we're going to talk through a couple of things today. So, if you were in our last webinar, we're going to kind of give an introduction to DataOps and the three pipelines, and I'll take about five or 10 minutes, and then we're going to talk about what is this innovation or deployment pipeline and why it's a problem, and then some ideas on what we can do about it.

And then we're actually going to show you sort of how it's fixed in DataKitchen, and then talk about another way that people have tried to attempt to do this, which is using some, what's called CI and CD from software and unit tests, and why that just doesn't work as well. And then some architecture and measurement considerations, and then we'll have conclusions and questions.

So hopefully we'll get you out of here in 45 minutes or an hour.

So,

in this deck, I'm going to have these gray slides, and they're really sort of a set of aphorisms, which are short phrases that help illuminate the ideas in the presentation. And the first one is, "What you do is much less important than how you do it." And I'm putting it here as a frame to the idea of what DataOps is.

And so, in my career, I've done a lot of technical things, built a lot of software, built a lot of databases and visualizations and models, and I've really been into doing that. But what I've realized is that the system around building those, the machine that makes the machine, the factory that builds the machine, is actually more important, the process that these things work in.

And it even came down to when I managed people, like I took over a data and analytics group in 2005, and we had lots of problems. And my first thing was to find people to blame, a specific cause. But if you actually go back and read Dr. Deming, 94% of causes are having to do with the process, the system that people work in. And so, this idea that what you do is much less important than how you do it or the system you work in, I think is an important idea.

And in data and analytics, we've got a lot of cool things that we do, models and biz and pipelines and governance and even the data itself. But the activities that happen on top of that, the how you deploy, how you develop, how you monitor, how you iterate, how you collaborate on top of those things, I think are actually more important.

And this is a really contrarian perspective that the process, the people, the operations are just more important than the tools and the technology and the data. And what does that mean, actually, that the process and the people and the operations are more important than the tools? And so I think it has to do with some idea of focus and mindset.

So the first focus that we're actually going to talk a lot about today is how do you decrease the cycle time of change, and how do you continuously change and do it

00:05:00

in a way that doesn't cause a lot of problems? So how can you go from the mind of your data scientist or data engineer or person doing data visualization, into the hands of your customer, and then iterate and improve quickly? That's one mindset. The one that we talked about last time a bit is, how do you lower error rates?

How do you actually stop having problems in production and worrying that when you walk into the office on Monday morning, you're going to get a Slack message that something's wrong? Another mindset is instead of focusing on an individual on your keyboard, how do you make the whole team work together? How do you have people who work for the same boss, or sometimes people who work for different bosses, work together? And how do you have less meetings?

So how can you empower people to change with safety and control? And then finally, how do you measure your process? How do you get analytic about the processes that you use in analytics? And so this is a set of-- I think of it as a sort of a wide-open space of data and analytics because we've got lots of tools, ETL, data science, visualization, data prep, databases.

There's just lots of tools out there. And so, one of the reasons why we've chosen to focus on this is just in my experience, that the times that people spend per week is not optimal. They spend too much time doing things like fixing errors or operational tasks and not enough time actually working on cool new things.

And that's just due to the complexity of data and analytics. And also, they just spent, in a lot of ways, data and analytics is kind of like the US auto industry. We're producing cars that don't last long, that have a lot of problems. If we want to upgrade the car, it takes years and years.

So we're sort of producing Pacers instead of Toyota Corollas. And that is evidence in just the sort of high failure rate in a lot of data and analytics projects as evidenced by Gartner and others. And at the end of the day, I think the teams who work in data and analytics aren't feeling great about themselves. They're suffering.

And this reflects my own experience that you're kind of caught between these three hard places. One, your data providers often forget that you exist and give you crappy, late, or broken data. Your customers always want something new and are intolerant of error, and will roll their eyes if something goes wrong. And then the sort of social structure of how you work within your teams, different teams want different tools. There's a team that you push to production, and that collaboration is hard. And it sort of makes for a beaten-down and sort of distraught and disempowered environment for our teams.

And what it means is there's sort of lack of trust all around. Your customers don't trust your data. You don't trust your data providers. Teams think the other teams are idiots, and overall, I think that we can do a lot better. And so, as we've talked about, and hopefully as evidence while you're here, we think that DataOps is the solution to that suffering.

And it really focuses on a set of technical practices, cultural norms, and architecture patterns that does focus on those things. How do you have rapid cycles of innovation? How do you have low error rates? How do you collaborate across all these tools and people and technology and environments? And how do you clearly measure and monitor the results of that process?

And so one way to talk about DataOps, and sort of that's the beginning, sort of why DataOps is, is to say, well, what do you do in DataOps? And so the way we've talked about it in this is just to say there's a lot of things you do, but let's talk about three pipelines.

And in those three pipelines, the first one is really about how you orchestrate data to customer value. And think of it like this, that anything that you do in analytics is an assembly line. Maybe your data comes in fast and streaming, maybe it comes in batch, maybe it's big or small, but it goes through a series of steps where it's ingested and stored and transformed and modeled and visualized and algorithms are applied, and there's just a lot of cool tools out there to do work on.

Every category from databases to the ETL tools, data management tools, data science tools, data viz tools, data catalog tools. There's 50 plus tools and categories in each, and it's a huge market, $100 billion market. And so at least one aspect here is your tools are great. It's not about the tools, it's about the system the tools work in.

And also these pipelines themselves, these abstractions, they're not owned by one thing. Most of them are sort of distributed. An IT team will own one. You'll have a self-service data team will own another part. And imagine an organization has hundreds of these pipelines that are being kind of virtually connected together. And if a customer says something wrong, well, how do you know which pipeline caused it and which team caused it?

And then we're actually going to spend a lot of time on this today, really, because if you think about it, when your team who's doing work, whether developing in R or Python or in they're developing something in Tableau or Looker, the

00:10:00

work that they create or the configuration that they create is conceptually, it's code in the same way software engineers are creating code. It's a bunch of complicated stuff and it has an engine to run in, but it's still code. And you've got this diverse set of tools and techniques and diverse people, and you've got to find some way to get that code from the person who created it into the hands of the customer. And that's the deployment process, the moving of code from a development person into production.

And so, you've got to be able to do both these things at the same time. You've got to be able to run a nice Toyota factory, and you've got to be able to pick up a piece of the factory and change and do it with low errors and fast cycle time and good collaboration and high measurement.

And so when we talk about the three pipelines, last time we focused on the value pipeline, which is what happens in production and sort of how do you run a nice Toyota factory. And then today we're going to talk about this innovation pipeline, how you actually deploy something into production. And then the third, our next webinar, is actually going to be about the environment pipeline, how you actually build environments that reflect dev and production, how you build sandboxes and all the complexity therein.

And so today, we're just going to focus on this innovation pipeline. So let's actually get to the topic and hopefully you guys who have heard this before weren't bored. So what is this idea about innovation pipeline and sort of why is it a problem? So let me start off with a very concrete case.

And here I'm going to read the aphorism first because I think it actually is a good introduction. "The problem is not that there's too much data. The problem that there are too many people, too many tools, and too slow processes to take advantage of that data." So let's see an example of that. So let's look at just a development life cycle.

And you can see there's columns here. The first column on the left is an individual development environment. The second is a team development environment, where there's more people. There's a third environment, which is your test and UAT environment. And then your fourth is your production. And so in some ways, we've got to make sure that we can move things from one to the other. And so who works in these environments?

Well, an individual may work, a data scientist may work in their development environment. But as a team, they're working with a data engineer who is doing the data work to feed their model. And then there's perhaps a UAT team or a pre-production team that they've got to work with. And finally, there's a production team who runs it.

So you've got these different groups of people collaborating just across these environments. And their development tools are different. Like for instance, a data scientist may use a Jupyter Notebook, but the data engineer may prefer SQL and Airflow or something. And so likewise, the code that they work on is different and especially the code version. And so, that Jupyter Notebook that the data scientist is developing in their development environment is going to be put together in the team development environment with the SQL that the data engineer's working on. And sometimes the versions get out of skew.

And then the way that people run the actual hardware they work on, the operating systems and libraries, and even the test data are different across these environments. And so how you move things from one to the other can be problematic. And so do you know, for instance, that

the Python library you use on your desktop is the same that's on the team environment, is the same that's in the UAT environment, and the same in production. So moving things into production is complicated and error-prone. And it's really kind of done poorly today. There's differences in environments between an individual development environment, a team, and a UAT. There's kind of a patchwork of sort of manual processes and scripts that make it happen. Sometimes people are copying things or mailing files.

It's kind of a hope environment that, okay, I'm hoping that what I've done in my individual development environment works in dev, and then I sort of hope it works in UAT. And I'm not proving that thing works ahead of time. And that creates a lot of complexity, a lot of confusion, and a lot of waste, a lot of slowness and time. And, what it means is that it's not smooth.

It's not a pipeline of moving things from a development environment to production. It's a patchwork and slow and process that eats away at the time that people could use to be doing something more interesting. And so here's an example of a large European telecom company. And this is not unknown. They've got four environments, a development, a system test, an EDW pre-production, an EDW environment.

It takes four months to move some SQL from one environment to the other. It's a manual process. They have a lot of meetings. They have a lot of checkpoints. They have a lot of documents that people have to do, a lot of review boards. And so, it's not that this team has just got an EDW.

They have lots of tools, but just to move EDW code into production, it's too slow.

00:15:00

And so you can imagine your business customer who says, "Hey, I want this new data set into my data warehouse." So I've got to source a table, land a table, integrate the table into some dimensions. I've got to be able to visualize that, and it takes four months to get into the data warehouse and another month or two to get the visualization. So six months have gone by.

And so you can wonder why your business customers sort of roll their eyes at the data and analytics team and keep saying, "Oh, they're so slow," or, "I'm going to hire a consulting organization to do the work." Because they have an expectation if they can get a box shipped from California the next day to show up on their doorstep, well, why can't I get a new data set into my dashboard tomorrow?

And so, we actually did a survey last year with Eckerson group. This should be May 2019, not 2020, on how long does it take you to deploy changes into production? And about 76% of the people were too slow. And you can see the chart on the right, a lot of people want weeks or months to deploy stuff from dev into production.

And

I think the other part of this is that the sort of deploying into production, we've got to remember the people and their motivations here, because in a lot of organizations, there's someone who's a production engineer, in this case it's Eric, and they're kind of a production perfectionist. Their goal is to minimize errors and chaos. Maybe they know a lot about the systems, maybe not a lot about the data, but they get really upset when things go wrong.

Yet you look at the people on the other side, maybe a data scientist or a data engineer, they want to create, so they want to do new things. And those are very opposite terms, but yet these people have got to work together. And the idea of kind of throwing things over the fence from a data scientist to a production engineer, it's just unacceptable because the production engineer gets unhappy.

The data scientist has a lot of models they can't put into production. And so these characteristics of how organizations have complexities in their environments, slowness in deploying, and the organizational challenges of the different people that work in the different locations, I think, contribute to this challenge that we have. And so, well, what can we do about it? So, there's a challenge.

Maybe it takes a few months to deploy. Your customers are unhappy. How can you get this working? Well, I think, the first thing is, again, here's an aphorism, and it may sound like a cliché in this, but it's not about a fear of making a change, it's about the removal of fear from making changes.

And so that's another idea of how you change your focus. And so let's think about this in a very simple way. And I'm going to have two slides that talk about what's the simplest way to think about this. So think of all the data and analytics you do. The data that goes in in production, the factory that happens with your Python code and ETL code and VS Code and et cetera, as kind of wrapped in a big pipeline, a big tube. Data goes in one side- Stuff comes out the other, dashboards or whatever.

And imagine there's a stoplight on the top of it that says, green is good, red, there's a problem, yellow is a warning. So in those tubes are not one pipeline, but lots of pipelines. And everyone who's developing or executing is working in a pipeline, whether they call it that or not, and that's a metaphor.

And there's lots of tools going in and lots of production data, and companies may have hundreds of these running around for the company in some form. And we just want a light to say, "Hey, look, things are working in production." And so, a very simple way to think of what we do in terms of deployment and DataOps is to say, "Okay, I've got this production pipeline.

I'm going to reflect that with my development team in a development environment. And instead of production data, I want test data. And I want to be able to run that in a development environment. I also want to have my red light, green light on top." And basically that says, "It's still working. The change I've added into that pipeline hasn't broken anything." And then have a safe and controlled process to take that from my development environment with my test data and my developers and put it in production. And that's really it, is we're just trying to make sure that when you have something in production, that you can change it and deploy it in a safe and controlled and risk-free way.

And so, what is this idea? We entitled this webinar Fast and Fearless Deployment. Well, first of all, fear is a good thing, right? Don't take a lot of risks. But fear is something that you should really listen to. And, when we wrote the DataOps Manifesto, the first three principles were kind of the basis of this idea of how to focus on fast deployment and fearless deployment. So the first is continually satisfy your customers.

And what that means is don't spend three or four or five or six months building something. Get something in front of the customer quicker, and perhaps when you're even a little bit uncomfortable, so that you can get feedback.

00:20:00

Because they may say they don't need it, and then you've saved all that months of work, or they may give you another direction. And I think people, business people or consumers, they need to see, sort of see and feel and touch analytics. And then get something working. Maybe not perfect, maybe not the best model, or maybe not the exact correct data, but get something working. And then change from there.

And then the idea is to embrace change. A lot of people spend a lot of time focusing on new features and get something in production, and then they're afraid to touch it. It's going to break. I don't know what's going on. And instead of embracing change, they're putting off change. And so these principles about focusing on your customers, get something working, and embracing change, really means that your team ends up maximizing learning.

Because that's what it's about. If you can iterate quickly, if you can get feedback on something that's working enough, and you can change it without killing yourself or creating a lot of technical debt, then you're in a good place. And so what are these principles to follow? Well, we'll first create a repeatable and reliable process for releasing data analytics. And as I call data analytics, a data set, a data science model, some schemas, some codes, some visualizations, all that stuff in the chart.

And then try to automate things. So what that means is, instead of doing manual processes or having meetings, have buttons to push or scripts to run. And keep all your work under version control. So the work that your data scientist or your data engineer is doing, that goes in version control. And even the processes that we're going to talk about here themselves should be under version control.

And here's something that sounds counterintuitive. If it hurts, do it more often. And that's the idea. If it's slow and it breaks, try to take things that break and automate it, make it more repeatable. And also build in quality through automated testing. And we're going to talk actually quite a bit about that, because that's the linchpin here. There's no magic in throwing code from a developer box into production and kind of hoping it works, and then babysitting the environment and waking up on Saturday morning with panic because something went wrong. But the trick here is to make absolutely sure in development that it still works. And then the philosophy here is trying to get people responsible for the release process. Because what it means is that done and complete means it's in the customer's hand. Done doesn't mean I've checked it in, or done doesn't mean it works on my box and not another person's box.

And to do that, you've got to kind of love your errors and also measure your cycle time. Have a chart that says how long, how fast does it take it for me to deploy. And so what does that look like in practice? And so, on how we work. And so I just want to give an example from our software of how that works and how you do deployments.

And there's different ways to people deploy, but I'm just going to walk through an example to make it concrete for you. So, one of our customers has this saying, "Want a real superpower? Have the power to say, 'Sure, we can do that. We'll have a first draft tomorrow.'" And so, that sounds great, but like how could you actually do that given the complexities of the world?

Well, I'm going to give a scenario. So let's say there's a VP of marketing, and he or she has a data engineer and a data analyst working for them. And they're working in the cloud, so the data's in Redshift, and their data work's done in Pentaho, which is an open source ETL tool, and the biz is done in Tableau. And for our sake, we're not really particularly bound to any ETL tool or ELT tool. It could be Pentaho or Informatica or Talend or Airflow or Oozie or whatever tool you like to do.

And then likewise, we're not bound to a particular BI tool. And a lot of big companies have multiple versions of these. What we do want you to do is be able to do this, that when a customer says something, be able to give them a segmentation tomorrow, but do it in a way that doesn't cause your team to get stressed out, doesn't cause a lot of technical debt, and doesn't create a lot of risk or an ungoverned nightmare.

And so those are great words, but what does that mean? Well, in practicality, sometimes when you want to do something, you've got to bring a new person in, and in this demo, we're calling it a data scientist. And that data scientist has got their own tool. This is a Jupyter Notebook, and they've got to do their work.

But in a lot of cases, sometimes a data analyst can just make a segmentation as a calculated field in Tableau, and maybe that's good enough. But a lot of times you've got to get everyone involved. You've got to change the database, add a model, change the visualization. So this whole team needs a place to work, and in our software, we call that a kitchen. And then I'm going to show you this process of working in a kitchen and then how you deploy from a kitchen into production and how easy that can be and how risk-free that can be. And so I'm going to switch my demonstration here and

00:25:00

actually go into the DataKitchen product. And so if I go look at this, our product's DataKitchen. And, since one of the things that we build in DataKitchen is this thing called a recipe, and this is a series of steps that happen. And so this is the one that's running in production, because on the upper left, I'm in this production kitchen. Here's how you build something.

So the data engineer's got something built in Pentaho, and they've run with it. And actually, they've used their favorite tool to do this, so they've got Pentaho on their desktop or on a server. And that Pentaho itself creates these files called KTR files. And these are just an XML file that are created by the Pentaho application. And some files create JSON, some files create XML, some files create executable code. But almost every tool in data and analytics has an editor and has an engine and has a file that actually stores the work. And so we run these.

We actually stick them in a Docker container. But the most important thing is when we run them, we test them. And here we're actually getting data out of the ETL process and doing a simple test comparison against it to make sure that it's right. And so when this system runs, we actually create this thing called an order run.

And so in production, we ran it, and it was completed successfully. And that's good because that means our analytics are working in front of our customer. And if I go and look at this, I see these are all five are all green. And one of the challenges in data and analytics is as things are going through the system, and this is a really simple example, right?

It's five nodes, it's all in a line, but most people have these complicated graphs with hundreds of nodes in it. It's hard to find out if something changed in the back or broke in the back, how did it affect the front? And so we do that by actually decorating this whole system with a bunch of tests.

And here's a test that we run in Pentaho to make sure it ran right. But we're also testing Tableau. We're going into Tableau and pulling out data from it to make sure that it's right. Because if your data provider gives you poor data, how do you know then that it's not where the focus of the problem is?

And there's logic or if then else or code in every one of these nodes in the system, and that's why we test. And so this is what happens in production. So now my VP of marketing's got to be able to go in and find out what's going on. And so, or we've got to be able to go in as a team and make sure that, do the exact task.

And so we have a concept in our software called a Kitchen. So I've got to create that place to do work. So here I'm in the production kitchen, so I'm going to create a child kitchen. And to do development in analytics is kind of hard, and actually we're going to spend our next webinar talking a lot about this.

But the environment that you do work is actually kind of interesting because it's got a lot of components to it. To do development, you need hardware and software and test data that are like production, but not production. Sometimes the environment can be spun up quickly and shut down, like

on an Amazon, or sometimes you've got a fixed server and you want to create a place in it. And we're going to do that here. There's some fixed servers. You want to invite people into your environment, maybe some of your team, all your team. You want to be able to run recipes and then not wake up your production team if there's a bunch of errors.

So you want to build a little, test a little, learn a lot like my old branch chief at NASA used to say in these kitchens. And because these recipes are complicated, you want to just zero in on the piece that you're changing. So there's in our software a bunch of sort of variables and controls that allow you to do that.

And finally, because you're making changes and you need to collaborate with people, you need to branch, meaning you need to have a repository of all your work and create a sort of virtual copy of it that then you can make changes and then merge back into the copy that it came from. And so all these things actually go into creating an environment, a place.

And so I'm going to give this a name. I'm going to call it Dev Sprint. I'm going to hit Next. And in our software, there's some work on how we set up and configure setting up the environment. This was done ahead of time, and I'm going to create a schema here. I'm going to pick just one recipe to work with.

And then another aspect of the environment is the authentication credentials of how you log into different things. And I'm going to pick a default here, but we'll talk more about that next time. And so I'm going to hit Next. And so now I've got to be able to go in and create two different

spaces and two different servers. One is Tableau and the other is Redshift. And I'm going to hit Complete Wizard. So now I'm going off and I'm creating the environment that I can do my work in. And in some organizations those servers are already running, sometimes this takes months in some organizations just to set the development environment up.

But we think that creating development environments should be quick and easy, and that when you go into your kitchen list, you can actually see that this production

00:30:00

kitchen has got a child kitchen where people can do their work. And so, I go into this Dev Sprint kitchen now. And so now I'm in my development environment, and you can see that I've only got one recipe here. And so, how does the development happen in DataKitchen? Well, someone goes into a recipe and they edit it, and I've edited it ahead of time here just to save us some time. And they add a node into our graph.

We've got different types of nodes that represent different tools. We've got nodes that can be templated, so you can be able to copy and paste or reuse existing stuff. We've got ways to have ingredients, which are ways to create reusable components. But they go in and the data scientist added in their Python Anaconda node, and it works kind of just like the one I showed before.

Remember, they're using their favorite tool, and here's the code from the IPython notebook. And so it ends up, here's our K-means cluster that it ends up running, and the same test framework when it runs. And so here's a key thing that when you want to be able to help people deploy quickly from dev to production, they've got to be sure that when they're running in development, that things work. And so to do that, you've got to be able to run these recipes or run these systems in a development environment. And the idea here is that if you run in a development environment, you need to be able to prove that what you've newly created works, but also prove that you haven't mucked up anything else that already is already running.

And so, the way that we do that is actually, if you think about it, I'm going to close this here. We've got these systems that branch and merge, and so I've stuck in these two Python Anaconda segments and load segments. How do I know that what I've done here hasn't messed up my Tableau reports?

One way to do it is just say, oh, have the smartest guy or gal in the company go in and look at it and sort of give it the blessing, do a technology review board. And so, in a lot of ways, in the idea of DataOps is don't use your best resources to kind of give blessings to design, saying, "Okay, you've done this, so therefore it shouldn't." Be able to prove that when you make a change, that change doesn't affect anything else that's already working.

So number one, prove that what you've done works in development. And two, prove to yourself that you haven't broken anything else. And so the way that we do that is you run in a kitchen. And if I drill into what's being run, you can see

that this recipe is progressing left to right here. Each one of these steps is going. And then if you look at the actual test results, and so, I added in this new feature, this Python Anaconda segment. Yeah, there's one test, and we could do more. But I've proved to myself that it works by that test.

And so, yeah, my new feature works. So am I done? Well, no. How do I know that I haven't inadvertently broken something else? So you need to rerun the tests that exist in production, at least some of them. And so here I'm rerunning this Python, this Pentaho test, the one that ran before. I'm rerunning the Tableau test to make sure I haven't broken anything.

And so this is a way that you can get confidence that in development, that what you've done already works by testing, and by testing not only what you've done, but testing every part of the system in development. And that can give you the confidence. And actually, we guide our customers to say 20% of your work should be these automated tests because you can run them in production, and they help you find out if there's errors before your customer sees it.

But also in development, they actually function more like regression and functional tests. So let's see this case here. So if I went into this demo and I made a small change, and let me just go in and change this description, and I'm going to change it to Webinar and hit Update. And when I prepared this demo ahead of time, it's already done.

And so I want to go back and make a change. And I always forget to do that in the demo. And I want to show you how we get the code from what we've done in development up into production. And so there's different ways that organizations handle the promotion of code, promotion of work.

The simplest way is actually to have the code just move from a development environment into production, and then the next time production runs, it picks it up. There are other cases, and we'll talk about this, where people have more stateful infrastructure, where you've got to actually patch. You've got a development environment, and then you've got a UAT or a pre-prod environment and then a production.

You've got to do deploy scripts or patch scripts to make sure that you've got the right DDL, the right stored procedures, the right code in each place. And we can handle both, but I'm just going to show you the simplest case here of how you merge code from dev sprint into production and make it available.

So here you've actually got to take the code and merge it. And what that means is, since we believe all analytics is code, you've got to be

00:35:00

able to go in and see the changes. And here I made a really dumb change that said Webinar versus today, because I already had this made out, but I do this, and this is that technology that we've wrapped called Git, which is a version control system. And so I've done the merge. I can manage my order runs, which is a way for us to manage statistics, and then I hit Complete Wizard. And so now I go back to my kitchen list, and if I go into that production kitchen, I can see in the history that this has changed, and therefore this really dumb code change is in production, and it will be run the next time the system's run.

And so what we're trying to do here is make it safe and automated to take code from a development environment into production. Not have technology review boards, not have the smartest people, not have a whole lot of documentation and meetings, although you can always have meetings and documentation, but sort of give people these buttons to push to make sure it's automated and testable and that there's artifacts that are created that you can tell what happened. And so if I go back to my PowerPoint,

that's our goal in DataKitchen is to make it fast and automated so deployments don't take months, they can take hours or minutes, and that when you deploy, you have a very low risk of failure, that you have a high risk of success. And-

And what I've heard at a number of customers is they've started to believe this, right? Everyone sort of believes that it should be fast to give work from my developers' fingers into production quickly. We all have our software people next to us who are talking about this thing called CI and CD, continuous integration and deployment.

They're talking about how fast they can get their JavaScript code from their box into production. And so I've seen this pattern of a couple of organizations that implemented automation and deployment and unit tests, and still had lots of problems deploying things from a development box into production. So this section's really about sort of why CI and CD and unit tests are not enough, or how to correctly, or how to go one step beyond thinking about what software developers do and apply those ideas into the world of data and analytics.

And so we've written quite a bit about the differences between DevOps and DataOps, and in a DevOps process, you're developing software. You're going through a build process where you're assembling all the artifacts and compiling them. You're testing them, and then you're deploying them perhaps to a web server, and then you're running. And so this circle of continuous integration is what happens in the development process. I check some code in, and it runs a bunch of tests and makes sure that the system is still in good shape. And then in some organizations, they go a bit beyond that. I press a button, and then it automatically deploys into production.

But this is CI and CD. In DataOps, it's a little bit more complicated because you've got this, a big deal in terms of managing sandbox. You still have the same development, but we've got a lot more tools to orchestrate, similar testing, similar deploying, but we also have orchestration and monitoring. So the DataOps process is a little bit different and a little bit more complicated than a DevOps process. And in a lot of ways, what makes DataOps different than DevOps are these ideas of the complexities of creating sandboxes, the orchestration of all these different tools in data and analytics, having testing and monitoring, and we're going to talk a little bit about that, the dual roles of testing and monitoring. And then finally, collaborating, because in a lot of organizations, there's not one data and analytics pipeline, there's multiple ones.

And so, let's talk about testing at the first point. So there's this idea that tests have a dual nature. And if you remember at the beginning, we said, that T diagram, and we said in production, your code, the Informatica, the Tableau workbooks, it's fixed, but the data is varying. And so when you're writing tests, you're monitoring the variation of the data to make sure that your customer doesn't have a problem.

In development, your code is varying, but your data is fixed because you're using a copy of production data or a test data set. And it turns out the purpose of those tests that you create in development is different. Instead of trying to monitor to make sure your customer doesn't yell at you, you're trying to do regression or functional or performance tests.

And it turns out these tests actually can be reused in a lot of places. Maybe not 100%, but the tests that you run in production can be reused in development, and their purpose changes. And so we've written a whole blog post about this, but there's this dual nature to tests and monitors, and that is really true in the world of data and analytics.

And the idea here is that in development, you should really, like we showed you,

00:40:00

kind of press a button that says, "I want to be able to deploy the feature." But before you do that, there's a whole set of types of tests that you want to think about writing. And so in software, they've got a whole nomenclature for tests, and some of them apply and some of them don't.

But a lot of organizations do something called a unit test. And they unit test their ETL, which means I create some fake data, I run it in and of itself as an individual unit, and I make sure that the logic is right. And that's good. I mean, it's not bad to have a unit test. It's just not enough.

Because you need functional tests that look at things as the whole feature development. If a feature is composed of six units, well, you want to check all six units together. And then a regression test, which is rerunning the same thing over and over again. And then even down to something called an end-to-end test, where you want to test the whole system as a whole.

And each one of these types of tests, functional tests, regression tests, performance tests, smoke tests, have a different purpose in doing it. And it's not so much that you need to get the nomenclature right, but the idea is that you want to make sure that you have a breadth of tests. And we've written quite a bit about a different type of nomenclature for tests that have to do more with historical balance tests, location balance tests, statistical process control tests that apply more to the data world. But these concepts apply.

But the point is that testing an individual unit of itself is important but isn't sufficient to prove that when you made a change. You've got to test the whole system. And so that gets a problem with the-- If you take that as true, that you can't test just a piece of it, you've got to test the whole system.

Well, the problem is that system is actually built based upon the organizational structure that the company has. And so you may have a production team at top here on the left and data engineers and data scientists working in it. And they may have their own development. The data engineer team may have, if you look at the sort of tree diagram with the tubes here, they may have a dev task that's done by Tom, Sue, and AJ, and another dev task that's done by AJ and Anne, that goes up to a QA team that finally gets into production. The data science team may have two different people, and so your team structure is kind of reflected in your pipeline structure.

It's called Conway's Law, and I think that applies to data and analytics. And it gets complicated when you've actually even got self-service on the time. You may have remote teams on the left that are pushing things to prod and getting it separate entirely from your enterprise production team. And so that's a challenge, right? Because your business customer kind of sees the sum of all these pipelines. And the pipelines are owned by different people.

And even how they work, the development environments, the kitchens they work in are different. And so how do you work? How do you actually deploy things to production in this world? Well, I think, the first idea is that, with a platform like DataKitchen, you get a unified way to address all these things in a commonplace.

And that's one way to do it. And in some organizations, they've tried to actually do-- They've got CI and CD working with a DevOps team, and so on the software side, you may have Angular and C++ and HTML. They may have a dev environment, a test environment, a production environment, and they're doing-- They could have tools like Jenkins or other ones, and they may want your data and analytics team to actually fit into that environment. And I think that that's fine, and you just need to be cognizant that the organizational structure of your data and analytics teams actually fits into the pipeline execution.

And so the idea here is that there's a structure to your teams that represent the pipelines, and that structure needs to follow how you deploy into production, so it can be faster. And then in some cases, you've got to actually work within a dev team. And then the final one, and this is perhaps a little bit abstract, but some organizations, on the left, have stateful environments. So you have a development environment, a test environment, a production environment. And each environment has state. It's got data in it.

It's got logic in it. And so you've kind of got to patch environments as you move things up. And so that creates complications. What I've seen some teams do is they actually work to a more what's called a functional architecture, where there's an immutable or an unchangeable data set. And if their data is small enough, in tens or hundreds of terabytes, they can actually rebuild the entire database from scratch.

And in that way, that functional architecture makes it in some ways easier to deploy from one to the other because you've just got to be able to. And then if you put some other ideas in there, like idempotency and immutability, you actually end up with a data architecture that is actually, I

00:45:00

think, a bit simpler. But I think deployment in both, whether you've got a fixed development environment, which is more of a stateful system, or you've got more of a functional architecture, works the same. So we've got a few more minutes left. And so I would like to talk about two more items in terms of deployment.

And the first is a data architecture consideration and a measurement consideration. So first- Everyone's seen a data architecture where you've got a production environment with a data lake and some data engineering, refined data on the left, and people on the right. And this is pretty common. And if you're an architect, you really want to think about how you deploy this automated deployment. It's a very central thing, how you move things from dev to test to production, and sort of think about the right to repair, the right to change, how you automate deployment, not as an add-on, but a central part of how you work.

And I think that's reflected in how people develop software nowadays. In a lot of cases, when people build new projects, they work first on these deployment. They work first on the factory, and then on the things that are made by the factory. And then second, if you're going to make a factory, you're going to help automate deployment. You've got to be able to measure the effect of that.

And so in

DataOps, we guide people, and with our software, to be able to measure the process. And things that you want to focus on measuring are your team productivity, how fast they deploy, how many environments they're running, and even the code and test coverage. And so what that looks like in our software, if you take a particular project, and I'm going to start at the upper left-hand side, you can count how much collaboration has happened by just counting the number of kitchens that people have used. Because a kitchen is a proxy for an environment.

And then you can count how many deploys happen from a development kitchen to production, or from a feature kitchen. And the amount, if you see more deploys, that means people are getting more value to production quicker. And then even looking at the risk involved. If you look, this is over time, looking at the number of automated tests going up in aggregate on this project. And then there's a graph here that has one thing called a ratio of the number of tests to nodes. Does every node have a test?

Does it have coverage? How many nodes are there in a recipe? How many recipes are there? So you can then get an idea of the coverage and complexity of the software that you're building. Because if you're a manager and you've got 40, 10 projects, 15 projects, you've got 20, 30 pipelines in production, how do you control the work and get some idea of is one team doing good work or not?

Is one team deploying quicker or the other? That's why you need to measure. And so let me

finish here with a couple slides. So the last aphorism I'm going to give you is, "Multiple takes lead to highlight reels." And I think that's kind of the idea here of why you want to do it. You want to keep trying, and if you keep trying, you're going to get it right. And so, I think that's the basic idea is multiple takes lead to highlight reels. And what we hope is when you apply DataOps principles in our software, that your deployment latency, how fast you can get something from dev to production, can go from weeks or months to hours or minutes, and that you can have low errors. And my own personal hope is that your data and analytics team is just happier and more productive. And so if you're interested in this and focusing on DataOps and these four things to focus on in DataOps, decreasing cycle time, lowering error rates, improving collaboration, and measuring the process, well, we've got a software product, and that's of course why we're doing these webinars.

If you're not there on the software product yet, well, we've written quite a bit about it. We wrote a book, we wrote a manifesto. We've got a blog that talks all about these things and kind of gives you the background and justification. And so the last thing and the last slide I'd like to do before I answer questions is to say we've got one more in the series of the three pipelines, and the third one is how you orchestrate your environment pipelines for reusability and scalability. And that's going to be on June 3rd.

And so sign up and register for the third one in the series. And so I finished it today, and hopefully we've got some questions that I can answer. Yes. Thanks, Chris. Thanks for running through that. That was very informative. I also just wanted to let everyone know, I put the link to the third webinar in the chat box, and you can access that there.

And in terms of questions, please enter those in the question box and we'll try to run through as many as we can in the next few minutes. So let's see what we have here. Okay, Chris, how are companies organizing their data teams? Is there a best practice or ideal team size for DataOps?

00:50:00

Well, I think there's,

in general, having teams that are empowered to be able to make change in the scope of a small team, five people, 10 people, is the best. So small teams, sometimes Amazon calls them the two pizza team, that they can talk to the customer, make the change, deploy to production, and even own the ongoing challenges that come up with from production is, I think, an ideal way.

Because small teams who are empowered to be able to do the work and have ownership of it, I think is my own personal belief, is that's a better way to run an organization. There's ways to scale that in an agile way. I don't think it's required to do DataOps. I think any team size can apply these principles to be able to do it, but the idea of working in a small team agile way seems to be getting some traction in the marketplace among organizations adopting basically sort of agile or scrum-based ways of working.

Okay, great. Thanks. Here's another question that just came in. What if my scope of influence is only across part of a pipeline? What can I do to implement DataOps? Oh, that's a great question. Well, I think, obviously buy our software. But no, I think you can start doing DataOps without buying any software. I think the first thing is if you even have part of a pipeline, put some automated tests in. And make those part of your, if you're doing ETL or doing ELT or doing some data science, put some tests in.

And make sure those tests can run in production and give you some alerts or some ways of tracking, and make sure that they run in development. Put the work that you do in Git. You can do that right away. And then another thing that you can do is just sit down and keep track of if there's problems in what you're doing, if you're part of a whole system.

Form what's called a quality circle. Just sit down and start keeping track of your errors. Maybe invite your other teams in to look at errors, because the mindset of instead of avoiding errors and feeling shame about errors, to count them and improve them is, I think, a fundamental way of working. So you can adopt these principles irregardless of whether it's end-to-end across the entire company or just within a small team.

Okay. Thanks, Chris. All right, here's another question. Getting organizational buy-in for new projects is tough in the current climate. Organizations are trying to cut costs wherever they can. How can companies justify investing in new technologies to build DataOps pipelines rather than focusing on the tools they already have? Yeah, well, I think that's one of the big challenges here in COVID land, right, is that we're in a more of a do more with less organization.

And I think there's been a great boom in data and analytics over the last five or 10 years, and you can't walk through an airport without seeing something about some data company or charts and graphs. And so the challenge is the promise hasn't met the hype. And a lot of business people may be looking to data and analytics teams for cost cutting. And so how can you meet that promise?

It's not going to be with buying another database, or it's not going to be with buying a better ETL or biz tool or data science tool. The fundamental problem is that it's a people and process problem. So you have to affect those things. That's what's going to be able to increase the amount of capacity for your system.

It's going to be how you do more with less, and how you keep yourself relevant in this time. So I actually look at it a different way. I think you have to do DataOps because you're not going to get the $3 million check to buy the big database and the big tool set. You need to start focusing on productivity and iterations and cycle time and customers. And that's what's going to get you through these tough times.

Okay, great. Okay, here's a follow-up on the first question. Can the DataOps model and DataKitchen work in both a federated model and a centralized model, and what are the advantages of each?

Well, if I could try to read into that question, there's a centralized model where people who do data engineering and data science and visualization, they all work for one boss, maybe a chief data officer. And that person's responsible for as- As the pundits say, doing data offense and data defense. There's other models where part of it may be the data warehouse team or the data lake team is owned by IT, but there's a federated model where there's different teams all around the organization with different bosses doing self-service data prep and data science and visualization. And I think both models apply to DataOps.

In fact, DataOps, the idea of our software was really built on that assumption that people are going to work in a centralized model, they're going to work in a

00:55:00

federated model, or work in both, and that the value in analytics is spread around an organization. And so I think, whether you've chosen to centralize it or you have freedom or you've got some mix of both, either way is

DataOps is, I think, the set of principles and our software can apply. Okay. We have one last question, it'll probably be a good one to end on. A lot of companies are using the term DataOps, but their offerings are very different. Can you help clear up any confusion there? Well, it's exciting for me, having been around DataOps for a while, that people are using the term, because three years ago, no one used it, and I went to conferences and people looked at me like I was an alien from another planet.

So it's good that the term's being used, but I think it's being used as a halo for a lot of companies that've said they're DataOps and have taken their existing ETL tools or existing five or 10-year-old tools and put the name on it. And I don't think that that's true. I think if you're not deploying into production, you're not doing DataOps.

If you're not doing multi-tool orchestration, you're not doing DataOps. If you're not automating testing, you're not doing DataOps. And so these pipelines, if you're looking at what DataOps is, I think our three webinars, you've got to be able to start automating and orchestrating these three pipelines. And that's really what DataOps is. And so, I think that's not uncommon in tech industries for people to take terms and use them. I remember when big data was just about cheap, fast or slow queries on parallel disc machines, and then it expanded to be everything in data analytics, and maybe DataOps is becoming that.

But for us, we think it's about deployment, it's about multi-tool orchestration and testing and environment management. That's what we really focus on, and I think the industry is starting to agree with those terms, and certainly Gartner and Eckerson and other ones are.

Okay. Well, great. Well, thanks, Chris. That was a really great overview, and I hope everyone found it really helpful today. Thank you all for coming. As I mentioned earlier, we'll be sending out the recording and the slides, so look out for that in your email over the next day, and we hope to see you in future webinars.

All right. Thank you very much. Bye.