Orchestrate Your Environment Pipelines for Reusability and Security

00:00:00

Good afternoon. Good morning to some of you as well. Thanks for joining us today. I'm Beth Beffely. I'm the VP of marketing at DataKitchen. Today, we'll be continuing our webinar series on orchestrating the three pipelines of DataOps. So welcome back to those of you who joined us for one or both of the first two sessions.

Today, we'll be discussing how to orchestrate your environment pipelines for reusability and security. So before we jump into it, just a few housekeeping items. The webinar is being recorded. As always, we'll email you the recording and the slides after the webinar, so please be on the lookout for that email in the next day or so.

For those of you who missed any of the previous sessions, I also posted a link in the chat box to those recordings. Also, we'll use the last 15 minutes to answer questions. Please answer your questions in the questions box on the control panel during the call, and we'll make sure we get through as many of those as we can at the end.

Finally, for those of you who are joining us for the first time today, I'd like to briefly introduce our speaker. Chris Bergh is the founder, CEO, and head chef at DataKitchen. He's the leader in the DataOps movement. He has more than 25 years of research, software engineering, data analytics, and executive management experience. At various points in his career, he's been a COO, CTO, VP, and director of engineering. He's also the author of "The DataOps Cookbook," "The DataOps Manifesto" and an in-demand speaker on DataOps at many industry conferences.

So with that, I will hand it over to you, Chris.

Thanks, Beth. I hope everyone's doing well today. Thank you for taking the time out of your schedule to attend our third webinar. So, as Beth said, this one is about environments and how to reuse them, how to make sure that they work for production, for development. And so, like the last two webinars, I'm going to start off with sort of a quick introduction to DataOps and the three pipelines for people who weren't able to attend the last two. And then I'm going to talk about what we mean by this term environment and why is it a problem.

Then we're actually going to go through four use cases, sort of a case of a development environment to a prod production environment, how you do multi-cloud environments, how you do multi-organization environments. And then we're going to go through a case study of what we call self-service analytic development environments. And then we're finally going to talk a little bit about sort of architecture and measurements and best practice considerations.

And then we'll take some questions at the end. So,

as an introduction to DataOps, I think the perspective of DataOps is

encapsulated in the sentence that what you do is much less important than how you do it. And so this comes first from manufacturing, and that when you're building something, you're building the factory. And in some ways, the factory is the main thing that you're doing in manufacturing. And sort of building the machine that makes the machine is actually really important in a factory, and I think that perspective comes through with DataOps. And then also for people who man or work on the factory.

In a factory, there's often problems, and Dr. Deming, who studied this, found that 94% of the time, it's not a person failing, it's the system the person works in. It's the factory process itself that's to blame. And so those two perspectives really set out the way that we think in DataOps. And in data and analytics, all of us have done things like a predictive model and run an algorithm and done some data preparation and visualization. We've all done a data dictionary and looked at a lot of different data sets. And so we're not going to actually talk very much about that in DataOps. We're going to talk about how you do those things, the development process, the deployment process, the monitor process, how you iterate, how you collaborate, how you measure.

And so we actually think the process and the people and your focus on very much giving the customer value is much more important than the tools and the technology and the data. And so what do we mean by that? Well, it's really a change in focus and mindset to focus on things like the cycle time at which you can get things from one environment to another, the cycle time of continuous deployment, how you can lower error rates in production, how you can have collaboration between people.

And this one's, in some ways, we're talking about collaboration between the technical environment that people work in, between a development environment and a production environment, and then how you measure the process. And so that's going to be our focus and our theme throughout this. And, as a result, what we've seen, and one of the main reasons why we do and think about DataOps is that data and analytics isn't being very successful because of these complexities in roles and organizations and tools.

And, in a lot of ways, data and analytics is more like the US auto industry.

00:05:00

And, in particular, moving things into production for most teams takes weeks and months. And one of the challenges that is it has a lot of errors. There's a lot of problems and slowness and frustrations. And so those frustrations actually come in in data and analytic teams. And so in a lot of ways, we who work in data and analytics are kind of caught between the rock of data providers who give us crappy data, the hard place of data consumers, and then the teams and environments that we work in. And that just makes for a frustrating world.

And so we've put together this idea of DataOps that talks about how to address cycle times and error rates and collaboration and measurement. And so there's lots of written about it, but that's a basic five-minute introduction to DataOps. And so in terms of the three pipelines that we're going to talk about, the first pipeline, the first one that we talked about was this really production pipeline, thinking of how all the tools that act on the data, the Python code, the ETL code, the R code, Tableau, kind of work in series, in order, in a kind of like a manufacturing line.

And how do you actually produce a good product from that manufacturing line? And then also how that manufacturing line itself is owned by different parts of the organization. And then in our second discussion, we actually talked about deployment, how fast you can move things from a development environment into production, how fast you can move it from the keyboard of your data scientists and your data engineers into the eyes of your end customer. And so we talked a lot about that pipeline. And underlying both those is this idea of how you have to do these things together.

You've got to run a good Toyota factory with low error rates, but you've also got to be able to deploy a Silicon Valley software team and also measure and collaborate. But underlying all this is this box on the right, that there is an environment that you do this work in, and that environment encompasses a lot of different things.

And that's what we're going to talk about because if we're going to talk about production, well, there's production in what? Deployment, it's from what to what. And so environments are actually foundational to all these pipelines. And so we actually think of this as a pipeline itself, that building and using these environments can be amenable to this idea.

And so that's what our focus is today. We're going to focus on production and environment pipelines.

So why is this a problem? Well, first of all, we did a survey last year with Eckerson, and we just asked people, "How long does it take you to create a new development environment?" And for most people, it's just way too long. Some of them are months, some were days, some were weeks. And to give it an analogy, if you look at a software team, and when a new software engineer who just gets out of college joins a software team, high-functioning teams want to have a development environment set up for them or for them to set up their development environment within a day.

And then you also want to have them be able to deploy some minor code fix to production that week. And so how many analytic teams have that same case where a person just out of college can get a development environment, successfully make a change, and deploy it to production on their first week of work?

And so I think that's possible here to do with DataOps principles, and I actually think there's business value in it, and people aren't doing it. And so, what do I mean by environment? So we're going to talk a little bit about exactly what that means, but the provisioning of an environment, the creation of it is a slow and manual and pretty high-touch process.

And it's a leading challenge for analytic organizations because when there's differences between a development environment, between a production environment, between one cloud environment and another, they end up not being fit for purpose. And to be able to reconcile those differences is a manual, slow, and pull your hair out kind of operations. And that leads to outages and instability.

It leads to expense, having more hardware and software and infrastructure and license than you need. And these differences between data in the environment, test data and production data, can cause deployment errors. And so this translates into impact on quality, data and analytic organization overspend, delays in your projects, and slow delivery of changes to your customers.

And so if we look at this, and I showed this in our last case, the columns here are environments. There's an individual development environment here, a team development environment, a test, and finally, a production environment. And if we go down to the last box, what are you trying to do in each one

00:10:00

of these? In your individual development environment, you're trying to develop a Python model or something in Tableau. In a team development environment, you're trying to see how that fits with what everyone else does on your team. And then in the UAT environment, you're trying to run it to make sure it's not going to break before production.

And the hardware is different. You may have development servers and test servers and production servers and databases. There may be data that's different, and you may have a test data set that someone does in an individual development environment that's very small. The team environment may be a cleaned, 90% version. The test one may be 100% version, and then finally, production environment.

And so there's different libraries and tools that run on each and different servers. And then there's data that's in each one of these that's different. And any time you have a case where there's a different version of a Python library between an individual development environment and a test environment, there's differences in test data and schema, it's an opportunity for something to break and those are really sort of hairy, annoying problems.

And so, they're complicated. And what we mean by an environment here is a little bit of a bigger term. It's not only the hardware and software that you use to run your analytics. It's the test data that goes into that. It is the right libraries of the hardware and software. It could be the network configuration that goes with it.

And also the code branch that you're working on, and we've talked about the importance of keeping of analytics and code, and all the tools that you're working with, Tableau or Python or Alteryx or R, and even the people that you're working in. So the environments, they're complicated, and they're in some ways more complicated than a software engineer because of the size of the data, the number of tools, the chain of the tools, the distribution of those across the organization, and the security concerns. And so it's just hard to get all these things together to build an analytic environment for you to do your work.

And so for us in our software, and we're going to demo this in a little bit, we've thought of this thing as a kitchen that brings together all these pieces, the hardware and software and test data, the people who work on it, the recipes that you're working in, and their Git branch, and the ability to work quickly.

A lot of parameters because you may want to run it on big data or small data, you may want to run the full process or a partial process, the Git branch, and even connecting to whatever kind of agile sprint Jira tool you're using. And so, a challenge in that, in building the environment is not only to be able to treat your environments at the hardware and software environments as pet, not cattle, to be able to instantiate them from creating code, but there's also a real challenge in creating test data, because test data itself is hugely important. And in some organizations, you can just copy production data, and maybe it's small enough or maybe there's no security concerns. But many organizations, you've got this challenge in the distribution of the data.

How fast can you get your test data from production? I've talked to organizations where the test data is six months out of date from production, and that means there's schema differences and distribution differences that happen in the data that all makes an opportunity for error. And then quality. Is the data that you're using like production?

Have you created fake data that doesn't have the same attributes in it? And then security, and there's GDPR rules, there's California privacy rules, there's risks associated with it. And so if you're dealing with consumer data, have you stripped out Social Security numbers or names or healthcare data? And so in some organizations, the differences between development or production data and a development data are profoundly different. But what we're trying to do is minimize errors to be able to make it happen. And also minimize costs because, in some organizations we've talked to, they have a 10-node development server that's costing them 30,000 a month to run on Amazon. And so does that need to be up all the time?

Does that need to be full of data? What's the expense in managing that? And also how do you manage this from a privacy and compliance standpoint? And so there's a lot of challenges in building and managing test data. So let me talk to the next case and talk about just the very base difference of how we manage environments, not multi-cloud or multi-organization or our case study, just development environments to production environments.

And so I'm going to run through a use case here and do a little bit of a demo in our software to talk about what I mean by that. And so let's take one simple case. So in every organization, there's

00:15:00

someone like Eric, who's kind of your production perfectionist. They run the production environment, and they've got a little bit of skill in everything, and their goal is to protect and perfect the daily grind of delivering data and analytics. They want to minimize errors, and they tend to be task masters, and they're right to be task masters because they want things perfect, because when things go wrong, they're the ones who get yelled at.

And we think that in using our software, there's someone called a DataOps engineer. And their goal here, and this is Chris, appropriately named, is trying to optimize the operations from dev to production, and they've got sort of DevOps skills and cloud skills and skills in our software. And then there's a whole different set of data doers.

Maybe it's a data scientist or a data engineer, someone doing BI, but their goal is to kind of create new features for customers. They want flexibility. They may have a whole bunch of different tools. And I'm only showing one person here, and a woman called No Prod because she has no production access, but there tends to be dozens of them in organizations.

And they have different environments. So production is managed by Eric, and that's separate software. In this case, it's a secure environment, so there's no access by developers. That's not always the case, but in some financial services company, they don't want developers to have access to production data. And so they don't even want to have developers have access to the network that production's on in some cases.

So it's a completely encapsulated environment. Maybe it has its own VPC in Amazon or its own Kubernetes cluster. And so to do that, we need to say, "I'm going to build something in development and push it to production easily." And so there has to be a different, what we call an agent in our system, a different way to run something in development in the production, because they have separate hardware and software, they're secure, and they have different access levels. And they have different management teams, and they have different credentials that go in. And so all those things are what we're going to talk about with how this interaction of people and technology work.

And so if you look at the production environment here in this example, there's a VPC that wraps the--

There's our DataKitchen agent running there. There's some SFTP servers, a Redshift database, an S3 database, a Slack channel for alerts, different places to put Docker and Python, and there's different development environments. So here's a very simple case of just trying to load some things in a database table, and you've got all these components that you've got to be able to organize the dance with them, people and their technologies, so that you can move things quickly from development into production.

And so what's key here in this diagram is these things that are in green and red, and these are what we call variables in our system, and they're pointers to what the configuration is. And so what we do is say there's, for instance, you're going to say DK implementation dev and DK implementation prod. They're pointers to variables in our system.

And those variables can be set differently. And so we're going to show how the kitchens worked. So the scenario that I just want to show you is like, okay, we've got a VP of marketing, and they say, "I want to see some new profitability data ASAP." So how do we get that from dev to prod easily, and with low errors and low ability to cause regressions into production?

And so what that looks like in our software is I'm going to go into our product, and I'm in DataKitchen here, and I'm going to this Kitchen UI. And we've got a production here, and I'm logged in as this woman, NOPROD, and she doesn't have access to production. She can't get into that kitchen.

And so doesn't have access from a network standpoint, doesn't have access from a DataKitchen standpoint, but does have access to this demo kitchen that she can work in. And so just want to show you what that means and how we've defined a kitchen. And so if I look at the recipe that she's working on, it's got a bunch of nodes in that do different things, some of which are Redshift, some of which-- Here's a Python up, Pandas transform. Here's S3 to Redshift. And these numbers represent the number of tests that happen in each node. But let's go and figure out exactly how we set up the kitchen in order to make this happen. Because when I'm doing some work in Redshift, I need to know, for instance, what the Redshift password is and what the Redshift database schema is.

And so how we set up a kitchen has got a bunch of different parts to it. So the first part is the users that have access, and different users may or may not. And you can see this woman, NOPRODUCTIONACCESS, her email is right here on the list of people. And then there's a set of what we call overrides, which are the way to abstract out the environment that you work in.

00:20:00

And so we've got a bunch here. And so if I look at this one, which is our Redshift configuration. And here's the name of the database, the IP address of the database, and the password, which is actually a vault credential, and the port. And so why is that important that it actually is a vault? Well, it goes in and says, "This is the password." And we have a vault as a server that stores encrypted passwords in our system.

And when we pull things from it, we're pulling it from in the context of the Demo Dev Kitchen. So we're getting the right password in development, and then in production, we're getting a same version of the password based on a different vault system. And so here you can see that we've got configured a different agent, which is running in that environment, different backup, and different alert configurations.

So all these things go into defining what it means to be a kitchen. And so the basic goal is that, so when you're doing some work in a kitchen and you do some development in this Test Merge kitchen, that when you push it up into-- when you do the changes up from Test Merge into the demo development environment, they're easy and fast and no errors. And abstracting your environments is an incredibly important way to do that, because that lowers the opportunity for errors.

And so defining your environments upfront is part of what we do in our

implementation process. So let's go on to another example, and I'll jump out of the software here. So the one I talked about before is there's a small team, it's a development to production, very common in central IT organizations. But a lot of people are trying to take advantage of the cloud and use the resources. And so they've got either an on-prem and then they're in Amazon or Azure, or sometimes they've got multi-cloud.

And so here's an example of what we've worked with one customer, and they were in both Amazon and in Google. And they were doing part of their work in Redshift and Talend and Python in Amazon, and then part of their work in using GCS buckets and BigQuery and then in Python. And there were data engineers on one side and data scientists on the other. And there's a story about why they have two, but you could take this and instead of replace AWS cloud, put on-prem, and see the same point is that they're two completely different environments.

But the customer gets the result at the end, right? And so the analytic is produced by the combination of both those. And when you have that model or visualization, you've got how do you make sure that each part of the process is coordinated? And then also since they're separate teams, how do you allow the separate teams to have different control over each?

And so one way that we do that is to make sure that these circles themselves, the AWS cloud and the Google cloud, have different agents running in them, have different variables and overrides that point to define what the environment system is. So we have variables that say what's GCS, what's Python, what's S3. And then the third part is we actually allow us to embed recipes within recipes so they can work together, and we call those ingredients.

And so the idea here is that you can have an overall recipe that says, run something in AWS, then run something in Google Cloud, testing across all of it to make sure that your customer does. And having each individual team in their own individual environment manage these sub-recipes is a way to have both freedom and control at the same time.

And so, that's one example of multi-cloud recipes. And so the next example I want to talk about is sort of a multi-organization environment. And so this can also be challenging because a lot of us in data and analytics face the challenge of how you deal with self-service. And so, in this case, there's kind of a centralized home office team that uses SQL Server and Python, and they're in Boston, and their data warehouse is updated once a week.

And then you've got these local teams that use Alteryx and Tableau. Alteryx is a self-service data prep tool, and Tableau is a visualization tool. And they're around the country, and they're doing what self-service teams do. They're being very rapid reacting to what the customer needs. But again, the customer sees the result, in this case, the combination of the result, the data and the models that are put in the database, and then the actual visualizations and data prep that happens. And so again, we've got this similar challenge, right? Like, well, if I change a schema, how do I know- Then I haven't broken a bunch of reports.

Or if I'm a self-service data analyst using Alteryx and I've mixed in a small dataset, how can I make that dataset available to everyone? How can I make sure that across all my different Tableau reports, the calculations are correct? And just how do I make this work when I've got some new data in a schema, an updated report? How do I coordinate across this?

Because,

00:25:00

in this case, the home office team is running in their data environment, and the local office team is using Tableau Server, which they pay monthly, and they're using Alteryx Desktop. And so, the same principle, I think, applies, is that each team has got to have their own recipes. They have their own shared responsibilities that run in their own environments.

And then they have to coordinate across each in the same idea that ingredients work. And so when we get a request from a customer that says, "Okay, I want to see my new target accounts." And so the first line of defense and the tip of the spear on analytics, and I think one of the primary benefits of self-service, is that that self-service team can then create a target account Excel file, load it into Alteryx, and mix it into Tableau and show the targeted accounts layered into the existing data.

And they can do that in a few hours and get feedback on it. And so what we want them to do is be able to work and create a local office kitchen for them to work based on the current version of the central data and do their work in that, see if that work actually provides value to the customer, add tests across the new data that they've put in and the new columns they've put in in Alteryx and Tableau, and then be able to take it and deploy that back into production.

And why does this all matter, and why would they even bother to do it? Well, it helps the home office team then know, well, I've got a new

file that I've merged in, a new Tableau workbook, a new Alteryx workbook that I've put into production to manage that. And any time I run something and have a change based on what's in production, I can then run tests against it. And so this way to have a home office and local office work together, have a

code that's aligned, but also have environments that are aligned for them to work together.

And so let me go through one more case study, and this is, again, a sort of same theme but a different idea.

So again, in self-service, there's a top five bank, and they really want to reduce the risk in self-service. And so they've got 1,000 non-IT users who want to use data to get insight from it. But the provisioning of the data that they need and the monitoring for compliance and risk is a huge problem for them.

And so they want to be able to say, "Here's some short-term use. Here's some data that you can use. Here's some tools. But I want to make sure that you've got the right data, that you have it for the right amount of time, that we know what you're doing with it, and that I can service those requests correctly." And so because, again, you're giving data about your customers or your suppliers to these people and how do you know that they're not doing something that goes against policy? So being able to create a data sandbox or a self-service sandbox for all these people, and it's a way to manage risk and apply compliance and actually just help the IT team out from doing this work.

And so what does that look like? So you've got a big bank, right? And they've got these centralized IT and data resources. And maybe in Brooklyn, you've got somebody doing small business loans in the New York area. And then maybe in Texas, you've got someone doing high net worth wealth management and trying to do some analytics or get some data.

And then finally, you may have some branch, Northwestern, the branch bank, regional marketing. So you've got these three different people, they're three different groups, again, it's a big financial service company. This is not common. And so they have different needs. So for instance, the guy in Brooklyn may need business data for the last 24 months.

He needs some DMB files loaded. It needs to be loaded in a SQL database. He needs Tableau access. So the guy in Texas may need, well, Texas customer data. It's got to be loaded in a database, but he's more technical. He wants Python and SQL access. And finally, the Seattle customer bank may need the DDA data, but they want it anonymized.

And they want to access it with Power BI and have some sort of data dictionary to understand what's going on. So there are three different teams, three different uses of data, three different cycle times. But they all share these kind of common set of challenges of, well, what's the logistics of giving people their data and tools? How fast?

What's the cycle time that I can do it? Are they allowed to use it? For how long are they to use it? What use case are they allowed to use it? And can they monitor that? Can they save their work? Can the IT team revoke access if the people leave the company? Can they reuse what they've already done?

And so self-service from an IT perspective means fast, low-cost support, and risk management of risk.

00:30:00

And so how does that look? Well, we actually helped set up a process for them where they could walk through a state of making a request and approving this analytic development environment. It takes time sometimes to get these things ready. Are they in error state? Is the user done? Et cetera. And we used a form front end for people to request in a simple way.

And then we actually enabled it with a bunch of DataKitchen recipes that allowed people to-- Some that did decommissioning, some that did provisioning, some that did dispatching, some that did monitoring, some that did failure detection. And all these different parts need to run in different cycle times and different work based in DataKitchen. And so the benefit here is it gives them, instead of spending three to four weeks to manually go through a huge IT checklist to build all this stuff and then different times and people, it's been automated.

And then this automation allows them both to monitor what happens and be able to automatically revoke access, reuse their work, and just take a lot less time and be able to manage this task support and risk. So that's the third. So we've gone through three use cases of development and the prod, which is the basic case, different cloud or on-prem to cloud environments, multi-organization environments, and then a case of self-service development environments.

So let's go through the last part and some sort of principles and architecture and measurement considerations across anything having to do with environments. So, we wrote this DataOps manifesto, and the 11th and 12th points were really about making environments reproducible. And so being able to say, "I want to quickly reproduce an environment from a development environment into production or from production into development and make it perhaps disposable, that I can recreate it." And this idea of reproducibility and disposability of environments, I think is really key.

And there's this phrase in DevOps, treat them like cattle, not pets. And there's a lot of treating of environments like pets, a lot of manual processes, a lot of go through this checklist, talk to this person, they know how it works. And so, the vision is you want to have a button to press that says, "Give me an environment," and have all that be automated.

And so the worth here is that then you can start solving that case of the 22-year-old with a CS degree who joins their team. Well, they can have a button that says, "I want to have an environment." They could do their work in that environment, then they could run tasks and then help deploy into production. And so it isn't so far-fetched to believe in these principles to make it happen in your team.

And so just some kind of core principles I think everyone should look for. So know what your environments look like, first of all. Know what a development environment is, what a production environment is, and be able to coordinate and communicate activity across those environments. And then all these tasks that go into building and abstracting environment, automate those.

You know that they're pesky, they're annoying, but they have real value. And actually, it takes put some good at engineering resources on them. And then make sure that this process of managing environments is in itself a source of data for you. And of course, keep everything under version control and even keep the scripts that you use to build the environments under version control.

And then if it takes time to build these environments, well, do it over again and do it more often and sort of bring the pain forward. And why is all this? Because there's always a challenge in technical work, what done means. And for us, I think done means it's in production. Done means it's gone through whatever number of environments you have, dev, QA, pre-prod, production. Done means in production.

And of course, give time to test data management. It's a real thing. And for some organizations, it's a more of an important thing than others.

And so let's just talk a little bit about architecture considerations. So everybody has a production environment, right? Everyone's got data on one side and customers on the other, and they all have a production environment that they work in with a whole bunch of tools that we talked about. And so when we think of DataOps as an architecture, we think of these things that you have to do, automating deployments, orchestrating and monitoring, and especially on the right here, environment creation and management. These are very important things to do.

And as part of that, you want to be able to make sure that you can identify where to run the environment. You need sort of a place to say, "I want to run it in dev. I want to run it in test. I want to run it in prod." Make it easy to say, "I want to run this Python code, this Tableau workbook, this SQL code. I want to run this in development." Okay, now can I take the same work and run it in test and make sure that

00:35:00

everything works? And so think of it as a hub and spoke. You want to have a hub, and then the spokes are each one of your environments. And the environments could be different as a data center with a dev environment that's separate from a production environment, or they could actually be literally different data centers.

And so here's sort of an architectural version of what we showed in these multi-team cases or multi-cloud cases, where there's one cloud or on-prem environment and another, and there's different agents that DataKitchen is running in each one of these. So you've got to be able to say, "I want to run it in Google Cloud dev," or "I want to run it in Amazon test" and be able to run which part of it and then have each system call.

And so there's a dance and coordination of these environments that you need, and you also need that sort of single pane of glass across them, where for a lot of companies, they may have some stuff on-prem, they may be moving to a cloud, and they have no way to tell what's running, what broke.

And being able to see it across these systems, I think, is an important way to gain efficiency and time back. And then another point is to be able to go in and think of it from a process analytics standpoint. And so, how many environments do you have? How many are you using? How many are people creating and destroying? And how fast are you deploying between these environments?

And so I think those things are very basic ways of doing it. And of course, there's more advanced things like how much is each environment costing you to run if you're in the cloud? Is there a way that you can turn them on and shut them off quicker so they don't have to run over the weekend and you're paying all the Amazon charges for them?

And so that's another type of metric that you can create. And so I think just to finish up here, we've talked about a couple of things. We've talked about why environments are valuable, why abstracting your development release, one cloud versus other, centralization versus local environments can really have value to your organization, and we gave four use cases on that.

We talked about some principles of managing environments and architecture and measurement cases. And then so finally, let's just talk about sort of why this matters. And so we think that by managing not only this one pipeline, the development pipeline, but managing the production pipeline, the deployment pipeline, you end up with a latency of which you can get work from the fingertips of your developer into the eyes of your customer and get it down from weeks and months to sort of hours and minutes.

And you can lower the rate of errors, both in production or in deployment, down very low. And I think what that ends up is a happier and more productive data and analytic team. And so again, we've have a software product that I showed you today. That's all focused on trying to decrease cycle time, lower production error rates, improving team collaboration, both teams and physical environment collaboration and measurement.

And so, if you're interested in our software, we've got a great software platform, and we've also spent a lot of time talking about these principles themselves, and we've wrote a book on it, a manifesto, that you can download and learn from. And so, that's it for my presentation today. What we're going to do now is ask a little questions and, Beth hopefully has been collecting some questions.

But before I go into that, we do have another webinar set up for June 17th with Rajesh Gill, who worked at Celgene, and talks about the story of building a billion-dollar product launch and how these sort of three pipelines and the application of DataOps principle made that success. And so I think now after that, I'm going to put it back to this slide. I'm going to share my webcam.

And,

Beth- Great ... is there any questions- Thanks, Chris ... I can answer for people? Yes. So yes, I encourage everyone to enter their question into the question box on the control panel. But to get us started off, we have a few questions. This is a good one, Chris: How does one manage technical conflicts between DataOps/data teams and software engineers, especially in environments where the product is software-driven?

Yeah, I think

both in software and in data and analytics, there are versions of these three pipelines, right? There's a deployment pipeline where you're taking something from a development environment to production. And so, a lot of times the software team is producing data that is being used by the analytic team. So simple questions come up. I add a new column to my OLTP database for my web-based application. That column then has to end up in a

00:40:00

DSS or a Redshift, an analytic database, which has to end up in a report. So how do I coordinate the release of all those together? And how do I manage the environments that those run in because they're very different technical environments. Of course, they may sometimes be together, but in some cases they're different.

And so I think thinking of it as the deployment pipeline, the environment pipeline, these principles apply both to a sort of a DevOps software environment and a DataOps data environment. And so, coordinating, for instance, if you've got a standard deploy process with Jenkins, for instance, being able to load and have your Jenkins process link both the movement of a new column in your OLTP database to a new column in your reporting database and moving that together from development to QA to production, running tests, managing the abstraction of the environment is both, I think is important.

And that's something that that DataKitchen can help you with.

Great. Thanks. That's a great answer. Here's another question. Do you have any tips on documenting data assets to enable fast access and easy reuse? I find we spend a lot of time rewriting existing code and duplicating existing models because we don't know what we have. Yeah, and I think there's a lot of tools to help build sort of dictionaries or

some people use what your data estate is, in Europe. And so what are all the data assets that you have that are available? And what are their state? Are they in development or production? And so the description of those data assets, the metadata that says, "Okay, we've got these four new tables that are existing, that have moved from our development environment into production." So you're creating four new tables with four new data sets.

You also need to deploy the metadata that goes with it, the description of those tables, the source, the lineage, along with the actual new tables. And so the deployment of changes into a Alation or a wiki, there's open source tools, I think should happen, along with the deployment of the SQL code and the DDL code that defines those tables.

And so if you think of a deployable unit as including the metadata that describes what you're doing, and the changes there, and it's not just I've added a table and a Tableau report and a Python model, I've added a table, a Tableau report, a Python model, and the description of the data so that people can find it in my data governance system, like a Collibra or an Alation or even a wiki.

Okay, great. Thanks. Here's a good question. To accomplish all this, do I need to hire a DataOps engineer?

I think the change in perspective of decreasing cycle time and lowering error rates, I'm a bit believer, I run a software company, so of course buy our software. But on the other hand, I do believe in starting small. And so, how do you lower error rates in production? One thing I guide teams to do is to just start with Excel, write down all the errors that you're having in production, and then every two or three weeks have a meeting and try to look for patterns.

And then when you find a pattern, implement an improvement plan to remove that pattern and take the shame and blame away from errors and put some positivity around it. And so I think there's things that you can do with someone who has the title of a data engineer or a data scientist who has the belief that focusing on the process is better than just focusing the results. And I've seen the lack of focus on a DataOps process. And I think as that momentum builds, you can see then that the investment in building these three pipelines, automating these three pipelines, managing these three pipelines has real value. And I think you'll come to believe that that actually, the idea of DataOps engineering is not just something that you do part-time, but actually is a full-time role and a full-time role for some of the smartest people in your organization.

And I hope someday we'll get to the point in data and analytics where high-functioning software teams, where those roles are paid the same as everyone else, as a data scientist. They're valued as much as the cool kids, the data scientists. And that becomes a great career path because I've seen that happen in software. It's gone from release engineers 20 years ago being paid less than Java developers to DevOps engineers being paid at least as much, if not more than your typical software engineer because of the importance of deployment speed and monitoring and the ability to build this great basket of this great factory to be able to manufacture work.

And that actually, I think is a good and worthy thing, but you don't have to

00:45:00

jump into it right away. Starting small, working on error rates and cycle times and then perhaps having people donate some time and then perhaps maybe hiring someone and then looking at software to help you, I think is the right path forward.

Great. Well, those were some great tips for getting started. We have one more question. If anyone else has any other questions, we do have some more time, so please enter them in the chat. But this is a good follow-on question to the last and probably a good summary. What are some of the ways you measure the benefits, value, and ROI of DataOps?

Well, I think always if you go to an analytics trade show, they all say, "We give you more insight, and it's faster time to insight, faster time to value." And I think that's true with DataOps, and I think perhaps that's more true than other, but it's very hard to quantify that. How do you get faster time to insight beyond sort of NPS scores with your customers? And I think there's really hard business cases, and I think you can look at developer productivity in terms of the amount of tickets or feature points they get out.

The amount of work that they do goes up. You can look at the cost of infrastructure, for instance. So if you manage your pipelines, you can then reduce the amount of unneeded servers that are there because you can start treating things like cattle and pets, and then there's the time that it takes to do that has gone away.

So the efficiency of the team goes up, the efficiency of the environment goes up. Those are cost reductions. And then you can also look at the enhancers, the insight generated. And so that's a little bit harder more to quantify on the positive time, on the positive one. But it comes down to at the end of the day, we think good insight can help you increase revenue, and application of DataOps principles can help you reduce costs and cycle time.

Great. Well, those are all the questions we have. So thanks so much, Chris. I hope everyone found this discussion to be really useful today. Thank you all for joining us and spending the time with us. As I said, this has been recorded, so be on the lookout in your email for the recording and the slides in the next day or so.

And please don't hesitate to reach out to us if you have any additional questions. All right. Thank you much. Have a great day.