Jumpstart Your DataOps Program with DataKitchen’s Lean DataOps

00:00:00

Welcome everyone. My name is Chris Bergh, and I'll be your host of today's webinar. And today's webinar is about Lean DataOps, and how to apply the ideas of Lean to adapt DataOps to have a program in your organization. And today we've got a special guest who's going to be able to do our discussion. And her name's Beth Pfefferle, and she's our VP of marketing. And she leads a team of B2B marketers here at DataKitchen, and she's responsible for evangelizing our new DataOps product category, educating the market about DataKitchen's DataOps platforms and services, and her marketing team does DataKitchen's branding, content, product marketing, and more.

And so before I hand it over to Beth, I've got a couple of intro reminders. The first is that we are recording the webinar, and we'll send out a recording in the slides in the next 24 to 48 hours. And then we're going to answer questions, but at the end of Beth's presentation. So feel free as you go to type your questions in there, and I'm going to put them together and we'll answer them in the last 10 or 15 minutes.

So, with that being said, I'm going to hand it over to Beth, and we're going to learn about Lean DataOps. Great. Thank you so much, Chris, and thanks everyone for joining us today. I'm going to turn my camera off to save some bandwidth, so hopefully everyone can see my screen now. So yeah. So as Chris said, we're here today to talk about Lean DataOps. So you may be wondering, what the heck is Lean DataOps?

I just figured out what DataOps is, or I'm still trying to figure out what DataOps is. Well, don't worry, this discussion is meant to simplify DataOps for you as much as possible. Lean DataOps is still DataOps, but it's our term for how to adopt DataOps in your organization in a real practical and manageable way. So hopefully by the end here, you'll have a new way to think about how to get started in your organization with DataOps, and you'll be inspired and confident to get started today.

So, first of all, just a quick overview. We'll talk a little bit about DataOps and Lean DataOps and what they are, and then I'll spend most of the time going through the four different phases of Lean DataOps.

So backing up a bit, why does your organization even want to do DataOps, right? The primary driver is that being data driven is really hard. 60% of projects fail altogether, 87% of data science projects never make it to production, and in a survey of companies describing whether they were data driven, that actually decreased from 37% to 31% over the last few years.

And beyond that, data teams are suffering. So we recently did a soon to be published joint survey with Data.World, and we found that data engineers are stressed, they lack confidence in their work, and they have low job satisfaction. Right? 52% hope and pray that things don't break in their daily job. 78% of data engineers said they were so stressed they would consider seeing a therapist. And 70% expect to change jobs within a year, and 79% have considered switching careers entirely.

So why is this, right? Well, anyone who's worked in data at a big company can probably attest to the fact that data teams are complex. You've got different teams with different roles, using different tools, working in different locations, both physically as well as different combinations of on-prem and in the cloud. Just managing all of this complexity can probably be a nightmare and often leads to unreliable and slow results.

So at DataKitchen, we subscribe to this value that what you do is less important than how you do it. So you could buy the next new fancy tool, which may be fun and great, and you may learn a new skill and have something to put on your resume, but as Elon Musk so eloquently states here, you really need to spend a lot more time and energy building the machine that makes the machine.

In other words, it's building the factory.

And that's really the fundamental promise of DataOps, right? It gives you the focus that aligns people, processes, and technology. This leads to faster cycle times and the ability to rapidly experiment and innovate. It leads to low error rates, higher data quality, and more customer trust. It allows for collaboration among complex sets of people, technology, and environments.

And it allows for clear measurement and monitoring our results. Now DataOps draws on the principles of DevOps, Agile, and lean manufacturing, and these ideas are relatively new to the data and analytics world, but they're certainly not new in the software and manufacturing industries.

So in the software industry, DevOps and Agile principles

00:05:00

have enabled companies to churn out millions of software releases every year. High-performing IT organizations deploy 200 times more frequently than others. DevOps enables this through concepts like continuous deployment, and Agile replaces traditional waterfall development methods. Teams can publish newer, updated analytics in short increments, and this enables the teams to be responsive to changing requirements and new customer demands.

So lean principles are also a central part of DataOps. In the manufacturing world, they've been following lean manufacturing principles for decades. These ideas originated in Japan with Toyota, and they put the focus on continuous and incremental improvements to product and processes while eliminating redundant activities. One particularly powerful lean manufacturing concept is statistical process control, and this measures and monitors characteristics of a pipeline, ensuring that- Variances remain within acceptable ranges, and factories that follow lean principles and statistical process control have dramatically higher levels of productivity and quality than those that don't.

Following these principles really transformed the auto industry and enabled companies like Toyota to eat up American-made competition like the AMC Pacer, which is one of Chris's favorite analogies, which I think is really a good one.

So what's unique about the data world is that you have two pipelines, right? You'll find the principles of lean, agile, and DevOps apply across both of these pipelines. So looking at this T across the top on the horizontal axis, you have your production pipeline, what we also like to call the value pipeline. And this is where you deliver continuous value to your customers.

So in DataOps, it's really important to monitor this pipeline following lean principles, so you don't learn about data quality issues from your customers.

You also have a second pipeline, which is your development pipeline, which is the vertical part of the T here, and we also call this the innovation pipeline. So this is like your software development pipeline, and applying the principles of DevOps and Agile here means that you can develop and deploy new analytics quickly and continuously, and minimizes development risk and ensures that you don't break anything when you deploy to production.

So DataOps delivers tremendous benefits across both of these pipelines, but we do get that many data organizations don't always have the budget, schedule, or buy-in required for DataOps. When it's thought about as a top to bottom, enterprise-wide transformational change. We talk a lot about the enterprise DataOps transformation, and we get that it can be intimidating or just hard to get started for many or most organizations.

Of course, that's where we all want to get to, but how do we get there in the best way? So we decided to put our money where our mouth is around the concept of lean and agile, and really thought about how would one implement DataOps in a lean and agile way? We asked ourselves, and hopefully you asked yourselves too, is what is the least amount of effort that yields the greatest benefits?

And that led us to this concept of Lean DataOps, which we're defining as a way to implement DataOps in small, incremental steps that builds upon existing workflows and data pipelines.

So we believe you can get started with DataOps right away by following these four phases. We often hear, "Well, we need to wait for this to happen or that to happen before we get started, or moving to the cloud" Or, "We need to get the entire organization on board." But in reality, by following this prescribed path, we really feel there is no reason to wait.

So any small team can get started easily with little to no disruption to their existing processes or workflows. So the first phase here, phase one, is production DataOps. So thinking back to the horizontal part of the T, the top of the T. And here the focus is on lowering your production errors. So this can be implemented by a small team or even a team of one, using the DataKitchen platform. No process changes are required, so it's extremely manageable to implement and start realizing benefits right away.

So once errors are under control and you start to see results, you're going to start wanting to move faster, right? And so to do that, you need to add development DataOps. Again, using the DataKitchen platform, this can be implemented by one team or across multiple teams with minimal process changes and integrations. And then once you've made progress in phases one and two, it's time to start measuring and improving your processes with measurement

00:10:00

DataOps. So this phase is for multiple teams and it can be achieved with some small process changes and some process data integration. And then lastly comes enterprise DataOps. So when you're ready, you can expand DataOps across your organization or business unit. Here's where you'll start to see the lasting organizational transformational change. And the DataKitchen platform supports DataOps for multiple groups and significant process changes. And if you follow the progression through phases one, two, and three, you'll really be in a good position now for enterprise DataOps.

So, now I'm going to talk a little bit about each phase. So really the best way to get started, as I said, is in phase one, and that's to focus on errors in the production cycle. So currently, most data teams spend way too much time finding and fixing errors, according to Gartner, 78% of their time. And in reality, that ratio should really be flipped and more like 80% of time should be spent on delivering business value. In our recent data engineering survey, 52% said that errors are a major source of data engineer burnout.

The number of data errors that teams are dealing with are really staggering. We did a recent survey, it was actually in 2019, with the Eckerson Group, and 79% of companies have way too many data errors. We define that as more than three a month. So 79% of respondents had more than three errors a month.

A whopping 30% had 11 or more errors a month. So numbers like this can have a really huge impact on the data team's productivity and dramatically reduce trust in the team's product. Now, if these errors could be reduced or eliminated, imagine the huge productivity gains that teams could experience.

Also, it's not just about data quality, right? Data quality is just one component of error rates. Quality is really about how your data looks at one point in time. Other issues can cause errors like code changes, processing issues, late data You may have heard the term data observability, which is the new trending hot term for something that DataOps has been advocating for a long time. So it's a set of practices that enable low errors, and it's just one component of DataOps.

But DataOps does a whole lot more, but making your system observable is really a good place to start.

So coming back to your production or value pipeline, you really need to think about your pipeline like a factory. So much like a factory that produces cars, data, or raw materials come in one side, they go through multiple processes and transformations. It's touched by many teams and tools, and value or the car is delivered out the other end into the customer's hands.

So a number of steps have to happen correctly to deliver value throughout the other end. And if any one of thousands of things are slightly off, the analytics can be negatively impacted and the team is held accountable. So the solution to this is automated testing. So at every step in production, you need to add tests that answer questions such as, are your data inputs free from issues?

Is your business logic still correct? Are your outputs consistent?

So the beauty is that with the DataKitchen platform, you can add automated tests at every step in your pipelines, the more the better. These numbers here in the circles represent the number of tests at each step. And there is a direct correlation between the number of tests in a pipeline and the number of errors experienced.

So as tests go up, we see the number of errors decline dramatically. And this is not something you have to do all on day one. You don't need 38 QA tests on day one, but it's something that can be built up over time. Every failure is an opportunity to add more tests and build up the robustness of your pipelines.

Furthermore, when using a DataOps platform, it's easy to get started because tests can be written in user's tool of choice, so there's no need for anyone to learn any new languages or tools. If your data scientist likes Python, they can go for it. If someone else likes to use SQL, they can go for it in those tools.

And there are an endless amount of tests that can be added, like statistical process control, location balance, historical balance test. And if you'd like to learn more about the different types of DataOps tests, we wrote a great white paper on this topic, and I highly encourage you to check that out.

So when setting up any of these tests, you can also set alerts that will notify you via email, Jira, Slack, or any other collaboration tool that you're already using if something goes wrong, so that you can get ahead of the problem and fix the

00:15:00

errors before your customers find them.

And then you can also classify errors by severity level and set failure modes. So for the most severe errors, you might want to stop the pipeline. Others may just require a warning and further investigation, while others may be informational and just monitor routine pipeline activity.

So with very little effort, you can implement a really important principle of DataOps and eliminate production errors. So when this was implemented at Bristol Myers Squibb, they went from no test per build to thousands of tests per build, and from frequent errors per build to zero errors. So this was quite an improvement.

And this really flips the narrative, right? By spending less time finding and fixing errors, the team has more time for the important work of innovation and delivering business value. This also leads to greater productivity and trust. So in the words of one customer who implemented production DataOps, "We reduced errors to about one per quarter. It's now been several years since we've had any major glitch. This has dramatically increased the efficiency of our data team and the confidence of our end stakeholders in the data."

So once you've improved your production cycles and built credibility in the data, what's next? The next question is to ask, "How can my team start delivering value faster? How can my team respond to frequent questions from its business customers and start iterating on results with them?" So the answer to this is to move to phase two and focus on your development and deployment processes. So here you want to focus on automating as many processes as you can, which will in turn lead you to maximize analytic development velocity, minimize deployment risk, and it'll also have a huge impact on collaboration within your team or across teams. And I'll talk a little bit more about each of these.

So coming back to your development or innovation pipeline, so now we're back to the vertical part of the T. This is where you need to start thinking like a software developer, right? And the key is automation. How can you automate the movement of deliverables from one environment to the next, especially considering that most deployments happen across diverse teams, using diverse tools, meeting the needs of diverse customers?

However, the reality is that manual error-prone processes are the norm. So in this real-life example, it took four months to deploy to production. So new analytics involve many different teams and tools. The data progressed through four stages to get to production. So from dev to test to pre-prod and finally to prod. These processes were manual, which introduced lots of complexity, slowness, and errors.

In fact, by the time new analytics were launched, the end users sometimes couldn't even remember why they asked for them. And that may be a situation many of you are familiar with. So overall, our survey results from 2019 support this. Most teams are too slow deploying new analytics to production. 70% take weeks or months. That's really very slow.

And then one big bottleneck is the ability to create analytic development environments. So in the same survey, we found that most teams really struggle with this. 38% take weeks or months, making it virtually impossible to deliver new analytics quickly. So this is not surprising, because environments and data and analytics are really complex. Creating environments involves a lot of manual steps and delays.

For example, you may need to get multiple management approvals, authenticate users who need a sandbox, provision hardware, install and configure software, replicate data. There's a lot to do, and we spoke to one enterprise recently that took 10 to 20 weeks just to complete these tasks.

So to address both development and deployment issues, DataKitchen created Kitchens. So a Kitchen is a sandbox environment where data developers and self-service users can do their work. So here you have a typical production pipeline, and say someone wants to make a change in this step here. They can branch off into a test Kitchen to work, and then their activity is segregated.

Kitchens can be spun up or down on demand, so taking what might have been a 10 or 20-week process down to minutes.

Kitchens really allow for risk-free development and empower developers

00:20:00

and self-service users. This is all possible because the technical environments underlying production and development Kitchens match. Development Kitchens include everything users need to create new analytics, like a security vault, Git branch, tools, test data, et cetera. And then users can safely work in their Kitchen without disrupting production or each other.

Testing is also an integral part of development DataOps. With the DataKitchen platform, automated testing is built into the release and deployment workflows. So testing proves that analytics code is ready to be promoted to production. So instead of a scenario which often happens today, where analytics are thrown over the wall to be released to production, everyone now can be confident that the new analytics are ready to be released and nothing will break.

Kitchens then automate the manual steps related to deployment. So when all tests passed, and because development and production environments are aligned, a data pipeline can be migrated from a development Kitchen to a production Kitchen with just a few clicks. So this eliminates endless meetings and a lot of the manual processes that really sap a data team's time.

So the use of Kitchens in development DataOps really greatly enhances collaboration. As I just described, a really common use case is just moving analytics from development to production. So even with a small development team, typically moving the analytics is a multi-step, multi-person, multi-environment process. Someone needs to set up an environment. Maybe it's a DataOps engineer, if you have one, or someone who's familiar with this and has the skills you need.

The data engineer needs to do the data work. These changes need to be reviewed and approved, and then this production engineer needs to deploy the changes. So this process is really simplified when you use Kitchens. Kitchens are hierarchical, so child Kitchens can be created off a parent Kitchen. Developers can then do their work safely in their own environment.

And when they're ready, they can merge their collective work up the hierarchy to higher environments and to production, knowing that nothing will break. So not only does this allow team members to collaborate easily, all these steps are automated, so it saves a ton of time as well.

When development DataOps is expanded more broadly to different teams, teams working in different locations or using different tool chains can now collaborate with ease as well. Each team can build their own pipeline in their respective environments. In this case, we have a team in New Jersey working in an on-prem environment, and they're coordinating with a team in California working in the cloud. So each team can build their own pipeline using their respective tool chains, and they incorporate test.

These pipelines can then be meta-orchestrated into one workflow by a centralized pipeline

that calls the local pipelines, and this enables seamless collaboration across the different locations, environments, tools, and teams.

So the results of development DataOps can really be tremendous. In this example, going back to Bristol Myers Squibb, you can see that the number of schema changes per week went from one to 12. Cycle time to publish new visualizations improved from weeks and months to the next day, and productivity dramatically improved as the number of data analysts supported by one data engineer increased from 0.5 to 12.

So tremendous improvements all around. And all of this also leads to dramatic improvements in innovation, right? When your teams are more productive, they can be more innovative and responsive to customer request. As quoted here, expressed here, "Executives want answers as quickly as possible. By using DataKitchen platform, we were able to mix and match data in new ways so that we can quickly offer answers to a question."

So moving on to the next phase is measurement DataOps. So once you've made progress in phases one and two, it's time to start measuring and improving your processes. We often find it surprising how many analytic teams are unanalytic about their data processes, but we really shouldn't be that surprised because it's really not the fault of the data teams because it's hard to collect this data.

So the DataKitchen platform gives you the tools you need to measure and improve what you're doing.

So in this phase, the platform automates the collection of system-wide process analytics

00:25:00

into one combined state data store for the analytic system as a whole. This enables you to track production, team and project metrics, as well as process lineage. You're moving up the lean DataOps hierarchy here, and this phase involves multiple teams and can be achieved with some small process changes and some process data integration.

So first you want to track how your manufacturing facility is doing and get some real-time insight into your operations so that you can remove bottlenecks quickly. So this report here is an example of our tornado report, which shows production issues and how long they take to resolve. Other types of reports are available that show you things like the status of builds, so are data sources on time and are builds on track.

So you can resolve these issues before they reach the customer. Measurement DataOps also allows you to measure and improve project and team performance. So here data gives you a bird's eye view, and you can tell whether projects are being completed on time or whether build times are improving or not. These analytics are also critical for building successful teams.

You can track whether teams are increasing collaboration, improving productivity, speeding deployment, meeting deadlines. So this really becomes a great management tool and a way for you to improve your team's performance.

Lastly, measurement DataOps will also help you track process lineage. So many companies track their data lineage, but they don't really know anything about all the processes that act upon their data, like the code, test history, timing steps. So access to this information will help you review execution, troubleshoot, and support those all-important governance activities.

So finally, once you've done measurement DataOps, all this data is really going to help you prove DataOps value to your boss, right? So by reviewing and sharing these metrics regularly with your team and key stakeholders, you'll of course, be able to continuously improve. But then you also want to showcase these improvements, and it will set you up to advance the cause of DataOps on a larger scale across the enterprise.

So lastly, we're at enterprise DataOps. So when you're ready, you can expand DataOps and everything you learned in phases one, two, and three across your organization or business unit. And here's where you'll achieve lasting organizational change. This step involves multiple groups and significant process changes, all supported by the DataKitchen platform. Here you'll also realize the full benefits of DataOps as it relates to collaboration.

So teams across the organization, no matter where they're located or which tools they're using, will be able to work together seamlessly, which brings huge benefits. We went over an example earlier of teams in multiple locations with a mix of cloud and on-prem environments working together in the development DataOps section. So you can imagine here those results on a much larger scale with teams collaborating across the organization.

There's a lot more to think about in this phase, right? So some key considerations are what's your overall corporate DataOps strategy? What cultural roadblocks may still exist? Another big consideration is how to organize your team. How do you organize your team to be agile? Do you need to hire DataOps engineers? Do you need a centralized center of excellence, a dojo or technical service organization?

There are many different models that can be successful. At the end of the day, how do you get your team to spend the right ratio of time on development and operations, right? So most teams today without DataOps spend less than 3% of their time focused on operations. But those that do DataOps are getting closer to 15%, which is a move in the right direction for sure.

In the software world, that ratio is even higher at around 23%. So a key part of enterprise DataOps is to keep moving your teams in this direction.

So the DataKitchen platform easily supports the transition to enterprise DataOps. In addition to the technology, though, DataKitchen is really here to help you succeed with some of the softer aspects. We've created a framework for establishing DataOps across the organization with six steps that help you get there. These involve educating your team on the value of DataOps, finding a first project, establishing a community of interest, demonstrating value in a short time, iterating on more use cases, and expanding to more use

00:30:00

cases across the organization. So we did a whole webinar on this topic, so I encourage you to check that out if you're at this phase. But the good news is that if you've followed lean DataOps principles up until this point, you're already way ahead of the game, and you have successes from the earlier phases that will make enterprise DataOps all that much more easier to achieve.

If you need more help, DataKitchen offers transformation advisory services to support your efforts. We have everything from strategic DataOps services to technical DataOps services. We have a maturity model assessment we offer. We can also help set up a dojo or other organizational structure, help set metrics or do agile training. The needs will really vary for every organization, but we're here to help.

In addition to the webinar I mentioned earlier, we have a ton of content on this topic. Here's a list of some of our most popular content, but foundational reads are the DataOps Manifesto, which you can sign, and the DataOps Cookbook. And then as we move toward a transformation, we have a new book, "Recipes for DataOps Success: The Complete Guide to an Enterprise DataOps Transformation." You can even take the DataOps maturity model on our website.

So these are just a few resources to get you started, and we can certainly point you to more as you move along.

So once enterprise DataOps is implemented across the organization, you'll start to see positive changes across all of these key levers. You'll start to experience better cycle times, more error-free days, better collaboration, more process measurement. And it's also not a trade-off, right? So fewer errors does not come at the expense of faster cycle times.

Before doing DataOps, we often find teams that think the only way to reduce errors is to slow down and hold more meetings. But that's not the case, and each phase of Lean DataOps builds on the next, so you can achieve fewer errors and also fast cycle times. And then ultimately, these improvements lead to lower cost, higher productivity, and happier customers, which is the overall end game.

So in conclusion, I hope this overview of Lean DataOps gives you some confidence that it's really possible to get started with DataOps today. You can start with production DataOps and start to realize real benefits and eliminate errors in your production pipelines. Then when you're ready to move to the next phase, you can do so at your own pace. The DataKitchen platform is designed to support all of these phases and will act as the process hub for all of your DataOps activities as you progress.

So that was a quick overview of Lean DataOps. I'd love to see if there are any questions.

Thank you, Beth. That was a great presentation, and thank you so much for sharing the ideas of Lean DataOps. So we've got a bunch of questions from our audience, and I'm going to go through them. So I guess the first one is,

can the DataKitchen DataOps platform be synced with the Cloudera platform? And I think that came up when you were talking about kitchens. Do you want to take that, Beth, or do you want me to? Why don't you take that one, Chris? Yeah. So Cloudera is a sort of a Hadoop and Spark platform, and it's got a lot of capabilities in, from streaming to batch to data science. And so, from that case, any data platform DataKitchen can be synced with.

And so we can actually create kitchens within the Cloudera platform, whether those kitchens have storage requirements or specific nodes or compute needs. We can actually build segregated sandboxes within Cloudera, or run jobs within the Cloudera platform. So I think there's a good integration that we have with Cloudera or Databricks or other large data platforms.

And so the next question is from Anuj, is how do you cater to the non-functional needs, volume, performance, stress? How close is the development environment to a production-like environment? And so I think that's-- I'll answer that one, Beth. It's sort of related, I think. DataKitchen is not a, I guess an unfortunately named company because it's not a data platform.

It's really about the processes that act on data. And so the volume of data doesn't matter for DataKitchen, nor does the performance. It's really the underlying system, the database, the server size, the network that actually does it. And so when you have a development environment, it becomes a question then of how do you size the development environment to be like production?

00:35:00

And so if you're in the cloud and you've probably got less than sort of 50 or 100 terabytes of data, you could probably spin up a development environment that's very much like production. In fact, the same specs. If your data is bigger, sort of petabyte scale, perhaps you want to have a smaller, filtered subset of data and a smaller environment to run in. And so those sort of non-functional needs, you need to...

What we think is there's either our consulting team or a DataOps engineer can help set that up so it's scriptable and repeatable and infrastructure as code. So I think one of the things is that we think that develop environments other than production are worthy of investment and worthy of being sort of driven from scripts.

And let me see.

So the next one, maybe this one's for you, Beth. Can you elaborate on the best practices for organizing DataOps teams?

Yeah. Maybe I'll let you take that one, Chris. Thanks, Beth. So actually, I had a sales discussion this morning, and so we did a webinar a few weeks ago about what a DataOps engineer is. And so there's different terms. Some people use the term DataOps as everything in data and analytics And so we have a little bit more restrictive view as it's DataOps and data engineers, their job is to actually help data scientists and data engineers and people who do data visualization and governance work better, so that those, the data engineers and the data scientists, can deliver insight to their customers.

So a DataOps team is really in charge of helping their customer, which is the data engineer and the data scientist, be successful. And in some cases, that could mean building environments like we talked about before, supporting them with testing, supporting them with production operations, and helping drive agile and iterations across the team. And in some cases, those DataOps engineers and those production functions are separate from the data engineers.

And in other cases, they are embedded with the DataOps engineer or with the data engineer or data scientists. And so it depends on your organization model. Some people want to have the sort of the Amazon two-pizza teams where you build it, you run it. Other organizations have a separate development function where data engineering and data science goes, and then a separate production function. And DataOps and a DataOps team can work with both. But we see DataOps engineering and a DataOps team that works with your data scientists as an essential way to drive agility and value for your customers.

And so I guess this one's for me again, Beth. Sorry. "Where does DataKitchen fit when there's an ecosystem of SQL Servers, Teradata, and GCP set up?" So,

I think that's a case of a lot of organizations, where they've got some perhaps SQL Server or Teradata on-prem, they've got GCP or Amazon AWS, or they have Azure. And so how do you deal with the fact that you're in this in-between state where you've got on-prem and both? And I think one of the challenges there is that you have work that you need to do that spans both.

You've got some of the work that happens on-prem, some happens in the cloud. You're transitioning work from on-prem into the cloud. And I think DataOps principles applies with both. From our platform standpoint, we have a sort of a hub and spoke or agent-based architecture where we have agents run in each one of your environments, both on-prem and cloud or different network-segregated environments.

And then the second is one of the best ways, and one of the biggest challenges that I faced in my career is when you have something that's working, perhaps something in SQL Server and Teradata, and you're trying to get it going on a new version of it on GCP and BigQuery, how do you make sure that the data matches?

How do you make sure that you haven't-- and how do you know that what you've built new in the new tech is equivalent to old? And they're sort of testing and balancing across systems, I think is really important. And so the idea of iterations and testing and meta-orchestration that's supported by DataKitchen, I think plays a valuable role.

And so, how about another question I see here.

So Beth, so here's a question. "If speed is more of an issue for us," I guess they mean sort of speed to deployment, "can we start the lean DataOps path with development DataOps?" Sure. Of course, that can be done. If someone is just starting out, we typically recommend that you start with production DataOps and focused on errors, because that's really the easiest path to get started.

00:40:00

However, if errors are not a problem in your organization and cycle time is a bigger problem, it's certainly okay to jump to phase two and we can certainly work with you on this. Yeah, and I think there's another question, too, of how does the sort of team structure go? "Can I implement production or development DataOps across more than one team or pipeline, or do you have to start with one team and then go from there?" Yeah, absolutely can be implemented across multiple teams or pipelines. As a progression, we recommend just starting with one team, but certainly easy to expand once you've started to multiple teams or pipelines.

Oh, fantastic. Okay. And some more questions are coming in. Thanks for those, people.

So there's a question on a lot of discussion is sort of testing in production, testing in development. And so how do you actually write tests? So,

does DK offer a cookbook or courses on how to write these tests?

And so, yeah, actually, I think what we're starting to do is we're starting a DataOps certification program, both from concepts and practicality. And so we actually have a whole set of best practices around writing tests, and some of which are embedded in our documentation, some of which can be implemented by our development team. We've got consulting engines that can build tests based on profiling data.

And so,

yeah, we've got a lot of best practices for the tests. And then I think the question is do you want to write those tests in production, or do you take those tests and build them in development first and then promote them to production? And Rob, I'm more of a believer that you always use development to develop.

And so whether you're developing a data transformation or a model or a visualization, the tests that you develop for that should be done in development. You shouldn't sort of try to, unless you really have to, muck with development systems. And so whether you're deploying a new test or whether you're deploying a new bit of code to do your data engineering work or data science work, I think the tests need to be our development activity in themselves. And mainly because they have code or configuration associated with them.

And then the last one here, here's Beth. How do you overcome the resistance from a risk-averse enterprise and a build-it-ourselves engineering mentality? Yeah, that's a good question, and we do find many organizations that do want to build this themselves using their DevOps or workflow tools. We don't recommend it, although teams certainly try. It does take quite a bit of assembly to get all these tools to work together as a complete system, and it could turn into a giant integration task.

So that's a bit contrary to the spirit of lean DataOps, and we've seen these types of projects overwhelm a team. You really want to be focusing on infrastructure development, not be focusing on infrastructure and development, and focus instead on process improvements. Yeah. Well, we're a software vendor, so we're biased on that, of course.

Yeah. But one of the things also is I've seen a conceptual mismatch between people. So they-- and again, this is a similar question I had in my demo this morning is, "Hey, we've got Azure DevOps, or we got Jenkins. Can't we just use that?" And I've seen organizations do it, and in some ways, tools like that are very much about the railway tracks, but not the signals along the railway.

And so I saw a group that did an amazing way to deploy code and wrap it up into Docker containers, and it was really a way to automate the movement of code, but it still took them six to 10 weeks to get that code, even though the movement was automated to get in production because they didn't have the proper testing, and meta orchestration, and kitchens associated with it.

So I think one of the challenges with a risk-averse enterprise is that I think you need to start small and you need to prove, because at a high level, what DataOps is asking people who've been in a data and analytics field to do something that they're uncomfortable with, is to go fast and that they cannot break things.

And so I think build a little, and learn a little, and iterate I think, and focus on a lean way of doing it can overcome risk-averse enterprises. And I think learning more about what DataOps is can help overcome the sort of build-it-yourself engineering mentality.

And so another question. If we're implementing DataKitchen using a brand new platform where there currently isn't a production pipeline, would you still recommend one to four order of implementation?

00:45:00

Honestly, when we built DataKitchen, I acted as a data engineer, and so I'm a believer that you sort of are always building-- you always are trying to get something working. And so the minute you sort of have your... Let's say you're building a database and a report. So you build one table, you write a test for that table, you deploy it into production. So in some ways, my method is to, in a green field, is to start doing DataOps development and production DataOps at the same time, in conjunction with how you're doing any of your work. And so when I found people who get more familiar with DataOps, it becomes part and parcel. Don't build anything that you can't deploy with a push of a button.

Don't write any SQL code without a test. Don't write a model without some tests. And so I think the order of one to four is generally based on the fact that mostly everyone that we talk to has something in production already and has a lot of problems. But if you're in greenfield, I think DataOps is an iterative development methodology.

So focus on iterations and wrapping automation in the iteration, and do it in production, and do it in development. So it's kind of like do one and two at the same time while you're building is my answer.

Let's see. I think we've got a few more minutes left. Let me see if there's some more questions. So Beth, this one could be for you. Can we implement a lean DataOps using the tools we already have, for instance, not using the DK platform? I think that's similar to the earlier question that was asked about building it yourself.

The DataKitchen platform is tool agnostic, so you can integrate with any tool you're already using. So that's what makes lean DataOps lean.

You could certainly try to build it yourself, but we've talked a little bit about some of the reasons why that kind of goes contrary to the principles of lean.

Yeah. Because I think that if you think about what lean is, right? Lean means just focus on delivering value to your customers and make sure that you are trying to learn about what your customers want by focusing on sort of cycle time and not having a whole lot of waste. And so one of the challenges I see data teams having is just not focusing on customer value.

They sort of say, "Well, we

built some database tables," or, "We built a model, and we're done." And how it's used, whether the customers like it, whether they want to have tweaks to it, isn't sort of part of their day-to-day thinking. And so I think one of the bigger challenges in DataOps is just stop having you distance yourself from what the customers want and really focus on is the end customer who's using the dashboard or getting the list or interacting with the application that you built, really using it and really getting value.

And that I think it tends to be

hard in some organizations because that value stream is broken up between self-service analytics and data science and IT and data engineering. And so but another idea behind lean DataOps is sort of think about the value stream and where along the lines you can actually improve it to make sure that your customers are successful in getting the value that they want and trust the data, and you're servicing their backlog of requests.

So I think that's it in terms of questions. Beth, I want to thank you for doing the webinar today. And to all the members of our audience, I want to have a reminder that we actually made a recording, and we're going to send it out with an email and the slides. And we actually have a great next webinar coming up with Wayne Eckerson, who's the head of Eckerson Group, at the end of the month.

And he's going to kind of talk about the market, everything you need to know about DataOps solutions. And he's going to help disentangle sort of what DataOps is, and what kind of solutions there are, and how to think about it from what's happening in the marketplace. And so I'm really looking forward to that.

And Beth, again, thank you for doing the webinar. And everyone, thank you for attending. Thank you, and bye. Have a great afternoon.