A Day in the Life of a DataOps Engineer

00:00:00

So welcome everyone. Thanks for joining our webinar today. My name's Beth Pfefferle, and I'm the VP of marketing at DataKitchen, and I will be the host. So our topic today is a day in the life of a DataOps engineer. We'll hear from two DataOps engineers and learn why this role is becoming one of the most desirable functions on a data team.

So we are thrilled to have two of DataKitchen's superstar DataOps engineers join us here today. We'll hear from Priyanjana Sharma, who has been a senior DataOps implementation engineer at DataKitchen since 2017. Welcome, Priyanjana. And, we'll also hear from Chip Bloche, who has been a DataOps engineering director at DataKitchen since 2018. So hello, Chip.

And they'll both tell you a little bit more about their backgrounds during their talks. Then, of course, it wouldn't be a DataKitchen webinar without Chris Bergh. Chris is the founder, CEO, and head chef at DataKitchen. He's a leader of the DataOps movement, co-author of "The DataOps Cookbook" and "The DataOps Manifesto," and he'll kick off the webinar today with some thoughts on the importance of the role and why you may need a DataOps engineer.

So before we get started, just a few quick housekeeping items. We are recording this session, and we'll send out the video and slides to everyone as soon as they're ready. So be on the lookout for that in your email within the next 24 hours. And then we'll also reserve the last 15 minutes of the webinar for questions. So if you have any questions during the webinar, enter them into the Q&A box on the control panel, and we'll make sure we have time to get through those at the end and hopefully have a really good discussion.

So, without further ado, I'm just going to hand this over to Chris, who will kick off the webinar. All right. Thanks, Beth. Thank you, Beth. And welcome everybody. I'm going to stop sharing my webcam now. And, so we're going to talk about the sort of what is a DataOps engineer first, and how does it differ from the other roles in data and analytics, and what sort of skills you need and sort of a way to think about it.

And let's just start off with the kind of the big picture is that DataOps engineers sort of own the assembly line that's used to build a data and analytic product. So that data product could be a chart or a graph or a database or a warehouse or some flat file process. And what we're trying to do is make that assembly line so that the people who work on it, the data engineers and scientists, can produce insight rapidly and do it with low errors and not spend a whole lot of time in meetings.

And so in some ways, the DataOps engineers own the pipelines that run that assembly line. And we're going to talk a little bit about that in the next two slides. And in some of our books, we've talked about the innovation and the value pipeline. And one perspective, DataOps engineers, they work on those pipelines, but not in those pipelines.

So they're in charge of the assembly line, but they don't work on the assembly line. That's the role of a different set of people. They're trying to make the whole system work. And so if you remember, and we've talked a lot about what DataOps is. It's sort of a set of practices and cultural norms and architecture patterns that really help people who do data and analytics deliver value quickly, work in iterations, produce insight with incredibly low error rates, collaborate without a lot of hassle, and then be able to measure what's happening with their processes. And so let's talk first just as a

preface to what we're talking about, what we mean by pipelines. And so, a pipeline is the journey that data takes from source to value, and there's a bunch of steps in between. And sometimes that's sort of what we mean by the assembly line. And in those circles in there are the work that someone who has the title of data engineer or scientist or analyst or governance has done.

And so that pipeline that's going across all those people's work and perhaps across departments, that's really what DataOps engineering is about, owning the value pipeline, but also being able to take pieces of those pipelines, whether it's a model or a visualization or a change to data governance and make a change. And we call that the innovation pipeline.

And for both these, what we want to be able to do is, number one, given the sort of long and convoluted journey that data takes from transformation to visualization to governance, we just don't want to learn about problems. We want things to work. We want to run our value pipeline, like Toyota makes changes.

00:05:00

And then we also want to be able to change that pipeline, pick up a piece of it, change it, and be able to iterate quickly and change our website or change our pipelines as fast as sort of Silicon Valley companies do on their websites, very quickly and easily. And so if we look at the role of the DataOps engineer, in some ways, you could take those circles that we showed and sort of put them around the role of the DataOps engineer. Because a data engineer is the person whose sort of product is data and builds data transformations.

A data scientist product is the model and a segmentation or a clustering of data. A data analyst product is a visualization. And so it's between all these roles, and it's really about making the collaboration happen. And so what the challenge with that is that in a lot of ways, the DataOps role to a lot of people who do data engineering or data science kind of isn't there.

It's not apparent. So if they're going to build something, like let's say I'm building a bunch of SQL, or I've built a new Jupyter Notebook, they kind of throw it over the pod. I've done my work, right? My definition of done as a data engineer is, it worked for me on my box and some magic process that I don't know or care about.

And a lot of times the challenge is that the people who are doing the work in data and analytics kind of focus on their little part and think the process of putting it in production is someone else's problem. And that's sort of a willful blindness, and it's very task-focused and not value-focused. Because I think what that means is what done means.

Done doesn't mean it works in my box. Done should mean it's in production. Done should mean it's in production, and your customer is happy with it. And so this idea of focusing on the project or the task as opposed to the product, kind of living in a world where I hope it works as opposed to proving, and then a lot of reliance on manual checks and sort of an unwillingness to be able to pull the pain forward.

And these are some of the challenges that I've seen in organizations who are trying to develop DataOps as a discipline and data engineering as a role, or DataOps engineering as a role. And so in some ways, DataOps engineering is about collaboration through a shared abstractions or a set of abstractions. And if you think about the work that someone does as a data scientist or someone in a data analyst, they create what we at DataKitchen call nuggets.

Think of those as code. Maybe it's ETL code that drives your Informatica job, or maybe it's a SQL transformation, or maybe it's some Python code or some XML code or whatever. That nugget is created and managed, but it has to be put into a bigger assembly, and that we call that a pipeline. And it has to be tested automatically, and it has to be run.

And its process of getting from development to production should be automated. And it should work across people because in some ways, DataOps engineers are not trying to solve the data problem. They're trying to solve the people and the process problem. And then, in some ways, what DataOps engineering is taking what's currently invisible and making it visible by providing these, taking these nuggets and instantiating them into a system that allows people to work quickly and iteratively. And it's also about automation.

And what we mean by that is don't-- if you're doing something manually three times in a row, perhaps that's the time to write a script to automate it. And these tasks that in general, we found that DataOps engineers do through automation are about orchestrating or meta-orchestrating all those tools in production. And then during production, testing, monitoring, observing the data and the artifacts that are created from data, like dashboards, to see if they're right before your customer notices a problem.

And then being able to help your data scientists or data engineer, being able to build a sandbox that they can play in that has test data. And then help them be able to do accurate regression and functional testing, so that if they make a change, they know that not only their change works in an automated way, but also how it affects every other piece of the pipeline.

And being able to deploy that with a button instead of meetings. And developing shared components and also developing measurements. And so in a lot of ways, DataOps engineering is about trying to take these invisible processes, pull them forward, make them visible through a shared abstraction, and then automate them. And a lot of the challenges with automation mirror some of the challenges I spoke of before. No one owns this, right?

It happens on Thursday evening, and somebody wrote a script that helps us deploy,

00:10:00

or Bob over here's got a couple of SQL things that he copies and pastes in every time we do a deployment. And one of the biggest challenges is that this overall process, that's incredibly important and incredibly valued, doesn't have an owner. And no one seems to care in some ways, because everyone's focused on, "My job is to produce SQL." And no, I don't think your job is to produce SQL or produce models.

Your job is to deliver value to your customer and then to iterate and learn from it. That's really what every data scientist, data engineer, and person working in visualization and government should have first. And there is some perception, I think, nowadays that the cool kids are all doing data science, and that this sort of DataOps work perhaps is for lesser beings.

And I saw that change incredibly in software, where it went from release engineers paid less than software engineers, now to DevOps engineers being paid more. And there's a perception that data's different, that you can't automate these things, which we at DataKitchen just completely disagree, and you'll hear about today. And this idea that you can't automate manual work is the way we've always done it. And so the jobs to be done in DataOps engineering and the automation that you're going to have to create are around building automation to create development environments and test data, and about being able to help your data scientists and data engineers modify and change production pipelines, and helping them create tests and building tests.

And also just advocating DataOps methods and speaking, giving a voice to this chaos and saying, "Hey, look, if we invest in this, it's going to make all of your lives better." And stop having the sort of the tragedy of the commons, where this thing isn't owned, and so we're just going to keep assuming and hoping someone else takes care of it. And so I think there is a degree, at least in this stage of the market maturity, of saying DataOps is a discipline that we need to invest in.

And so, what that really means is, and pardon my French, is automate the s**t out of things. If you're doing something manually three times, write a script, fix it. And I apologize for the word "s**t" here, but a lot of times, automation and writing a script that helps with the deployment or automated testing reduces waste. It lowers errors.

It allows people to work in a version-controlled way, and also maintains security, and also maintains reports on the process. And so this automation goal, I think, in addition to collaboration, is really part of the job. And if you look at it from a skill set, like when we hire DataOps engineers, we look for a bunch of skills, right? One is, do they know a language?

Can they script? Can they automate? And that's usually sort of Python or Bash. And then do they know at least one good data language like SQL? And do they understand source control? And of course, since we produce a software tool, it's great if they already know tools like DataKitchen. And then are there tools that do DevOps configuration like Terraform, or Puppet, or Chef, and Docker, and Kubernetes?

And then is there some process skills? Are you familiar with Agile methods and tools like Jira? And then finally, just the familiarity of the process of taking data and giving insight. What do data engineers and scientists and people in governance do, and trying to understand and be part of that culture. And so it is like its brethren role or its friend role, the DevOps engineer, it is a variety of skills. And that's what's great about the role, but it does require a number of skills in a number of different areas.

And so the other part in the role is if you think of it as a manager and like, "Well, how many of these people do I have to hire?" And one way I think about it is spend 15% of your time on these automation, and testing, and pipeline building tasks. And if you spend that 15% of your team's time, perhaps in a small team, it's all done with a day a week for a few people. But then after time, you can hire a role and right now, barely any time, maybe 3% of the team's time is spent on these sort of automation, collaboration, production deployment pipeline tasks.

And if you do that, you actually end up being able to do more work faster and better and happier. And I compare that to this lower graphic on the left in the slide. In software engineering, 23% of the time on average is spent on DevOps tasks. And so I think this is something that we just really underinvested in and we've been paying the price for years. And so DataOps is about owning the process.

And so if you do that, one of the things that we've seen DataOps engineers do is

00:15:00

that they change the percentage of their customers, the data engineers, the scientists, the analysts, the end customers. And the end customers end up getting more value. And they do that by saying, "Okay, spend more time doing good stuff. Don't spend time chasing errors or sitting in meetings, and spend some time reducing your technical debt, being able to modify something in production.

Don't get locked by, 'I've built something, I can't change it.'" And of course, the other benefit of DataOps engineering is being able to deploy quicker and faster and improving team efficiency. And so before I give it off to my partners in crime who do this day in and day out, if you are interested, there's a lot of resources about DataOps and data engineering that we have on our website.

We've actually got two books now, a manifesto, and we've become a great resource for people to learn about what DataOps is and what DataOps engineering is. So I'm done, and I'm going to actually hand it back to Bas to go on to our next presenter.

Okay, so Priyanjana, you should have control now.

Okay. Just give me a moment to share my screen.

All right. Hello, everyone. My name is Priyanjana Sharma, and I'm a DataOps Implementation Engineer at DataKitchen. I have been with DataKitchen for around four years now, and I have a data engineering background with a master's in information systems. I started my career working as a data engineer and then transitioned to a DataOps role. I currently work with the implementation team at DataKitchen, and I've had the chance of working with multiple customers on enabling the data teams to adopt DataOps. On my recent DataOps implementation project, worked with the engineer, business analyst, project managers, data enablement teams, as well as IT teams to automate their existing processes regardless of the technical architecture.

I have worked with teams with applications and infrastructures on-prem or any of the cloud platforms, including AWS, GCP, or Azure, or even using a multi-cloud or hybrid architecture. As for the tools, I usually use a programming language like Python, and a data language like SQL to work with various databases, ETL tools, and BI products that are available in the market.

Along with that, knowledge about some DevOps configuration tools like Docker or version control systems is mostly used in this role. So I believe for any DataOps implementation project, there are three major aspects. Number one being understanding the existing challenges of the data teams and getting to know the underlying architecture and tool chain. And with that understanding, establishing the definition of done.

Then the final step is designing a DataOps solution and its implementation.

Now, a DataOps project begins with listing down the challenges. And having worked with multiple customers on different use cases and scenarios, as well as a completely different tool chain all the time, I believe data teams are facing similar challenges. In majority of the cases, the biggest challenge is breaking data pipelines due to highly manual processes.

One such project that I have worked on involved dependency between multiple teams and various roles. If you look at the graph here, a business analyst consolidates data into Excel from some public websites, SFTP server, downloads some files from emails, and then applies some calculations in an Excel spreadsheet. Now there's a data engineer who's waiting to receive the process file from the business analyst and then loads the data to a database.

He or she then runs a Talon job that does the ETL to dimensionalize the data, produces a data mart, and then upon execution, sends an email to the BI team, to publish the Tableau report or refresh the Tableau dashboard. Now, there are a lot of challenges in this process, because there are no automated tests.

There are communication delays because there are multiple stakeholders belonging to

00:20:00

different teams, all having their own priorities, ultimately delaying the delivery of the reports to the senior management, who's responsible for making important business decisions. There's a fear of making changes as it might break production, and there is no process to spin up an isolated dev environment to quickly add in a feature, test it with the actual data, and deploy it to production.

Moreover, there's no monitoring to track the actions performed by the developers, its timeline, and the final output. Now, we talked about establishing the definition of done. When do you think your project is ready to be pushed to production and you can declare it done, keeping the learnings from DataOps principles in mind? I believe it's done when you're able to orchestrate all the tools, team environments, and processes in one single pipeline.

And you can manage these different environments, be it dev, test, production, SID, doesn't matter how many environments you have, and tool chain across all of these environments. All your code that you have written is version controlled in a system like GitHub, so you're able to maintain history and track the changes. The code is parameterized so that it is reusable across different environments and teams. The DataOps pipeline you have built, this is the most important one, it has enough automated tests to catch errors.

It's done when you are able to decrease the cycle time of change and add in new features quickly, as well as deploy these changes easily. Finally, in the end, when your implementation is complete, are you able to track and measure your process?

So as a DataOps implementation engineer, in order to transform the use case I just mentioned into a DataOps solution, I started off with gathering analytic nuggets from all the teams. And by analytic nuggets, I mean business rules from the business analyst, the SQL scripts from data engineer, and Talon job from the ETL developer, Tableau report from the BI engineer.

I then did a quick design of the DataOps solution to orchestrate all the tools, teams, and environments in a single pipeline.

During the implementation, my focus is to automate all the processes so that the data engineers, BI engineers, and analysts can spend more time implementing new features and less time worrying that their processes will fail and fixing errors. So in this project, I automated the extraction of data from SFTP, from the public websites, and email to be published to an AWS S3 data lake.

I used the business rules I got from the business analyst to process the source data in Python. I then orchestrated the SQL scripts, the Talon job, and the Tableau dashboard refresh all in one workflow using the APIs these tools provide. All the code, the Talon job, and the BI report are version controlled and wrapped via Git. The pipeline has automated tests at each step, making sure that each step is completing successfully.

There are some additional data quality tests based on the business rules, that check the dimensional model after the ETL job has completed. And in case of a failure of any test, if anything errors out, I have alerts configured that go to email, or you can configure it to go to Slack or Teams or any other collaboration tool your team uses.

Now, while implementing a DataOps solution, we make sure that the pipeline has enough variety of automated tests to ensure data quality and to leave time for more innovation and reduce the stress as well as fear of failure. And here is an example of a historical balance test, wherein I wanted to make sure that the total number of orders are always increasing.

I'm testing a current output of the total orders with a historical value of the same variable. And in case something goes wrong, I have configured my pipeline to stop and not execute any further. Now imagine getting a call from your CEO because your sales were off by a million dollars. And with automated tests, we have been able to catch errors like these even before the data was pushed into production, and it showed up in the reports.

Some other examples of tests that should be added in a DataOps pipeline are location balance tests to make sure that the number of rows in the data match the

00:25:00

expected value or threshold at each stage in the pipeline, or to make sure that if you're moving some files, they're not corrupted. You should also add some business rule-based tests to make sure that the business assumptions are always correct, as well as some basic data correctness and completeness tests. And these tests, they could be in any language.

It could be SQL, or it could be a variable defined by a Python script, or a variable defined by a shell script.

In this solution, the SQL code as well as the Python code, it's completely parameterized to ensure it is reusable and can be deployed across environments without any changes. If you look at the SQL code here, I have used Jinja templating, which is a Python templating language, in the SQL code itself. So the same query can run in different databases or schemas, in a completely different cluster by updating the connection information for just a couple of variables. Essentially, you are building a template and promoting code reusability by introducing parameterization into your code. And the same code we wrote to move data from S3 to Redshift, from SFTP to S3, or publishing a Tableau workbook, it can be used across teams, projects, and environments by updating just a few variables.

Just for example, I can reuse my published Tableau workbook code by just updating the name of my Tableau report, Tableau project, or even two different Tableau servers by updating the parameters, just the Tableau password or the database password that the reports are being pulled off from.

Adding a version control system like GitHub, it enables a way to manage environments, and innovate and deploy new code faster. Now, if I want to add a new feature, all I want to do is I can create a feature branch and configure it to connect to an isolated environment, say a different S3 bucket, a separate dev database cluster, and a dev project in the data source.

I can develop my new feature in my feature branch without worrying about breaking production, test the code in development, and finally merge it to higher environments and run the same tests again. And this way, automated testing is built into the release and the deployment workflow, and it's ensuring that my pipeline is delivering high-quality analytics every time it is being run. For data teams, monitoring and tracking is a very important feature that they're looking for to adding into their processes these days.

So I'm keeping a track of all the instances of the job being submitted and its metadata using the DataKitchen DataOps platform. And here you can see when the job was submitted, who submitted the job, when did it start, what was the duration, was it successful or not? I also record all these metrics and analyze it to build out a DataOps report to measure the progress in collaboration, keep a track of production error rates, and keep a track of the data error rates, test coverage, and the SLAs. And finally, I would just like to say that by introducing the change in our mindset, taking inspiration, as Chris mentioned, from methodologies like Agile, DevOps, and Lean manufacturing, we're able to streamline the workflows, catch errors much earlier in the process, increase the productivity of the data teams, and deliver high-quality analytics faster.

And in some cases, in some projects, in use cases, it has helped us save hours, weeks, and even months' worth of work. And that's it for me. I'm going to switch it over to Beth. Yes. Thank you so much, Priyanjana. That was awesome. So now we are going to give Chip the floor.

So Chip, you should be all set to go.

How's that? Looks good. Okay. So thank you so much, Chris and Priyanjana. I'm a panelist, but I feel like every time I listen to both Chris and Priyanjana, I learn something, and I appreciate that.

I guess the question that came to my mind as I was asked to do this, as someone

00:30:00

who has had a long and meandering career is, okay, what's so different about a DataOps engineer?

My career has progressed from work as a traditional database and interface developer, where I collected data, and I played a BI reporting role where I analyzed data, and I played a data architect role integrating data sets. And now I lead a team of DataOps engineers, where our role is really to integrate data flows. So each step is kind of built on the last and involves a larger perspective. And I wanted to highlight some interesting changes that I've seen over time in the last, oh my God, 30 years.

A DataOps engineer curates processes, and I'll talk about each one of these. A DataOps engineer shepherds process flows across complex corporate structures. And really what's different about this is not the idea of shepherding process flows and

catalyzing collaboration as Chris mentioned, but the fact that the complex corporate structures have changed so dramatically over time and over the last number of years. Amazingly, over the last single year, the change has been, of course, particularly dramatic. And the other thing I wanted to point out is that a DataOps engineer runs toward error. And you might think, well, we all deal with error. What does that mean?

And what I'm going to say is that we can't eradicate error, but we seek to manage it as part of a system.

So let's talk about curating processes. For a data warehouse, the product is the data set. For an analyst, the product is the analysis that they deliver. For a DataOps engineer, the product is an effective, repeatable process. And so we are less focused on the next deadline or the next deliverable. We're looking to create a process that works every time that it's repeated and run.

And I think as Priyanjana was saying, individual nuggets of code are kind of fungible from our perspective. Success is all about,

really, reliability and consistency.

So in order to do this, what we try and do is to leverage a common framework. Our goal is to seek out opportunities for reuse in the work that

people share with us. And it can be something as simple as encapsulating common joins, unions, filters, creating views. And it can also be something as challenging as carving out reusable steps from individual sets of processes that people share with us. The goal is to identify any of those first steps that lots of different people in different roles have to do in order to do their work. And if we can provide shortcuts by doing those steps, we can really give them some advantages.

In addition, what we do is to create and apply what I call meta tools. So we have tools, for example, that support data governance that can be shared among all of the processes that we manage. So we do timeliness tracking for data sources and builds and jobs pretty much as a matter of course. We have automated tools that create documentation of data structures, change logs that automatically are updated every day.

00:35:00

We have data profiling tools that we run automatically to compare versions of datasets. We have automated testing and a system for exception reporting, where tests identify issues that need to be addressed. And all of this serves to increase transparency, which is really critical to increasing trust in the process and trust in the result.

I'm sorry for this. This is a big slide. But the second thing is a DataOps engineer shepherds processes across the corporate landscape. The flow of data is something that is pretty fundamental. You pull information from wherever it's generated, you transform it and summarize it, and then you have to redistribute it to where it's needed for people who have to make decisions and act. The Critical thing here is that we're not just focused on our narrow swim lanes.

We have to focus on crossing boundaries. Often, the path of information doesn't follow the lines of authority or the traditional hierarchies of an organization. We have to follow that data wherever it goes. And it's harder than ever, right? Because we see more siloed business structures than ever, and these data paths will span across different departments, different buildings, different time zones, companies, cultures, and countries.

So that's, I find, a very interesting aspect of the job, but it's very challenging. You're serving as a catalyst to bring people together, and sometimes we've talked about the definition of done. Sometimes the definition of done can be perceived by others as a definition of ownership. And, people don't like that. And I think that's one reason why Chris was talking about making that distinction between owning the assembly line and owning the individual steps along the way.

It's not always as clear, and people can be very concerned about their own territory. And the impact of this, I think everyone who's dealt with it knows it. It's less transparency, it's less reusable logic and business rules, loss of control over source code, and it's harder to QC methods and data that have so many different statements in so many different places.

There are more points of failure, and ultimately, that leads to less reliable results.

One solution that I think has worked really well for us is this concept of building a process hub. And a process hub is a coherent, common framework. It's a shared workspace, a shared set of services, to amplify the value that other partners bring to the different processes. And an example of one of those services is an engine that we built to generate QC rules from baseline data, and then generate scripts that themselves create test scripts to actually perform QC tests based on those rules. And it's quite a nice way to, in as automated a fashion as possible, throw a real standard, large set of small tests at data that comes our way for different processes.

The users can take advantage of that. The common framework implicitly encourages collaboration. And inherently, importantly, it's a pipeline from a development process to a production process because they're really on a continuum, with users who are working within the process framework. And what I like about it is it leverages what people do best. The idea of bringing stuff into production is not something that analysts need to focus on.

00:40:00

Real advantages of this, rapid deployment, but when workflows are on the grid, you can see it, you can track it, you can measure it. You can do some of the great

graphs and reporting that Pranjal was talking about in an automated way. And the result, hopefully, is a single version of the process to support a single version of the truth.

The third thing is that a DataOps engineer runs toward error. And this may seem like a kindergarten slide, but this is really hard, right? Error is key to the feedback loop that makes complex processes reliable. And there is so much complexity in these larger data flows that cut across so many different structures that we can't possibly anticipate everything that will go wrong. We try.

There's always going to be something that's going to be a challenge to have to face. So each error is really an opportunity to go back and improve reliability. And I think of this in terms of machine learning. If you've got a model that works perfectly with training data, you don't necessarily see its bias and variance.

Then you subject it to the real world, the real population, and then you start to see things that are a little bit different, and you have to react to them accordingly. And then finally, the idea that errors are data, just like any other data, and need to be

understood and analyzed in order to do your job more effectively.

Now, here's my secret on how to avoid error. All my insider tricks here that you really don't want to do.

How to avoid error and fail as a DataOps engineer. Meet the spec. And by that, I mean if you narrowly define success, you can declare victory. And it doesn't necessarily mean that you're

going to do anything about errors that occur. So we had a situation where we had a partner who was performing updates based on matching identifiers and then updating data based on the data set that we provided them. And they reported with every update, they sent us back an email saying that the process had been successful. And in fact, it had been successful.

They ran through it. It did what it was set out to do. But what they didn't tell us was that there were a large number of non-matching records that we had provided that didn't match their data in their system. And the spec was to exclude those. We didn't know about it, and consequently, there were a significant number of updates that didn't take place because of an upstream error. But within their small universe, they met the spec and it was successful.

Another way to avoid error is to demand perfection. You're not going to get it. What you're going to do is stifle feedback, and you won't find out about errors that occur.

You can also resist disruption, which keeps the system static and avoids change. You can protect your secrets, and nobody likes scrutiny, and I say that as someone who doesn't appreciate it myself. But one thing I found at DataKitchen has been the value of transparency to improving processes for everyone, and it really makes a difference.

Another way is to QC the end product and not the process. You can confirm and validate that you've made one deliverable work successfully, but if you have to repeat it again and again, you need to QC the process every step of the way. And finally, everyone knows how to make problems disappear, and some of my favorite SQL does this. And we all know about it.

00:45:00

Select distinct to make duplicates go away. That may solve a problem in the short term, but it doesn't necessarily make the data- ... more correct for the next person who might have a different use case for that data. Another classic example of a Redshift has a clause when you're doing ingestions called max errors.

I know an excellent

DataOps engineer who recently set max errors to 1,000 because they were dealing with a problem with a data source format. And in fact, they missed another problem because of that, because they solved their problem, and they let 1,000 errors get into the system.

Alternatively, I think what we try and do is to love errors to death. We want to manage the possibility of errors as part of the process and improve systems. Every time we see an error, we try to address it with a new automated test. We try and create systems that expect people to make mistakes.

We don't like them, but we want to be able to catch them, and we want to empower people and not blame them, because we all need to know when these issues arise. And we also curate mistakes. On our team, we have a checklist of silly errors that we try and make everyone aware of because we've all made them, and we want people to be aware of them, and we want people to not make them a second time. We're also testing every step of the way instead of testing just at the end. You can catch more errors that way.

You can also recover faster in an urgent situation, right? You're doing a build, you want to know by testing every step of the way, you can identify immediately exactly when an error was introduced into a complicated process. And in this, I found sometimes a lot of simple tests can be better than a few complicated ones because you can't anticipate all of the different things that can go wrong.

So a wide net of simple tests can really be effective to tell you as early as possible that there's a problem. Another thing is to test what you already know is true, because you may be certain it's true, but it may not be true the next time because the data inputs may change or have errors in them, or somebody changed the process upstream of you, and therefore there's a failure or a logical flaw.

And then, also seek out feedback really matters. I was involved with a system where they didn't apply an overhead factor to a financial calculation. It should have applied the factor, but the downstream users realized it and applied it on their own in their individual end products. Then at some point later, someone on the team realized that the factor wasn't being applied, and it was a bug.

They considered it a bug and they fixed the bug, thereby breaking all of the downstream processes that were already applying the factor on their own. And because we didn't know about what was happening downstream of us, we didn't get that feedback, we actually made something worse in order to fix it. So,

finally,

it's all about collaboration. It's all about reuse, creating tested, reliable functionality and using that as building blocks to build other processes. And that's where processes build on processes and you can successfully minimize error and create a culture of transparency. And that's what I got.

Thank you so much, Chip. That was a great overview. So now we have a bunch of questions, so if anyone has any, please enter them in the control panel, and we'll get through as many as we can in the next 10, 15 minutes

00:50:00

or so. So, Chip and Priyanjana, if you want to turn your cameras on so we can see you. Let's see Chip.

Great, there you are. So Chip, a question came in during your talk, so I wanted to get to this one first. How did you build the shared workspace? I think it's referring to the process hub. Can you go into more technical details about tips for doing so?

I think you're on mute

. So we had a great shortcut. We have the DataKitchen platform, which really, I have to say, so I'm an employee of DataKitchen.

Basically, my function is as a customer of the platform. And I have to say that there are

aspects of the platform that implicitly make so much of this easier, and it's because it really creates that framework. There are certain things that I really like about it. I love the collaborative aspect, the fact that it's source code control, that users can have individual workspaces, that those workspaces can be merged, your work product can be merged via source code control into a shared staging area and ultimately a production environment.

Testing is built in as part of the process. But on our team, we've also extended testing in different ways. We've tried to create a testing engine that makes it easier for users who are less skilled to create rules on a template for certain kinds of standard tests. And,

by capitalizing on the platform and also adding some of our own stuff, we've really been able to-- The idea is to kind of build a bus system for data flows, where if you

put your data flow onto the process, you get a lot of these extra things that come along just as part of the deal. Okay. Thanks, Chip. So we have several questions about where the DataOps engineer fits into the data team. So is it in support or continuous improvement or architecture? And another one very similar, considering... Let me find it.

Considering that a DataOps engineer shepherds processes across the corporate landscape, where is the best place for this role? In the data office or enterprise architecture?

Who wants to take a stab at that? What have you guys seen or what suggestions do you have?

I'll take a shot. I think when you've got three or four data people together doing some data work, you need a DataOps engineer. Right? And so, because they're off doing their work and building their nuggets, and they forget the tragedy of the commons, and no one owns the processes between them, and someone's got to speak that voice. So I think having a DataOps engineer at least part-time aligned to those teams, I think is incredibly important.

And so I'm a big fan of small cross-functional teams. A couple of people, five, six, they own the result, they own the product, they work with the customer, and having a DataOps engineer as part of that makes it incredibly, those teams very successful. And I'm also a big fan of domain-oriented design or data mesh, where small teams can do a lot, who have ownership and can really focus on what their customers need and understand their data and those complicated end-to-end data flows. So, I think it's important that they be part of every team, and maybe not full time, at least aligned to them and be worked.

And what we've seen is some organizations are setting up centralized DataOps functions where that team can be utilized by one or more sort of smaller teams who are doing the data work.

And another related question is, how does a DataOps engineer fit into an agile squad composed of other engineers, a scrum master, or the product owner, et cetera?

I think it's the same answer, really. They're part of it. Right? And whether they're full-time or part, they should have a seat at the table because what I've noticed is really good DataOps engineers have this, it's almost this ew

00:55:00

reaction. They go, "Ew" to things. And they're like, "You deployed that to production manually? Ew. You didn't do any automated testing? Ew. We're having another three-hour meeting to go through a problem? Ew." And after a while, the whole team starts to internalize that and go, "Well, why are we having a three-hour meeting? This is dumb.

Can't we just automate it?" And so I think really, they're also an influencer on those teams, and maybe it sounds weird, the ew comment, but that sort of focus on what the team can do better to work together and fixing this sort of central space between everyone that no one's taking care of now, it needs a voice, and I think the role of a DataOps engineer, and I think he or she can help the team do that and should be part of any agile or scrum team.

So, a comment that was made, which I think is a really good one, and I'd love to get all your comments on this. So the fear of the existing data engineering staff is underestimated. "You're changing the way I do my work so you can eliminate my position." The answer is 100% no. We want to integrate these tools that we can free you up to work on more interesting and challenging issues.

Chip and Priyanjana, have you encountered this in your work, this fear?

Absolutely. Among a lot of different people, and because the role cuts across so many areas, I think everyone has that concern initially. I think the beauty of it is that we really give people time and faith in an underlying system so they can focus on what they really do best. They bring value to the organization by doing what they're empowered to do, not all the

scut work of... I think of it almost like a golf caddy. You help people select the right clubs, and they're the ones who are really doing the-- they're the ones who are playing the game. Yep. Another thing I think I... Go ahead. Yeah, go ahead. No, go ahead. Okay. I think I agree to what Chip said.

So we as DataOps engineers, we're not replacing data engineers. We're not replacing BI engineers. We are taking what they're building, applying best practices, applying DataOps principles, and improving their processes, adding in automated tests, making their code reusable, orchestrating all the independent pieces, and putting it all together so it's easier to track, it's easier to monitor,

so that the data engineers, the BI engineers, we free their time, and they can use their time to implement more features to make their code more better, and just improve on the project. So we are working with them, we're not replacing them. Yeah, and who wants to spend a two-hour meeting when something goes wrong and having 20 people on the phone call finger-pointing about, "Oh, is it the raw data?

Oh, was it your data work? Was it my data work? Was it the visualization? Did the model? Was it the server?" And everyone's finger-pointing, and you get two hours of your day wasted with people finger-pointing, and why can't you just have a system tell you it right away? And we all make mistakes, and then find a way to write a test so it doesn't happen again.

And so there's a lot of waste in data and analytic teams, in meetings, in concern and anxiety that doesn't have to be there. And I think in some ways, the role of a DataOps engineer can help change that. And I think technical people are creative, and I think data engineering and science and visualization are very creative activities.

And so, the DataOps engineer role is to be of service to those people. Just like data engineers and data scientists and people who do data viz are of service to their customers, trying to get them to see the value in data and make changes to their business or university or organization. And so it's this line of thinking and being of service that really helps, and also knowing where you can play and where you don't play.

And so successful DataOps engineers don't jump in and start writing a new predictive model or don't start-- they focus on the system and don't try to do other people's jobs.

Right. So we are at the top of the hour, but we have a lot of questions, so I want

01:00:00

to get one more in. So how many DataOps engineers are staffed for a data engineering team? Is that a separate role or something that you expect data engineers also to develop DataOps skills?

I'll take it. I think the answer is yes to both. I don't think you can be a good data engineer and not write automated tests. Right? And you can't be a good data engineer and not think about how the path that your work does to get into production and how you're telling if it's working or not. I just don't think you're very good if you do that.

And those things, yeah, they're DataOps functions. Right? And I think there should be DataOps engineers, and as well as you should consider it part of your job, and I don't think we need to be so drawing circles in you're in my circle and not in your circle. There's work to be done and customers to be made successful.

And so, I think

just like I spent 15 years in software, and when you start drawing circles around your role, that means you're not owning it, and that means you're really at the end. The thing that you want to own is making your customer successful. And who's your customer? Your business person, the consumer of analytics, and they need your support to get there. Because I did a webinar recently with Randy Bean, and the number of customers that are reporting that they're being data-driven at the highest level is shrinking. So I still think one of the reasons we're not being successful as an industry is because we're drawing a bit too many lines.

And yeah, you've got some DataOps work to do. And you know what? It's great. If you get someone to help you out full time, fantastic. You get to do more of what you do well. And so,

maybe I'm on my high horse, but I don't know, Chip or Pri, what do you guys think about that?

Couldn't agree more, and it's a kind of mindset that really benefits everyone, right? To think outside your swim lane, to think beyond the next deliverable, to think about creating an enduring, resilient process.

So we are out of time, but I'm going to sneak in one more question because I think it's a good concluding question. So when no DataOps architecture exists, where and how do you recommend to start besides understanding the business and needs? And this is looking more at the technical level.

I guess I'll take it again. So yeah, I think the first place to start is

where you have problems in production because that causes the most obvious pain. And your development process, being able to iterate quickly, that's important, too. Measurement's important, too, collaboration. But man, if you're having problems in production and your customers aren't trusting the data and things are broken, let's start there. And if you can't get that right, then you lose credibility.

The data that you're all working on loses credibility, and then the whole data and analytic function becomes kind of a joke because your business customers are doubting the data and then just following their intuitions anyway. And so, I think that's a good starting point is just to start loving your errors in production, start fixing them in an automated way, and then use that as a springboard to working on some of these other sort of cross-functional processes like deployment or environments, measurement.

Great. Chip and Priyanjana, anything to add there?

I would say don't forget to convert your manual test to automated test.

That's great. Well, sadly, we're out of time. There are a lot more questions we didn't get to, so I apologize if we didn't get to your question. I know people have to run to their 1:00 meetings. So, if we didn't get to your question, we'll follow up with you directly after the webinar. I want to thank everyone for taking the time to join us today.

An extra big thanks to Chip and Priyanjana for coming on today and sharing your insight. We really loved having you, and maybe we can do this again sometime. And thank you, Chris, as always. So, for all the attendees, we'll be sending out the recording and the slides to you within the next 24 hours, so please be on the lookout for that email. And if you have any additional questions about the presentation, please don't hesitate to reach out to us directly.

You can reply to the email that we send or send an email to Beth at DataKitchen.io,

01:05:00

and I'll get it to the right person. So again, thanks everyone, and have a great afternoon and evening. Thanks, everyone. Thanks, everybody. This was really fun. Thank you.