10 Tips to Overcome Data Engineer Burnout

00:00:00

Hi, everyone, and welcome. I'm Emily Pick, product marketing manager here at Data.World, and we are so glad you're here. Before we dive in, let's quickly spin through some housekeeping. You're probably already familiar with most of this, so I'll go quickly. First off, this is a pre-recorded broadcast. That means you'll be able to enjoy a presentation that's free from technical difficulties or unexpected interruptions.

It also means that you'll be able to chat with our speakers throughout the broadcast. Just reach out using the Q&A button on your screen. Also, this webinar will be available for on-demand replay in about 24 hours. You'll receive an email with the link, just in case you missed any of this or want to share with colleagues. We also have some excellent resources available for you to check out in the webinar console. We hope you enjoy the presentation.

So welcome, everyone. Thanks for joining us today. My name is Beth Pfefferle, and I am the VP of marketing at DataKitchen, and I'll be the moderator for the discussion today on the results of our 2021 data engineering survey. So we're very excited to share what we learned with everyone. Hopefully, you all find it extremely interesting and telling.

To do that, we have Chris Bergh, the founder, CEO, and head chef at DataKitchen joining us, as well as Brian Jacob, the founder and CTO of Data.World here, and they'll chat about the results, share their own experiences as data engineers and managers of data engineering teams, and most importantly, they'll offer up some solutions. So welcome to both of you.

Thank you. Chris, would you mind giving the audience a brief little bit of background on you? Yeah. Thank you for the introduction, Beth. I'm a technical guy, and so I did software for 15 years, and then I got the bright idea to do data and analytics, and when I left the software field, I thought it would be easy. I thought I would be home by 5:00, and life would be roses and the days would always be sunny. And for many years, I really had a hard time producing data and analytics that were trustworthy, that could change with the needs of the customers, that could satisfy my team and their need to innovate.

And so I think a lot of those things actually show up in the survey, and so I'm excited to talk about it. Great. Thanks, Chris. And Brian? Yeah. So I'm Brian Jacob. I'm the CTO, co-founder of Data.World. Similar to what Chris said, I've been in software for my whole career, and as data has become more and more central to the operation of software systems, a thing that really started to become obvious to me was that a lot of the practices that we have for managing software and for managing people who work in software haven't yet been applied in the field of data.

So a lot of what we're doing is basically trying to take those same practices, those same ideas, and bring them to folks who are data practitioners through the use of our platform and through the practices that we talk about. Great. Well, thank you both, and welcome to both of you. We're definitely looking forward to getting both of your perspectives on the data.

So first, before we jump into that, why did we even survey data engineers? I'll just spend a few minutes giving a little bit of background. So for several years now, the elephant in the room has been that data and analytics projects are failing. In fact, several years ago, Gartner estimated that 85% of big data projects fail. More recent data from NewVantage Partners, shown here, show that the number of data-driven organizations has actually declined to 24% from 37% from several years ago, so that's a pretty big drop. Only 29% of organizations are achieving transformational business outcomes with their data.

In addition, only a third of companies have an established CDO role, and the average tenure of the CDO is only two and a half years. So when we add all these facts together, this definitely supports the story that something is amiss in the data world and that processes are broken. Yet, among all this, one area that hasn't been studied is the data engineering role, so we felt that it would be really interesting to look at how data engineers are doing under these circumstances.

Are they thriving, or are they feeling the impact of failed data projects? So we surveyed 600 data engineers, including hundreds of their managers, to understand how they're faring and feeling about the work that they're doing.

00:05:00

So the top-line result was that 97% of data engineers are feeling burnout. So to me, that was actually a very surprising number. I thought there would be some burnout, but that's pretty high, so I'd love to get the panelists' reactions to that number. Brian, did you find that surprising? Anytime you see a number that high, it's somewhat surprising, but not really.

The notion that most folks in that field are feeling burned out doesn't surprise me. I think there's not nearly enough kind of support infrastructure or expectation that data engineering... Data engineering, it's software engineering, but a lot of times it's not really managed or treated like that. It's something that folks are doing without a lot of the same support infrastructure that exists, and it's kind of well-honed over decades for software engineering, and we could go into a lot more detail on that.

But that to me is the number one biggest reason for this, is that folks are kind of expected to just make it work, and like a lot of things in software, that works until it doesn't. And I think what a lot of these folks are feeling is that they've kind of hit that point of scale where just kind of ad hoc processes aren't keeping up.

Yeah. And Chris, would you agree with that? Yeah. Excuse me. If you go under it a little bit, I think, why does anyone feel burned out, right? Because they don't feel successful, and there's an unending wave of stuff heading their way. And you could call that wave customer requests, you could call that wave broken systems and errors, but a lot of people in that role, they feel under a tsunami of crap, and they can't get out of it, and so that just leads to burnout.

And so, to me, I'm not surprised because the role itself is, in some ways, I think it's fantastic that we've invented the term data engineers, and I used to call them ETL engineers, and someone called them the data guy, but these roles have often been filled and seen as cost minimization roles and not as generators of value. And so data engineers, the term, the professionalization of it, has actually been a great boon in my mind because it actually represents that they're a true source of value. However, that hasn't actually changed their perception in a lot of organizations.

They still have a lot of requests, still stuff flows downhill to them, and they don't have the means to get out from under the requests, and then people look askance at them when they aren't working night and day to get things done. And they end up fixing the same problem over and over again.

I think without that support infrastructure, a lot of that kind of continuous expectation of just work day and night comes from, they don't fix problems and keep them fixed. They aren't empowered to do that. They aren't trained with the skills to do it necessarily, and they aren't given the kind of support to go treat this like engineering, where I want to fix this problem and fix it with a system so it stays fixed.

Yeah. And there's a cult of heroism in data engineering where I'm going to work really hard, I'm going to pull all-nighters, I'm going to get up on Saturday morning and fix it, and that's my role, and it's sort of like John the Baptist hair shirt of data engineers, and that's what you've got to do.

And I sort of reject that. I think that does lead to burnout. It leads to employees-- And I've had the situation where people just suddenly they're productive, and then they're just up and gone, and they take a new job, and they say it's because of the commute, or they say for whatever reason, they're nice about it, but they've just got to get under and start over again.

And so when you see people leaving your ... your company, they're taking a whole bunch of knowledge with them. And in a lot of cases, that knowledge is only in their head, and they've built something that only they can run. And so burnout leads to all sorts of business effects of having things that are brittle and broken and increased cost and lower efficiencies.

Great. Well, thank you both. So, we also dove into the reasons that data engineers are burned out. So we have some of the top five here, so I'd love to get your thoughts on these as well. So these aren't in order, but the first is that 50% of respondents were feeling a relentless flow of errors.

Chris, do you want to comment on that? Yeah. Your data providers don't know that you exist. You're kind of an afterthought. I've spoken at giant DevOps conferences, and I say, "Hey, this relationship between dev and ops is great, but you also have a relationship between the software team and the data team. And you know what?

00:10:00

They're your customers." And they're like, "Oh, that's really true." And you have thousands of people building software-defined systems who don't understand that there's a whole team of people trying to analyze data from it. And so we get tables that change, schemas that change, data that's forgotten. And so, your data providers don't care that you exist, honestly, and they probably won't. They should, but they probably won't.

And then, we live in a technical world, and servers happen, and

everyone's trying to work hard, but people are working at their max capacity, so they put code in the system that breaks. And so you have to deal with... And the data itself changes in ways that is sometimes unpredictable, and it could be syntactically correct, but the semantics change so much. And so at the end of the day, you put something out for your customer to use, and you're kind of hoping it's right. It's like, "Oh, is it going to work?" And you wait until they yell at you. And if you don't hear anything for a day or two, you start to feel good that you did it right. And I don't know, that just seems...

I don't really want to live that way, hoping that things are right. And it's one way to deal with the relentless tide of errors, but hope is not a strategy.

And Brian, has that been your experience as well? Yeah. Everything Chris said really resonates, and just, again, relating it back to software. If somebody else is building a piece of software that I'm expected to integrate with, we're going to have a contract, an API between those two systems. If that breaks, immediately, I can't use this software anymore.

I'm not going to even try. If this breaks, it'll be known, and the onus is on the person providing that interface to fix it. Your schema, your valid data, that's your API into your data, but we don't tend to treat it like that. It's treated as this ad hoc loose contract, and so when you get these errors, whose fault is it? Is the data being written wrong, or is it being interpreted wrongly?

If that's not clearly defined, then it's an unfixable problem, it's nobody's problem, so it's everybody's problem. Great, and we're going to talk a little bit later about some ways to fix this. Yeah. You guys ready for the second reason? Yeah. Okay. Manual processes crowd out innovation. So 50% of respondents were feeling this pain.

Brian, would you like to comment on this one? It's that heroism that Chris talked about. It's almost always faster right now today for me to just manually fix the problem and make it go away so I can go to the next thing on my queue. The problem is if I fix it manually today, the same problem comes back tomorrow and the next week, and it comes to the person who takes over my job when I leave.

And, without kind of stopping and saying, "The fact that this is broken means a system is broken, let me go fix that system so that this particular flavor of problem doesn't happen to me again tomorrow or to the next person in a week." This absolutely resonates because I think, when you're in that hero mindset and you're just solving this onslaught of problems, it's always faster to just make it go away until tomorrow.

Yeah. I think a lot of people, they see their life as a queue of tasks, and their job is to service that queue. And the faster they can take things off and check them off, the more they feel done. And they don't stop and reflect to say, "Wow, this task, I'm doing it repeatedly over and over and over again. And you know what?

If I automated it..." And I actually when I first started my career in 2005, I worked for a doctor, and he had some experience with Deming and Taylorism, and he actually, we brought in a consultant to do time motion studies of ETL engineers to see why it was taking them so long, and looking at all the tasks they did. And that sort of Taylorist mindset that it's a bunch of checklists, and if you service your queue, you're going to be better is not the right way to think about it. And I think we're buried under the view that I'm a data task doer. And more tasks done, more value to me.

And I don't think that that's really what we should be focused on. It should be focused on the one most important question, is my organization, are my customers getting value out of the data? And then the other case that I think software engineers take pride in is what's the least amount of work I can do to get there?

And how can I get this done so I have more time? And how can I shrink my task queue through automation?

00:15:00

And those are different perspectives, that go against the "I'm just servicing a data task queue day in and day out." Great. Thank you both. So the third reason number three, a steady stream of half-baked request. So again, nearly 50% of respondents were feeling this. Chris, was this your experience as well? Yeah. Okay, this bugs me. And maybe because I'm a little older.

All requests are half-baked. There is no such thing as a baked request, and get over it. And your customers don't know what they want, and that is the way the world is because they're busy, and because our talents in data and abstractions and coding are a bit weird, honestly, and not everyone in the world has them.

And they can't express them clearly. And it's your job to iteratively pull out what the most value of the requirements are. And so half-baked requests are norm, and get over it. And the other part that I think is a struggle is people defensively say, "Those requests are half-baked, therefore they don't need to get on my queue, and I don't have to think about them." And so half-baked requests are where insight's generated, where value is created, where companies are getting value. And so that dialogue between you and your customer to service those half-baked requests is incredibly important.

And the faster you can go through that cycle, the more you can learn. And then the other part of it that's in the spirit of agile and software is you actually do less work because you learn what really matters. And, when you take half-baked requests, the other aspect of half-baked requests is people take the half, they fill it with what they think the customer wants, which is often perhaps not what they want, but what you want to do technically, and then you overbuild, and you end up building too much that's not really needed.

And so the idea of iteration, seeing the world as a dialogue with your customer, a river of questions that's unending, as opposed to building a house and walking away, I think that's the essential part of sort of focusing on value, focusing on customers. So, this is the way the world is, and honestly, this is good.

This is an opportunity for you

to show how great you are as an engineer.

That's some great feedback, Chris. Brian, would you agree with that? Yeah, yes, for sure, and I would relate it to just the notion of engineering and product thinking.

Chris said every request is half-baked. The discipline you kind of learn thinking about building software products is, your customers come to you with problems. It's your job to craft a solution, right? It's like the old Henry Ford, if he built what everyone wanted, he would've built a faster horse, not a car, right? And that's the place you find yourself in as a data engineer, is when people come to you with problems, your job is to solve that, to come up with a solution.

The problem is why requests seem half-baked is usually customers come to you with their problem wrapped up as a solution. They usually come to you and like, "Oh, just add this field to this data feed. Oh, just fix this." Well, that's just taking that and saying, "Well, I'm going to go do that," and then complain that it's half-baked. That's the wrong approach.

You step back, you say, "What is the problem that this person is packaging up with this request? How can I solve that? How can I improve the system so that that problem gets solved? But I'm also thinking about solving the next 10 questions that are going to be in that same

vein." And that's that product thinking that again, this is what you kind of learn and build a discipline around software. We need to bring that to the notion of data engineering, because it's another flavor of the same thing. Okay.

Reason number four, blaming and finger-pointing. So 63, so a pretty high percentage of the respondents were feeling this. So- Chris, yeah. Or Ryan, you can take this one if you like. Oh, sorry. This one really resonates because, I think, again, in the software world, you're using source control. The more you kind of shine a light on proper providence of how things happen, the less you have blaming and finger-pointing.

The more it's kind of an opaque ball of, well, something's wrong, and no one really has clarity in how that works. I think you don't see as much blaming or finger-pointing specifically because, in the software world, you're probably using Git. Git literally has a command called Blame.

00:20:00

But what Blame does is basically tell you exactly who changed this line of code and when, and how did it get to be where it is. So, in the software world, that's kind of shining light on the providence of, how did this piece of software get to be in the state it's in? The more you do that in data, the more you kind of have transparent systems that show this is how everything moves state to state, the less there's blaming and finger-pointing. It's pinpointing, this is where the problem is, and this is what we can fix.

And it's less important who did it, when, and how. It's that this is where the problem is, and we can go deal with it. That makes a lot of sense. Chris, your thoughts on that? Yeah. If you think about the social structure that data engineers work in, right? In one case, there's a data engineer and a data scientist, and someone doing business intelligence or visualization as one team, as one unit.

But oftentimes, the data engineers are their own unit. And then there's people who are doing business analytics in another part of the company, right? And the real customer of the data engineering team is trying to enable those self-service tools like Tableau to use it. And then the customer itself is the third one. And so,

you've got to kind of know who your customer is, right? And a lot of times, sometimes for data engineers, you're on a team, sort of a full stack team, and your customer is a business person, wearing a suit and tie. But oftentimes your customer is a data scientist or someone doing business intelligence. And so, we're all part of a system that the end customer, who really is paying our salaries.

And so, when you are pointing the finger, it's easy sometimes to point to these other people over here who don't understand. And there's a lot of Hatfields and McCoys in organizations where there's a central team in line of business analytic teams go on, and there's no awareness. Problems happen. The guy or gal in the suit finds a problem in your dashboard.

You're on a phone. You've got three different line of business groups. You've got your centralized data engineering team. You've got your data governance team, and you don't know where it lies. And so people are always saying, "Is it your fault? Is it my fault? Is it the data's fault?" And so, to me, this is a symptomatic of a greater problem, and that we don't know that they're there, right?

There's no way to point and find out where the problem is right away. And instead of instrumenting our systems to show where problems are, or perhaps even better, find them before the customers see them, we're having phone calls and meetings, and often it's the people who are your best people on it. And that all further exacerbates the bottleneck issues that some team has.

And if you're on these meetings where you're talking to other people on the team or your customers, and you're trying to diagnose something with 20 people on a phone call on a Saturday, that's a symptom that you need to improve the system. And that's not the way it is, and you're not being a hero.

And honestly, some organizations are great. They like conflict. Other organizations tend to be more passive-aggressive, and then you sort of complain in silence. But when you're blaming and finger-pointing, that's an indication that things need to improve.

Okay. And last but not least, 69% of respondents feel they have restrictive data governance policies. Ryan, do you want to comment on this one first? Sure. I think,

generally, I would classify this more as lazy data governance policies, right? And not kind of product and value-oriented data governance policies. The default position of most folks in data governance is they're incented to make sure that there's never a misuse of data, which is a fantastic goal and is absolutely necessary. But they're incented only by that, and so the easiest way to implement a data governance policy is to just lock everything down and make it very, very difficult to achieve access to anything. And that's one way of knowing that only the people who should have access to something get it.

You have to kind of invert that and say, "What I want is to have the maximum value out of this data. This data's an asset. It's not a liability. I want to actually have a data governance policy that figures out, okay, who can have access to it, and make sure those folks can easily discover and get access to it." So I absolutely would agree that restrictive data governance policies get in the way of data engineering work. And I think that comes down to incentive structures, how governance has traditionally been done.

00:25:00

This is a problem for everyone. This is a problem for the whole company that data governance is done in this fashion. And Chris, your thoughts? Yeah. I think there's always a river of change, right? And giving insight. And if you look at the work that people do, that change is expressed in perhaps SQL code or Python code, or a Tableau workbook, or an Alteryx workbook, but it's also expressed in a data catalog or data lineage or the rules that govern data security. And I think those artifacts themselves are worthy of iteration and improvement. And so I think what's happening now is data governance 10 years ago or 15 years ago was always the cranky 50-year-old who kind of got out of doing ETL, and knows where all the bodies lie in data, and that was governance.

So governance was a person. And I think governance is a process, and I think honestly, governance as code is probably the best way to do it. And so I think these restrictive things actually stop people from innovating on data, and that's why we're here and why we got into this field. Okay. So how can we make it better, right? A lot was covered here.

It feels a little bit dire, so what can we do about it? So here we have 10 tips to overcome data engineer burnout. So the first is, don't suffer, build a system. Chris, do you want to talk a little bit about that? Well, yeah. It's the last 15 years of my life, so yeah, I guess I can talk about it.

Yeah, I suffered for many years, right? And my wife thinks I started this company just as a whole, it's therapy. She basically thinks you started a company because your life was hell for so long. And in some ways that's true, right? Because- I didn't like waking up in the morning and worrying that the data was wrong. I didn't like people calling me up saying it was late.

I didn't like my customers rolling their eyes at me saying, "You big, dumb tech nerd, you're too slow." I didn't like having employees go with me and complain that they couldn't try something out quickly and learn. And that's not a fun position to be in, and I think everyone has those problems. And so to me, I think the process I went through internally was to reflect and then say, at first, I thought it was something particularly wrong with me, as a person, as a manager, and I read books.

And then I started to realize, no, I needed to actually think about the system, and I started to read Deming and think about it's a factory, not just an artifact. And from Deming, I started to read more stuff about DevOps, and how we actually build the system. So, the answer here isn't to hate yourself or hate your team or quit and go on. It's to say that you have an opportunity here to transform that suffering into system building, and that can actually help you and everyone else on your team. And to me, I like this phrase because it's a lever.

It says, "Okay, I see the problem." These characteristics of burnout and lack of data projects being success, that's a pointer to building systems to solve them. That's not an indication of personal failing or that the field is broken. It's that we need to do some work, and building these systems around our data tools, and reflecting on how we work as a team, I think is part of that.

And Brian, would you agree with that? Yeah, absolutely. We

talk a lot in software engineering around there's a lot of power in being the person whose fingers are on the keyboard, right? Chris was like, "Oh, you big, dumb tech nerd, you're too slow." But at the end of the day, you are the person who actually can make this happen. You are the person who actually has the technical skills to get this done, and you can empower yourself to do exactly what Chris just said, which is when you're faced with a problem, you don't have to just take the today path of least resistance and do the manual task one more time.

You can step back and build yourself a system that makes your life better. You're just as heroic. You solve the problem just as well for your customer, but you actually make the whole system better, and over time, you can make your life and your customer's life better. You don't need permission. You are empowered because you're the person who's got fingers on the keyboard solving the problem, so solve it in the way that scales and makes the system better.

Okay, so great advice from both of you. So number two, don't agonize, automate that s**t.

00:30:00

Brian, do you want to take this one? Yeah, but I would mostly just repeat what I just said. Yep. Don't agonize. Don't wait for permission. You do not need permission to solve your problems in a way... You don't have to treat it like taking a queue of manual tasks. You are empowered to turn your job into automating this.

Don't solve a problem manually. Build a robot that will solve it for you every time it comes up from now on. This is the way. Yeah, this is the way. Yeah, exactly. Like "The Mandalorian," this is the way. Chris, anything to add there?

Yeah. From a leader's perspective, you need to make time for your team to automate stuff, and you need to celebrate automation, and not see it as crap work, but see it as value generation work. And the transformation that's happened in software was when I last ran a dev team in 2000, we had one release engineer in a 35-person team, and that release engineer was paid less than everyone else, and we had a SaaS product.

Now that has transformed into a DevOps engineer, and most organizations have 25, 28% of their team devoted to DevOps, devoted to automating that s**t. And why? Because automation gives velocity and helps your team, who are actually creating value, get it done. And so automation is a velocity generator, and so this career path of automation, I think, is really important on data teams.

And what we mean by automation, again, is not the hands on the keyboard help you automate building fast-writing SQL code. In some ways, I'm a little dismissive. I've done SQL, I've done Python, I've done a bunch of languages and a bunch of different tools, and yeah, they're better, one tool's better than the other.

But if you can build a system around it and automation around that system, you get 10X the leverage. And so devote time to it. It's not scut work. It's not for lesser beings. And what's happened in software is your DevOps engineers are, in a lot of cases, actually paid more than the sort of back-end and front-end software engineers because automation provides leverage.

Great. That's a really good point, Chris.

Leadership needs to embrace this and run their teams this way. I 100% agree with that, and that's the trend. That's where this will go. But in the meantime, don't wait for permission as well. Exactly. If your organization hasn't caught up to the need to work this way, work this way anyways. Great. So, the next tip is run toward errors. Chris, do you want to take that one?

Yeah. You live in an error system, right? Data has errors in it and will always have errors. It's never going away, and so don't hide them. Don't forget about them. Errors are opportunities for improvement. Errors are opportunities to automate. And so in a lot of teams I see, they're embarrassed by errors, but it's really a place for you to shine, to automate, to improve the system, and it's an opportunity for you to learn. And so I think, don't avoid errors, just run towards them. And it's a really counterintuitive thing because most of the time we're embarrassed. Does it make us look bad?

Are we going to lose some positional authority in our organization? Yada yada. But on the other hand, this is really a great place for you to shine. And Brian? Yeah. Errors are opportunities to improve. Yeah. Every time you see an error, it's an opportunity to build a system and make sure that error doesn't come again.

Okay. So fully embrace this. Once you know you have a problem, fix it. So next is don't be afraid to make a change. Brian, do you want to take that one? Sure. Maybe I keep getting ahead of myself in answering these points earlier because, again, power the keyboard. You are empowered to make this change.

There is nothing saying that you must solve your day-to-day problems with manual labor. You can solve them with automation, and you should, and you shouldn't wait for permission.

And Chris, your thoughts on that? Yeah, a

lot of agility is the balance between sort of fear and heroism, right? And finding the right balance. And yeah, there's some cases where you want to reduce that fear and reduce the heroism at the same time, and get the right balance. And so "don't be afraid to make a change" is really reflective of that.

Okay. So next is done means live in your customer's hands.

00:35:00

Chris, what does that mean to you?

There's this notion in psychology of sort of putting a wall up or being defensive, and a lot of data teams say, "My job is to get to X," and at some arbitrary point on the journey, that data takes the value. My job is to fill a database with a schema. My job is to get the model done.

My job is to tweak this dashboard. But at the end of the day, what done means is your customer's getting value out of it, full stop. Right? And that's what done means. And so, focusing on products instead of projects, tear down those walls in your head, and really focus on making your customers successful.

I think it's easier sometimes to make an arbitrary intellectual line in the sand and saying, "I'm done at that line," but done really means it's in your customer's hands, and they're getting value. And do you agree with that, Brian? Yeah, for sure. The definition of done is one of the kind of hardest principles to get into any kind of engineering team.

And, it's like that last 20% that puts it in your customer's hand is often the thing that you'll shy away from and maybe is the less fun kind of polishing and finishing work, but it's the only thing that matters. Getting it most of the way there and then letting it sit doesn't do anyone any good, so this is definitely- Okay. So next, we've already talked about this one.

Don't be a hero. Make heroism a rare event. Brian, do you want to elaborate on that one a bit? Sure. Yeah, and I think it is important to call out. We keep talking about heroism. Heroism is great. Heroism is going to save the day sometimes. And sometimes you just got to put in that long night. Sometimes you've got to go in.

Sometimes it's definitely the right decision to do the thing that doesn't scale and just get the job done today, right? We've talked a lot about how you shouldn't do that, and as a general practice, you shouldn't. But, part of being in an engineering kind of role is having that judgment for, "Actually, my customer's on fire. I need to fix this as fast as possible and put a solution in their hands." It's then the discipline to then circle back and say, "And then tomorrow I'm going to go back and start working on how do I make sure that this particular fire that caused me so much anxiety and required me to be a hero doesn't happen again?" So, yeah, heroism, it doesn't go to zero, but if that's the only way you ever get your job done, then all you're going to be is an exhausted hero, and you're going to burn yourself out.

So number seven, seek out opportunities for reuse and sharing. Chris, can you tell us a little bit more about that?

Yeah. No, I'm sorry. The work that we do in data and analytics is code, and you don't want to copy code from someone else and tweak it a little bit and then create. Copy and pasting is not an engineering strategy. And so if you can build, reuse a couple components, you can share those components with other people, you can then scale because we're in the complexity business. And so, software engineering has kind of dealt with that for years, but we have large distributed systems, lots of different tools, lots of different data sources. And the idea of abstraction and grouping things together into functional units, making them reusable, giving them APIs, is really helpful and can provide leverage to your team.

And so technical design at a high level, architecture design, scalability, reuse, are important features that we shouldn't dismiss in our pursuit of delivering data value for our customers. Okay, great. Thanks, Chris. Number eight, practice agile data governance. Brian, can you tell us what this is? Yeah, sure. This goes to the comment we made earlier about restrictive data governance, but again, data governance is very important, right?

You should not read my comments as saying you should be laissez-faire about data governance, and it doesn't matter who has access to things. It matters. It matters very importantly, but we need to change the value we ascribe to data, and governance should serve that value. So, for me, agile data governance is applying agile principles the same way we apply to product development, the same way we are thinking about your data as a product and being agile in how you define who can have access to this, what kinds of access are appropriate, how can this data be used to maximize its value while constraining how it can be used to make sure it's

00:40:00

not used inappropriately in ways that are unethical or contrary to regulations or are strategically fraught. Okay, great. Thank you, Brian. Number nine, measure your processes and improve them. Chris, can you tell us a little bit more about what that means?

Well, if you look at our slides, we talked about errors and overwork and how slow it is to change things. And we're in the measurement business, right? We're trying to measure the organizations we work in. And so I think we should apply that to our own teams, our data engineering teams, our data science teams. Measure the work that your team does and look at its output. And I think you're going to be surprised at how

much time people spend in meetings, how many errors you have that go into production, how long it takes you to actually make any changes. And I think these key measurements of error rates and cycle times and productivity are something that we should apply our analytic skills to how we work as a team and how we create value for our customers.

And that's going to be very insightful, and it's actually going to point out some of these problems that we see today. Okay. Thank you, Chris. And lastly, number 10, don't put your head in the sand. Focus on value delivery. Brian, what are your thoughts on that? I think it's the summary of a lot of the stuff we've already been talking about, right? Don't suffer. Build systems. Automate these things.

Don't be afraid to go ahead and make the changes. Done means done in your customer's hands. Being agile with data governance and making sure that we're maximizing the value. The whole point of all this is treating your data like a product, focusing on that delivery of value, and then doing the things that help you engineer toward that value.

I think if you take the long view of this, just as Chris was saying with how DevOps has become so core and respected, this is going to happen in this space. Engineering is engineering. I think there's a lot of notion that data engineering happens on this fast time cycle. We don't have to do all those engineering processes because they slow us down. They don't slow you down, they speed you up.

It's the saying I've heard a lot of my colleagues kind of throw around in this, which I love, why do you have brakes on your car? So you can go slow? No, it's so you can go fast safely. All of these engineering processes and automations take a little bit more time each time you set them up, but they save you huge amounts of time downstream, and as this practice matures, it's going to become an engineering practice just like any other software engineering practice. You might as well get ahead of the curve and be part of that.

Great. Thank you, and I love that analogy of the brakes on the car allows you to go fast. So, great. Well, thank you guys both for sharing all these tips to overcome data engineer burnout. So one last question for both of you. Frankly, when I first read the results of the survey, I found it a little bit depressing. So it's great to hear all these ways that it can be overcome.

Do you both feel hopeful? Chris, do you want to answer that first? I do because there's a set of ideas on how to run teams who are working on a shared, technically complicated thing that were created, honestly, in factories 30, 40 years ago, right? The Toyota production system, lean ideas. And then some people saw the same sort of problems in how we build software, and the sort of DevOps and Agile movement was born.

And the same problems are cropping up in data and analytics systems. It's a technically complicated thing. It's a team working on it. And so I'm optimistic because I'm deterministic. I just think these ideas are going to win, because the ideas are right. And, collectively, I've seen we drive cars now that don't last 50,000 miles.

They last 150,000 miles, and maybe electric cars are going to last 500,000 miles. Things improve. And, I think this is really a huge opportunity for people's careers. It's a huge opportunity for the market. And so these problems that we talk about may seem depressing, but you peel and there's an enormous opportunity to change.

And, I'm excited about it because I'm happy to see that this... When we first started talking about these ideas seven or eight years ago, most people thought we were just aliens, and looked at us askance, and the ops term, while being fashionable, has become much more accepted, and people want to be agile, and we're finding CDOs who have DataOps or DevOps for data or however term they use as part of their goals.

00:45:00

And I'm excited by it. I think it's a big opportunity for your career in the future. It's a big opportunity for us to help move our field forward. And really, the burnout is a symptom, and sort of focusing on agility and DataOps is the cure. Great. And Brian, how do you feel? Do you feel hopeful?

I am, yes. I'm 100% hopeful. I'll be brief because so much of what I would've said, Chris just said very well. So the only thing I'll just really add to that is it's absolutely inevitable. I have zero doubt that systemically this will be the trend. The things that we talked about today is how these practices get done.

I think the question, if you're one of the 97% of people who today would say you're overwhelmed, is like, do you care about fixing that problem for yourself? Eventually, this will be the norm. This will be how things get done. It doesn't happen automatically, right? It takes people building software and systems and talking about these processes.

It takes people empowering themselves, but that will happen. It's already happening. It happens in every kind of engineering discipline. It will happen here, too. It's just how early do you want to be on that curve, and how much do you want to make your life better right now today by jumping on the bandwagon?

Great. Well, those are awesome responses. So we are out of time. I want to thank both Chris and Brian for joining us and sharing your experiences and advice. This was super helpful for me, and I'm sure it was for the rest of the audience. I want to thank all the attendees for joining us.

We hope you found this webinar useful. These were just a few highlights from the survey. We will be sending out the entire survey results to all the registrants as well as the webinar recording within a few days. So, be on the lookout for that in your email. If you have any additional questions, please don't hesitate to contact Chris, Brian, or I. We'll also send our contact information in the follow-up email.

And then we also have some additional resources here as you think about how you're going to move forward and overcome some of these challenges. Data.world has a great white paper on agile data governance, and DataKitchen has a DataOps cookbook, which tells you everything you need to know about DataOps. So with that, I will sign off.

Thanks again to our speakers, and I hope everyone has a great afternoon and evening.