DataOps in Practice, Swedish Style

00:00:00

Good afternoon and good morning. Thanks everyone for joining our webinar today from wherever you are. I know we have attendees from around the globe. My name's Beth Pfefferle. I'm the VP of marketing at DataKitchen, and I'll be the host. Today, our topic is DataOps in Practice, Swedish style. We have a special guest joining us, and as you might have guessed, he's coming all the way from Sweden, Lars Albertsson.

Lars is the founder of Skling, a firm that provides data value as a service, customer-adapted data engineering, analytics, and data science. He's a recognized thought leader on DataOps and a frequent conference speaker on big data technology, privacy, and data strategy. Before founding Skling, Lars worked at Google, Spotify, Schibsted, and as an independent consultant.

Today, he'll share with us his DataOps experiences at Spotify and other organizations, as well as provide some general DataOps best practices he's learned along the way. Chris Bergh is the founder and head chef at DataKitchen, and he'll also join us today. He'll jump in occasionally for some discussion with Lars and to highlight certain points. So welcome, Chris.

Hi. Yeah. Before I hand this over to the two of them, I'd like to cover a few housekeeping items. You're all on mute, so we'll use the last 15 minutes of the webinar to answer your questions. Please enter your questions into the box on the control panel, and we'll collect those and answer them at the end.

The webinar is being recorded, so we'll email a recording to all participants. So please be on the lookout for that in your email in the next day or so. We'll also send you a link to the slides in that same email. So with that, I will hand it over to Chris, who's going to get us started.

Yeah. Lars, thank you so much for joining us today. And I actually met Lars through Twitter because I found one of his presentations on SlideShare. And there's a lot of stuff if you Google the word DataOps out there, and honestly, a lot of it's not that great. And when I read Lars's presentation, I was like: "Wow, this guy's really smart, really technical, and I've learned something." And so I complimented him, and so we formed this Twitter friendship and then actually met.

And so I just think he's one of these unique individuals who can go to a very technical level as well to a business level and back and forth. So I hope you enjoy his discussion today. I think he's a very smart person with a really great perspective on how data and analytics can improve.

Thank you. I am flattered by the introduction. So you've heard my name and so forth already. As you've heard, I worked at Google and Spotify and so forth. Those were pivotal points in my career because I joined Google in 2007, and I realized how much value you can get out of data. Google were, at the time, the only company doing big data, essentially.

I joined Spotify later when I realized they were doing some interesting things into data. We're now at 2013. And that's where I learned much of the trade, much of the data engineering that I know. Chris approached me and said, "Could you come and hold this DataOps in Practice presentation at our webinar? And it would also be fun to hear a bit about your learnings from Spotify and so forth." So this presentation is like a mix of those two topics, basically. What I've learned has colored my work after Spotify, where I've been freelancing and now run a data engineering startup.

And I've learned equally much from the freelancing as from Spotify because I've met so many different kinds of organizations, and often the organizational issues are much more of a challenge than the technical ones. But this talk will be quite technical. So I'll cover how we came to start with DataOps at Spotify, and our first initial steps. And all of this is very old information. I no longer work for Spotify, and I am not up to date with what they're currently doing.

And likewise, this is information that has already previously been published and presented in conferences because I'm not allowed to give out any non-disclosed information. And I'll give you some

of the knowledge that I've picked up on how to be successful with DataOps and give you some glimpse into the toolbox that I'm using on a day-to-day basis.

So Spotify started very early with its data journey. Hadoop was released in 2006. They installed a Hadoop cluster in 2007, I believe mostly because it was a new shiny thing and they were having web-scale data at the time. And this picture on the right is Jon, one of the first employees of

00:05:00

Spotify, and I believe this is the second incarnation of the Hadoop cluster. And Jon is very happy because he has just fixed a problem by installing curtains in the room because some early recommendation jobs were overheating the poor machines. At this time, in the very early stage, there were basically three main use cases.

There were the recommendations, as mentioned, and also product insights. Spotify was very early with using A/B testing for determining which product changes would be useful or not and so forth. And the reporting, which we will get into, which I will mention during the presentation later. Reporting is a business-critical thing at Spotify because there are contracts that mandate that insights and reports be sent to the record labels.

And the reporting pipelines are also used for royalty calculations. So they have significant financial impact At the time, some of the factors behind the early success of data at Spotify were more or less purely cultural. Those who have viewed presentations on the Spotify model might have heard of the autonomous teams. There was this culture that each team should be enabled and autonomous and not have dependencies on other teams.

And that sort of organizational design affects much how we work at Spotify. There is also very much a lean culture of eliminating waste. If something doesn't look useful or is unnecessary, just throw it away and make things as small, as simple as possible. And Spotify is also very much a learning organization that is easy to trial, fail, learn, and adapt. And this is very much in the DNA of the company and has been. Those factors were strong factors behind the data success.

So, the installing Hadoop early had some perhaps unintended effects of pushing us very early towards good data processing and DataOps. Because at the time, most data processing was done in databases, where you have your data in table and you run a program that mutates the data, so you have a new version of your data and the old one is lost.

Now, when Hadoop was installed, it could do very little at the time. The early versions of Hadoop had very few features. No indexing, no transactions, very weak security, so all the data was readable by everyone. And you can only do transformations in batch. So these mutation-based workflows didn't matter.

Which, as an effect, pushed us into what we today know as data pipelines and data factories, where instead of mutating data, you store the raw data and it's cheap, so you can store it forever, more or less, and then you transform your data in series of steps rather than mutating it. And each refined set of data is mutable because they're so difficult to mutate, so they might as well be immutable, which means that you can share them within your organization.

So by accident, we got into a pipeline-based workflow. We got democratization of the data. The data was shared within the Hadoop cluster so that everybody could read more or less. And since it was immutable, we could run new experiments with very low risk. Now, later adopters of Hadoop did not get these benefits, ironically, because fast-forward five, 10 years, and Hadoop has got all of these features with indexing and role-level security and mutation of data and so forth. So the late adopters I see strive to keep their old workflows even though they're in Hadoop.

So why does this matter? Well, I'll try to illustrate that. Now, I would like to give a reporting use case example from Spotify at this point, but I can't do that. So I will diverge a little bit and present some slides that I recently presented at a conference, and the topic is COVID-19 and data analytics.

It turns out that there are a lot of similarities between these problems and the challenges we had with reporting at Spotify. This is a graph from the Swedish Health Authorities, the number of fatalities in Sweden from April timeframe. And if you look at that graph, what is the conclusion? At least to me, the obvious conclusion is that the pandemic is over, that nobody's dying anymore. Now, as you probably have guessed, this was not the case,

00:10:00

because this is an artificial effect of slow data collection. We just haven't got all of the data yet. So if you look and draw the same graph every day, you see that for every day, given the information that you have, the graph points downward because of the slow data collection. So every day, if you look at the graph, you could draw the wrong conclusion.

And this is a very common problem that I see, or a very common mistake when people are using traditional BI and databases. They collect the data, they put it in the database, they do a query, and they get a graph, and then they might very well jump to the wrong conclusion without noticing. So, what should you do? Well, it turns out you never want the latest data, which people often look at.

You want data that you can actually compare. So this is a richer graph of the same data, where the delay in collection is color-coded. So you can see that it takes a week or two for all of the data to properly appear. Now, you don't want to wait weeks until you draw conclusions. So you have to decide, either you wait for a long time and you get very high-quality data, or you do a quick analysis, but only look in a comparable data that had the same time to propagate. So you have to choose your patience.

So you can, for example, choose a patience of two days and get this red line here, or choose a patience of six days and get the blue line here. Or you can do smarter things and look at what data might be missing and forecast from previous observations. Now, this is not the graph that we see from the health authorities.

They aren't doing this. And they know how to do this, of course. But the reason they aren't is that it is too expensive for them. And this is very similar to the reporting challenges we had because the mobile phones, they report data late because somebody turns them off or they're in flight mode or wherever.

But we nevertheless were required to report all of the songs that had been played both quickly, because we want the insights, the record labels wanted the insights, but also accurately because they were used for financial calculations. So how do you solve that? Well, you do multiple calculations. And in order to push the cost down of doing multiple calculations and raising the quality of data, you must move from seeing the data refinement as a craft, something that you do by hand with manual tools, to a process, a factory where you instead improve the process step by step.

So in this case, we realized that one step of processing or one chain of processing is not enough. So we do parallel processing in multiple time windows. Now, we don't know without looking at the graphs whether this is sufficient or not. So we might want to add another pipeline to sort of assess the data quality and trigger us whether something is wrong with the inbound data quality, and likewise with the outbound data quality. And I worked for a couple of years with reporting at Spotify, and we were all constantly surprised by the number of data quality issues that would pop up.

Every month, there was a new thing that we didn't suspect. So we had lots of these pipelines monitoring our assumptions and validating the data. And if the data is bad, you add new steps to repair the data so that you remove the duplicates and filter out what seems to be fraudulent events created by bots, for example.

And all of this quality work that you do, if you are in a democratized environment like the one we had, which was Hadoop-based, these become reusable to other teams as well, so that they can benefit from the work you do in improving the quality of your data.

In some cases, it requires more complex processing to repair the broken data. We had some cases where the phones were logging the wrong things, but if we joined the faulty events with external datasets, we were able to repair some of the data and improve the outcome and quality. And we said we wanted forecasting.

Whether you're dealing with COVID-19 or with reporting, this is something that is valuable from an analytics perspective. So you might look at what did it look

00:15:00

like last month? Let's forecast based on that for this month. But forecasting is difficult, so you don't know whether your first forecasting is right. So we add multiple pipelines to do multiple forecasts, and then you measure the success after the fact and see which one is good and which one isn't. So in a data-mature environment, the cost of doing all of these extra jobs to improve your one single use case has been pushed down so far that you get return of investment.

Whereas in most companies, the cost of adding all of these jobs and pipelines and maintaining them and operating them is still way too high. So it's not done, and then you get the graphs that you saw from the Swedish health authorities. So you can assess the data maturity of an organization by looking at the number of pipelines they operate and the number of datasets, because each of these provide additional business value by raising the quality or giving complimentary information and so forth.

And this is even more important if you are into machine learning, because with analytics, you can probably guess whether something is right or wrong. But machine learning is much less understood by humans, so it's even more crucial to measure all of the properties of your machine learning pipeline in order to make it robust.

So this is sort of one of the goals of DataOps, to be able to do that with low cost. We weren't there when I joined Spotify in 2013. This is a conversation that I had one of the first weeks at my job. "So I want to change this pipeline. How do I test it?" "Well, test it sort of next to the real production pipeline, but do it manually. And don't do any mistakes when you're changing the past, because then you will overwrite production data," and so forth.

This was a high-risk environment. When I joined, there were just a few teams doing data processing in Hadoop. I was in the analytics computation team. There was a collection team, ACDC, typical Spotify humor. There was a recommendation team, and there was one guy doing reporting and maybe one or two more teams. In order to run pipelines, a high level of competence was required, and it was risky operations and so forth. So most teams that had data needs, they came to us and said, "Could you implement this feature?" and so forth. But there weren't enough of us.

This is roughly what the production environment looked like. There was a Hadoop cluster. It was a few hundred nodes at the time. The data was there, the computation was happening there, and there were edge nodes that were controlling the actual computation. There were redundant edge nodes so that if one went down, another one would take over. And it was all orchestrated with a workflow orchestration tool called Luigi made by Spotify.

I'll speak more about Luigi later.

They were kicked off by standard cron jobs, and typically on a higher frequency than the actual job. So if you had daily jobs that were supposed to calculate something daily, they would kick in every hour or every 15 minutes and try over and over in case something failed. So one day

something went wrong. These jobs were deployed with Debian packaging, very much old style, and pushed out by Puppet. And all of these were edge nodes, were like pet machines, machines that we cared about on an individual basis, and they had names, so forth. So one day, my manager, he saw this runoff process, and it had been running for a long time, wasting CPU. So he killed it.

And we didn't know that. But suddenly one pipeline were calculating the wrong things. And it turned out that this was running on an edge node that wasn't part of the Puppet list. So no new packages had been deployed there for six months. But it was still running. But since it was so slow, it had always been slower than the other redundant nodes, so the others were winning the race to do the computation all the time.

But now it was happy and healthy again, so it eagerly grabbed a job and miscalculated the data. So this event took a while to figure out because we weren't aware of this node's existence. And then we started to realize this doesn't

00:20:00

hold anymore. We need to move towards a more factory process. So we started this journey to what is today called DataOps. Now, I believe that the term DataOps was coined back in 2014, so this was one year before that, and it didn't have a name. DevOps was fairly new, so we imported some of the ideas from there. So what we wanted to do was to get to a situation where we had no pets, only cattle, and all of the processes didn't have a lot of state, which was managed by Puppet, but something stateless. And instead of knowing all the folklore and being very competent, there will be a simple way for anyone without strong data engineering competence to deploy pipelines.

And the goal was to being able to deploy a new pipeline from idea to production in less than a day and then fix it in less than an hour. This golden path concept is common at Spotify because of the autonomous culture. So, as part of the autonomy, it's part of the deal that you cannot mandate that anyone does anything in particular unless there is a very strong reason.

Instead, we aligned by providing a recommended way of doing things and making that way the easiest way. But if you have a strong reason, you can choose another path, and maybe that path is better, and then we have innovation from the bottom. So we set out to make a golden path for building data pipelines, basically.

So this is part of what we coined up. Instead of these Puppet-based and Debian packages, we made a continuous integration, continuous deployment pipeline, where the code was tested locally rather than manually in the production cluster. And we packaged it up to some kind of standard artifact and just dropped it in a

standard way on the edge nodes. And then there was a little loop that looked for new

packages, new versions of the pipeline, dropped them instead. And in that package, we had all that was needed, the workflow orchestration, the Java code that was doing the actual computation, and also the cron scheduling. So it was just self-contained packages. Now, you might think this sounds like a container or a container image. It does.

This was, however, before Docker was actually useful. Docker was around at the time we evaluated it and concluded that it was not near and ready for prime-time production. So we made our own little self-contained plan. And then all of the edge node workers became cattle instead, and they would just pick up whatever was in the artifact store.

And this worked very beautifully

and was really simple, just a few hundred lines of code. So what we concluded here is that it's common to use functional programming paradigms when you're expressing data processing. Now, it is perhaps even more important to use functional principles and paradigms on the architecture level. Like the whole data pipeline is a functional thing where things are immutable and transformations happen.

And also the CI/CD deployment pipeline. These machines that used to be pets, they were no longer holding any state that we needed to mutate. They were stateless. We could just throw in another machine and things would boot up automatically and so forth. So both artifacts and datasets are immutable. And both the processing and the deployment became more or less deterministic and reproducible.

And one of the Factors that made us successful in this is the experience and knowledge and implementation we had regarding workflow orchestration. This started early. Like many early Hadoop adopters, Spotify started out using Oozie, and then someone decided it wasn't good enough, so they wrote another XML-based workflow orchestrator called Rambo, and then Builder was written to replace Rambo. By then they realized that, hmm, maybe complex things like graph, like work or pipeline dependencies shouldn't be expressed in a markup language. So they switched to Python and built Builder 2, and based on that experience, eventually Luigi was produced.

So there was a lot of learning in that process.

00:25:00

Now, it seems that some of that learning has gone backwards, so you see these markup languages coming back again. But having this in Python was definitely a success factor. And Luigi, in comparison to the competitors, is super, super simple. There are lots of things that it cannot do, and that's one of the main benefits, that it just does one thing really well.

So what happened was not so much that we could do technically interesting things, but that we enable the whole company to use data efficiently and to work and innovate with data. And that holds much more value than any of the technical fancy things or machine learning and so forth. Spreading data and enabling everybody to work with data.

The real value of the big data hype was not web scale data or super analytics. It was a new way of working and a new way of collaborating with people.

One of the things that made me really proud about the things we did was that a couple of years later, Discover Weekly was made, and this was made by a team in New York. I have never met the people. They had no contact, as far as I know, with the infrastructure teams. They were completely enabled to build this fantastic product from the data being available and there being very minimal friction to throwing out new ideas.

So it started out as a Hack Week project, and they claim that they had to do very little work because it was piggybacking on things that already existed. But in order to piggyback on things that already existed, they need to be available to you. So in this picture here, the Discover Weekly is just a fat thing here, but they could reuse much of the work that was already present in the recommendation pipeline because the data was democratized.

And this was really just a few engineers that did bottom-up innovation. There were no strategic decisions that we should do this kind of recommendation thing. There was no six-month project. It was a really quick hack. And this is, I think, the most popular machine learning product that has been built by a European company.

One of the principles that I think made us successful at the time was that everything that we were building and using were very small components, sort of Unix philosophy. Do one thing well, integrate lots of little things in pipelines.

And build complex system from individual bricks that you can easily replace and remove, that don't overlap with each other. I believe that the cloud vendors is the best example of this success, where in a cloud, you can use some of the pieces, but you don't have to. You don't have to use a big table. You can use Cassandra instead if you're in Google Cloud and so forth. They don't overlap, and it's not our way or the highway.

And that's very much how at the time we built systems at Spotify. And if we found some internal pieces and there was a hole in the middle, we saw that as a challenge that could be solved, usually with a few lines of Python code. Whereas if you take the more larger scope enterprise style tools and combine them, you will have some overlap.

And in that overlap, you have complexity. That's complexity that you will never get rid of, because every time there's a new version, you will see, do they still fit together? Do they still harmonize and so forth? And that overlap requires quite a bit of integration. And one example is Airflow, which is similar to Luigi, does workflow orchestration and orchestrates the execution of pipelines.

But it has a different philosophy in terms that it has a wider scope and so forth. That scope overlaps to some degree with Kubernetes. So this in the corner here is the ticket for Kubernetes integration in Airflow, and it's a ticket that has been open for three years, produced a few thousand lines of code and so forth.

Because these systems, Airflow does scheduling and I believe log aggregation or log collection and so forth, and these

00:30:00

systems need to play nicely with each other in order to work together. Whereas in contrast, Luigi doesn't need a Kubernetes integration because it has no overlap. It just runs inside Kubernetes without any integration whatsoever.

So there are pros and cons. I will show you some pros of Airflow as well. But that is just one principle that we adhere to and that I still adhere to.

And since we

were building things from small bricks, when Docker finally was production ready, what the

team that came after us did was to replace our own poor man's container deployment and put in Docker and Kubernetes instead. And they only had to rip out a little piece and then put in another little piece that would schedule the Luigi jobs on Kubernetes instead. And I'm also very proud of how easy it was to throw away the things that we built, because that makes you innovate forward.

Whereas if we had built something large, holistic, monolithic, complex, this would have lingered for much longer.

So what happened after the DataOps journey was that all functionality that could be done in Hadoop gravitated towards Hadoop. And there was no mandate, there was no strategic decision to build a data lake and put everything in there. It was just natural gravity decided by each of the teams because things were simpler when expressed in Hadoop rather than when expressed in online services as had been the case in many cases.

So examples of things that weren't originally in Hadoop that moved there was search and metadata management and a couple of other important functionality. Why did they gravitate to Hadoop? Why weren't they still in their microservices, which they had been? So let's look at some trade-offs in integrations of distributed systems. If you have a distributed system, there are basically three main ways to integrate them. You can either do the sort of the service-oriented architecture microservice pattern where the services peer to peer call each other when they need data, when they need someone to process data and so forth.

This is the quickest way, lowest latency of data going back and forth from one place to another. At the other extreme, you have the offline processing where you put things into files and you ship the files around, or a file goes from one job to another job. This is the slowest form of processing.

We were able to push the latencies down to about a minute or two with batch processing. But if you have quicker or higher latency requirements than that, then you're better off with a quicker type of processing. And in the middle, nowadays, at the time, Kafka was very new, so we didn't use it all that much.

But you have this stream computations, where you don't do things point to point, you publish things to a stream similar to Hadoop, but for real-time data and any process, anybody can consume it and pick it up and so forth. And that stream processing works fine if you can tolerate a few milliseconds of latency.

But if you will need to look at data for that over a longer time period, then batches is usually preferable. So what we saw happen was that functionality that had been in the online world moved to the offline world. The reason is, I believe, is operational complexity in the online world. Let's look at some of the operational maneuvers that you need to do in a distributed system. So one of them is you need to upgrade.

You roll out a new version. That's a known procedure. It's fairly straightforward. Perhaps you do a blue-green or canary deployment and so forth. But this has a high risk, right? If you do it wrong, you will have immediate user impact somewhere. Someone will see it immediately. So you need to put in a lot of quality assurance to lower the risk sufficiently, depending on whether the impact is high.

Then if something goes wrong, you will have a failure of the service. If you have a failure of an online service, the

00:35:00

functionality will go down. If the service crashes, you cannot no longer use it, right? So this has an immediate user impact. If the service is processing data, at this point, you will have a data loss because you will have these RPCs queuing up or the queues building up in between, and eventually data will drop on the floor.

Even worse, if you push out a new version that has a bug, so it doesn't crash, but it miscalculates data, then that miscalculated data will spread throughout the system, and you will get a cascading corruption effect. And this is very difficult to recover from. In the general case, you have no tooling to help you.

You have to put down a lot of hard manual work if it goes really bad. Now, if we compare this with batch processing, where things are put in files, then it's not just one event going back. It might be a million events in a file. So in the upgrade case, when you roll out a new version, it will next hour or so, it will pick up the new file and so forth.

This is something you can do with low risk because you don't have a immediate effect on the user Likewise, if the batch application crashes, then you will have a delay in the pipeline, but the files are still there. So when you have fixed the problem, you can redeploy and you have no data loss. It just catches up.

And in the case where you have a bug, you will get a downstream data corruption. But the original files are still there, so you can recover from it. So it's a temporary data corruption.

So let's zoom in on how this latter case is handled in the types of systems that we built, where Luigi was orchestrating these pipelines and so forth. So it happened a couple of times that we had these problems. If these pipelines were serving something important, then we would always keep a couple of old versions around. So if it was a recommendation index, we would keep some old versions of the recommendation index

in the serving database. So when a bug was discovered by some kind of measurement in the ecosystem here, we would figure out the bug, write a test case that triggered it, and then deploy a new version. And then since the upstream data was stored, remember, they're immutable and so forth, so we could just remove the invalid data.

We first switch to using the old index, and then we remove the invalid data. And eventually, Luigi, who orchestrates this, will say, "Oh, there's some missing computation here to be done." We'll recalculate with the right values, and then we will switch to the new index. Debugging problems can, of course, take time, but recovering from an error like this with data corruption can, in the best case, be really quick.

If it's a really important pipeline where you don't want to risk this and be very careful, you can run multiple pipelines in parallel. So you can deploy a pipeline, run it in dry mode, do a comparison of the outcome for-

Lars seems to have frozen up there. Let me see if we can get him back.

Okay, Lars, we're back. Splendid. Thank you very much.

So next slide, please.

So if you have a pipeline that you really care a lot about, as I said, you can deploy it in parallel with the original pipeline, compare them for a long time, and when it's mature enough, you decide, you switch to using the results of the new one, and if something goes wrong, you can switch back to the old one. This is something we did for financial calculations a couple of times because the numbers are so critical.

And this means that you don't actually need a dev or staging environment. I've seen a couple of large companies insisting that they should have multiple Hadoop clusters and so forth. This is usually unnecessary and also undesirable because in order to evaluate these new pipelines, you need the real production

00:40:00

data. You cannot evaluate a data pipeline without data.

So next slide, please.

So if we go to streaming solutions, they fall somewhere in between. So an upgrade is similarly fairly simple. You can run parallel streaming pipelines and compare the result, just as we saw in the previous slide. That works for streaming as well. Likewise, they're also fairly resilient to service failure, because if a stream process goes down, then if you're using a stream storage such as Kafka that has a long history, when it comes back up again, it can catch up from where it was, and you only have a brief outage, but you don't lose data in the process.

On the other hand, if you have a bug and produce incorrect data, then you will have data pollution downstream. And now you have a much more difficult problem than in the batch case, because in the batch case, if you have a bug that goes on for five days or five hours, then you might have five faulty dataset and a couple of more downstream. And that's a human scale, but if you have a stream process that went over, you might have five million bad events, and it can be quite hard to figure out what has actually affected what.

So that's a much worse situation to be in. So streaming is operationally somewhere in between batch and online services. Next, please.

So in order to illustrate the complexity of handling stream faults, what you see on the left is the blog post from Confluent on how to handle replace in reprocessing Kafka Stream. So if something goes wrong, you can go through all of these steps in the blog post, and as you can see, it's fairly long, to recover and sort of do recomputation on what went wrong. Now, unfortunately, this only works for a single job. It doesn't work for a full pipeline.

Next slide, please.

So all in all, this means that you can choose between faster data speed, lower latency, or you can sacrifice the latency. If your use case can survive minutes of latency, you can cut down on your operational overhead and accept a higher technical risk when moving ahead, which means that you can innovate faster. And this is, I believe, the underlying reason that all of the functionality that could go into the data platform, into Hadoop, went into the data platform. And this is a trade-off that I feel is very poorly understood. The swift latency always gives much better demos and looks much better, and we want real-time analytics and so forth. But I think this is a fallacy, and one of the principles that I've adopted is to use batch processing wherever possible because of the operations are so simple, or they can be very simple. Next slide, please.

So one of the things we learned was that in order to have these properties, you need to have a very clear separation between what's online and offline. So these are examples of some outages we had. We were on a daily basis dumping the user database, and this was backed by Cassandra at the time.

Spotify were at the time very heavy Cassandra users. And we had a program called Sqoop, which is like parallel database dumping on MapReduce that would reach out to Cassandra and suck in all of the data of the user table. And since it was parallel, it put a very heavy load on the Cassandra clusters. They could handle the load. That was fine.

But as the number of users grew, it grew longer and longer and longer, this load spike, and eventually it covered 25 hours and was run on a daily basis. So then we got a double spike, and this took out the user login service. And the login team solved this by putting a firewall between the login service and the Hadoop cluster, and then we had to figure out what went wrong.

So likewise, there was another outage when one of the recommendation teams decided that we now have a great new recommendation index that we have recomputed, and they pushed it to Cassandra, and Cassandra was deployed in multi-region mode, so it would replicate to all of the other regions across the Atlantic and so forth. And that saturated the Atlantic link so that other traffic, which was more important, couldn't get through.

00:45:00

So the learning here is that batch jobs are very bad neighbors. They saturate one resource at a time, and your online

activities are more important. So you need to separate very clearly between offline and online so that these worlds don't meet. Next slide, please.

They need to meet, of course, because you get the data. So you have to take very much care when you're sucking in the data from the online world, and likewise when you're egressing the data from the offline world into the online world again. So instead of going to the production database, perhaps there's a replica that you can dump from. Perhaps you can use the daily backups to restore an offline database and dump from that, and so forth.

Next slide, please. So quality. You

remember back in the earlier slides, I mentioned that all of these extra jobs and extra activities were done to improve quality of your data. So there are a number of quality dimensions that we care about. There is timeliness. Is the data produced when it's supposed to be produced? Like is my daily report here in the morning or not? There's correctness.

Given the input data, did we process it correctly? There is completeness. Do we have all of the data, or do we have these downward-pointing graphs without realizing it? And also consistency. If we compare a couple of data sets, are they referring to the same things or not? And I will show you a bit about the toolbox that we'll be using and explain how we handle these different dimensions.

Next slide, please.

So we ended up in this DataOps transformation building a standard sort of testing development or test harness that looked similar to a standard unit test harness, except that it tested one batch job at a time. And since we bundled it in standard testing frameworks, it runs nicely on local developer machines. It runs nicely in CI/CD and so forth.

And this actually turns out to be really simple because batch jobs are so easy to test. They are very deterministic. They have no state. You just provide some input, modify, run the job, check the output that it makes sense. Some batch processing frameworks don't scale down and require a cluster to run, so I tend to nowadays avoid these frameworks.

Next slide, please.

Now, in order to validate that you haven't broken anybody else's job, for example, if you change some format by accident, you might break downstream jobs. You will need to test longer Pipelines, sets of batch jobs. And there are two simple or two straightforward ways to do this. One is to use the same harness as before, but you make an artificial job that actually consists of many smaller jobs. And that is easy to get to run, but then you have to maintain a separate job, and it's not exactly what you're running in production.

So what I usually do is I go into the workflow manager, and I tweak it to, when in test mode, rather than going to the cloud or going to the Hadoop cluster to get its data, it goes to local files. And then I spit out a file locally, run a series of jobs and read up files and verify that the data matches.

Now, this doesn't work so well with all workflow managers.

Most of them have no built-in support for this. Luigi hasn't. But Luigi is quite easy to tweak in this manner, or it's possible at least to tweak it. I've heard that is more difficult with Airflow and other workflow managers. So here I expect to see some innovation in this area eventually. But right now you have to put down a bit of work.

I won't mention how you test streaming jobs because it's very similar to how you test standard transactional systems. So, going to the next dimension, timeliness, basically monitoring whether data sets are present or not. And this is one of the benefits of running Airflow over Luigi or other workflow managers. They have this monitoring built in.

What we built at Spotify was one of these little feature, little components that do just one thing

00:50:00

and start to do it well. In the upper right corner, it's called Datamon. That's an internal tool. It's not open source. And you can see in the middle lower picture that Airflow looks quite similar. It might have been inspired by it. I don't know. This is an area where I hope also to see more innovation pop up because there's no really great open source tool.

The one to the lower right, Twitter Ambrose, seems to have gone defunct. What I do in lack of one, when we're building

or when we are going to clients building systems or for our own platform is to do a very poor man's data monitoring, basically looking at the lake, looking at the files, and just measuring the time should they have been here or not, and then mapping that to a standard database. And then using Grafana or something, just some standard tool.

So poor man's data monitoring. Next slide, please.

So then there is the

correctness of the data and completeness and consistency of the data. Given that you trust your computations, you will need to measure whether the data you have makes sense on its own and in comparison to other data as well. So the Hadoop and Spark and similar frameworks here provide a feature called counters that is sort of an out of bound result of a computation that is metadata about the computation.

So usually when you bump into an old code path that you didn't expect, you bump a counter, and then you export these counters, and you see them as standard monitoring or just like you would in an online system, monitor the number of 500s. We would perhaps monitor the number of orders that didn't have a matching user or something, which we will see an example of in the next slide, please.

So this is some code example. It's a bit small. Sorry about that. This is how it would look like in code in order to generate a count. And this is a tiny little example I have from-- I run some data engineering courses, where if you weren't measuring the data quality and you were to join orders in an e-commerce shop with the users, you might do a standard inner join, which would silently drop the orders that didn't have a corresponding user.

So in order to make sure that you monitor if things go wrong, in this case, if an order doesn't match a user, you do a left join, and then when you have a null in the matching right side, you do discard the order if you cannot repair it, but you bump the counter so that you know whether something goes wrong or not.

And this is easy to do in MapReduce or Spark and so forth. This is really hard to do if you're processing with SQL, which is an interesting characteristic. And one of the reasons that we decided not to bet on SQL processing for our main workloads when we did the DataOps transform at Spotify. Next slide, please.

Because we were doing an evaluation at the same time we were doing this transform, we had to reevaluate the way we processed code because we were doing Python MapReduce, which is an odd beast. And we were the largest company that were doing Python MapReduce, so we had to do the innovation ourselves. We couldn't piggyback on someone else's, and it was clearly the wrong direction at the time.

Nowadays, Python is popular in data science and data engineering, but at the time, data engineering was completely Java and Scala dominated. So we wanted to go with the ecosystems and wanted to find something else. We evaluated using Hive and expressing our pipelines in SQL. And

SQL is fairly popular for doing these kinds of processing these days. Most people that come to SQL, they have a new problem that they want to do, and they get very quickly up and running with some SQL. If you're used to it, it's easy to express your things. We came from another direction because we had pipelines in production that were really critical and really important.

So in order to bet on using Hive for processing, we needed to be able to translate those pipelines into SQL. And it was clear that it wasn't nearly expressive enough.

00:55:00

And Hive at the time translated SQL statements into MapReduce jobs, and it was doing that fairly clumsily, in part because of the lack of expressiveness. So job pipelines that had three jobs ended up being seven, eight, ten jobs long, and that would require too much resources for us. And also, we couldn't live without these quality measurements because if you are to measure these detailed quality things like, the order doesn't match a user or the played song has no known user that it matches to. In order to measure these things, which is necessary to detect fraud and so forth, for each of these, if you were to express it in SQL, you need to write another complete statement. Whereas we had code that was fairly business-wise complex.

It turns out that royalty contracts and so forth have a really business complex logic. So deep down in the code, we would have an if statement that would have an else clause, and then we want to bump a counter. This is not viable to bring up to SQL. So we ended up with building on Crunch, a Java framework, and Scalding.

Nowadays, Spotify still has a lot of Scalding jobs and some Crunch jobs that has mostly moved to Cheio, which is a wrapper of Apache Beam. So one other thing that we

implicitly sort of decided was that at the time, we did not aim to do a sort of a low-code or self-service platform for analytics. That is fairly common to see, and I believe that we made a better choice by instead organizing the teams so that all teams had proper data engineering support. I'm sure there were some teams that didn't have sufficient support out there.

But that was the general idea, that each team should be autonomous and master their own success and have sufficient technical capability to do everything that they wanted. So the platform was much more technical than most that you see. This was fairly easy to do at Spotify because people moved between teams, and we were able to reorganize and evaluate organization fairly quickly. In other environments that I've seen, this is very difficult to pull off. So you often see these, "Can you engineering build a self-service platform for us in business over here?" which is, I believe, much less efficient than just mixing teams and doing cross-functional teams.

Next slide, please. Hey, Lars, this is Beth. I just wanted to give you the heads-up that we're at the top of the hour. We may lose some people as they drop off. We can certainly keep going, but just wanted to give you that heads up. Oh, okay. No problem. I just have a couple of more, I believe, and then I'll wrap up.

Okay. So sometimes the counters are not sufficient for evaluating consistency and quality. So whenever you need to compare more than one dataset, we ended up building separate pipelines that will compare dataset. Is the new version of the pipeline better role than the other one? How do they compare with the history? Do we have unexpected changes over time and so forth?

So this is a tool that we brought out when the counters were insufficient. Nowadays, there are pre-made tools for this, like Great Expectations and DQ from Amazon, but we didn't have that available at the time. So there are lots of custom measurement pipelines like this. Next slide, please.

So overall, back to the machine learning scenario. This is at the time what sort of machine learning pipelines in general looked like. There were never one pipeline training model, almost always multiple pipelines trained various versions, trying out different parameters and so forth, creating multiple models. And then sometimes some of them were just evaluated. Some of the

machine learning scenarios might

choose dynamically on the model, depending on the situation, depending on how much history there was for the user, for example. And this is definitely an outdated slide. There has been so much progress on machine learning at Spotify now, but this is sort of what it looked at the time, and this, I believe, is much more mature than what most companies that you would see out there are currently at.

01:00:00

Next slide, please.

So lots of things went really well. Spotify is sort of a data leader in Scandinavia today. But not everything went super well. We were bitten a bit by the autonomous culture because the things are difficult to harmonize and coordinate So I believe I counted to 20 or 25 different ways to store a timestamp, for example, and this is just unnecessary heterogeneity that makes life difficult. And likewise, we never got these pipeline end-to-end tests to work properly during my time there because it required too much coordination across the teams. We did make some effort, but we never really pulled it off.

So that was challenging, not for technical reasons, but for culture reasons. And of course, we made lots and lots of mistakes, but that's part of being a learning organization. So next slide, please.

So what has happened since is that many companies that use data heavily have been very successful in Scandinavia. So we have an interesting cluster of unicorns and companies that use data heavily on a day-to-day basis. But all of this is limited to a few very high technical companies. The top data engineering and DataOps competence does not move very much from these companies. Next slide, please.

So what has been driving me is the realization that data and AI gives some kind of superpowers. I want to spread that to a wider range of companies, and that has been driving me personally in my mission in these last few years. Because there are so many companies in Scandinavia that have values that are very close to mine and that I want to help succeed by providing these powers. Both these are the examples here are large companies, but there are many small, similar companies as well.

Now, this turns out to be really difficult because what you need to do is not install technology, you need to change the way you work in order to get value from data. That's what the whole DataOps and big data is all about. Next slide, please.

So what I have been trying, first as a freelancer and now as a startup, is to see how simple can we make the technology so that remove everything that is unnecessary and just focus on the flows, the ways of working, and the ways of collaboration, and focus on generating business value. In my last freelance project, I was

privileged to be able to test that out at Bonnier News, one of the media houses here, with a small little team and see how simple can we make a data platform, and let's focus on getting the first pipeline out. And we got it out into production in two weeks, and we're able to give the data to customer in three weeks and have a steady flow going and a happy customer. And if you compare that to the larger companies that run similar big data Hadoop projects, they have been taking between 6 to 24 months, in my experience, in order to get the first value out from these projects.

We took that further when we started Glyng and changed the business model so that I'm not freelancing anymore. There are a few of us, and our business model is to do it as a service instead. So for the first retail client, which also funded the development of the platform, we were between one to three people, and after one year, we had 70 pipelines up and running, spitting out data to about 200 ingress points or 200 different dashboards, graphs, reports, integration points with all the systems and so forth. And we're producing 3,400 data sets per day.

And if you compare that to your typical analytics department with 50, 80 people, that's more data sets than they're typically producing. They're, of course, doing more complex data sets, but by numbers. It's not as much as Spotify, who's currently in 100,000, 200,000, 300,000, somewhere in that range, but they also have thousands of people working with data on their databases. And at the far extreme, Google said that they were producing billions of data sets back in 2016.

So this measurement serves as some kind of measurement of how much value you are able to squeeze out of

01:05:00

your data.

Next slide, please.

So this is sort of our business model that instead of going in and helping a customer on-site build out the capabilities, we say, "But let's collaborate, but we host and we build the pipelines instead." So we set up flow, your data flows to our cloud instances. We collaborate on building the pipelines, and then the customer get refined data back.

It can be simple things as report dashboards and so forth, or complex things such as machine learning. And we do all sorts of simple or complex things. We do use TensorFlow every once in a while, but usually we strive to focus on business value and do the simplest solution possible in order to maximize the value. Turns out, actually, to be fairly hard to sell value.

It's easy to sell hours or features or shiny objects, but it's a bit confusing for customers to buy business value. So I'm very grateful for the contact I've had with DataKitchen because they are one of the few companies that are sort of value-focused when dealing with data, and I'm learning a lot by watching them and how they operate.

That's it.

Great. Thank you so much, Lars. That was an awesome presentation. We are a little bit over time, but we do have Chris here, and I know he brought up some good points during our little break. Chris, do you want to add any thoughts before we wind down here today? Yeah, it's more a question for Lars.

It goes back to this sort of huge patterns that people sort of miss. And I think one pattern is the thought of a lot of people who do data and analytics focus on doing data and analytics. "I got to do the data transformation or the model or the visualization," and they neglect these things like that errors are going to happen, and how do you recover from errors? Or they neglect thinking about, "Well, I'm going to have to change things, and I'm going to have to build the change." And these operational things get to be secondary in people's mind. Because they're all focused on doing the cool new thing.

And in some ways, if you tilt your presentation a different way, you're saying, "Well, think about errors. Think about deployment speed. Think about how you want to respond when things go wrong, because things are going to go wrong, and build to reduce that from the beginning." And you use the term factory, and I think that's a really appropriate thing. And people sort of aren't building factories.

They're focused on kind of building the cars that come out of the factory. And so why is that? And how do we change people's minds to focus on those ideas? Yeah, I completely agree with you. I mean, much of the presentation is focused on when things go wrong, right, and iterating. I believe that this was easy for us when we were at Spotify because we were used to this iterative mode from the beginning. We knew that we were, not only for data, but for everything, things were not going to go right.

And what happens, and how do we learn from it, and how do we correct when things go wrong and so forth. And there was, or is at Spotify and many similar companies, a product mindset. And you have product owners and product managers that own a thing over time, and that it is never done. It is organic, and it lives on, and it is supposed to be iterated on.

So you must measure and adapt. Whereas if many traditional organizations have a project mindset, right? There's a project manager who has this finite thing to do, and when it's delivered, it's done. And I see a bunch of that in the companies that I meet, that they still have this idea that we're going to do a project and we're going to do a machine learning project, and then we deploy the model, and then it's done, and then we'll move on to something else.

And I think that's a key difference. Yeah, and I think this sort of the agile culture, the iteration, the love your errors and being able to improve them, and then kind of seeing analytics as a product or a river of value that sort of is never done, right? You're always continuing to tinker and improve.

And I know Spotify had those values, and some companies are trying to, I've heard people say we want to do the Spotify model in terms of how they do scaled agile, and it's just interesting that

01:10:00

perspective. And so a related but a different question that's a little bit more technical for you is there's...

A lot of DataOps is sort of a reflection of ideas that came from different areas. And so, if you look at the sort of ideas in lean manufacturing or agile and DevOps, they, I think, have applied into the world of DataOps from an intellectual heritage. But in software, there's this different ways that you organize complexity.

And when I started as software engineer, there was a whole object-oriented ideas, and you encapsulate a complexity by building objects. And in the last sort of five years, there's been a trend against that to say, "Okay, that creates some hidden complexity that is hard to handle in big distributed systems." And there's this idea of function where you start with something and you apply a function to it and apply a function to it, and you get to the end. And the similar ideas is, I think you brought up in a few slides in data, where you start with an immutable data.

You do transformations, which are idempotent, you end up with more immutable data, and then you add things onto it. And that sort of functional way of working, I think, has also been enabled by the fact that disk is cheap and cloud computing resources are cheap. And so how prevalent is this idea that I'm going to build a functional data system?

And how do we get people to think more in that way? Because that's also in the spirit of let's build to reduce complexity so we can find our errors quicker and iterate more and move from project to product.

Sorry, I lost your last sentence. Can you hear me? Yeah, I can hear you. Okay. Can you repeat just the last part of the sentence, please? Yeah. So how do we get people to do sort of more functional data engineering, more think of their projects in a functional way as opposed to the sort of stateful

way? How do we do it? Yeah, I think DevOps has helped here, because it has illustrated the value of building container images from scratch and deploying them on discardable machines. And that is a example of a successful functional paradigm on an architectural level. So I often, because people tend to sort of accept that more easily than accepting that you should Build data factories and not have spent too much work in databases, because they're so used to the databases.

So I tend to connect to that. And also then these operational examples that I gave, how do you recover from an error? Wouldn't you want it to be easier? Those are the tricks that I've used. Okay. Do you have others up your sleeve?

I find that the typical way that you've built a Kimball data warehouse in larger companies with things like slowly changing dimensions is built on an era where disk was expensive and you had to talk to the Oracle salesman when you wanted to do something different. And now disk is cheap and compute's cheap, and you can make parallel versions of things with just a script.

And so the cloud sort of frees it up to say, "Okay, well, you can do more." You don't live in a world of scarcity, you live in a world of surplus, and that enables you to follow a different paradigm. But a lot of organizations are already sort of deeply into treating things like pets and not cattle and having sort of deep stateful systems where if you've got to move something from their development system and their test system and their UAT system are pets that need to be moved and deploy scripts.

And it's very hard for them to see a different way of doing it, because of that sort of, "This is how we deploy. We do a deploy script, and we've got to update the QA database and update the pre-prod database." And so given all the challenges they're in, and so it's been a harder idea, and I'm just wondering if the community needs to talk more about these functional principles as a whole on how you build data systems and if that's useful for people. And we've been throwing around and tossing this blog idea on it for about a year, and we just haven't had time to do it.

Yeah. I guess in a way, I have given up on that because my toolbox is quite technical and I have technical processes and technical things in my toolbox.

01:15:00

I often come to organizations, and I realize that, "I don't have the tools for you. What you need is DevOps coaching and CI/CD consultants and so forth. You need that more than the actual data work." And, so I decided when we found Xqlang that we need to explore new ways of collaboration, rather than us coming in and building up something for you, but then we just leave in the technical legacy.

We don't change your ways of working. How about we take your data out instead and we demonstrate how we work, and you can come and work with us. You can come and learn in our environment, rather than changing yours. And then, I would see it as a success if one of our customers would learn, be with us for a year, two, three, and then say, "Oh, we've now figured this out.

We can do this on our own." We would lose a customer, but that would be a successful story, I think, because that will spread the word.

All right. Well, that's all I had. I think we went 20 minutes over, but Beth, were there any more questions that you think we should cover? There was one. Here's a good one. "Thank you for such a detailed and thorough presentation that dives into the actual implementation of DataOps. In your presentation, you cover traditional data processing focused on technologies in its early days, such as Hadoop." I didn't warn you this was a long one.

"How about in today's world of available technologies and advanced features, databases with multiple features, as well as add-on applications that have data quality, data lineage, data replication across geos, data on the cloud?" All right, it's a little long. "I still see traditional data management taking place even with these advanced technologies being a part of the tool chain.

What is your point of view on this?" So, I think that the new innovation in data processing is, in a way, a double-edged sword,

depending on what has been driving it. Because for the Hadoop systems and the cloud systems, we've seen a lot of innovation that I think is driven by the traditional enterprises say, "So Hadoop seems great, but we'd like to keep our old workflows, so can you please implement transactions and mutability and all sorts of things?" And then you just get the worst of both worlds.

And so some innovation just makes the landscape more foggy, I think. And then there's some innovation which is driven by a new need of data pipelines and DataOps and so forth. Lineage is one such example, that we have been doing without, but hey, there are actually quite a few things that are important and that can be solved with better lineage tracing.

But the platform that we're operating on and that we've boiled down to is very simple. We just work with standard object stores. We work with Kubernetes, where we run everything. We don't even have compute clusters because you don't need to scale out for 99% of your applications, one machine is enough. You can have 12 terabytes of memory these days.

We still use Luigi because it's so simple, and we use relational databases. And there are very few cases where that is not sufficient, and then we handle it on a case-by-case basis. Sorry, BigQuery is also one of the more important innovations that have made quite a big difference in how we work with data.

So I believe that really the big changes that have been happening and that has enabled the successful companies is ways of working, ways of collaborating, and rather than technology. Great. Well, thank you, Lars. I think we could probably discuss this all day, and I hate to be the one that ends all the fun and this good discussion.

But I think we should wrap up. So I want to thank everyone for taking time to join us today. There were a few questions we didn't get to, so we'll follow up with you individually if we didn't get to your question. Like to give an extra big thanks to Lars for speaking and sharing his wealth of very relevant insight with us today.

To everyone else, we'll also be sending out a recording of this webinar and the slides in the next 24 hours, so please be on the lookout for that in your email.

01:20:00

If you have any additional questions, please don't hesitate to reach out to us at DataKitchen or to Lars directly. So everyone, please have a great afternoon and evening. Thank you. Thanks for inviting me, and if there are questions, just post them on Twitter, LinkedIn, or in an email to me, and I'm happy to answer.