Your Model Is Not an Island: Operationalize ML at Scale with MLOps

00:00:00

Good afternoon, good morning, and even good evening to some of you. Thanks for joining us today. My name's Beth Befferly, and I'm the VP of marketing at DataKitchen, and I'll be the host for the webinar today. Our topic is on how to operationalize machine learning at scale with ModelOps. Our speaker today is Chris Bergh, who's the founder and head chef at DataKitchen.

For those of you who don't know Chris, he's the leader of the DataOps movement. He has more than 25 years of research, software engineering, data analytics, and executive management experience. At various points in his career, he's been a COO, CTO, VP, and director of engineering. He's the co-author of "The DataOps Cookbook" and "The DataOps Manifesto," and a regular speaker on DataOps at many industry conferences.

So before I hand it over to Chris today, just a few quick housekeeping items. The webinar is being recorded, so we will email a recording and the slides to all participants, so be on the lookout for that in the next day or so. Also, you're all on mute. We'll use the last 15 minutes of the webinar to answer questions, so please enter your questions in the box on the control panel, and we'll collect those and get through as many as we can during the Q&A session at the end.

So with that, I'll just hand it over to Chris, and he can take it away. Thanks, Beth, and welcome everybody. So my name's Chris Bergh. Thanks for the introduction, and we're here to talk about models. And so

let me turn off my webcam so you get a little bit more screen real estate. And so the subtitle of this is sort of how do you do AI or ML or data science with DataOps, and really this perspective that no model is an island. And in order to walk through that perspective, we're going to go through a couple of things to talk about today.

So the first is we'll begin with this almost a philosophical perspective on operations, model, data, whatever, which is what you do is much less important than how you do it. And then we're going to go through eight key success factors in how to do operations with models. Then we're going to talk through a couple scenarios that we do with our customers.

I'm going to give a quick demo, and then we're going to conclude and talk a little bit, have perhaps a little bit of time for questions. So let's get on with the philosophy. So I actually began my career, I studied AI, graduated out of graduate school in 1990, and at the time, AI was not a cool thing to study. And I went and worked at NASA Ames and did air traffic control automation and built a bunch of models to help air traffic controller sequence in space aircraft.

And then I did some more software products, but then about 2005, I changed over to what I thought was the easier data and analytics world, and I realized it was much harder. Not because I didn't know software, didn't know AI, it's just things kept breaking. We had data science models in production that would break based on the data that was wrong. It took a long time to get anything into production, and my first temptation was really to blame people and say it's a person's fault. Maybe that engineer's wrong or that data scientist is wrong. And of course, we didn't call them data scientists in 2005.

We called them advanced analysts. But I began to look at a little bit different perspective of all the work that we do in data and analytics, and really, it started off with this idea that the stuff that we focus on in data and analytics, the model or the algorithm, the stuff that I had really been excited about in graduate school, or the work that we do to prepare data or the visualization or how we govern it, or even the data itself, the thing that we spend all our time on isn't actually as important as the system in which we work.

How you deploy, how you develop, how you monitor, how you iterate, how you collaborate, how you measure. It's almost as if the factory is more important than the car. And in some ways, I think that's a sort of a manager's perspective, but it's also a bit contrarian that says you've got cool technology and cool models and algorithms and data, but the people and the process and the operations actually have a lot more to do with the success than the particulars of those, because those things change.

There's different algorithms, different models, different tech is always changing. And so what we're going to talk about today is a theme and how that plays in in whatever you do in analytics is that by focusing on a different set of perspectives, that how fast you can get something from the mind of your data scientist or your engineer through their keyboard into the hands of your customer and then monitor it and improve it. The cycle time of deployment is really important, because what you're really trying to do is maximize their rate of learning.

00:05:00

Running something with very low errors or models that don't predict accurately once they're in production is also incredibly important. And a lot of data and analytics is done by a team sport, and those teams, unfortunately, don't all live in the same room anymore, if they ever did. And so how do you have a data science team and a self-service team and a centralized data warehouse team not end up having problems? And then finally, all this work and cycle time and error rates and collaboration, that's actually a data set, and you should measure how you work, and that can actually help you improve.

And so the real problem is that I think a lot of people don't have the sort of operational or lean or agile perspective, and we're kind of making data and analytics the same way I grew up in Wisconsin in the '70s, and there was a company called American Motors that produced really terrible cars, and they actually went out of business, and one of the main reasons they went out of business is the Japanese started to make cars that followed lean principles.

They were better and cheaper and lasted longer. And so I think these ideas come from lean manufacturing, total quality management. They've come through software, and now they're coming into data and analytics, and I think the result of these Lack of these ideas means that a lot of our projects cost a lot and end up with a lot of unhappy customers.

There's a lot of failure rate in data and analytics. In fact, some people, Gartner says 60%, sometimes 80% of all data and analytic AI projects fail. And so we think that that causes a lot of problems with the team. And looking at people's emotions, I think I've suffered sort of dread going into work, where, is something going wrong?

Are we going to have bad data? Did something break? Is the model not working anymore? And so thinking about it from this operational perspective, by focusing on cycle time and error rates and collaboration and measurement as a perspective on how you do analytics of whatever form, and I'm going to use the word analytics pretty broadly here.

It could mean I'm preparing data, I'm developing features in order to train my model, I'm deploying a model, I'm visualizing a model, I'm governing the whole process. That whole set is, I'm going to use the word data analytics or just analytics to cover it. And if you can think about how to deploy it fast, how to tell if it's going wrong, how to work and get people to collaborate better, and how to measure and monitor, I think you're in a better place.

And so how do these ideas play out into the world of data and analytics? And so the title of this presentation is "No Model is an Island." And so we have this perspective that there can be some harmony that can be achieved by thinking of the process to deliver analytics of every form together. And I think the challenge and that lack of harmony really comes out in a survey that Gartner did recently in sort of what are the barriers to AI.

And I'm going to use the word AI and ML and data science kind of interchangeably here, and some people may or may not agree with that, but it's these algorithmic ways of making predictions, either in batch or in real time. And Gartner says not only do a majority of models never make it to production, it takes nine months to get models from pilot to production. And I think that's a challenge, really because you need feedback from your customers.

You need to deliver value to your customers quickly or else they're not going to care. And also, if you wait nine months from when you think of something to when you get into production, you're going to have to context switch. You might get it wrong, and it's really hard to predict what your customers want.

And some of the challenges that I see out of this is that a lot of people are scared of errors in their models, that models are part of a system, data and training data is hard, and also it's

hard to prove that models are working. And so there's been a bunch of investment, and if you've seen the amount of words of AI come in or machine learning or data science, and there's a lot of fashion that's gone into this and a lot of people have entered into it and into this world of data science. But I think the idea of operations is actually gaining a lot of traction, and whether you call it ModelOps or DataOps, actually, we see a good increase in the amount of search growth over the past two years from looking at both ModelOps and MLOps.

And so there's a term ModelOps and then MLOps, and they're kind of used interchangeably. I'm going to use the term ModelOps here. And then Gartner's seen about 1,000% increase in the last two years in the analyst inquiries. And the word DataOps is ending up on the hype cycle. And so these ideas out there, the Ops ideas that are having an impact on

00:10:00

the data and analytic world. And so what are some of the key, given that background, that there's challenges, that the industry is increasingly seeing the operations side as a way to have value, and our definition of what the operations side is around cycle time and error rates, et cetera. What are some of the keys to actually doing ModelOps?

And so I'm going to go through actually eight of them in different slides. And I guess the first part is that models are not unique. See it as part of a bigger picture. And I've gotten pushback from a number of data scientists on that that say, "Oh, it's different. Models are special." And I guess having done the work, I don't particularly believe that. I think there's uniqueness to it in the process, but at the end of the day, models are part of the analytic continuum, and they should be treated as such. And I'll talk about that.

And then the second is test and development and automate deployment to production. The third is monitor everything in production, including your models. The fourth is DAG your models and put everything in Git, and I'll talk about what DAG means. It's not a sandwich. Treat your experiments as variations of production. Treat real-time models similar, almost, to batch models.

Don't build your own ModelOps solution, and see ModelOps as one of the many parts of DataOps. So the first part is models are not unique. See it as part of a bigger process.

So what's that process? Well, let me walk through it and go through what I mean by that. So when we do data and analytics, we work in teams, and there's people who kind of put data together. They're called data engineers or ETL engineers. There's people who run algorithms over it. They're data scientists or AI engineers.

There's people who actually try to take that work and explain it to people. Maybe they do data visualization or self-service BI. There are data governance people who try to make sure that we use the same terms and we can tell where it came from. And then there's the people who take that and run it day in and day out in production. And so if we look at it kind of from a tools perspective, there's just a massive amount of tools out there.

There's great tools to do data visualization, like Tableau or Looker or Cognos. There's great tools to do data science, like doing Python or Or R code. There's SAS, there's Dataiku, there's DataRobot, there's a bunch of tools out there to actually run algorithms on data. There's data catalog tools like Alation or Collibra. There's 50 different ETL tools, Informatica, Talend, Airflow, and there's all sorts of cool data places, including ones like Redshift or Snowflake, which had an amazing IPO recently.

And then there's lots of data sources. And so there's just lots of tools, and there's probably in each category, maybe there's 30 to 50 in each category. And so, one of the things I've learned managing data engineers and data scientists is people love their tools, and I sort of never in my life want to get between two data scientists arguing about whether R or Python is a better language. Likewise, I don't ever want to get involved between a data engineer, between the people who like to do ETL and the people who like to do ELT. And so I think the tools are only as good as the process that they work in, and so that's part of what I'm bringing up here. And so if you look at it as an example, a data engineer sources some data and maybe a data scientist runs an algorithm on it and builds a model that segments it.

And then after that, maybe there's a self-service person who uses a tool like Alteryx to source some small data file, like who's the owner of these accounts, and matches it in and does a

visualization of it. And finally, the data governance people tries to say where it came from and the production person puts it into operations. But if we look at it from how those teams work, and this isn't always the case, because sometimes teams are together, but they all have their own process per team. The data engineers have a process that tends to be typically more like a software waterfall process.

The data viz people tend to have a very quick process because they can push a button and get it to production. Production operations tries to standardize across, and data science has its own unique process in terms of creating data, developing features, building and training models, deploying models, monitoring models, and we're going to talk a little bit more about that today.

But they're all unique and interesting in their own part. And so if we look at it just to finish this out, is that most people who are doing data and analytics source data from a lot of internal systems or external systems through APIs and files and syndicated open data. And our perspective in DataOps is that in production,

00:15:00

data comes in, and whether it runs streaming or in batch, whether it is big data or small data, structured or unstructured data, there's a factory going on where each one of those tools that we talked about, pipelines, ETLs, models, catalogs, are kind of adding, describing, and assembling the set of artifacts that go out to the data customer. And that could be a file, it could be a

series of charts and graphs or an application. And so this perspective that in production, we're running a factory, and the same methods that apply to producing good results in a Toyota factory, things like safety culture or statistical process control or focusing on lowering errors can actually apply to our data and analytics work. But what's interesting is a Toyota factory is kind of all in one building, and for us doing data and analytics, we've got this overlay of an organization structure where sometimes the data warehouse team is owned by one part of the organization, sometimes there's a data science team or more, and there could be a lot of line of business people using the results of both. And so we've got this idea that the work that is done in production is owned or created by different parts of the organization, and they have different roles. And so this idea that Conway's Law applies to it, and we don't have time to talk about that today.

So the first one is just see what you do as models, as part of a factory, as part of a system. It's not an island. And that's a theme throughout this, is that you need to take your blinders off. You can't focus solely on your own task if you're developing a model. You've got to see it as part of the system.

And what we're trying to say here is that seeing as part of the system has to do with being able to see a bit bigger picture. And that goes on to my next part, test and development and automate deployment into production. And so what do we mean by test? Well, first of all, when you're developing something, when you have an environment that you're developing it in, a test environment, a dev environment.

And that environment itself has got a lot of complications, and I'm going to talk about that in my next slide. But what we want to do is be able to take things from a development environment and get it into production quickly without errors and without having a lot of meetings and people to talk to.

Because we're all engineers and technical people here, and having a lot of meetings and having a lot of Word documents or Excel documents to do the checklist that you should do to get into production isn't great. And there's really a couple requirements here. The first is that you need to be able to abstract out the environment that you work with. And so that could mean hardware, software, tools, the code that you're working on, the right hardware and software versions, even the test data sets. And so the sandbox, the environment that you work in, abstracting it so things can move quickly from one place to another, I think is actually really important.

And what it looks like is, if you think about it, I'm going to switch a little picture here, and if you think about what a lot of architectures do, or data architectures do, they focus on that box on the top, the production environment. And so what- We want you to start thinking is a little bit bigger. Think of all that work, that factory, but you're going to have to figure out a way to automatically, without typing things in, without going through worksheets and manual checks, deploy that from a development environment into production. And to do that, there's really two criteria. One is that you need to have tasks that run across it, and we're going to talk a little bit about what that means.

And second is that you need to abstract the environment that you work in. And so this is kind of a architecture diagram that says, think in your mind, not only about what's in production, but actually work on how you deploy into production quickly. Continuously deploy if you can. Continuously test if you can. And to do that, you need to abstract the places that you work in.

So let me go on to the third step here. So once it's in production, we'll monitor everything in production, and including your models. And so what does that look like? Well, so let's say the first case here, I want to go in and I want to build a model. So I get some historic data. I may train the model, I may build the model. It sort of depends on the algorithmic technique.

But at the end of the day, I have a model which is perhaps some code and some configuration that goes along with it. And then I put that in production, and it makes predictions. And how it does that is we apply live data to it, and that live data could come from a scoring of a live

00:20:00

database on a batch model. It could come from a user's click or a form that they just filled out if you're doing a risk model for

helping an insurance company pick a policy. And so the model makes predictions. But as the model's predicting, you're getting feedback from that model. And you can use that live data and that feedback to retrain the model, or improve the model. And so that feedback is, in essence, a monitoring system. And so you need to be able to think about how you can monitor what's happening to your models in production, if they still are accurate, if they still are predicting, and if the inputs and the outputs of that model, the data that's going into it, or the outputs that are coming from it still make sense.

And so these models themselves are part of a greater system, but how can you take that second model and deploy it quickly? And so monitor, so you can then retrain your model.

So, and one of the ways that we think about this, and here's a picture from our product, is that this graph is we call a recipe, and I'm going to show this in the demo part, but if you look at it, it's a graph, but it's a whole graph. It sources data. It builds a database table, applies a batch forecasting model to it.

It publishes Tableau workbooks, updates a data catalog. It does everything that you need to do to be able to see the whole data and analytics function as a system. But it's not just about monitoring the model, because you need to monitor and test the data going into it. You need to monitor and test any data transformations that happen before the model. You need to monitor and test any visualization that's after the model. And if we think about this as just a set of tools, and this is ELT, it could be ETL. This is a Jupyter notebook, and it could be R or some other tool.

We have these design decisions when we're building analytic systems. And let's just say I have an if-then-else statement that I need to put into this. And I could put that if-then-else statement in my Tableau workbook, or I could use data to derive that if-then-else statement, or I could put it as part of the attribute of a dimension in SQL, or I could put a script on data load. And there's actually good reasons why they should be in every place or could be in every place.

Having it built from data, that's a great thing. Or having it being able to do within two minutes in a Tableau workbook so I can get feedback from our customer, that's a good thing. Have it as an attribute or dimension so it's consistent, that's a good thing. And so we have these decisions about where we put things.

But at the end of the day, that's logic, and that logic itself needs to be monitored so as data's flowing through your system, you don't suddenly end up with 10,000 more values in the third clause of your if-then-else statement, wherever it is in the flow, which suddenly causes your customer to say, "Hey, this looks weird." And so you need to monitor everything, including the models, and we'll show this a little bit.

And so it's monitoring data, monitoring artifacts that's created from data, monitoring the tools that are acting upon data and the models. And I find it hard to separate these two because models are only one source of error. If you feed crappy data into the models, if someone has taken the model, the result of the model, and mixed it with something in your logic, in your visualization tool, it could be wrong. And a lot of organizations end up having problems where they have to figure out how they get this-- who did what.

Now, I'll talk about that in a little bit. So number four, DAG your models and put everything in Git. So a DAG is a directed acyclic graph. So it's kind of a series of steps. And so, our perspective is that there's a lot of DAGs, a lot of processes or workflows that people do as part of it.

And if you look at this diagram, there may be a DAG that happens in your data engineers, which they could be using a tool like Airflow to do it, or they could be having a DAG that's equivalent in their talon work. A data scientist could have their own DAG. There could be a data visualization DAG and data prep.

And so there's a lot of data science DAG tools out there. There's tools like MLflow. Netflix has got Metaflow, there's Airflow, there's Kubeflow. And I'm not sure why they all end up in flow, but the idea is that there's a DAG, a directed graph, and maybe you like to use those tools, fantastic. Or maybe you like to use data science toolkits like DataRobot or Alteryx or SAS. Fantastic.

Maybe you like to code directly in Python and R. Fantastic. And so there's a great set of tools, but all of us, there are independent teams, all of which have their own DAG.

00:25:00

And maybe your data engineering DAG has to be fed into your data science DAG, it has to be fed into your data visualization tools DAG. But everything, that perspective, that there's lots of tools and there's lots of workflow or DAGs in those tools, and those tools themselves have to be stored in Git. So that model, the code, the hyperparameters, the random seeds, all that stuff should be put and stored in Git.

And because you want to be able to revisit it and get back on it. And just like that, you also want to be able to treat your experiments as a variation of production. And so the management of variations is actually really important, because we've got all these DAGs and all these tools running together, and what we're talking about here is being able to think of it as a system. And so I showed that diagram before of where every group has got their own process, and data science and AI has their own unique set of challenges.

So for instance, you need to develop features and prep data specifically to train or build your model. And that could be very different than the set of attributes and dimensions that could go into your typical set of charts and graphs that go for business reporting. It could be based on the very rawest data that's not even in a data warehouse.

It could be based on a very different data set. And that process of quickly experimenting and seeing what data is predictive, if not, is a challenge for people and data scientists. Prepping the data, developing the feature, building the model, and evaluating the model, then deploying the model and monitoring the model. And these steps, I think, are unique to a data science team, but they're not unique to data scientists forever in particular because a lot of these things are just variations that you need to manage.

And if you think of your process as one big system, maybe a DAG of DAGs or a set of workflows, you can parameterize these to run based on different data inputs, different model inputs, different feature creation steps. And you can kind of make a time machine. And why would you want to make a time machine?

Well, if your model's in production for two or three months, you need to retrain it. So you should be able to go in and go back and say, "Is the model still working the way I intended it?" And if I adjust it based on new data, based on new tweaking, is it still predictive?

And so if you think of what you do as a variation, instead of something unique, you think of your processing as one production process where you can manage variations of it. One for perhaps feature development or hyperparameter optimization, really the development process, and then the other is a production variation. It gets easier to start thinking about the complications of this system.

And again, we think that models are part of an entire system of data and visualization. So being able to treat these as variations of productions, we think actually makes it easier to deploy as opposed to sort of living with blinders on and just focusing on the model itself. So some models are in batch and some models are in real-time.

And so, I think you need to treat your real-time models, the ones that respond to queries similarly almost to batch models. And so, I think most data scientists that we run across are delivering batch models that are scored, that are ending up loading data in the database. But more and more people are delivering real-time models, i.e., a model that's got a query response time. And that could be, as I said before, it could be for an insurance or for a website or any activity.

And there's lots of good tools to actually serve your models. TensorFlow Serving, SeldonCore, KFServing, or if you're a Python fan, you could just take your model and put a simple Python Flask API on it. But the idea is that instead of having a model run against a set of data and then add things to that data, it's running as an API and you can call that API. And that actually makes it interesting because the response time of that API actually becomes really important. And so there's typical DevOps response times like latency, throughput, consumption that you can get for any API.

And there's also sets of operational metrics like alerts and notifications that that API isn't doing. And again, most companies have a mix of mostly batch, but some real-time models. But I think you need to think about when you're doing MLOps, where you're starting. And so some people who are more advanced, maybe they're focused all on real-time models, or maybe their domain is focused on real-time, but a lot of

00:30:00

companies are just focused on batch models.

So part seven, don't build your own MLOps solution. And so, yeah, there's a lot of Cloud platforms like Microsoft and Google and AWS who've got a bunch of tools that have really come out of software. They have tools for storing code, tools for doing continuous integration and deployment. And they have tools to do Kanban boards.

And so they kind of say they do ModelOps or DataOps, but unfortunately, they don't quite. They've just sort of taken an adaptation of tools that software engineers use and said, "Okay, we can bolt these onto a data process" and it doesn't quite work. And I think for a couple of reasons. And the first is that data people are not software people.

And I think for the main reasons that, having been both and continue to work with both, software people are much more likely to want to have four to five to seven tools and two or three languages to work with, that they're comfortable with that, but your typical data engineer and data scientist isn't. And we've talked a lot about the different pipelines that you need to think about.

The production pipeline, the deployment pipeline, the environment pipeline, and you need to automate those and test those. And the automation and testing of these pipelines is particularly missing from any of these cloud providers. And then DataOps, ModelOps deployment is not the same as CI and CD. There's actually much more steps that you need to do to deploy with models. And the fourth is that the metrics that your system throws off about, and we'll give an example of this, about how much you deploy, what the error rates, how you monitor models are particular to the data domain and model domain.

And then fifth, there's the complexity of the organizations where you may have a centralized team, a decentralized team, and then the complexities of the sandbox environment. So basically, they don't do ModelOps or DataOps, or they do do it, but you're going to have to build a lot of stuff around it to make it work.

And then the last part is eight, see ModelOps as one of the many parts of DataOps. And so we've heard, and sometimes I laugh, there's a lot of ops suffixes out there. There's DevOps and DevSecOps and AIOps and NoOps and AppOps and CloudOps and MarketingOps. And yeah, it's all sort of marketing kind of fluff, right?

But I think of it in this way, that there is a set of, at the highest level, there's a business management concept that has to do with lean and learning organization that comes from Deming, and those set of principles are played into different ways that you actually organize the people. And I think in

software and data and analytics, there's things like Agile and Kanban and Scrum and pair programming, et cetera, and those are good ways to organize the team and the people. In manufacturing, there's sort of Six Sigma and Total Quality Management. But if you apply those to different organizations, like if you apply those ideas to a software team, you end up with DevOps or DevSecOps. If you apply those to a data science engineering team, you end up with what we think of as the group of DataOps functionalities, DataGovOps, ModelOps, AnalyticOps.

And so what does that all mean? Well, if you're doing something like data science, you should think of what you're doing as ModelOps or MLOps. If you're doing data engineering, well, you could call it data engineering ops, or some people call it DataOps. If you're doing data visualization, well, you've got to think about how you do the operations for self-service, self-service operations.

And one of the interesting things, a month or two back, we gave some webinars on DataGovOps and DataSecOps, and what's interesting is they were actually really highly attended. We had sort of 700 people at them, which is large for a typical software company webinar. And there's an operational side of it. And if you think about what's the common thread of all these ops terms, well, yes, it's marketing. But really the common thread is to think of your process that you work in as an independent being. Think of how you automate it, how you script it.

Think of things as code, as an important metaphor. And so our perspective, and this is the way DataKitchen thinks of it, is that like data and analytic teams are moving together. There's a chief data officer, a data and analytics role, and they're tending to more and more have all the pieces in. They're having the data part, the data governance part, the data visualization part, the self-service. We think that you need a sort of a common platform to think to

00:35:00

do all these ops together, DataGovOps, DataOps, ModelOps, self-service ops. And yeah, there could be--

And by having a common platform, you can start to address these common perspectives that are across all these areas and really help accelerate your adoption of DataOps, which is actually going to be the subject of a forthcoming webinar on why you'd want to do DataOps and how to do a DataOps transformation.

So let me just go through a couple scenarios that we've seen with our customers. I'll do a quick demo and then take some questions.

So why do people focus on this ops stuff? Well, they're finding that it takes forever to get stuff done. It's slow deployment. When they get it into production, there are too many errors, or the people are fighting with each other. There's lack of harmony between your data scientists and data engineers, and people are doing visualization.

And so that came up in this transportation company, and so they had a really complicated infrastructure. Some of which were streaming and batch data. They had data coming off their sources. They had ... models that were running against it. The results were ending up in an Oracle database and in Redshift. They had Jupyter Notebooks being acted upon Tableau.

And Monday morning when they came in, they were having problems. Their data customers were saying this was wrong. And the lack of harmony between the teams ended up... Because each team, the data scientists own the notebooks and this predictive model part, there was a team that owned NiFi, a team that owned Kafka, the data warehouse team, the cloud team, and no one knew where the problem was.

So Monday morning when someone said something was wrong, they couldn't find it, and they would end up trying to, because everyone had their blinders on, "I only own this part." And they had no visibility to the greater system, nor did they have any idea if there was a problem ahead of time. And so what we did is we took our technology and sat on top of all those models and transformations and visualizations and notebooks and streaming and batch to be able to tell ahead of time if something's wrong, because if something's wrong, you should know.

So for instance, if your vehicle

set has been giving an average of a million rows every new data every week, and suddenly you're getting 100 rows that week, hmm, maybe something's wrong with the sensor. Or a model that's acting upon it, maybe suddenly it's not being predictive. Maybe its prediction accuracy is wrong, and maybe you should notify it. And so by having something that sits on top of it and testing and monitoring all the artifacts, actually digging in the data, digging in the results of the model, the inputs to the model, you can actually write tests or monitors, and that's what you see on the right, and you can send alerts.

And so this helps in this sort of holistic view of the end-to-end system, actually helps you resolve errors faster, helps you stop from having errors in the first place. And again, don't see your work, no model's an island, just like no data transform's an island, just like no visualization's an island. It's all part of the big picture that you need to monitor and automate so you don't end up having the horrible Monday morning and the blame game email that goes around.

Now, let's take a second case of an example, and here's where there is a data science team, and they developed a really cool model that actually did sentiment analysis. And the marketers liked it. It was a small team. They had done some work in Google and done some work with Tableau, and the data science team wanted to move on to the next thing. Right?

They had sort of done their work, and now they had to hand it off to a team who could run it and who could make sure that it's in production and make sure that if it does go out of whack, that they are able to find out if something's wrong. And so what was done was they took the model and all the associated artifacts that go into the model itself, the how the data was scraped from Amazon, how the sentiment analysis was done, how it was put in BigQuery, how the visualization's done, and abstracted that again into one piece.

And so the idea here is that by having it run and monitoring the production, the production people could notify that there's an error. And maybe, the data side could find out if there's a data problem, or the data scientists could then go in and say, "Hmm, maybe it's time to train the model." And so, one of the big challenges if that data scientists and people who work with models report is that they struggle to get something in production, but once it's in production, they have to babysit it. And that babysitting is something that you shouldn't do, and how do you get that off your plate? How do you work with a team?

Because you need visibility as a data scientist into what's happening in production. And the production people and the IT people all need. So you need sort of a plane that you all can work with, and that's what we've built here. We've been able to say, if you think of no model as an

00:40:00

island, you can deploy the whole model system out to your IT team, to the production team, and you can all have a common view of the problems, and you can fix it if it's not working. And then the second last case I want to talk about is,

data science and model development is not done in one place. Sometimes it's a data science team, but sometimes it's actually done way out in different lines of business. And so, there's a big bank that we're working with, and they've got a huge amount of data, but it's used by different people. There's in New York or Texas or the branch banking, and they have different skills.

So for instance, there's a data scientist in Texas who wants some data, who wants to build some Python models, wants some SQL access, and that's different than the business person, the business marketer who wants D&B data, or the branch bank person who wants DDA data. And so how you enable self-service is really important to this bank because they don't want people to be able to go off and do things that they can't monitor and test.

So how can you build a data science sandbox or a self-service sandbox where you can kind of give data, give tools to people, but then be able to monitor what happens to them if they're doing something that you don't think of and then take back or delete their access to the data. And so giving them sort of, you could call an experiment workbench, a self-service sandbox, some way for people to be able to get at data and the tools they need on data, I think is important. But it's good, but it's not quite enough because you have to think about how you move those things back into production.

Because what if that data scientist has something great? How do you actually take it and not have to recode and redo the whole thing? And so because you don't want to have... If someone develops a great Python model, perhaps, you want to be able to wrap that up in what we call a recipe or wrap it up in a basket where you can test it and deploy it and put it in Git and follow these steps that I talked about before, so that you can run it as it is or run it with a few tweaks.

And of course, there's other cases where that model doesn't go into production, or there's another case where maybe that model actually does need to be added to a central database as an attribute or a new table. And maybe it does need to be recoded. And perhaps that's fine, but at least you have an opportunity to see, you could either not You can either not deploy it, put that model in a nice basket with all the data sets that it has, or be able to deploy it into production as part of your own core technology stack.

And so I'm going to skip the last one here and kind of go on to just kind of just give a quick demo of our product here to sort of talk about what I mean. And so

one of the things I like to show in our product is this idea of a recipe as a series of steps. And so I showed this before, and I think it's actually kind of interesting in that there's a lot of different ways that people actually build models. And you can see, as I brought before here, there were some data transformations, some ELT in a forecast model and updating a Tableau workbook and updating the data catalog.

What's interesting about it is that I've noticed that a lot of people like to use their own technology to do models. And this one is actually done in a Jupyter Notebook, which is fantastic. Some people like Jupyter, or some people like to write SAS. And here's actually the model that's been developed. And so here's this IPython Notebook, and here at line, I think it's 111, is the actual forecast prediction. And so you may like a Jupyter Notebook, you may not.

You may like to write Python, you may like R or SAS. You may have your own engine that runs this, and that's fine. But the way we think about it is it's a nice little basket where we pull data into it, run it in a Docker container, parameterize it, push some inputs into it, get some outputs from it, and then test it and monitor it.

And that's incredibly important to be able to go in and look at the model and see what happens. Because when the model runs, we create this thing called an order, and every time it runs, we create one order run. And you can see here in our UI that the sales forecast model ran successfully. And we can look at that because we can say, "Okay, here's the test," and you can see here's where the forecast model ran, and its tests were correct. But we're also testing other parts of the system.

So we're going in and testing, for instance, the Tableau workbook to see if it's working right or the data transformations. And here's these numbers refer to the amount of tests that we have. And so what's interesting is that this is just a point in time. This is one run, but there's a set of statistics if you look at every run to be able to understand what's happening over time.

And so if I go back and look at the set of runs over time, we can

00:45:00

start to make interesting discussions, interesting

judgments. So one of them may be just looking at the data, like what are the invalid data sets going in? You can kind of see it's going up, and then it gets this little triangle here, which we call a warning. And we actually will send you an email or an alert or a Slack message to tell you that something's wrong. You can also configure it to be kind of your production line Andon Cord to stop the assembly line, so this is so wrong, you don't do it. And some of them are based on data, some of them are based on actually how the model's working.

So if I actually go in and look at this next graph, here's one where we're actually doing the root mean squared error of the forecast and looking, and it's pretty good, and then it suddenly gets high but still not quite high enough to send an alert. And so you need to look at whether your models are predicting and even whether your system's running in a timely manner.

So looking at the time over time to see if it's right. And so

one of the main points that we want to bring off, and the reason I showed this, is that thinking of your system end to end, being able to run it across all your tool chains, being able to test it in production, and then being able to use the history of those tasks to inform the decisioning about whether it's right or not is, I think, an important part of doing it. And this works based on batch or based on an event. It works in various different ways. And

these tests themselves, we think are an important part of being able to judge whether. And so if anything that you take out of this message is tests or monitors, think of your modeling activity end to end, no models in island, and make sure that you test and monitor your island and make sure that it's right both in development and production. Because these actually form a double duty.

This is telling you something's wrong in-- This tells you something is wrong in production, but you can also run a lot of these things in development to make sure that when you make a change or when you retrain the model, that it's still correct. And so tests are incredibly important.

And so let me go back to my presentation and just get to the conclusion and questions. So

we started off with this philosophical question about sort of how you do it is more important than what you do. And so if I think about how data science and analytic teams are working today, we spend a lot of time on sort of the tools and the technology, the data, the algorithm, hundreds of blog posts. We learn about it, we like it, and we only spend sort of maybe 3%, this is my guess, time focusing on the people and the process, the operations.

And I don't think that's quite right, because if you actually look at the software industry, and back in 1999, I managed software teams, and we could ship software every three months, and I thought I was great. Now I wouldn't even get a job, because three months is way too slow. It's three hours or three days.

And also the investment that you take in the operational side. Most software teams are up about 23, 25% of their time ... are working on people who are working on the system that allows other people to deploy quickly and safely, deploy new ideas into production, measure things. They're working on what software calls DevOps.

And I think we need to apply those ideas here to data and analytics. And whether it's 20% or 15%, we need to devote more time to these operational tasks and sort of give them some, I think, respect, because I saw that happen in software for a while. The software industry had teams work for me of 40 or 50 people.

We had one release engineer, and that was it. And now most software teams have 25% of the team is focused on those DevOps tasks. So I think there's a lesson to learn here in the amount of investment and time. And I think if you actually do make that investment, you actually end up in a better situation.

You can take all these things-- Because a lot of times there's these people who've been in the data and analytics industry for a while, they assume that you can never push all four of these things in the graphic equalizer up. If you go fast and deploy to production, they're going to assume that you have to have more errors. Right?

The amount of error-free days go down. And if you have people collaborate, well, you're going to have to slow down your cycle time because people got to go to meetings. And there's just no way you can measure the process. And I think what we find is the best-in-class organizations can push all this graphic equalizer up. You can go fast, and you

00:50:00

can do that with low errors, and you can do it with high collaboration. And we're not on drugs here. And the result of that actually means that your team is more productive and you have less unhappy customers. And so it may sound like a pipe dream, but this is a direct analogy to the same principles about why Toyota keeps producing excellent cars, why it's now the largest automaker in the world, why American Motors is no longer in business, why companies that have embraced DevOps are able then to both ship software faster with lower errors and be able to do it with hire people right out of school, and they can ship to production that week.

And so we think there's a direct analogy between those and our experience with our customers. And of course, if you want to learn more about DataOps and ModelOps, we have a book. We wrote a manifesto. And, also if you're interested, we also have a software product that does DataOps and ModelOps and some of the other ops.

And you can visit our website. And so that's it in terms of my presentation today. And I wanted to leave just a few minutes for questions, and Beth is going to help me out here. And- Thank you, Chris ... Beth, is there any questions that came up from the audience? Yeah. That was great. Thank you. So now it's time.

If you have any questions, you can add them here in the box on your control panel, and we'll try to get through as many as we can in the next few minutes. So here is one to kick it off, Chris. How can ModelOps or DataOps help you govern your models?

Yeah. Well, I think, there's a question of what govern means, right? And so I think it means the first thing is which model is it? And so I think by having your models and all the configuration and parameters that drive the model stored in Git, I think is the first case. And then another aspect of data governance is sort of understanding the provenance of data that goes in it, the results of the data that goes out.

And I think by taking the models and treating them as code and then being able to add the results of that into a tool of your data catalog tool, I think is important. And there are some unique things in governing models that we didn't have time to talk about. So for instance, are the models biased at all?

Is your model have some inherent bias? And that really has to do with the kind of training data and the kind of tests that you have based on it. And so we've seen a lot of cases where people had done, they thought, a good job with a predictive model, but the data that they used had some inherent bias in it, and it was

treating people differentially, racially or by sex or some other criteria that didn't have its intended results. And so I think if you look at model governance from that perspective, detecting bias in models and also help trying to give some explainability to models, I think could also fall into that category.

Okay, great. Thanks, Chris. Another question is, what is a good first step to implement something like this, meaning DataOps or ModelOps, at our organization? Well, I think when I started this back again in the mid-2000s, if we wanted to focus on lowering error rates, well, the first thing that we did is open a document and wrote down every error that we had.

And then every three or four weeks, we'd sit down in a meeting with representatives from every team who did data science and engineering and operations and say, "Hmm, what's the pattern here? Is there an error that we see over and over again?" And then how can we automate that away? How can we write some script or code that that never happens again?

And so a quality circle, a way to focus on errors as a thing that you should do, I think you can do that right away. You don't need to have software to do it. And then likewise, just trying to measure how long it takes you to deploy a model into production, because there's a lot of tools to help you build a model.

There's great suites that can help you collaborate with other data scientists. There's AutoML solutions that can help you tune your hyperparameters fast. There's nice UIs of many different tools. But really how fast, once you develop the model, can that get into production? And sort of looking at the problems and steps and how you can deploy it, I think is important. And so a lot of it starts with the perspective of like, well, if I am focused on cycle time and error rates and collaboration, what are the problems? Let's make a list. Let's try to work through them.

And by having that ideal as a way to work can actually force change or help make change in your organization. Because I run into a lot of people who are in some ways sort of beaten down, and these ideas I think can help them

00:55:00

reclaim control, and be able to start doing great work again and not have to start something in January only to see it get into production in September.

Great. Thanks, Chris. Well, unless anyone else has a question, believe it or not, that is all the questions that we have right now. So I want to thank Chris for giving us that great presentation. Thanks everyone for taking the time to join us today. As I said at the beginning of the call, we'll be sending out a recording of the webinar and the slides in the next 24 hours.

So be on the lookout for that in your email. If you have any additional questions, please don't hesitate to reach out to Chris or to I, and we'll point you in the right direction. And so that's it. Have a great rest of your afternoon and evening. Thank you all.