Connecting Your Data Mesh with DataOps

00:00:00

Hello. Welcome, everyone. Thanks for joining us today. My name's Beth Beverly. I'm the VP of marketing at DataKitchen, and I will be the host for the webinar today. So we have Chris Bergh, who's the founder, head chef, and CEO of DataKitchen. He joins us to talk about an exciting topic, which is how to connect your data mesh with DataOps.

So for those of you who are new to our webinars, just a little background on Chris. He's a leader of the DataOps movement. He has more than 30 years of research, software engineering, data analytics, and executive management experience. At various points in his career, he's been a COO, CTO, VP, and director of engineering, and he's also the co-author of "The DataOps Cookbook" and "The DataOps Manifesto." So just a few housekeeping items. If you have any questions, please enter them in the question box on the webinar control panel, and we'll use the last 15 minutes at the end to try to get through as many of those as we can.

And then lastly, the webinar is being recorded, and we will send the link out to all registrants after the webinar, so be on the lookout for that. So with that, I will just hand it over to Chris. Oh, thanks, Beth. So, welcome, everybody. I'm going to turn off my webcam to make the screen bigger.

And so we're going to talk about this exciting idea of a data mesh today. And so, I'm really excited because I think in a lot of ways, the field of data and analytics, data science and data engineering has gotten complicated over the past 10 or 15 years. Lots of tools, lots of servers, big data, small data, AI, et cetera.

And so people have been working on how to deal with complicated systems for a long time, and that's actually, the software engineering field has done it. And so, I'm really happy that people have started to talk about this idea of a data mesh, which is kind of an idea of how to deal with complexity that's come from software. And of course, DataOps is another idea on how to deal with complexity and building and deploying systems that's come from software.

I think both those things can really help, these ideas can really help data and analytic teams. And so that's gonna be my talk today. So I'm going to talk a little bit about what data mesh is. There's a lot of great resources out there. I'm going to start by talking about the people side of the data mesh.

What's the team structure like? And then I'm going to go into the tech side, and then I'm going to do kind of an extended example of how you build a data mesh. And without using the name data mesh, we've been following the sort of domain-driven design at my previous company and this company for the past 15 years.

And so, we're going to talk about that example, and then we're going to go into kind of how to construct and connect a data mesh using DataOps, and then some concluding thoughts. So why a data mesh? Well, I guess the big idea is large, complicated, centralized systems fail. And because of complexity, lots of things going in, lots of things going on, it's hard to understand, change, modify, and complexity is a killer. And so a data mesh is a way to break the complexity of systems into smaller pieces.

And then second is when you build systems, the idea of your team focusing on a particular skill. Like I do schema design, or I happen to do transformation, or I do data loading, or I do testing. Those skill-based roles have a really hard time responding to rapid customer requests. And I think one of the big reasons is that knowledge of the domain, and I'm using the word differently here, yet knowledge of the data that you're working on really matters.

Because data is very different. Data has complexities. And so, a schema is a schema, and so you may think for every dataset in your company, I'm going to get a schema designer and they're going to design every schema. But actually having some knowledge of the actual data helps you do your work. And the idea that there's sort of one overarching pattern that covers everything, I think is another prescription for failure. And I just think we've seen a lot of data and analytic projects failures, and we talked about that with DataOps. And I think, another way to deal with complexity is to take the big problem and make it into a group of smaller problems. And that's sort of what a data mesh is.

And think of it as an organizing principle for data systems. And the work that you do is organized into discrete domains. And we're going to talk and sort of define what a domain is.

00:05:00

And the idea is that a team can work autonomously in each domain. And they get to know their data, they get to know the customer, and they get to focus on what the customer needs. And that way, the team actually gets really good knowledge of the data or the things that they're building from the data.

And I think that actually can empower them to be more agile, have more ownership, have a better feeling of control, and be more responsive to the customer. So instead of I've got someone who does my schema design and they've jumped from one area to another area to another area, their schema design is sort of embedded in the team as well as different techniques. And it's really inspired by this idea of domain-driven design in software. And again, software's a complexity business.

And one of the failure patterns of software is the big, complicated, the term is monolithic systems. And it's hard sometimes. If you can make discrete pieces, it's better. And so the main idea is take a best practice from developing software and apply it to data and analytics. And If you've been listening to us talk about DataOps, we certainly are believers in that.

And so the next thing I'm going to go into is sort of team structure. And I think one of the ideas of agility is this idea of flow, that you are able to do your work uninterrupted. And by sort of partitioning work into pieces, maybe ingestion, processing, serving, and different tasks that you do, it actually ends up slowing down the work.

And these diagrams are from a company called ThoughtWorks, who I'll have later. But they end up sort of building barriers between teams because I've got my specialty, you've got your specialty, but we don't know the context of the task. And a lot of times in data and analytics, the context of what you're working on is hugely important because your customers are on a journey to get insight, and they may not know exactly what they want.

And so being able to make a change in some part of the system in order to quickly iterate and give an answer to a customer is hard when you're all sort of, "I do this part. You do that part. She does the third part, and the customer wants this thing." All three of us have got to gather. We've got to organize our work.

We got to send emails and documents. And so if you can get the team to work smaller and kind of think of an idea of a data mesh as almost an organization mesh. And the idea that data engineers are kind of not perfectly fungible. That means they can't do everything. And assignment of the ownership of a domain to a small team that's cross-functional, and it has the capability in that team to do what it needs to get the job done.

And so a data mesh is, think of it as an organizing principle for teams as well as a way to structure data and build composable systems. And there's something I think really important about ownership and staying with things and being determined. And so if you own the datasets and you own the processing of the datasets and you own the changes to the datasets, you're kind of the product team, and you own it.

And that sense of responsibility is empowering. It's hard sometimes because you own the result, and sometimes the result isn't great and you got to change it. But this sometimes in agile terms, they call it product thinking. I think of it as ownership thinking. And it gives the right incentives because you own it and you're trying to make your customer successful. You're not trying to work in an abstract software development process where you're sort of building all the sort of rituals and things to do. You're trying to own the result.

You want to make your customer successful. And that mental model, having a team focused on that is very empowering and actually is just such a much better way to work. And if we look at the human side, again, it's like in software, can you get small teams, five people, two people, nine people, and they're organized around the domain that they work in.

And then think of a domain as a set of data, for those people who don't know it yet. And the best world is that the operational part, running it and the building of it are kind of part of the same team. And there's a phrase in software, you build it, you run it. And this idea of product thinking, that you own it.

And if you live and work with a finite number of datasets, you get to understand the domain, the questions, and a lot of times, data and analytic questions are representation problems. Should I represent this in a schema? Should it be configured in a different way in the chart? Do I have to tweak a model? And these things are better when you have small teams that can talk to each other.

And so these organizational boundaries, I think, are trying to take it away and

00:10:00

focus really on the customer. And so in some ways, your domains are a way to organize your team to be customer-focused. And so that's the first idea, the sort of human side. And now let's sort of go into the technical side of what a domain is. And then we're going to go into an example.

And so, well, if you've been in the data field for a while, there's sort of the Inmon data factory, there's Kimball and star schemas. There's different ways that people arrange the data vault pattern, which is a great way to store organized raw data. There's philosophies on how you build systems. And so this is saying it's not a replacement of that. It's a way to say, "I'm going to do a piece of it, and I'm going to focus it on a certain area." And those could be based on kind of sources of data.

And I think of it as things you may want to master, like entities in your business or subject areas like customers and products. Or it could be like you could organize a domain around a complicated source, like a user interaction history or web logs. And that way the team gets to know what's going on with that domain. They understand, for instance, if you've got web logs that come from two or three different web systems and user interaction history.

Their domain is to get a good representation of what the users are doing across the tools that you have. And then there's another way to look at domains is they're more aligned with how people use the data, the consumption of data. And think of that as like an integrated dataset. And maybe that's a star schema or a series of tables or aggregate views.

And this is really meant to get the data in a way that is useful to customers, and maybe it's both. It's a star schema and a bunch of views. But really, it's focused on getting the data trusted and used by the people who are going to take advantage of it. And so that domain's aligned with customers.

And so, if we look at what's in a domain, and you sort of would draw a circle around it, well, obviously there's data in that domain, right? And then there's the things that are created from the data, maybe the views of that data, and I think they also should have reports or dashboards, artifacts created from the data. And the other part is code that acts upon the data. The pipeline, the tool chains, the code that's acting upon it to produce it.

And in the domain, there's also that team, and also the sort of metadata that's associated with that data itself. And so if you think of domains, one of the idea is, okay, instead of having a giant data lake that has a series of tables in and you've got to navigate, well, instead of doing a big data lake, I'm going to have several domains.

And that sort of has the question of how does one domain relate to another domain? And so I think the idea is that they've got to be kind of controllable and composable, and that the output of one domain could be the input of another. Some domains are built from raw data,

and the output of other domains. And I think that's a really key order of operations challenge in building these systems, because composition is a first order thing here. And

how you manage that is important. So for instance, I have a list of mastered customers here, and I have these two domains using it. And so there's an implied producer-consumer order of operations relationship between these two. And so we're going to talk actually a lot about how you can make domains composable and controllable. But also there's another idea of domains as a thing. Well, what's outside of that thing?

What's the interfaces? And I sort of put a bunch of Ws on it. The first on the right side is sort of like, where is it? What's the data? How do I access it? How do I get at it? Sort of the where. The other is kind of the what. What's the description, the catalog? What's in it?

Another W is sort of like, think of it as the when. When was this built? What test results were there? What was the timing? What's the status? And then the how is kind of the code that was used to build it, the configuration, the processing pipeline. And then sort of the with is kind of the raw data that went into it.

And kind of try to think of these as a bunch of URLs that your system has. That okay, for this domain, I want to get at my catalog. How do I get access to that? How do I tell its status, whether it ran? Can I get a URL?

00:15:00

Where can I get at the source code that was used? And having this as interfaces to a domain allows you, and being intentional about that, allows your team then to better collaborate and coordinate together. And so,

what that means, and so what do you want out of a domain? And when you have a series of kind of independent work. Well, you want to trust it, right? You want to know that the data is correct and the artifacts are right. And trust has got to be earned. And so that's one of the reasons why having these sort of points of introspection or access points, you should be able to ask these domains some questions and get some answers to it.

And it's kind of got to be usable, right? So that means you've got to have a way to have a catalog to understand it. And then it's got to be sort of findable and discoverable. And it's got to be described, and it's got to be secured and permissioned. And I think this idea that the domain is URL or API driven, that you can get at it.

And another way is just more humanly, I think of it kind of like you want to have a single throat to choke. So a domain is a team, and you don't want to put an abstract, okay, I'm going to go and some random data engineer's going to fix it, or some random person. I want to know the team that's doing this.

And if I have a change or a problem, I want to go and have a single throat to choke to make that happen. And maybe it sounds rude, but I do think that's an important part for customers, because sometimes they don't know who's doing it. And the relationship sometimes between the people who are working in the domain and their customers is really important because you want that cycle of question, change the data in the system, question, change the data in the system to be as fast as possible so you can maximize the amount of learning. And maybe that's a better way to say single throat to choke and having a single point of contact.

And so what this kind of means is, if you think of it, there's sort of a change in focus on how you work. And so think of the domains as sort of a first order thing. How do you group your work into small teams and partition them instead of sort of one platform where people are just servicing any request that comes in?

And so that actually becomes an important part to think, and has to do with how your customers use the data and what the data is. And then if you think about it, what services are you providing your customers? And we talked about these API services, the data, but that sort of service orientation and customer focus.

And then really trying to focus on helping people get at and discover all the data that's there because the domain is a partitioning mechanism, so what partitions do they want? And then I also think this idea of decentralization is quite important in data mesh. Instead of having

perfectly fungible people, you've got teams, and they're focused on it, and they're centralized on a set of problems, a set of datasets. And I think that gives people freedom to innovate over sort of a central control. And some software teams go so far in as to say you control everything, even the tech standards, the stack you work on. And I've had experiences, software teams who have different One part's written in Java, another part's written in Scala, another part's written in Python, and that may be a little bit too much.

But if you take the ownership of you build it, you run it, you own it, and your number one job is to make your customer successful, decentralization is a great thing. And then I think of it more of like you're kind of building an ecosystem of these domains, and they've got to be able to relate together.

And so that's sort of a quick technical introduction to domains. And domains and their constitution and a data map. And there's more out there that's written on it. But I think one better way is just to sort of talk through an extended example and an example that I know quite a bit about because I've been working sort of in commercial pharma analytics for 16 years now.

And so, if you're from the US, you know we have a privatized healthcare system. And the people who produce pills or biologics are commercial pharma companies. And they have a lifetime patent of that if they make a pill or an infusible, an injectable, they get exclusive rights to that for a period of time.

And so in order for the companies to do-- They use that because they want to make money and use that money, hopefully, to invest in building better drugs. And so if you look at the commercial life cycle, it's this kind of graph of pre-launch, growth,

00:20:00

maturity, and then sort of end of life, and that's measured in years. And there's different phases of a commercial pharma's life. One is the sort of launch phase, where they're sort of preparing and trying to figure out who's going to buy it, what we're going to do, and trying to get the number of people for the second phase, which is the physician phase, where they're trying to grow the number of doctors who are writing these products. So that growth curve is actually really important.

And then in a mature phase, after a few years, it really becomes more about how the payers, because in the US, your doctor writes you a prescription, and then it's paid for by a payer, a insurance company. And so in that phase, it's really about the discounts that you get. And so there's this complexity and different-- It turns out their teams are organized differently.

There are what are called NPP, non-personal promotion teams and sales teams. They focus on ads and digital websites and they matter a lot sort of pre-launch and launch of a product. And then there's kind of a physician domain team and sales. And if you've ever gone in the US to a physician's office and you see kind of that good-looking man or woman in the corner, those are your pharma sales reps. And they're delivering samples, talking to doctors, providing information, and that's how a lot of sort of branded pharmaceuticals are sold in the US still.

And this actually really matters in the first few years because you're trying to get more doctors to write prescriptions for your product. And then during the later years, when there's a payer domain and this part, in essence, kind of controls the price of a product. And the price is set kind of through to the rebates that are given and the arrangements that are given to the payer.

And those are things about the rebate contract sort of being on formulary, the tier and the co-pays, these sort of contracting mechanisms actually really matter. And so payer marketing matters more during the mature phase. And there's overlap on these, and it's a kind of a generalization. But the data in these is actually kind of very different. And so

for instance, there in the US pharma market, there are domains of data that they buy data from companies like IQVIA. They have internal data sets from Veeva. There's a whole multi-billion dollar industry of data providers, and some of which are like census data, some of which come from calls and samples. But what's interesting about pharma data and how we arrived at it is a lot of the data itself is real data, but it also is mixed with projected data. And so it's hard to mix some of these data sets together, even though they may be about the same physicians and payers, because they come from a different data source, they're sourced differently.

They really tell sort of a different story about the data. And so these domains tend to tell stories about physicians or payers. And there's a typical sort of processing architecture here, where you have the data sets that come in from internal sources and external sources. They're updated sort of daily, weekly, monthly. There's a processing to a warehouse and then users, salespeople, marketing execs, the rest of the company.

And these are very important because the amount of marketing and sales effort on a product can determine how much money a pharma company can make and invest in new drug development. And so the way that we've thought about it is kind of thinking of it in a domain layer standpoint, and one is that there's sort of a mastering layer that is typically done by IT, and this is sort of what are your physicians that you call on? And there's about a million physicians in the US, but the call on universe may only be 40,000 or 10,000 or 100,000. And they're trying to get that right because sales compensation is set. And then there's other sort of small files, like what's your product hierarchy that goes with it.

And then There's sort of a data warehouse, or think of it as an analytic data mart, where people are trying to put data sets together to be able to answer business questions that come up from the sales and marketing enterprise. And then typically, and this has happened a lot in the last few years, there's sort of a self-service and data science layer, and they keep their kind of own data set. Maybe they have their own tools, maybe they run some SQL code, or they do some Python work, or they use a tool like Alteryx, and they're building reports and views and PowerPoints off this. And so there's kind of a...

In all these layers, there are data and some actions, artifacts being created from

00:25:00

the data. And if we look at it from a layer standpoint, on the left, you have these raw data. And then there's these different domains. There's kind of a mastering domain, maybe around physicians, another mastering domain, maybe around product market baskets. There's kind of the integrated data sets. Think of these as star schemas and aggregate tables or views that people are using, maybe data engineers, and they're kind of broken up into this payer and physician and NPP.

And then there's sort of self-service tools. And so one of the reasons why they need to be broken up is that the data on this side comes in at a very different timing. It's not like, "Hey, all the data arrives on Tuesday." It sort of drips and drabs in and there's some comes in on Tuesday, some comes in on Wednesday, some comes in and drips in multiple times in a week.

And so there's a relationship between the refresh of the data, the update of a master file, the update of the data in a domain, and then the reports that go to it. And so for instance, if your physician, there's a file that the AMA has, sort of a physician file. If that changes, that could change your mastering, which could change the physician dimensions into your star schema and the physician domain and the payer domain, and would affect your sort of brand team reporting.

And so how do you manage this complex relationship when you've got different domains? And if you believe in the idea that you should try to have people work independently and have ownership, which is a good idea, you've got to think about these sort of processing relationships, order of operations relationships, and the technical relationship of the data.

And because in this case, the physician, the schema, this has a physician dimension in the physician domain, and the payer has a physician dimension, and you want those to be aligned and have the proper IDs on. And so there is a sort of data sharing that happens between these as well. So thinking intentionally about order of operations and how people work, I think is important and was a key to our success in being able to do this.

And then, of course, having different teams work on the physician and payer and NPP domain and different expertise of that data. And so what happens is you actually end up with kind of different processing steps. In each one, you have maybe the IT team has sort of an MDM system, master data management system.

And the people doing the data work have their own tool chain and the people doing self-service. And so you've got these domains. People are in control of their domains and with their own tools and their own process. But again, that relationship is important. And so,

it comes down to a couple of questions. It's like, okay, if I update the data and each domain's got its own processing information, and perhaps it may have their own tool chain, then each layer's got its own series of steps, its sub-steps, and each layer wants to know the results of the processing. And what causes the update of each domain?

Sometimes the simplest thing is you're done at 2:00, I start at 2:30. And that order of operations as schedule can work, but it also doesn't work sometimes when something accidentally goes wrong and your thing that you thought was going to finish by 2:00 actually goes to 2:45, and so you've kicked it off and broken it and chaos happens. And so being able to order of operations these things, sort of meta orchestrate these, is important.

And in some ways, you need to kind of think of it as a master DAG, directed acyclic graph, a master workflow to run them all. And in order to do that, you kind of got to think about the sort of inter-domain communication links. And there's some questions that you have to ask in building and setting up the relationship of those domains. And I've sort of classified them here. And one is, I guess the first two are, think of them as queries on the domain.

You can ask the domain, "When was the last time you were updated? Was it successful? Was there warnings?" Is the data good? Have you run tests on it? Can you prove it with some test results? So can you ask the domain itself questions about its recency and accuracy and testing? And this isn't describing what's in the domain, it's just saying, have you processed it well? Do you trust it?

And then the next two are kind of process linkage, like, "I'm done. You start." Or, "I'm done. Here's some more information about what I've done, like where it is and what's the schema, what's new, and here's a bunch of parameters that you want to do." There's a process linkage. And some companies will do process linkage with kind of an event bus in the middle.

They'll say, "I'm done," and then they'll put it onto a Kafka queue and then say, "Here's whoever else in a pub-sub way of doing it."

00:30:00

And then there's kind of a data linkage, like we share common tables, or the output of your data is the input to my data. And that's very common. And then you can almost think of it as a development linkage. "Okay, I'm making a change. Can I recreate your domain in development? Can I modify your domain in development and my domain to see if this whole new feature works?" And is there sort of a path to production for our changes? And so I think thinking about these linkages between domains is important, and it also goes to why having these sort of ports or access points on a domain that you can go and get at it, and it can answer these questions.

Because then if you have a URL-based way to do this or you can then ask questions of these URL and parse the answer and be able to set up these interfaces in a more logical, programmatic way, rather than just having a Word document that someone has to run. And so, let's see. So we're at 1:30, so I'm going to have five slides, and this actually doesn't have anything to do with data mesh, but it has to do with our example. And I think one of the reasons I'm saying this, this is actually very helpful because one of the ideas of a domain is it makes the work smaller. And if your data is terabytes of size as opposed to petabytes of size, you can actually start to think about the idea of reprocessing all the data in a domain from its start.

And that becomes a doable thing. And what that means is you start with what's called immutable data. It doesn't change. And the work that you do on it, think of it as pure. If you put the same input in, you always get the same output. And nothing else happens. You put in the input, you get the output, just like the function machine you learned about in third grade.

And then the other thing is that you can run it over again. It's this word idempotency, and it means that you can run the same thing over again, and you get the same thing. And it's actually a really useful principle, because a lot of the work that's done in building those domains has dozens or hundreds of steps. And can you say, "I just want to rerun the middle steps over again at a breakpoint, at a checkpoint, and then I can get the same result and run it over and over again." And that allows you to build more of a composable system, which allows you to test and develop and patch faster.

And so this example, we actually sort of built with this sort of functional programming one, and it allowed us to be able to make reproducible ways of working. And I think that's sort of critical from a sanity standpoint. If things go wrong, we were able to just start over and start the build from scratch.

And it enables sort of a complexity reduction. And also because the cloud actually makes this really easy. You could just spin up another server and start from a whole set of S3 buckets again and run it. And it ends up meaning because you've reduced complexity working in a functional way, you get faster time to value.

And I think, again, this data mesh, DataOps, functional programming, there are ways to deal with these complicated systems. And the fact that now we don't have to worry about calling the Oracle salesman, and there's a couple of good articles on the internet about sort of building functional data systems that we can provide links to.

And so the last thing I think is when you have your domain, you can kind of think of it, and this is the way we did it, was to think of it as a big function. And so I'm going to go take my production data, and I'm going to run it in my domain, and I'm going to be done. And my test passed, and things are good.

And that's sort of a big red, green light saying, "My customers are going to be happy." And then the second is they also make it easier to deploy. And the trick that we used is since you start with the same production data, perhaps you call it test data, but perhaps it's the same as production data or slightly modified. You can run the whole process again, and then you can kind of compare the results between the previous and the current, between the development and production.

And the path to production here, once all your tests passed, is not a merge or it's a flipping the DNS entry. And so what this means is you could build a Redshift database, a space in Snowflake. You're building a complete copy of what's in production because it's based on immutable data, based on a whole functional architecture.

And once that tests pass, you know that it's good. And the path to production isn't about patching a system and doing deploys. It's really just about flipping the DNS. And that actually makes it very easy to not only have quick go to production, but have lots of versions of production,

00:35:00

run a production that has 1,000 people and a canary version that has two people on. Have lots of different development versions. And so thinking in this functional way, working with domains actually makes it act quite quick and being able then, and especially thinking of these sort of access ports or ways to get at it. And so this sort of DNS flipper idea of pushing to production has been very effective for us in building systems, domain-driven systems.

So the last part, and we've got a few more minutes left. So we talked about, okay, things in data and analytics are complicated. This idea of don't do everything, break it down into domains, and then what is in a domain, and then how do domains relate to each other. And then we gave an example of a couple domains in commercial pharma around payers and physicians and how those domains relate to one another. And so now I sort of want to pull in the idea of how you construct and connect data meshes, and that's sort of the DataOps idea. And then I have a few pointers and concluding thoughts to talk about.

So as you know, we've talked a lot about DataOps. And again, it comes from Agile and software's DevOps, so at least two of them are sort of pure ideas that come out of software and then lean manufacturing, again, which is honestly another way to deal with a complicated manufacturing line. And the point of DataOps that we talked about is, can you

experiment and innovate quickly? And do that with low errors in production and development, and enable collaboration. And so Data Mesh is a really nice fit with all this because it fits quite clearly with being able to do this. And now you aren't required to use a Data Mesh in order to do DataOps. You can do DataOps on a non-Data Mesh architected system.

It's nicer if you have a Data Mesh, just like DataOps is sort of nicer in the cloud, but certainly works on-prem. And so let's just talk about the development process. And so thinking about how you're going to take your domains and change one part of it and see the effect. And so it's hard enough in a single domain where you've got some data that you're ingesting, some data you're transforming, maybe you're a model in visualization.

How do I know that I haven't broken anything, even within my domain? But it gets more complicated when a domain has got a relationship to another domain. And so how do you allow this sort of ability to, in development, make a change and see its effect and sort of pull the pain forward? And this is a challenge in software domain design as well, is that you, okay, people do work and your piece changes, but you've got to be able to put that piece together with the other domain and run them together as a system. And I think that's an important part of domain-driven development, is we've got to test the system and not just test the piece. And of course, you've got to test the piece.

But sort of development-level system and unit and functional tasks based on realistic data, I think is important. And that's where the key questions are: how do you allow a local change to a domain and sort of global governance and control? And that's something I think is important to do, and not only within a domain, but between domains as well.

And of course, we have some software to help that. And so the two aspects of our software, one is that within a domain,

the each steps that you're using to build the work, with whatever infrastructure and tools you're doing it on, need to be tested. And so you need to test each step and we've talked quite a bit about what makes a good test. How do you test across your tools? And it's not just sort of workflow like Airflow or Control-M. It's really about being able to have a test-infused workflow. And then it's not just a single workflow, it's a workflow of workflows. It's a sort of a meta DAG, or a master DAG across those. And I think those components are what you need to have a successful Data Mesh rollout. And then likewise, we talked about these interfaces, and our software has these, at least two of them.

Our software does not tell you or describe the data. We are not a data catalog. There are plenty of great tools to do that, but we do describe, with a single URL, what happened. You can go get at the order run. You can tell when it ran. You can tell what its status is.

You can tell what code was applied to it, timing, test data. And so then you can ask these

00:40:00

URLs questions about, hey, is that data good? When was it last updated? What tests were applied? And that actually gives what's the history of it. And having this URL-based way allows you to then start building programming abstracts, not on the URL, on the payload from the URL. And then likewise, trying to understand what work was done in a domain and what code was there, and being able to get at that code. And so one of the ideas of having it have a URL is that it's sort of constant.

You can always find out what code was run, and then if you want, you can then go and visualize it and see it and learn from it. And then for advanced organizations, maybe you could create a kitchen and change it in development, and you could then allow people to make some changes with control in your own team's domain, which may be scary, but it also is a really exciting possibility and sort of brings the open source model to, I want to create a

branch and make a change, and then you approve it being coded into the individual domain. And then this idea of composition. And so we have a way to take our recipes and group them into something called ingredients, which are ways to create composable units. And then each sort of domain can change independently, but they can also be orchestrated together and sort of our features of kitchens and a way to this idea of an ingredient allows you to create-- It's sort of a compositional device. Allows you to create reusable components that have an interface to them that is defined.

And then finally, this sort of idea of inter-domain communication links. We do almost all of them. We don't really do the data linkage. There's plenty of tools that access data and store data. But the sort of ability to query a domain comes from our order runs. Ability to do the process comes from calling an order run or working, running a recipe.

The event linkage and the development linkage all are sort of features that we can support in DataKitchen. And so just to sum up, I think it's important to think about structuring your domain, but also thinking how to develop and how to deploy and monitor. And so our recipes themselves could be the master DAG as well as the DAG inside the DAG.

And I think tests and monitors are really important to assure that the data is trustworthy and truthful, from your customers and your suppliers, and also sort of aids in diagnosis and detection. And then one of the things that we also do is a lot of The work that you do is just managing variations.

And when you intersect the sort of functional architecture, creating a production one, a canary one, development version one, development version two, being able to handle all those variations of changes in the domain and being able to run and check those, I think actually enables a great degree of agility and also enables people to be highly productive. And the last thing is we haven't talked a lot about the infrastructure, because in some ways, the idea of a domain design isn't really about the infrastructure it runs on.

It's a design pattern that runs in an infrastructure. And if you saw our discussion last week of a data fabric, a data fabric is kind of the infrastructure that you could build a data mesh in. But being able to do any work in reality does involve having infrastructure for you to do your work, and being able to do that sort of on-demand, I think is an important feature for people to have.

And then finally, how do you help your team get to this? It sounds great. Customer focus, product focus, domain design, data mesh. Wow, that sounds great, but building a system to make that happen, building the factory as well in order to make great cars, I think is a really important part of the mindset that people need to have.

Or you're just going to get caught with the potential and you're going to build a lot of technical debt, you're going to have a lot of heroism on your team, and people are going to burn out and leave. And so whether you're using DataOps principles to build a data monolith, or whether you're using DataOps principles to build a data mesh, or using DataOps principles to serve up a single table and a single report.

Building the system, the machine that makes the machine, is really important. And so applying DataOps principles to data mesh or data monoliths or even small projects can help you in all these ways that we've talked about in previous webinars.

00:45:00

And so just a couple concluding thoughts before I finish. So data mesh, a cool new paradigm. Lots to read about. A way to deal with complexity and systems, and its focus is kind of on the data and the architecture, and not so much on the operational process. And sort of the DataOps part of the mesh is equally important.

And the sort of linkages and compositions between domains are important. And balancing sort of central governance and local domain independence, I think is really important and sort of states' rights versus federal rights. And our software's got a bunch of features that can help you apply DataOps to data mesh. And if you want to learn more, I've got a bunch of links that I've found, mainly from people at ThoughtWorks.

And so I've listed them here that you can go and follow. And there's more and more people talking about this design pattern. And then lastly, of course, we're a software company, so we have some software to help accelerate your work of data mesh and DataOps. And then we've written quite a bit about DataOps itself, and so we've got a manifesto and a book for you to follow and read.

And we're actually coming out with a new book in the next couple months on how to help your organization transform to work in a DataOps way. And so I want to thank you for your time and, in the last few minutes, see if there's any questions I can answer. Thanks so much, Chris. That was really great. Yes.

So to the audience, if you have any questions, now is the time to pop them into the question box here in the control panel, and we'll get through as many as we can in the next 15 minutes or so. So, to start it off, Chris, how do you ensure a robust governance structure with a data mesh domain-driven approach? Is there a particular structure required for effective data governance? Centralized or decentralized structure?

Is one of those best? Yeah.

Data governance is a big topic, right? And there's security, being able to store the metadata about a process, data catalog, data lineage. Those are important tasks that happen in governance, and I think the way that

we touch it at DataKitchen is we think those things should be done as code. And so data governance as code, data security as code, data catalog as code. And I think the idea is in a data mesh, let's say you're changing one table and adding a column in that data mesh. Well, that column could be affecting a dashboard, and that new column should be in your data catalog. And you should also, if you've got column-level security, that should be in how you apply security to that table. And so maybe that's database-level security. And so these things shouldn't be done by people touching the keyboard or typing.

Ideally, they should be deployed as one complete unit. Here's the new column, here's the new data that loads the column, here's the change to the visualization, maybe the tweak to the model, here's the update to the data catalog, and here's the new security

that we're going to add. And so this idea of as code, governance and security as code, I think are a really important part of building any system. And whether you've designed it as a data mesh or a data monolith or just one table. Thinking of it not as a bag on the end, not something that you do manually through meetings, but something that can be automated, and deployable, I think is a better way to work.

Okay, great. Along the same lines, Chris, within DataKitchen, I do not see any reference to metadata. Is that a function that is encompassed in the product? Having data be discoverable seems to be a key need. Yeah. Well, we're not a data system, right? We talk to the tools that talk to the data. So our software doesn't have any representation of data.

And so as a result, why? Because there's plenty of tools that do that. There's data transformation tools and ETL tools and data lakes and all sorts of tools that you use. And so, we're not adding a lot of value by giving yet another way to transform data. And so, there's a question of what is metadata? And so the description of the data itself ends up being in a data catalog, like what's in the rows and columns or what steps happen to it. That tends to be sort of where it came from and lineage. And so we have a different set of metadata, and I think of it's more of processing-level metadata.

So for instance, if I'm a customer,

00:50:00

I want to go and say, "Okay, where is this data? How do I access it?" If I look at it from a customer's perspective, and I go back to this little odd diagram here. Hold on a second. I'm a customer. I want to get at data. Well, how do I do that? Well, the first thing is, where do I get it?

What database is it in? How do I connect to it? That's one connection point. And then, well, what is this data? What's the description? What's in it? Where did it come from? And that's another. And then the other is, can I trust it? When's the last time it's been refreshed? What are the test results?

And we handle that part. These two things on the right here, the access the database, at the data, the data catalog, that we don't. And then if you're more technical, well, what code was used to produce this data? And you can have access to that. And so I think thinking of it in this sort of constellation of endpoints actually gives a better way to then be able to have your customers sort of trust what happened. And we've produced simple reports off this that say, "Okay, here's the data sources. Here's when they last refreshed.

Here's all the endpoints. Here's when they last refreshed. Here's their test results. Here's a link to a wiki page that describes it. Here's how you access the data." And all this stuff can then be available for people to interact with and learn. And

again, all these endpoints aren't really data. They're all sort of metadata. The code that acts upon the data, the results of the processing, the description. I could argue that everything outside of this hexagon or in these data sources, everything in these circles here, these green circles, are metadata. So that was a long-winded answer, Beth, sorry.

It's all right. Good. Thank you. So the next question is, so we already have a data lake built. Can we extend this to a data mesh?

Well, I think so, yeah, because I think a data mesh is a design pattern, right? So you could instantiate a data mesh in whatever technology that you've used to do your lake. So let's say you've got a Spark cluster, or you're using your data lake as a bunch of S3 buckets. So the data in a lake can be used as source data to build different domains.

There's a question of, isn't a domain just a data mart in some ways? And it is a little bit, right, but it's also about being able to have this level of composability of the mart and being able to have the sort of team intentionally work on it. And so, I think really that mix between the human way of working, having a team focused on a domain, having a team focused on a specific customer and making them successful, and sort of thinking of your infrastructure, your data lake as just a tool. It's not really an endpoint. It's just a tool.

Your end result is you want to focus on your customers being happy, not focused on, "I did a data lake." And okay, you did a data lake, so it's sort of, so what? The end of the so what in a domain design always ends with your customer is happy and satisfied. And so that's what you're shooting towards.

And that goes to this sort of idea of product thinking and being able to focus on making your customer successful.

Okay, great. So which domain team would own a data warehouse?

That's a good question. Well, so that's a question is can you have... I've never really seen the sort of ideal, sort of centralized data warehouse that answers all the questions that every customer has. And so in some ways, I'm a little cynical and believe that any of these you build it and they'll come sort of scenarios of a warehouse or the universal, and I think that's also hidden underneath this idea of a data mesh, is that one size doesn't fit all, and you got to break your teams up into smaller groups, and maybe it's not a data warehouse, but it's a mesh where there's each one of these domains is focused on answering a set of specific business questions.

And so is that a warehouse or a set of warehouses? I think there's some language that has to be mapped between these DataOps principles and the older or newer sort of data warehouse, data mart type, logical data warehouse language. And I was thinking about doing that, and I'm not sure I would've had time to describe it in this, or that I really understand it enough.

Great.

00:55:00

Great. Thanks, Chris. So the next question is, what's the place of a BI team in a data mesh?

Well, I think the jury has different opinions on that. So I think of the BI team as just another group and just another domain in the data mesh, and that's the example that we gave here, where in this one, the people who are doing the reporting, the sort of Tableau reporting, they're also a team. Because in my experience, self-service teams are not just doing charts and graphs. They're integrating data.

They're making, shaping data in a form. They have small data sets that they're managing. So they're a domain in and of themselves. And so, to me, I feel like that's the way my experience is. Of course, it doesn't have to be that way. You could have pure self-service and all they're doing is pivoting on the data. But again, I just haven't seen that to be true.

I've seen self-service teams do self-service data prep, do self-service data ingestion, do self-service visualization, and now self-service data science. And so they're their own domain. And thinking about the relationship between perhaps a more technical one who's building generalized schemas and a more function-specific one is, I think it's an important idea to think about. And I don't know if my particular view may be divergent on self-service from what other people have written.

Okay. Thanks, Chris. So next, so there's often high control on the production database. So as a data scientist or operations researcher, how would I convince IT that I was being very careful with the production instance?

Yeah. So the question at the beginning was, there's a lot of high security and concern on production data. And in certain companies, that's really true. And so, I'm not sure where that refers to, and so in general, the idea, there is still a development process in domain-driven design. It's not like development goes away and development databases go away.

And so, oftentimes, organizations will have only read privileges to customers of their data warehouse, and they can't actually add ancillary tables. And so I'm not entirely sure how to answer that question because it kind of has a whole bunch of contingent sub-questions that I need more information on before I can answer it. But the broad answer is, data mesh doesn't mean that you're doing development and production.

It still means you have a development process. It just means it's scoped more on a domain as opposed to everything. Okay. Great. So what's the most important tip regarding microservices teams in DataOps? Well, I think the term is--

Well, if you mean microservices in the software sense, or you mean microservices in the data mesh sense? What do you think? Yeah. I can't tell.

And it's all terms here, right? Because the core idea of a microservice and a data mesh team is the same. Smaller groups, more focused on the task, more focused on understanding it can better serve their customers quicker. And that's the main idea. And the main challenge is, yeah, by making things into smaller pieces, you've got to think about how they all connect.

And so, that's similar whether you're doing a software microservice or doing

a domain design. The challenge also becomes, like in any transactional software system that has data that's persisted in an OLTP store, the data and analytic team is extracting data from that to do analytics in. And so there it becomes complicated again and of what microservice you're pulling things from to be able to push into your data mesh to be able to do the data work.

And so that's where-- and it's not anything different than what we've always seen, where it's a monolithic transactional system and a monolithic data warehouse. It's just by breaking those up into smaller pieces, you hope that you can actually do more work, but it also increases, I think, the amount of collaboration, and that you have to work between these teams, and the need to handle that collaboration is, I think, between your software development microservices teams and your data mesh teams, becomes more and more important.

Okay. Thanks, Chris. So we are running at the top of the hour. So we have a ton of questions. So if we didn't get to your question, we'll follow up individually. But maybe just to close it off, Chris, we have two questions here about just organizational structure. So maybe you can finish with this. So one is: Is it possible to implement a data mesh thinking without a reorg?

01:00:00

And then the other one is: This model suggests both centralization and decentralization. Within many organizations, the pendulum between centralized and decentralized swings between extremes on a five-year cycle. How much do you view this as a conceptual issue within organizations?

Yeah.

So there's two questions. So the one about the swing between centralization and decentralization. Yeah, I think that's true. And I also think there's another swing between we're going to do it in-house and we're going to pay a bunch of people outside. And so I've seen data and analytics functions be brought in-house and then go out of the house and then they are aligned to lines of business and not aligned.

And so,

I think a lot of those things, in my humble opinion, is because people are working in a waterfall in a non-DataOps way, and if the teams had the right level of agility, that they could actually be able to make their customers satisfied. And you wouldn't have to have these big swings between, we're going to do it centrally or not, because people wouldn't be complaining.

And so- One of the keys I think is trying to think, and the key idea of a data mesh is that get smaller teams and get them empowered on a specific domain, and then they're going to be able to better make your customer successful. And the DataOps idea is automate the work, automate development, automate deployment, automate testing within those domains, and then be able to automate the kind of connections and collaborations between domains.

And so automation and automate testing as well as grouping people on domains, I think can be a helpful way to handle that complexity and hopefully avoid the swings of in-house versus out-of-house or centralization versus decentralization. Because I think Gartner and other people are right. Our analytic customers at the end of the day are not happy.

They're not getting what they want. And we've got to change how we work, and how we work now, it's too complicated, and we've got to find a way to reduce the complexity and add automation to it. And that's why we've big, complicated systems, and they're just as complicated, if not more than software systems.

And so these design patterns of data mesh, DataOps, functional programming, hub and spoke, are all ways to help deal with that overarching systemic complexity.

All right. Thank you, Chris, very much. This was a really great webinar with lots of great questions. Thank you everyone for joining us today. As I mentioned, if we didn't get to your question, we'll follow up with you directly. We'll be sending out recording of the webinar and the slides in the next 24 hours or so, so please be on the lookout for those in your email.

And lastly, if you do have any additional questions, please don't hesitate to reach out to Chris or I at DataKitchen directly. So thanks again everyone, and have a great afternoon and evening. Thanks, Chris. Thank you.