Data Products and Data Mesh

00:00:00

Everyone welcome to today's webinar. So my name is Chris Bergh. I'll be walking you through two topics today one about data products and the other one about data mesh. So what I'll do is I'll start off with a kind of a background on these ideas. And so just to kind of set a background on me. My name is Chris Bergh. I wrote a company called DataKitchen here and outside of Boston and Massachusetts. And so I've been in the software industry for a while and and the data industry for a while. And so I have a I think it interesting perspective on this and so I'm more of a first principle thinker. So this is more of a first principles discussion on the ideas of data products and data mesh and from a logistics standpoint.

Feel free to type answer questions into the question box. What I will do is kind of during the meeting or definitely at the end. I'll try to go back and revisit the questions and for today, we're going to spend about 45 minutes on this presentation. So let's begin and I will post the slides and the recording of the webinar on our website this week.

So let's start. So the theme of this is really complexity. Right and I think the theme of this presentation is that complexity kills and it just makes everything harder and this is Ray Ozzy. He's the guy who invented Lotus Notes has a long career in software and I think this complexity idea applies to software and applies to data and analytics and so it makes it makes things hard.

And so what kind of examples of complexity are there in data and analytic teams? Well, I think first of all, there's complexity in just having large teams, you know, multiple teams doing multiple roles and of course having thousands of different data sets of different types and shapes and having lots of customers each with their own unique needs.

And then there's also complexity and sort of each team member having their own functional specialty or sub-functional specialty, you know, for instance your data scientists your data Engineers or people who do data mastering versus doing ETL. And then there's a sort of long interconnected project plans are also really hard for doing major work and then a lot of organizations have a very convoluted and manual sort of release to production process.

And then the architectures that we build today in the cloud are typically multi-tool multi-server data architectures. So and then everything runs on in data analytics like software everything runs on the code and that code may run in a SQL Server database or may run on your ETL tool, but we just have thousands in lines of code.

So we have lots of complexity and that complexity makes things hard. And so if we go look at how software has done this, well, they've invented some terms and I'm not gonna talk a lot about these but I'm only gonna talk about the second too. So for instance DevOps and my company does data offs, which is application of lean and agile and principles to date an analytics and then domain driven design software products test-driven development observability event driven architectures apis and functional programming and all these are really great ideas and help reduce the complexity burst burden of the people who develop and run software systems.

And so today we're really going to focus on the bottom too. Right these applications of these Concepts these first principles in domain-driven design and software products to data products. And of course, I will throw in some more ideas about how the applications of DevOps or DataOps But there's this isn't the only thing that's kind of going on and one theme in the data and analytic industry over the past five or ten years is this bringing in ideas from software, you know, and so instead of things like test-driven development. We have data testing.

Instead of observability. We have data observability instead of an event driven architecture. We have streaming and the so-called capped up Kappa or Lambda architectures instead of an API. We have a data contract instead of functional programming. We have functional data engineering and these are all really good ideas. I won't go through what the meaning of all of them is, but if you're frequenting our webinar, she'll hear us talk about many of these ideas, especially observability and testing.

And so let's just talk about definitions. And so what do I mean by a data mesh, so I mean that it primarily it's a way to organize your team's work into chunks called decentralized domains and the benefit is sort of instead of boiling the ocean, you know, instead of boiling your data ocean you focus on fewer data sets and fewer customers and get more done.

00:05:00

And so it's a way of taking the complexity out of lots of data sets lots of customers and that many to many problem and making it smaller. and then data products are similar and sometimes related idea that all of our work is part of an ongoing ever improving flow of value and that your success is not defined by completing tasks on your project plan, but by the actual usage and ever expanding delivery of insights to your customer.

And so data mesh think of it as a way to get complexity down and data products talks about this word flow how to get the flow of value to your customer over and over and over again and keep improving it. So let's we're going to talk about data mesh and data products separately. But in each we're actually going to kind of go through four chunks sort of the people side the tech side and we're kind of look at it from the inside and the outside. So the first thing that we'll talk about is is data mesh.

and so a lot of data teams are kind of organized functionally. So you may have a team that does ingest and then a team that does ETL and then a team that does visualization a team that does data science and in some data teams, there's people who do schema versus ETL some people who do Architects and so the flow of work is kind of compartmentalized by each team. And and so what happens is the work has to go from one person to the other.

And that organizational structure sort of builds walls and when you make a change you need to coordinate and so that's that that's hard for a lot of data and analytics teams. And so the idea here is I think it's really simple is that data people are not perfectly fungible. So if you think of an organization that has a thousand data sets, it's very hard for someone to know all thousand data sets. So it may be easier to assign 50 or 100 data sets to a smaller team and the customers that know them so they know the data they know the customer and then that organizational principle allows them also to work in a tighter a tighter Loop. And so really I think of data mesh from a person first principle standpoint. It's the application of domain driven design which is a way again and software to instead of having what's called a monolith having smaller components of software and to do that having domains is also means map the organization to The Domain. So you're you're teams work in a domain.

And the idea is that your teams will kind of own that project. They'll understand those data sets. They'll understand the customer and that learning will help them iterate faster and work better. And so as an example, so like what does this mean by domain or how do you cut up your data worlds into pieces well.

In my experience, there's just several ways like one is you could do a buck kind of mastered versus unmaster data. So it's aligned with your sources of data like where it came from. And so one could be like what's the sort of facts on the ground of data and maybe that's things like weblogs or user interaction history others sort of entities like customers or products or the other way is to sort of have them align with consumption data. Like okay. We're we built a star schema for one thing or another set of reports for another one. And so the idea of domains is that they're they're not perfect right they um is that they need to have interrelationships with them. And so that actually adds the work and some thinking that has to be done in data match.

And so like what is a domain? Okay, so think of in my mind I think of a domain as a unit, so maybe it's some integrated data Maybe it's some raw data. It's the artifacts that are created from data. So for instance the schema The Fill models the views the reports the dashboards and I think of it as the code that's acting upon that data. Maybe you've got some SQL code or a data visualization. It's also the team and it's also things like the metadata for instance your data catalog or lineage or the processing history Etc.

So it's it's almost like a complete analytic system. That people are working in because at the end you wanted to deliver value to your customer on this and maybe your customers and other team, or maybe it's a business user. And so you know what like if you're going to go in in this way of working. What do you want out of a domain or a data mesh, and I guess the first thing is that it's independent. You can work independently of other groups and that the data that comes out is trusted. Right and it's usable and also that it's discoverable and find of findable that

00:10:00

it's understood that it's secure dedicated and number six is important that it can interoperate with other domains. And lastly, I think a lot of organizations. They don't know who's responsible. Right and they don't have a throat to choke for customer problems. And so having sort of a domain architecture says, okay. This team is responsible for this set of data and then they can know who to go instead of having to have tickets and it goes into sort of the it black box and that's also you know, the idea is the team the Data mesh team is smaller so they can have more customer intimacy.

And I think of you know with the domain or the data mesh having interest to having interfaces, right and I've broken them up in here into five pieces. So the first is the width it comes with data, it could be raw data or it could be data that's already processed by some other domain because a it could be when like, when did it process What's the timing? What's the test results? It could be where where to find it.

How to access it how to you know how to get at it. Maybe it's the connection string or the URL to the Tableau report. And then there's the what sort of the description of the data the catalog the lineage and then I think of it as the how all the steps and code and configuration and Tool chain the pipeline that get there. And so I think all of these are interfaces to The Domain that actually make it quite interesting to understand what happens and so for instance and I also depends on your view like I think people who are more of an engineering sort maybe actually interested in the tool chain and the processing the the customer may actually may be interested in the what or perhaps the win on where it goes and the where

somebody say

so let's keep going and the idea of a domain is that they're composable and this is an interesting idea that the work in one domain could be the input of another domain and this is I think one of the ways to gets its power is that you can sort of Break Your World down into smaller pieces and one team can use the output of another team's work and I think that idea of composition actually makes it very powerful because it allows people to both control their own world in a specific data domain but utilize other people's work.

And that composition is important, but it also requires what I'm going to talk about a little bit some some observability. So in some ways it's also a change in Focus, right? So I think a lot of data and analytic teams are sort of. Have the kind of Lord of the Rings you of the world that there's one ring to the rule them all that there's one platform to rule them all and we've seen that over the years with sort of data lakes or data lake houses or data warehouses. We put all the data in and magical happen. And I think that's unfortunately not true that by partitioning your work in a smaller teams and those and by giving them some ownership over the partition and customer focus they get more done.

And in some ways the focus isn't on the project plan. It's really on the services that you're providing to your customer. And so it's rather than I'm loading. This data is like well what value am I giving to buy customer? And it's really about Discovery and it's also about this idea of decentralization. And so by having well-designed domain interfaces, you can get the freedom to innovate and instead of having central control and having okay. I'm an ETL engineer.

I work on my thousand data sets. I've got to switch context from you know marketing to sales to manufacturing to finance day in and day out that context switching is also a cost and then it's also thinking about it as a whole whole ecosystem link together. and so That being said let me let me give it example here sort of an extended example of our experience here at DataKitchen's use of a data machine and we've been kind of doing it without that name for almost a decade and a half and and let me talk about why that is in the example.

so in commercial Pharma Analytics in the US people company spend multiple billions of dollars bringing a product a drug to Market and they have lots of Problems doing that but once it gets approved they have a period of time where it's protected by Patent seven or eleven years and in there there's a bunch of data that they can use to understand which Physicians prescribing which payer is paying because in the US there's it's it's commercial. So they're they don't work for the government and there's this growth

00:15:00

curve right where product sales grow up and then when it gets to maturity and it goes to generic it goes down. and so In this case, the domains are kind of a combination of data and use case because different parts of the Pharma product life cycle. Kind of grow at different times. And so there's what's called non-personal promotions and PP which are things like emails and websites and radio ads.

There's physician level data which are about sort of sales and claims and anonymized patient data and then they're sort of pay your own plan who pays for because in the US mostly everyone has insurance and so these domains are kind of reflective in the teens that work on it. And so there's a lot of times a sort of a non-personal promotion marketing team If you're sort of watch a watch a TV show you've seen the the ads that go on and then there's the sort of the physician domain if you ever gone into a doctor's office in the US and you find some good looking man or woman in the suit that tends to be someone who's out detailing Physicians and then there's people who are actually in charge of Contracting and and making contracts with the people who actually pay for what are called payers your insurance company. And so there's different teams. They have different use in the life cycle, but the but they're there and so on.

This is lots of different data that people use to understand launch in a different times. Right and then some of it is purchased through syndications people like IQ via it put that data together. Some of it is internal like that comes from their Salesforce some of which is really available like Census Data some of which is very specialty providers like stocking data or source of business data and so the purchase and integration of these data sets are really interesting because they don't actually come at the same time. They have different frequencies and then each domain is it's characteristic and so for instance in one area you may have In one data set you may have 100,000 Physicians and another you may have 300,000 Physicians and and they also can't that a lot of times the data is sparse and there's projected data that's mixed in so it's complicated.

And so how do you do this? Right when these different data sets lots of different data sets. A typical pharmacological I've sort of 50 to 100 different data files that are coming in and how do you make this work? Well, the way that we thought about it is that there's sort of Master data sets right that are sort of small files that are in and of itself a thing. And so what are Mark are called on Physicians or what's our region District territory hierarchy and then of course, there's there's sort of the integrated data set right the sort of the main data warehouse or data Lake and then there's kind of self-service and so let's look at this in a diagram. So what comes in here and these are these circles represent different domains. There are sort of the thousands of files that come in on it.

Daily weekly hourly basis, right? And then there's sometimes a layer that comes in that sort of an MDM layer like who is the exact physician? And who are our Target lists or product Market baskets and sometimes this is data isn't very big but it's actually really important because people get paid on it and incentive and buy it and then there's sort of think of it as integrated data sets right sort of raw data comes in and in some cases facts are dimensions. And in this case, there's a payor domain which has to do with data that comes specifically around payer. There's a physician domain and then there's the NPP or not personal promotion domain and why these are our different is because sometimes like and the NPP domain you may only have an email address.

And in the payer domain you may not be able to link the payer data with physician data sometimes. And then finally, they're analysts who use it and they have maybe a brand Team. There's field sales who use it? Sometimes there's data science teams. So these domain layers are actually work quite independently of our other of each other and then the data sets you you know, sometimes they'll be a data engineer to within each one of these domains working independently and likewise. There's field sales reporting and branding so in bigger companies, they can be completely separate.

And so what happens here is like if you're going to change for instance.

A data set right and you get a new data set in well that data may have a new physician record in and that new physician record may be added to the master which actually may affect both the physician domain and the pair domain and effect the brand Team domain. And so these domain processing relationships are actually quite important because if you have a new mastered physician, that means you have a new ID for them that means there's a dimension that have Physicians and you probably should have that mastered ID,

00:20:00

which probably should show up and the brand Team reports. And so this complexity of of the Ripple of changing data sets, you know has to be addressed enough. in a it's sort of a data mesh processing world. And so what that means also is that each processing step actually each build process happens kind of independently, there's different tags here for doing the work and doing the steps to get it out. And so the customer doesn't know all this right that at the end. They don't know that there's some cell service tools using ultrix and Tableau. There's some data Engineers, you know, writing some SQL and elt and then there's some it team to doing MDM and there's a bunch of pieces in between and this all's not apparent to your business customer, but these by having these different people working in the data mesh, you can actually get really really great productivity, but you have to be cognizant of these links between domains.

And and that's really something that's kind of important is that these inter domain communication links can are I think a criteria for Success? So for instance saying, okay. For instance, I'm done you start right the very simple case. There's a process language or I'm done you start and here's some parameters. And in some cases there's a linkage on data you we share a data table?

And then there's other cases are like, you know, when's the last time you're updated? Can I trust you with different different test results? So these domains have relationships to each other that have to be thought of and in order to make this process work, but there's a finite number of relationships. And so kind of getting to the the end here. So what we've talked about is is one of the reasons people do data mesh is complexity from the way I touch the elephant. It's mainly because of people

that data engineers and data scientists aren't completely fungible that they have to they work better when they know the data and the customer we get went through an example of in Pharma analytics. We're having separate teams of people working in domains. And then we talked about the challenge of having one domain work with each other and that kind of domain linkages.

The other side is like when you have a bunch of domains Each domain is a data Journey because each data domain itself may have one or more tools one or more data sets and a data. Journey itself is something that tracks and monitors sort of all levels of the data stack, right? The data demand could have its own database its own ETL tool. It's own business tool and the idea is that in that processing of it's setting a set of expectations about what should happen.

so you can know if everything ran on time or without it errors and so data Journeys are sort of the operational context and status of what's happening within a domain that can actually be shared and provide some information to those linkages that I talked about a little bit. and the problem also is that when you have multidate domains, you have these sort of composite data Journeys and because each data Journey each day domain can exist independently, you need some way to actually look across all the data domain processing and sort of take data Journeys and make them composite make them part of one day to journey is embedded in another or is related to another so and why do you need to do this? Well, because

In in data Mash or domain driven design there could be two different teams with two different tool sets and two different processing architectures. And if you're dependent upon what another team does you want to make sure that it's been updated that you trust the results you want to know if it's fresh and all those things or if there's any data quality problems. So all those things mean that you need to have sort of these composition of data Journeys across different data domains, I think is essential because you're having a very public processing step and and making that shareable and observable I think is a is really important to this architecture.

And so by the way, we have a tool that does this called our our DataOps observability tool it allows you to sort of build data Journeys and composite data Journey. So, you know, each data mesh could have its own data journey, and then you can build composite Journeys across multiple data across multiple data meshes. And so we I think that this is an essential part of success and in doing a data mesh And so let me stop here and see if there's any any questions that people have.

So the first question is, how do you manage when a data domain encompasses multiple sets of microservices in the operational

00:25:00

process? and so I think To me. I think it is sort of good interfaces make good neighbors. And so if your microservices sort of a source of data for you and then that data changes and I think from me I think the the idea of testing or in production testing of each one of those data domains and so it actually goes back to observability and data testing to make sure that the inputs are good as well as the processing steps are good.

And so yeah, and so that was one question from Krishna.

All right, let me keep going. So so let's talk about this other related idea about a data product. And so again from a first principles approach. I'm going to talk about the people and attack and and sort of the ideas of how DataOps intersects with that. And and so if we go to the next question, so

A lot of times at one time about five years ago. I talked to a large company's data team and I said well what defines success? And they said well what defines success is we've completed all the tasks on the project plan. And then ask them the question if is what you've built useful and they couldn't answer that question because success was sort of walking through each task in a very complicated project plan. And so In some ways data products are the opposite of that. So instead of focusing on the tasks and a project plan your focused on the flow of change to your customer. So you're always focused on getting something to your customer.

Learning iterating again iterating again and so in some ways of projects have a beginning middle and end the products don't they're a continuous stream of value to your customer and that continuous stream of value could mean you've integrated more data or change to that board or change to data representation. And so from my mind, it's really about thinking this project to product thinking is really the core of what data data products mean and so iterative development and focused on the flow of value to your customers.

And so another way to think about this is is that you know, you've got a project which has got a project plan and then you sit in front of your Gant chart and you say okay, you know, this person's gonna work on this borrow. This person's gonna do another and then you know, the idea is that people are perfectly fungible right? You can pull them from one domain to another And a data Engineers data engineer. So well, he's gonna work on on Tuesday. He's gonna work on the e-commerce website and on Friday, he's gonna work on financial data and then the other cases think of it this way in a product more work is brought to people so they know the product they know the customer.

and so that idea is that instead of people get to know the team that they work with. And so by having products with people assigned to them think of it in this way instead of having people assigned to a project there's teams and the products are brought to them work is brought to the people.

And then another case and think of it this way that a project has a lot of and a lot of waste in data and analytics is done from waiting or wait time or context switching. And so for instance in a typical waterfall process you go from design to defining to development to QA to release and there's a lot of waiting in between right to to triage rugs or to have QA and and that waiting also means context switching because people don't sit on their hands right at develop our works on a different project. And so those perfectly fun to fungible people are switching context and moving from one area to the other kind of working their tickets and not really having any context for what they work and again and the idea of a product methodology they get to know the product they get to know the customer and they're always kind of tweaking and innovating.

And so the idea is small features deliver quickly in order to get feedback from the business. And this I think is a really essential part of this idea of working from projects to products. And then lastly I think you know from a sort of big big company standpoint, right? I think a lot of companies will have projects and they'll have annual funding funding for we're doing 15 projects this year and we've divided our our budget up and these are the projects we're doing Right, and I think the timeline is really in a more project or product orientation. There's more

00:30:00

frequent checkpoints and there's more cases to reallocate funding from projects to projects because data products the the a lot of times they increase and work and they decrease in work. And so and also the timeline, you know with a project it's got to start an end and then you go into the next thing with products. It's ongoing and it's managed through its life of the product.

And then in data projects, it's really role-based. I work on this role and I work on every task in the company doing you know, I'm a I'm a SQL design or I'm a schema designer. And I think what's nice about the sort of product approach is that people are working in smaller more cross-functional teams, and they get to learn more and that's actually I think a positive thing for a lot of data people and I think just wrong measurement standpoint, right? What is successful success as I completed my task on time on budget.

and did it provide value, but maybe maybe not. But I think the data products in some ways is about how much value is the customer gotten value on out of it. Are they happy are they utilizing? It? Hasn't had an Roi. And so it's a very different sort of customer Centric perspective on data projects where getting value to the customer is what matters?

And so I think with hard part in data and analytics is a lot of us sort of work in these waterfalls ways right where we've got some data sources and then we do sort of some modeling of the data. We find out our business rules. We develop some stuff we tested then we deliver it and this sort of labor intensive more monolithic more waterfall way is is pretty typical and in the data and analytics world. And so I think some of the ideas that have come from software and of course from lean manufacturing are about applying Automation and observability and testing and also to think about your work in delivering small values. So what happens is you instead of working for four months on one thing you break it into smaller pieces and you deliver value and you learn and you know, my favorite example is you you talk to your customer they ask To 10 things and you know in three months later you give them the 10 things and by the when they look at it, they say thanks, you know, these four are great. But you know these six we don't really need and guess what now? There's three more that we need.

And so that's where the waste comes from a lot of data analyst teams. They're not lazy. It's just they're it's really hard to find Value. For your customer and and spending many months thinking about it just doesn't work and to the extent that you can deliver an incremental way through Automation and observability and testing. I think it that helps.

And so another way to look at this is that data products? They're never done right? You're having small incremental releases. It increases the learning of your team. Um, your team gets better idea of what your customer wants and your customer has more opportunities to help you learn and understand what they're doing. And so this focusing on cycle time and focusing on the flow of value is I think very important and that's what this is about. It's kind of like Max it's kind of think of it as maximizing flow and I think of it as the flow of value from the people who are doing the work and data analytics to the customer.

That flow is I'm taking the ideas from your data scientists or your data Engineers had getting them into production. And that's the number one goal. It's sort of finding value. Creating flow and and so like what is the flow? Well, maybe that's a new data set. or maybe it's some improved model or visualization or you know data table.

Maybe it's some new code or configuration. And so we're always by trying to improve the way at which you get new code configuration data sets to your customers and and the measurement gets to be different here, right because it's about cycle time like how fast can you deploy? It's also about your success rate like how efficient are you in getting things? And it's and how much work has done. How much are you getting done?

And so think of it in terms of this value stream and continuous delivery over that value stream and thinking of what you do as a product and you're always improving the product. And those ideas of I'm working in a value stream and trying to identify. How hard it is to deploy? All the steps. You need to deploy.

00:35:00

And automating that work. And so it really becomes if anything, I think the biggest thing in data products is focusing on the flow of value and it's not the data flow. It's not how fast or it's not streaming or batch. It's the flow of the work your team does and that's primarily the work your team does is code in some ways code or configuration. So it's really about maximizing the the value that you're team creates your data scientists and data Engineers as expressed in code or configuration and and put into production with low errors.

And so let's talk about that and we've talked for a long time about the idea of of DataOps and and in some ways, you know, I've been talking about bringing some software ideas to data and analytics for a decade. And you know and data off. In a lot of ways, it's really about trying to enable this idea of Rapid experimentation and Innovation.

So it's really about trying to enable that flow of work from development into production. And it's also about running production with very low errors through observability. And it's also about trying to automate and collaborate and then it's about measurement and observability and you know Gartner are not two reports has talked about by applying these principles, which I think are very much align and related to data mesh and data products. And and I think there's sort of consumed conceptual overlap in this and between what a data product is and what a data match is, but you don't have to use a data match to do DataOps nor do you have to do data products, but this idea of experimentation and error rates and automation.

And you know, what's what's really interesting about? A lot of these work that they that happens is that this pushing from development to production. So you have a production system here on top with a lot of problems in production, but it's also hard to actually push things into production to go from an idea to development into production. That's also hard and maximizing that flow is really about the data product idea and then trying to get a team to focus on these fixing errors and implementing new features in a data domain or a data match is part of the data mesh idea.

And so lastly, I think what we've seen is that by focusing on things like product errors driving your production errors increasing the flow of work. You can actually have a huge increase in productivity. And so we've got two products that do this. We have an observability product that can observe your data mesh and can observe your data products. And we have an automation product that can help you deploy help help you iterate and help you test your data products in your data mesh.

And so what's the conclusion here? So I actually think the data mesh and data products. So there are very exciting new paradigm and it aims to really reduce complexity in your the team and I'm not sure you can reduce the complexity in your data, but you can reduce the complexity and how your team works together and it's also about maximizing the flow of work from your team to your customer.

And there's a lot of talk about data mesh and data products. I don't think data mesh or data products are data virtualization or Federation. I think those are tangential Technologies and I think a data product is not as different than data as a product and I think data as a product is on sort of a way to monetize data, and I think it has a different set of ideas.

And I think the sort of data offs idea of an observed observability and automation are sort of equally important to this. So if you want to learn more, we've written some books on DataOps and data cookbook. And if you want to learn about our software, give us a call. And so that's it in terms of my discussion for today. Let me go and see if there's any comments or questions I can answer.

But this was valuable to you.

Let me see. So there's a question here. so yeah, feel free to ask some questions, but like I think

I think Christian has a comment on facts and dimensions. Are there any other questions I can answer before we finish up today?

All right. Well, thank you for your time. Like I said, I will be posting this video and the slides and if you want to reach out to me or my company just go to DataKitchen.io. And of course, we're happy to give you a demo or software. Oh wait. There's one more question. Thanks for sharing.