AI 101 For Data Quality And Governance Professionals

00:05:43 Welcome, and what this session covers

Chris Bergh: All right, welcome everybody. My name is Chris Bergh. And I want to introduce you to Chip Bloche, our data quality expert, and he's going to give you a webinar on AI 101, data quality and governance professionals. Take it away, Chip.

Chip Bloche: Hey everybody, glad to have you join us here and I'm vice president of data engineering and quality solutions at DataKitchen. I'm the originator of TestGen, our automated data quality testing tool for the AI-enabled world. Happy you could join. And this friendly little title screen here is AI generated. And I like it because it shows this happy picture of all these happy people working together with artificial intelligence.

00:07:36 What's really on our minds about AI

Chip Bloche: But that's not the way it is in the real world. What's really on our minds. I can tell you that my work has changed completely in the last year. And it's a challenge, right, for all of us to deal an exciting challenge for all of us to deal with all of these changes. We're all concerned about you know, are there going to be job cuts? Are there going to be new jobs? AI is your new assistant, your new coworker, your new boss. It's going to help you. It's going to hurt you. It's your therapist. It's making you dumber. It's saving the economy. It's going to destroy civilization. There's a lot to metabolize here both for us as individuals and for us in our companies. And as you know for those of you who are data quality and data governance professionals that's really it's really critical that we try and understand what kinds of changes AI is bringing our way.

00:08:52 Data quality in the AI wave, and the DGIQ workshop

Chip Bloche: We're like it or not the AI wave is here and it means lots of change and it means greater risk and data quality is in the thick of it and is really going to be essential. I think what we're discovering is data quality is going to be essential to the success of AI in the enterprise and cannot be ignored. I just want to point out that this is a preview of a workshop that I'll be doing at DGIQ in Providence next month with Danette McGilvray who is the author of Executing Data Quality Projects and Ten Steps to Quality Data and Trusted Information and Danette has just been an amazing collaborator and I just want to give her a shout out. As a teacher, as an advocate for data quality, an advocate for data quality people in organizations and she's she has taught me a lot and I'm grateful for that. So, I'm really looking forward to doing this presentation with her.

00:10:11 More pipelines, more people, more speed

Chip Bloche: This is going to be a what you're going to see is a is a little piece of it but it's a it's a much longer tutorial. So if you happen to be going to DGIQ, please join us or say hi to me. I'd love to hear from you. So what's happening here with AI? It's it clearly there are going to be more data pipelines. There are going to be more people with their hands in the mix and it's all going to come at a faster pace if that's possible. Right? Because we're in a in a field where the pace is always fast, but I think we ain't seen nothing yet. Let's talk about the different kinds of pipelines that you might see in as your company moves forward with AI. First of all, we have conventional pipelines that are now more and more becoming AI authored and this has been happening for some time. It's it means faster development.

00:11:16 Shadow pipelines and interactive pipelines

Chip Bloche: It means more deployments of pipelines and it means that pipelines are all over the enterprise in places that you might not expect because more people are empowered to automate more things. Some of these are governed in traditional ways and some of these are shadow pipelines that are happening within departments or areas of your company where they're where pipelines haven't been produced before. We're also seeing more and more interactive pipelines.

Chip Bloche: And by these I'm the AI in real time. And these are questions that people ask questions of their data. They might be ad hoc questions. They're unpredictable questions and they're as part of a chat and they're often unpredictable results. It's very challenging to govern and assess these kinds of pipelines because the process itself is transient and the results can be untraceable. So what do we do in this situation where we can't possibly manage and verify every single pipeline that is now proliferating in our organization.

00:12:53 Govern the process, because manual testing can't keep up

Chip Bloche: You have to think of it in terms of risk management and you have to think of it in terms of risk mitigation versus controlling every little detail. It's just it's just not possible anymore. You have to focus on governing the process in order to improve the product. And I just want to say that's why you know DataKitchen built TestGen the whole purpose of TestGen is not because we don't like manual testing. We don't like people-powered testing development. It's just that manual testing can't keep up. So, we try and take the easy testing off of people's plates and automate that process for people. So, all right, I've said we can't control every variable, but we can manage and mitigate risk. One of the interesting things here in order to do that is to try and understand the kinds of convergence and even confusion that AI is bringing to companies. It's blurring boundaries and I think of the functional boundaries between what is data, what is code and what is documentation.

00:14:20 AI puts everyone in each other's swim lanes

Chip Bloche: Also the workflow boundaries. What's a data source versus what's a process versus what's a deliverable and also the role boundaries of people in our organizations. What who is doing data engineering? Who is doing analytics and who is doing data quality and governance? Our goal here in data quality is to try and disentangle and trace through these tangled up converging roles and identify checkpoints where we can assess data quality, we can identify root causes, and we can continue to monitor as a as we go along. And one of the one of the fascinating things, I don't know if you're seeing this, I'm seeing a lot of this in organizations. We're basically all in the same soup here. We're all in each other's swim lanes. AI has made everyone an analyst. You don't have to write SQL anymore in order to pull data out of your enterprise systems. It's made everyone an automator because with AI you're empowered to create repeatable processes and it's also made everyone interestingly it's made everyone a kind of a governance person because it's it makes everyone a curator of quality and good documentation and effective governance.

00:15:58 Capability versus incentive: the creator isn't the validator

Chip Bloche: If you're working with AI, you're constantly asking yourself, well, how can I improve my prompts? How can I improve my outcomes? And you end up spending a lot of time on tasks that are really governance tasks. The problem here, I think, is that everyone starts to think that they're they can do each other's job. And I think we can mix up in that case capabilities that AI gives us versus incentives because if someone is just because someone has got new capabilities, it doesn't mean that they're the right people to do particular kinds of tasks. That is the creator isn't necessarily the right person to be the validator. And that's actually true inside of an AI process as well as inside of an organization. The opportunity here is that there's new understanding of what other people what challenges other people are dealing with. And there's really an opportunity for new insight and improved collaboration. And if I if we can do a better job through this convergence of marrying tech potential with business value then it's all for the good and I what I really believe is that your role as a as a data person data quality and data governance really can make a difference in shaping this pathway and bringing people together for the good of the organization.

00:17:49 The SIPOC model, and how AI obscures the process

Chip Bloche: So, watch it happen. Let's see what happens. What we're seeing is smaller cross-functional teams, forward-facing engineers in your in your organization and interestingly enough the empowerment of the generalist as opposed to someone who's just specializing in pure tech. So what's a traditional data pipeline right we've all seen this where this is the SIPOC model and kind of a common representation we have a supplier who provides the information we have inputs that go into a process and the process produces outputs and the outputs are then read and interpreted by a customer. Everyone knows this. We all have this kind of thing in our organization. How does how does this survive AI? Well, AI obscures the process. First of all, the process becomes this big black box. And people will tell you things like and you hear this all the time, it's too complex. It's not consistent. You never know what it's going to do.

00:19:04 The black box that wants all your inputs

Chip Bloche: Nobody really knows how it works or what's happening inside of an AI model. Not even the people who developed it can understand how an AI model maps information and certainly you'll never understand it. We have to shine a light on this. Another thing that I think is happening here with this process is that the process is really taking over. Everything is another input to this very demanding black box. It want it wants your data but it also wants your documentation. It also wants your best practices. It will pull all of that into its process and you don't know exactly how it's using it. And it will come up with these results that you know not only does it produce outputs but it also takes over right it interprets your outputs for you. It can write them up. It can make them look great in the slide deck. This process this black box is doing everything. And it's that's why it's really important to try and understand what is going on inside of there.

00:20:30 The VISTA assessment

Chip Bloche: Danette and I have come up with something which we call a VISTA assessment. And the idea here is to identify points inside an AI process where you can assess data quality and assess process quality. And we'll be sharing that at DGIQ. I don't think I'm gonna have time to give you a sneak peek here. I was hoping to. We're still working on it anyway. So, so give us a holler and if you're interested, we'll be happy to send it out to you. It's basically something that we're going to just give out. And the idea is to help you find where the bodies are buried. I just wanted to mention it. We're not we may have to split this into two sessions here to even talk about the things that I wanted to talk about. So I know we're not going to get to this in this session today, but let's talk about what an agentic process looks like.

00:21:33 The changing definition of a data pipeline

Chip Bloche: And I really want to focus on some of the basics here. We've all been exposed to pieces of this. But, I think as we look at it, it will help us understand as we go over some of these details, why does AI act the way it does? What why are, you know, what are the vulnerabilities to error that we face in working with AI processes and where can problems arise? That's really the way to think about this process as we go forward. One thing I can tell you is that the very definition of a data pipeline has changed. Right? So we have both predefined processes and we have inferential processes as we've said we've said non-traditional data is a critical input and interestingly enough the interpretation of data is actually a process or can be a process within the pipeline itself. So there a lot of changes here. Let's take a look at a what an agentic process looks like.

00:22:51 Anatomy of an agentic process: orchestrator, agents, LLMs, context

Chip Bloche: This is a real simple example of one. It's just basically an analyst working with an AI system. It might be something like Databricks Genie or it might be Claude Code or OpenAI Codex and the analyst has a prompt. They're asking a business question and the AI is made up of you know on the on the AI side the AI consists of an orchestrator a bunch of agents that the orchestrator organizes and LLMs that are the large language models that the agents use to do their work and there's also context Right? That the context is the information about the organization. It might be documents, might be enterprise data or it might be enterprise applications that are available to the LLM and the agents. And the result can be an analysis. Maybe that's run one. Or it can be reusable so that the analyst can ask the question again and again and get a reusable you know and get a useful response.

00:24:11 Agents turn talk into action

Chip Bloche: Let's look at real world at these real world processes. Let's start with the agent itself and the LLM. So the agent as a whole that you work with may consist of an orchestrator that supervises a number of agents and the agent will make use of the LLM but also the agent is able to do things with the answers that the LLM provides. So the purpose of the agent in all of this is to turn talk into action. Orchestrators actually started as a separate project separate product and in your organization you could be using an orchestrator that's independent of agents. What we're often seeing as time goes on is that these agentic tools are kind of taking over. They're including orchestrators in their in their own operations and there's kind of a convergence into a kind of a harness to manage multiple agents on a task. What we're going to be talking about today is basically how all three work in concert.

00:25:34 What an LLM does: arranging tokens to complete patterns

Chip Bloche: But first, we want to talk about how each piece works or at least the agent and the LLM work together and what they provide. So, what's an LLM? Well, everyone knows what an LLM is because we're all working with these chatbots now for a few years. We all have a basic working idea, but it's useful to take a step back here and try and understand just what it is that it's doing under the hood. And basically, you can think of it, and this comes from information theory, it's basically generating information, right? It's matching patterns based on the prompt or the information that you provide it and it's generating new variants. It's arranging tokens and tokens are these units of data. It could be a word fragment, not even a complete word. It can be pixels for image generators. It can be sound waves for music generators. It can be little blocks of video.

00:26:47 Emergent capabilities and reverse invention

Chip Bloche: It's arranging these tokens to complete a pattern. And as the model scales, as the information that is working with becomes vaster and vaster. There these capabilities that start to emerge and it seems to infer, it seems to reason, it almost seems to be human and you can think you know it all comes from that pattern completion and you can you when you think of an LLM you think of it as the inference engine of the agentic model process. I find this interesting because an AI is really you know this emergence concept is an information theory idea and an and AI is it wasn't so much an innovation as it was in some ways a phenomenon where these capabilities started to just appear like sudden and sometimes surprising the people who developed them at some point it developed you know AIs couldn't do math very well and suddenly after they continued to scale they were able to do math and so it's a it's a we're kind of here with this reverse invention process where the tool comes into existence and our role is to figure out what to do with it.

00:28:25 Managing an LLM like an employee

Chip Bloche: It wasn't a tool invented to solve a problem. It was something that does these tasks really well and we have to figure out what does it do well, what does it do poorly, how do you put it to use. I kind of think of it like managing an employee. You want to understand the skills and talents of the person in order to put them into the best position to succeed. It's the same thing that you do as you as you see the this these emerging skills of the agent which are continuing to emerge right with each new version they can do more and more things they're more and more capable but only in you know in certain ways they're not capable at all as I said what is the agent really doing every time for to as all of these skills emerge It's completing patterns, right? It's using associations of tokens in its in its own memory to create assertions that it returns to you.

00:29:40 Trained to be plausible, not accurate: the concept space

Chip Bloche: And the interesting thing is it's as we've been discovering it's trained to be plausible. It's not trained to be accurate. There's nothing that intrinsically calculates accuracy in the within the LLM's process. It's calculating the likelihood that tokens go together and that therefore words go together, but it's not calculating the likelihood that a particular assertion that it makes is actually right. This memory area or this library I like to call it the concept space. You hear a lot of different terms for it. The thing about it to know is that you can't change it, but you can influence how the model navigates through it and comes up with the sets of associations that are most beneficial to you for the purpose that you're that you have. And we'll talk more about that, but that's really one of the levers that you have over how an agent works. The you're controlling how it navigates through its concept space.

00:31:00 Deterministic versus probabilistic processes

Chip Bloche: Let's see here. Looking at time, we can go I'll go past this one. The ways an LLM completes a pattern. Again coming from information theory, it's actually surprisingly simple what it's doing. It these are four tricks that it that it uses to create its response. It's important to know that we have deterministic processes, right? And we have probabilistic processes. Deterministic processes are the you know if you the traditional processes in let me go past this the traditional processes in programming they're predictable clear traceable cause and effect in instructions that are very explicit and predictable consistent outcomes. The superpower of an LLM is that it's got an element of randomness, right? It's not always going to regurgitate the most likely pattern. People sometimes say, well, why, you know, if I have a task to do and I always want to come out want it to come out the same, why can't I have an LLM do it without randomness consistently?

00:32:23 Why LLM randomness is useful, and where prompts give you leverage

Chip Bloche: So, so it I have a predictable result. But what happens there is you lose the real advantage of an LLM, which is that it can explore, it can look at different possibilities. It can try different things and find what works best to solve the problem that it's been given. And it can also interpret ambiguity both in your questions and in its possible responses. And that comes about because of the randomness. The what it what it means is that the you can influence the result. You can't completely predict the result but you can influence the result by improving your prompts. That is the more accurate the associations you can get from it and the more grounded the your inputs are to reality, the more grounded the responses are going to be to reality. So with the prompt, you gain leverage over the AI's process. One thing that's important to remember though is that LLMs are not human and I think we often anthropomorphize these things, but those mechanisms that they use are all working to achieve a deterministic outcome.

00:33:54 LLMs maximize a score, and the score can reward plausibility

Chip Bloche: They're trying to maximize a score or sometimes they call it a reward and it's a quantitative measure of success in responding to a prompt or solving a problem. This is this is all the way it's trained in its post-training using a score and that's really what it's after. It's trying to get the best possible score and sometimes that score is really obvious like did something work or did it not work? Is math correct? And sometimes it can be more subjective and harder to pin down, you know, like is a is a poem pretty? That kind of it turns out those kinds of questions, the more subjective it gets, the harder it is to get good responses out of an LLM or ultimately an agent. And sometimes a score in the training process for the LLM didn't capture the right goal. And this is something to be aware of. It may capture a plausible may get a high score because the response was plausible, not necessarily because it was right.

00:35:07 Deterministic levers, and what an agent is

Chip Bloche: So you want to look for your deterministic levers here by clearly defining the goals that you want an LLM to reach for and very clearly providing a way to evaluate the result. And one of the things I don't like about anthropomorphizing these models is that we kind of give up our responsibility for the process, right? We kind of forget that we have a lot of leverage over the results. All right. How do agents come into play here? So, an agent is simply you can think of as an LLM that's been empowered to do stuff. It has tools at its command. And the LLM may create code, an agent can execute the code. An agent, an LLM may create a command an agent can execute that command in your operating system. So, it's sort of like a an action printer for an LLM. And ultimately, what this turns into is a software system that can not just come up with a goal, but can also perform it.

00:36:27 The agent control loop: plan, execute, verify, revise

Chip Bloche: And do tasks for you. Whether it's simple things like organizing files, it could be writing and executing programs to accomplish tasks that you want, answer emails, make dinner reservations, you name it. But what does it all come down to? What an agent is doing is using a control loop. And this is the way this is the way agents work. It's just a simple loop that repeats itself over and over again. So you an agent will plan, it will execute a command or the text, it will verify the result and then it will try it again. Right? So these are within an agent. These are typically very short iterations. They're easily verified and it can use tools to improve the quality of the result. The kind of the perfect application for the control loop is programming. It turns out that programming is kind of the dream use case for LLMs because programming is something where it's easily readable, it's easy to execute, it's verifiable, and it's reversible.

00:38:01 Why coding suits agents, and what a prompt is

Chip Bloche: So all particularly that verifiable because remember we were saying that there are some things that are easier to verify, some things that are harder to verify. Code is whether code executes or not. That's really easy to verify and an agent can keep doing loop after loop after loop until it gets it right. But remember if the harder a task is to verify objectively the less you know the more difficult it is going to be to get correct answers from agents. So what kinds of tools do we have here to work with to improve the effectiveness of agentic processes we have prompts and we have context and that's what I want to talk about now there the these are actually factors that you have control over in order to get better results. Now, what is a prompt and what role does a prompt play? We talked about this a little bit that a prompt acts as input into an agentic process. Does that mean it's data?

00:39:15 Is a prompt data, code, or documentation?

Chip Bloche: A prompt acts as instructions for how to do a process? Does that mean it's code? And a prompt can be an explanation for what the process is all about. It all can go into a to the LLM. Does that mean it's documentation? These are challenges for governance. Who's going to govern this? Whose role is it to make sure that these things are correct? Whose role is it to verify them or keep them up to date? It can become very difficult as the kind of the fog of AI descends over all of these all of these previously clearly distinct roles. So the I want to talk a little bit about how a prompt steers you through the concept space. You know depending on what you ask it, it's going to go to a particular place where it finds associations that relate to them. So, if you're talking about poetry, it might go to one spot where there where there are language associations.

00:40:25 How prompts steer the model through its concept space

Chip Bloche: If you ask it to draft a technical specification, it might go to a spot that's not too far away from poetry because it's trying to use language to express something. If you're if you're talking about translation from you ask it to translate from French, it might go to a spot that's maybe not too far from where it goes if you ask it to translate from COBOL to Python. There are things that relate in these kinds of tasks and they're things that are very different. And by your prompt you're deciding where or not deciding but you're helping the LLM go to particular places where it can answer your questions. It actually all becomes very interesting because prompts can steer you in surprising ways. For example, I think I never realized that you can actually tell an LLM to think in another language. So for example, if you say think in Japanese, there might be information that is more available in its training data that's in Japanese.

00:41:40 Think in Japanese: prompts that steer in surprising ways

Chip Bloche: So, if you're interested in the Toyota quality system or continuous improvement or Japanese cooking, you might tell the prompt to think in Japanese and you unless you know Japanese, you want to give it an answer, give you an answer in English, but that could improve things. On the other hand, there are other cases where there's less information in Japanese and for the most part the strongest set of information in the concept space because it comes from the internet is in English and that actually can be an issue if you are Japanese. There are a whole bunch of different ways here that surprisingly prompts can steer. I won't go into all of them, but I can say you know you have to watch out for bias. So if you say for instance act as an older programmer, this one is near to my heart. They might say evaluate this SQL code as an experienced programmer and that could be very positive because the it will go to a place where there is a there all kinds of useful tricks of the trade in its concept space but it could also be very negative because there may be a lot of information on the internet or conversations let's say on Reddit where people complain about older programmers and say they're inflexible and they're

00:43:22 Subgroup bias and global bias

Chip Bloche: Rigid and they're tied to old technology and maybe it will it will give a response that could be condescending or limit your options. We call this subgroup bias. These are associations that are present in the internet. They're really there, but the model turns them into a universal rule and ends up amplifying that bias and sure enough it creates a self-fulfilling prophecy. Another example, you're you know this is one that I find interesting. You're a middle-aged Caucasian, highly educated English-speaking male with centrist Western viewpoints. Well, it turns out that one is global bias because it's sort of it actually has zero effect on accuracy because it's the default perspective found in the training data. It's the ocean we're swimming in. And it's one reason why different people in different cultures in different countries are fighting to have their own versions of LLMs. You know, you'll hear people talk about sovereign AI because their those versions of LLMs might reflect more effectively their cultures.

00:44:41 What makes the best prompt for data processing, and why context

Chip Bloche: What makes the best prompt for data processing tasks? Well, I can I can tell you based on hard knowledge, the clearest information that you can give is the best. So, clear goals, detailed specifications, deliberate calibrated language, very clear deliverables with clear constraints on what you want your results to look like and step-by-step instructions and as I said before, verification steps along the way. What does this sound like, by the way? What kind, you know, the writing a prompt for a data processing task? Turns out it sounds an awful lot like programming. All right, let's talk about context. Why do we need context? It's because all of that training data in the concept space that's mapped to be so useful of the magic of an LLM, it's not going to be enough for you and your company and your use. What's missing? Well, basically anything that wasn't on the internet. So, your company's policies, your boss's preference, right?

00:46:01 What the concept space is missing, and the Spirit Airlines data

Chip Bloche: Your interestingly representations of three-dimensional space that's not on the internet. And that's why it's been so much harder to train self-driving cars than it has been to train large language models because there hasn't been this proliferation of information about the three-dimensional world that is that is rich and free and available on the internet. Unwritten rules in your organization and uncaptured knowledge. There's so many things that are assumed but not said that you have to spell out to an LLM that are just not in this training data. And my favorite common sense, right? Some sometimes they're getting better and better, but a lot of times you can just be shocked at a at a response that you will get from an agent. This was really interesting. If you notice over the summer, there was an article Google bought out the business data of Spirit Airlines that when they went bankrupt for $10 million. Why would they buy the business data of a bankrupt company?

00:47:17 An LLM with no context is an intern

Chip Bloche: It's because they're trying to get it this kind of information, right, that wasn't on the internet. They want their emails. They want their Slack messages. They want the kind of inside information that shows how a typical company deals with real world problems and they want to include those as they develop their agents and models. Consequently, what's an LLM with no context? I say I say meet your new AI overlord. Basically an intern, right? This is this is someone with no experience in your real world. They're full of information, but they can't judge the relevance of that information to your real world scenario. They're very quick to agree. They don't they're not judging the likelihood of being right or wrong. As I said, it's all about plausibility. And it ends up being very difficult for them to keep priorities in perspective based on what your real needs are. And this is why AI results can often come out so badly.

00:48:36 Where your company's context lives, and why its quality matters

Chip Bloche: So where is your company's context? Context is everything that the agent needs to know that isn't in the prompt and that isn't in the concept space and this is where you're going to find it: in written documents. It's going to be in your enterprise data and it's going to be in your enterprise systems. The role of context engineering is to take all of that information and turn it into something that's useful for the agent. So context can be static, it can be slowly changing and it can be real time depending on how you provide context to your agent. It can be in the form of structured knowledge. It can be unstructured knowledge that is then structured through storage in context engineering like RAG and it or it can be instructions and governance and the quality of this data is vital to every one of these. This is this is a data quality issue here. The when people are creating context for AI systems the quality of that context is crucial and has to be managed.

00:50:12 Data health as context, and the Model Context Protocol

Chip Bloche: There are also other areas that are kind of like little cubby holes inside of the agent where context is stored and I won't go into the details here but there all kinds of little areas that you have to be on top of and in fact the health of your data can even be context and this is something that we do with TestGen where you can wire in your data health into the context that's available to the agent and it can use that information in order to help guide its decisions. And another example of context is Model Context Protocol, MCP. You've heard about this and this is actually also another set of cubby holes. Where an MCP server can include its own data retrieval, its own instructions, its own background, and its own set of actions. And this turns out to be a really great way to help give an agent clues as to the best way to do certain tasks and to create a more deterministic outcome by limiting the rules and clarifying that the instructions that the model has to work with.

00:51:42 MCP as a division of labor, and data quality dimensions for context

Chip Bloche: So what you're what you're really doing here by using some of these tools like MCP is you're creating a division of labor where if there's something that can better be done deterministically, you're doing it deterministically. And this is really important and valuable, right? Don't let the agent reinvent the wheel. If there's an opportunity to provide an agent with deterministic results, you're going to get results ultimately that are that much better. You're introducing more predictability. You're making your whole process deterministic where it matters. Go past this one. What are some of the challenges to getting context right? Well, they may sound familiar to you if you're a data quality person. Does a rule reflect what's really happening? Well, that's accuracy. Is a requirement still current. Timeliness. Does a procedure conform to guidelines? All of these things are data quality dimensions that are very relevant. Applied to context as well as other kinds of data.

00:53:00 Treat context like structured data: grandpa's roll-top desk

Chip Bloche: In fact, you can think of context like structured data and this works very well, right? Context has a schema. There's a structure that's expected and that's important. Is that is that intact? Right? Context has boundaries. Are different kinds of context clearly delineated? Context has relationships. So are the hierarchy and scope of context clearly preserved? And context can drift, right? Has the structure changed unexpectedly or have the rules not kept up with changes in the real world? And context has to be retrievable. The agent has to find the information that matters. I think of context as kind of like you know good context as kind of like grandpa's roll-top desk. And maybe in your case it's great grandpa or great great grandpa. But grandpa knew where everything was, right? He could always find his glasses. He didn't need technology. He didn't need a search engine.

00:54:06 Code repositories and Databricks as organized context

Chip Bloche: He had a rule. Everything has its place. And that's how AI works best, when everything has its place and it can find the right place for everything that it needs because otherwise it's just going to make stuff up. What's a great example of grandpa's desk? Well, a code repo, right? Code repositories organize context in a very predictable way that it follows conventions. It's all self-contained. All the information you need is within the repo itself. It's highly organized and the best part is that because repos are all over the internet, agents are very well trained on what to expect. So this is a this is kind of a great example again back to that programming use case of context that works really well. Databricks is an example. Databricks has been trying to kind of replicate that highly organized context that you might find in a coding repo and they've had a lot of success with it. And this screenshot is an application that we generated actually to manage Databricks context.

00:55:28 Context is data, code, and documentation: govern all three

Chip Bloche: But you can see how Databricks has got these little cubby holes, these clearly defined spaces where it stores information about tables and columns, instructions on how to do certain kinds of queries, particular filters and fields, highly organized context that you fill in. One challenge was with all those cubby holes, it can be hard to manage and version control it and that's why we built our own tool internally to bring that all that context together into a single place and we found that to be very helpful. Again, you know, these tools continue to improve and they're getting a lot better at doing governance tasks and it's nice to see I think Databricks is just continuing to improve what it makes available. So yes context is data, context is code, context is documentation and for good AI results you have to govern all three. I think if there's anything that you take out of this, it's the idea that data quality people have got to play a central role in curating and evaluating context and to ensure results.

00:57:05 Context is not a private workspace, and how good agents go bad

Chip Bloche: I think a lot of times you know bottom line people who are doing the development of these things can sometimes think of their the context that they're putting together as their own private workspace. And this couldn't be further from the truth. This really needs to be under the purview of data quality and data governance in order to get effective results. I think that's really the bottom the bottom line on context. There's a lot more to this. I think we're low on time. I'm not going to get started on the next section, but to give you a preview, maybe we can do another one of these, Chris. To give you a preview. The next section is how good agents go bad and we'll be talking about a lot of the failure modes of AI enabled processes and then we'll be talking about how you can use an assessment tool like VISTA to identify where those failure modes might be lurking in your processes.

00:58:24 Your turn to ask good questions, and Q&A on TestGen MCP

Chip Bloche: But for now I would say it's your turn to get started and ultimately I hope this has given you a place to get started to start asking good questions. That's really the key. As a data quality person, you may not have all of the answers. None of us do. But if you start asking good questions about all the places where information is stored and used inside of AI enabled processes. And make sure that you're a part of those discussions. You're going to be ahead of the excuse me, ahead of the game and not running behind. And there you go. So if there if there are any questions, I'd be happy to take them.

Chris Bergh: Well, Chip, thank you so much. That was a great presentation. I like your picture of Torres del Paine in Chile, that mountain range. So Chip just how do you think tools like TestGen and TestGen MCP kind of help

Chip Bloche: What I really like about MCP in general and with TestGen, we've tried very hard to achieve this goal has been to take things that could that should be deterministic and make them available as reliable, deterministic processes and information for AI tools to work with.

01:00:07 Q&A: TestGen MCP fills the test gaps from ground truth

Chip Bloche: Let AI do what AI does best, but limit that inference to areas so that you're not reinventing the wheel. You're not asking AI to reinvent the wheel each time. That's the way you're going to get the most out of AI. And a and with TestGen we've been able to use MCP to automate the process of generating and executing tests but also where AI fits in it's able to see the tests that have been automatically generated and it's able to fill in the gaps. So using our data catalog for instance, we in TestGen AI will read from our data catalog and it will identify relationships between different kinds of data points that might be in different columns or even different tables. And it can build suggestions for business rule tests that can be pretty complicated and you know amazingly good based on that ground truth information that the MCP provides. Now, if I go and just tell an AI to create a data quality test against data all by itself without any of that ground truth, it can do it, but it's going to what it's going to end up doing is writing all this SQL, trying to come up with relationships in a different way each time.

01:01:46 Q&A: MCP governs access to enterprise tools, and close

Chip Bloche: It might give you some really good answers. It might give you some bad answers. It'll probably give you both, but it's not going to be consistently good. And that's where I think MCP really helps. If you have an MCP interface to any enterprise tool, you're really better governing access to the enterprise tool. You're providing specific data that's the right data that the AI needs and you're providing specific instructions that help the AI achieve your goals for you. Oh, I can't hear you, Chris. I think you're on mute. Yep.

Chris Bergh: Wow, that was too bad. Chip, I can't I lost my window. Thank you so much. Well, thank everyone for attending. We'll send out the slides and a recording of this webinar probably today or tomorrow. And appreciate everyone showing up. And thank you, Chip, so which is a great webinar, very interesting and a great introduction to AI and LLMs and agents and context and all these buzzwords that you hear about and it's a great way to and MCP and a great way to set it up for data quality people. Thanks everyone.

Chip Bloche: Thanks everybody. Take care.