10x Your Data Engineering With AI

00:00:00 You can’t just point Claude Code at an ETL process

Chris Bergh: Data engineering ing with tools like Claude Code today. And we're the the main idea that the takeaway is you can't just point Claude Code at an ETL process and expect to be successful. There's a bunch of principles that you've got to have set up that we're going to talk about and there's three of them here. DataOps, something called FITT, which is an architectural design pattern and data testing. And those are the three sort of necessary conditions to have success while pointing Claude at your data engineering your data engineering work. And so, like I said, we'll share the slides and the transcriptions. If you've got questions, put them in the chat window. And yeah, and like I said, I'm Chris and then Eric.

Eric Estabrooks: Hey everybody. Yeah, Eric Estabrooks. I'm I run managed services over here at DataKitchen. So a lot of this stuff is yeah, it's it's good good stuff.

00:01:02 Claude Code, Genie, and other coding agents

Eric Estabrooks: So I like to hear about

Chris Bergh: Yeah, he's he's the real expert here.

Eric Estabrooks: It.

Chris Bergh: I'm just I'm just gonna I'm just going to jump in.

Eric Estabrooks: Chris is better presenting.

Chris Bergh: Yeah. So, I I think this is really about how to make yourself more productive as a data engineer, right? And and so and I'm going to use Claude Code as the example here, but it could be other coding engines. I know Eric has got a belief that and I think Claude Code is the best right now, but there are other things like Genie code within Databricks which is kind of a a skinned version of Claude Code. I know OpenAI is putting a lot of money in their their agent. So, so coding has become a really big use case for any kind of LLM applied to data. And and so what is that? Well, for those of you who don't know, you you install, you know, Claude's a large language model.

00:01:56 Why a UI-driven ETL is a problem for an LLM

Chris Bergh: A large language model, you give it text and it gives text back. And so in this case, you give it text saying, you know, I want to integrate a a I want to integrate this table. Can you write the ETL process for me? And what Claude Code does, it has a bit of software that runs on your disk and then from it it will produce code. It produces text. And so one of the most important things is if your ETL is primarily UI based, you're kind of so you've really got to have a textbased or a good API or a good textbased representation of your ETL process. And so the the challenge here is that it's like anything with an LLM, what you put in is not always great and you need to tweak it and iterate it. And so there's a process of trying to put in and work with an LLM that is a little bit more sophisticated than just your local chatbot. But that cycle of iterating and trying things is incredibly important.

00:02:55 Write the code, run the code, see what it broke

Chris Bergh: And also the cycle of how you spec things, how you break things up. There's but this idea of that Claude Code can be very effective with data engineering and there's a whole set of practices that we have one slide on that we'll talk about and Eric can do but but I think the the main idea of this presentation is is sort of pointing Claude at if your codebase I it can write code you don't know if the code's right and you don't know if it actually works and so what you want to be able to do is have Claude write code and then run that code and see if it actually is giving the correct result and see if it's actually broken anything else. And so this sort of I want to change something small and be able to see its effects on my codebase and on my data is an incredibly important concept and it's very similar to what you get when you hire an intern, your 19-year-old intern. And so, I I I like to think of LLMs as incredibly powerful interns who have no idea of the context of your organization.

00:04:10 An LLM as a powerful intern with no context

Chris Bergh: And so, how do you make an intern productive? Well, you put a system around that intern so that if they type something in and and they know immediately that they've done something wrong. And that's really what we're going to talk about today. How to it's it's how to make an LLM productive, but it's also how to make your interns productive. And so, that's the metaphor that I'm going to use. And I think, you know, what this means is that your how you build your data warehouse, your data and analytics system is is different, right? Because I think previously it's very much a craft environment where it's human operated. People are hitting buttons, there's knowledge in people's heads. You you know, you have to check things manually. And if you do that with an agent where there's knowledge in people's heads or you have to do things manually, you're going to end up where agents hallucinate, drop tables, make mistakes. And so, how do you get this sort of agent ready infrastructure?

00:05:17 AI-ready data infrastructure, not just AI-ready data

Chris Bergh: Lot of people talk about AI ready data. We're talking about AI ready data infrastructure to be able to make it work. And I think the the paradigm here is that everything should be available through an API or available through text that lives on your disk that the agent can talk to. And the second idea is that you should be able to when we say agent here I mean LLMs but LLMs can spawn multiple agents. So you should be able to try multiple things. So having isolated environments and being able to run things in those environments in a sandbox way. So, so trust and being able success with using tools like Claude Code doesn't part of it is actually the technique of prompting but a lot of it is the underlying infrastructure on how it works. Eric, do you have any comments?

Eric Estabrooks: Nope. No, you're This is This is great.

Chris Bergh: Okay.

Eric Estabrooks: Yeah.

Chris Bergh: And and so like okay so the argument is I can't just point it at my my warehouse that three that one guy knows he's built for 20 years and you know expect expect it to do great things right because that person's got all this knowledge in their head.

00:06:43 Data engineering as a process problem, in three layers

Chris Bergh: Well then how do you have a system where the knowledge doesn't live in people's heads and an agent can inspect it. An agent can evaluate it so if it tries things it can keep working. And so we've broken that up into sort of three words or three patterns. And so the main idea here is that we're treating data engineering not as a data problem but as a process problem. And so you're operating on the process. Think of a data engineering as not data engineering. It's it's coding on data. And so really what you're trying to do is think of it as a coding problem or a a how do you get your code and a step-by-step process to be able to do this. And so we've broken it up into three layers. One is we've talked a lot we have you know written books on DataOps and the thing that we're going to talk about with respect to that is how you provide environment isolation. The the second is we're going to talk about something called FITT, which is functional idempotent two-stage and tested.

00:07:48 FITT, data testing, and why isolation comes first

Chris Bergh: That's a that's another set of architectural principles and how the code operates. And then lastly, we've got a little bit of a duplicate here, but we're going to talk about data testing and how that how you evaluate it. And so all these things I think are important to be successful. And in our work where we've used Claude Code and in our own managed services we use Claude Code every day for every customer. Now luckily we've sort of built our infrastructure like this because for years we've tried to tune that we could drop in a young data engineer and have them be productive quickly. And so if you think about that metaphor, how do I drop in an intern and have them be productive? It's the same idea of how do you drop in Claude Code and make them productive. And so the first key idea is isolation. And that is you want to be able to isolate the environment that you work in. And so the first thing is what does Claude do? Well, it writes code.

00:08:49 Isolated code, isolated data, isolated compute

Chris Bergh: So you want to isolate your code. You want to have a git branch or a a a partition that you work on on a separate codebase. And then you want to have sort of a separate set of data. In tools like Snowflake, you can do zero copy clone. You don't want to sort of write things into production. You don't want to step on someone else. So, you want to have kind of an isolated data environment and an isolated compute environment that you can work on. And so, I've got separate code operating on separate data on separate compute. And so, what happens is when you work, you kind of enter this kitchen, this place that has all these characteristics, and you run this sort of trial and error loop. I'm I'm trying something. Did it work? I'm trying again. Did it work? And you're not touching production at all. In fact, you're not even touching a the development environments here are ephemeral. They're just you you turn them on.

00:09:47 Ephemeral environments, and Claude working through an API

Chris Bergh: And so you need this idea of version control, separate data environments, separate compute environments, and and for this to work. Make sense, Eric?

Eric Estabrooks: Yeah. Yeah. And that that isolation is is key. So it makes things a lot safer and easier both for just people as well as in in this case because of the presentation most importantly for the agents.

Chris Bergh: Yeah. Yeah. And I think what's also working here is again the assumption that we didn't talk too much about is that

Eric Estabrooks: So

Chris Bergh: Claude is operating on an abstraction, right? An abstraction could be a bunch of text files on your disk or it could be an a rich API and one version of an API is called an MCP. We just have in our automation product just a plain old regular API which works as well as well and so the idea is that Claude is operating on these on these systems. It's writing code, running code, in a separate environment.

00:10:51 Throwing the work away without consequences

Chris Bergh: And and in a way that like if it doesn't work, it can throw it up, throw it away and start over again. And one of the if if you've done any work in coding and Claude, one of the challenges is like you get it's pretty easy to get ahead of yourself. And then you're like, wow, I've done all this stuff. I have no idea. And and what the idea here is you want to be able to throw away all your work and not have anything affected.

Eric Estabrooks: Yeah, just a a bit of a nuance just a lot of people know it, but just I like always like to stress it like the LLM, right? It's good. It's good coding companion, good like coding employee for you. Most of what we're talking about is it's building processes that run on some other substrate like some kind of, you know, dbt or our DataKitchen automation, right? Like Claude is building those things. It's judgment.

00:11:45 Claude builds artifacts that run without it

Eric Estabrooks: It's making decisions and all that, but at the end of the day, you're left with a code base that runs like independent of Claude. And that's important because you pull Claude in to help you with the development. You maybe have it in your workflow somewhere to help make decisions, but for most data things that's rare. It's like it's building artifacts that get run somewhere else. It's not Claude that's going to go build your data assets later. It's going to help you build a process that gets run. So,

Chris Bergh: Yeah. And then that process is code, right?

Eric Estabrooks: Exactly.

Chris Bergh: And so that's why this is really this notion is incredibly important. And and why is that? Well, because you're you're sometimes doing multiple things at once, right? And and in some way you can say, I want to try three parallel things. I want to try option A, option B, option C. And then I want to evaluate to see which one does.

00:12:39 Running approach A, B, and C in parallel

Chris Bergh: And so you can see here I've got my kitchen, right, which is I've using approach A, approach B, approach C on my separate environment on my separate data. This works for looking at a value multiple approaches, but it also works for just trying two different things at once. A lot of times things in Claude are long running. It takes it a while to think and you're like, "Okay, go do this. Come back in 20 minutes and then I'll go and I'm going to do this in the interim. And so the idea of that I can have multiple you know whether you have multiple agents or multiple Claude windows running in parallel is great because that way again if it doesn't work you can throw it away and start over again. If it does work you can evaluate it on other options. You can even ask Claude to evaluate other options. Say, "Well, I've tried three things. Which one should you do?" Which is another best practice of sort of treating Claude like an intern, but talking to it with sort of the Socratic method.

00:13:44 Ten sessions on a good day, one on a hard one

Chris Bergh: Asking it questions, asking it to clarify. Eric,

Eric Estabrooks: This.

Chris Bergh: You you look like you want to say

Eric Estabrooks: No, no, this is I love this diagram,

Chris Bergh: Something.

Eric Estabrooks: Right? Because this is and we do this all the time, right? We may have well, what if we do it this way or this way, right? Before it would just take days sometimes to do any one of those iterations. But now we can just in a meeting basically take on everything and have an answer by the end of the day. This isn't something you run start with on day one, right? You start on day one with one Claude session, one Claude Code session. But what you'll find is you're waiting for it to think like Chris said. And so sometimes you're like, "Okay, well I'm gonna pop up another window. I'm gonna do a user management task, right? You got to get that guy doing something." And then you may have another task going.

00:14:31 Managing Claude the way you manage employees

Eric Estabrooks: And so you you get to this state where you start to have multiple Claude Code windows a lot of times working on different tasks. But instead of you kind of having to do the work in each one, you're more like popping between them, checking up on the status of your employees and kind of giving them the direction or nudges they need to head to the right direction or check their results. And so you can evolve into this. You know, there's there's a good days where, you know, I'll have like 10 sessions like legitimately cranking away on stuff and then other days something's a little more complicated and I'm down to one, like narrowly focused on that one. So this is a you know this is what's possible and depending on how much you have to get involved with the actual task depends how many windows you're using.

Chris Bergh: Yeah. And I I think what you said there is really key.

Eric Estabrooks: So

Chris Bergh: You're you're you're managing the you're managing Claude, right?

00:15:26 Own the what, not the how

Chris Bergh: You're not quite doing the work, but like a good manager,

Eric Estabrooks: Yeah,

Chris Bergh: You want to say, "Here's you want to describe the task in a way that that is sensible to the person." and sort of give them what they have to do and not how they have to do

Eric Estabrooks: It's funny you say that because I just had a discussion this morning.

Chris Bergh: It.

Eric Estabrooks: I was supposed to so I use Claude to build Jira tickets right so I intake requests from customers I make nice tickets I send them off to the team and and that's accelerated honestly tenfold from what I used to do but what I did yesterday I did a I did a mistake I had in my head how it should be done so instead of me handing the ticket off like 18 hours ago it took me till this morning and like 30 seconds with one of my data engineers looking at what I had written, they're like, "Don't do it that way. That's dumb." And they like re redirected it.

00:16:22 The Jira ticket a senior engineer discarded in 30 seconds

Eric Estabrooks: So I I I broke my own rule, right? Like you know, I run teams of data engineers. I train people. I try to set the goal for people. I don't tell them exactly how to do it, but I tell them kind of avoid deleting production and don't check code. You know, I give them like guidelines, but then I say, "Hey, here's a goal. Don't break stuff and here's what not to break." But then you figure it out. But I broke my rule yesterday. Told them how to do it instead of just what I wanted and then how to prove and verify. Right? So 80% of what I sent over in the ticket like just got discarded in 30 seconds by like a senior engineer looking at it. Whereas if I had just written the ticket the right way the first time, you know, would have been fine. And that's he's going to use Claude to do it as well.

00:17:08 Give the goal, the guardrails, and how to verify

Eric Estabrooks: But it's sort of that same metaphor. It's like you know give it the goal, give it the guidelines, tell it how to verify its own work, but then let it deal with you know doing the implementation.

Chris Bergh: Yeah. Every junior engineer,

Eric Estabrooks: So

Chris Bergh: Every senior engineer who becomes a team leader for the first time gets all up in the implementation of their employees, right? And they hate it, right?

Eric Estabrooks: Yeah

Chris Bergh: And then they'll come to and I've seen this many times where then they'll come and they'll say, "God, he's like he's telling me what to do. He's micromanaging me." And and managers own the what, not the how, right? And so you guys, hey, what what is it? And then let let the agent or let the person and it's the same principle here. You're managing LLMs from a team point. And think of them conceptually like a like a again like an intern. And you've got to talk to the intern and coach them, right?

00:17:59 Kitchens, Databricks asset bundles, and Snowflake DCM

Chris Bergh: And help them along and check in and just you don't trust an intern to do the right thing. Like so having the verification is really important. So we've had this idea of a kitchen in our product for a dozen years. And so some of the other bigger cloud providers have caught up a bit. So Databricks in the past few years has something called declarative automation bundles which was a Databricks asset bundle. Snowflake's got a DCM project, but the idea is can I have an abstraction of my code, my tests, the data parameters, what data I'm using, the environment. Can I script that up so I can create an environment? And here in the Databricks diagram, they've got development workspace, staging, production. Think of these not as static, but as like I want to create 36 development workspaces or three or one, and everyone's creating their own development workspace. An AI kind of needs to cut and paste your work from one workplace to another. So, a kitchen is a metaphor, but it's it's been instantiated in other places.

00:19:05 Never hand-edit the output: fix the process or the data

Chris Bergh: And again, think of this as something that you script that it's it's done through code, not something that you click and do manually.

Eric Estabrooks: The a little bit back to the artifacts and a few things. Yeah. Claude's code specifically or the coding agent is building for you. You know we we never deliver data that a human has like interacted with like for us it's raw data plus the process gives you the output that goes to your customers. Customer being whoever is getting value from it. So if you don't like the final output or there's a problem with the table or even a data issue, you don't touch the the output. You you fix the process or you get new data and then you run that through again. So, it just kind of ties in with what was saying here

Chris Bergh: Yeah. Yeah. Never. I I think the thing is there's such a temptation to do things manually like go and fix a table,

00:20:00 Everything as code, because code is what agents speak

Eric Estabrooks: About

Chris Bergh: Do some tweaks, but the more everything should be as code, right? And and that that's a best principle that we've talked about when with people, but it's just it's it's essential with AI because that's what they speak. They speak code, right? So like you you can't do anything else. You can't say, "Hey, can you go eye up this report?" It just doesn't work that way, right? You could do that with a senior engineer, but you can't do that with an LLM. So everything has to be as code infrastructure and all the work that you do.

Eric Estabrooks: One one little note on that and this you figure out early on usually before it bites you too bad is when you tell like a Claude Code or any of the coding agent to do something for you. Even if you've been very clear about engineering standards, use this platform to orchestrate processes. Sometimes it's so interested in just getting the job done. It'll go do it in an ad hoc manner and then you'll realize a day or two later that all those queries that it ran, it actually just kind of ran them ad hoc and didn't put them in the process.

00:21:16 When the agent runs queries ad hoc and doesn’t save them

Eric Estabrooks: So, typically there's a bit of a learning phase. And when I say learning, I don't mean truly like model learning. It's more building up your data engineering context that lives on your machine. Making it clear to the agent that it needs to be writing code and if it's going to do something ad hoc, it's kind of communicating with you about what it's doing. So that's it hasn't happened in months and months and months, but you know, there's been a few times where like, hey, where'd that code go that we did the other day? And it's like, oh, we did it and it's done, so we didn't save it.

Chris Bergh: Yeah. Yeah. You lose track. Yeah. So, I'm going to go on to So, the first step is like,

Eric Estabrooks: Yeah.

Chris Bergh: Hey, I have to work in someplace separate that's driven by code. Because Claude this parallelism of running agents just like the parallelism of having a you know infinite interns is is part and parcel of what makes it successful and and we've there there's probably more DataOps ideas in there but that we've sort of for the purpose of this this is the biggest the biggest lift.

00:22:23 FITT part one: functional and idempotent

Chris Bergh: Now the second part is is what we call the FITT architecture and I think the the the first two are what I'm going to focus on functional and idempotent. So the idea here is that I want to try I want to give Claude commands and have them try different things, right? And I want them to retry it. So that's a really important thing. And what retry means is that's the I idempotent. I want to be able to run it over again and get the same result. And so this is really incredibly important now in the world of LLMs. And so why? Because like SQL or data things is just full of side effects. You you alter a table and it's altered and then you come back again and the altered table is already there. So you need to write things in a way that it can back out to where it was. And so there's in software there's functional idempotent languages, but you want to think of all your ETL process as chunks of functional and idempotent chunks that operate on the data and then if you run it again and run it again, you'll get the same result.

00:23:34 Two-stage: raw to final output, no intermediate layers

Chris Bergh: And so this principle I think is made is something again that we did trying to make interns successful but it becomes incredibly important because if I I want to run things and then I want to test it and so that's where the testing comes in and the two-stage comes in just from a simplicity. We we're not a big believer in having because we put everything into the process. There's no need for intermediate bronze or medallion. It's sort of take it from raw and get it to the final output. There's no intermediate layers. I mean, we can if we want to, but like it's a it's a much more simpler way to do things. And again, this is

Eric Estabrooks: Yeah, just to clarify on this, this this fits an enormous amount of use cases that people are going to have. Obviously there's you know some tons of data cases where this doesn't fit but like you know across multiple industries we've seen you know huge numbers of data sets where this is a pattern that'll work you know and the edge cases where it doesn't then that's something else but like for most stuff that most data engineers run into this is this is a pattern that's just going to make life easier.

00:24:54 Optimise for iteration speed, not server cost

Chris Bergh: Yeah. And then the idea is that we're you want to have a well-engineered system.

Eric Estabrooks: Yeah.

Chris Bergh: So one of the aspects of engineering is cost or storage, right? And so yeah, a two-stage has more processing power. It takes more takes more time. But once you but we believe the most important things that you solve and that you absolutely need

Eric Estabrooks: Heat.

Chris Bergh: To solve for things like Claude Code is building an infrastructure where you can iterate rapidly on try ideas where you can deploy to production quickly with low risk. And if you work on those two things then when you have certain small bits of data that are up updated every 30 minutes you can just patch those in. You don't have to rebuild the whole database.

Eric Estabrooks: All

Chris Bergh: And so optimize optimize for speed of

Eric Estabrooks: Right.

Chris Bergh: Iteration of your work not cost because the time on your team most of the cost in every team is the engineering time like 80 90%. If you make that better, you'll be able to always fix the processing and your CPU and your credits.

00:26:08 Iterating when the agent gets the SQL wrong

Chris Bergh: So we've written a couple of blogs on this and have a webinar that talks about it but but this is the main idea is that if you can think of there's this FI chunk a functional idempotent chunk of data where you can run it and then you can run it over and over and over again. So think of what what you're you're typing in something to the agent you say I want to take this table and combine it with this other table and join it to a third table and produce something else. And so you write that and the agent interprets your words, translates it to SQL, and screws it up royally, right? It's like, I have no idea. Well, what do you want to do? You want to be able to say, no, that was wrong. Try this. No, that was wrong. Try this. And iterate through it. And so, I found, you know, in my Claude work that I tend to do a lot of iterations.

00:27:07 Writing a spec first, and zero copy clone

Chris Bergh: You know, there's always a trade-off between building a more defined spec up front, right? That that you can work. And that's also a a best practice is saying, okay, if I'm going to do sizable work, I'm going to talk to Claude and I'm going to write the spec, have Claude write the spec, have Claude review the spec, read the spec, and then you have less cycles here. But you still need to be able to run it and evaluate it. And so being able to, you don't want to have a situation where like you run it once and then you run it again and everything's messed up. And so it what's really great about things like zero copy clone on snowflake is that well you can just throw it away and make another clone and it's zero copy and then you you they don't charge you for that. And so I think this idea of working thinking of your work in a functional way and idempotent way enables you to do this. Eric,

Eric Estabrooks: Yeah.

00:28:09 Restarting anywhere in the DAG

Eric Estabrooks: No, I I can't agree

Chris Bergh: Yeah, it's good. There there's a bunch of tricks here on how to do this with SQL because SQL is the opposite of a functional net and

Eric Estabrooks: More.

Chris Bergh: A potent language. It's like it's this it's a language that is built for side effects,

Eric Estabrooks: Right.

Chris Bergh: Right?

Eric Estabrooks: Yeah.

Chris Bergh: And so you have to think about how you write your SQL in order to make this work.

Eric Estabrooks: Yeah, Chris, the you know, and this is for you know, most data engineers, you're familiar with a DAG, right? You've got a bunch of nodes, You know, that connect to each other, right? The way we always build our stuff is that that that whole graph is you can start from anywhere but if you restart somewhere in the graph you just got to make sure everything downstream builds. So everything ends up being responsible for kind of recreating itself expecting it may have been left in an inconsistent state and having to deal with it.

00:29:04 Drop and recreate, merges, and small chunks

Eric Estabrooks: And a lot of times with data right it's just a drop and recreate the table. You know, sometimes though it's like if you're doing merges, you got to go in and delete before you then do your updates again. So, that that kind of that re-entrant idempotency bit like Chris said, you got to do it explicitly with SQL. But, you know, the nice thing is if you architect the the flow of your data in small understandable chunks, it's very easy to kind of do that. You know, I've early on, you know, I know I had some giant fact table and I was like adding all this stuff to it, but what I was doing, I kept mucking with like the main fact table is having all these intermediate stages. So, every time I needed to fix something, I'd have to go way upstream on this like massive lift to get it done. And so, little bit of refactoring, you know, and I could just start it, you know, lower down in the process.

00:30:02 Idempotent chunks in Python versus in SQL

Eric Estabrooks: So,

Chris Bergh: Yeah. Yeah. And it it does it's good stuff like in you know you can restart later and so it's easier. So these this idea of a functional idempotent chunk or is really easy to do in programming language like Python, right? It's and and so you can write it that way. But the idea is I'm working in this world in a SQL database and being able to make it happen and and it's much easier if you just do the whole thing in Python, right? You suck the data up, you do it in Python. And then you know it takes your inputs and gives you an output and if your output's not right, you do it again and again and again. And so that may be another way to implement this idea of a functional idempotent chunk. We happen to like think that the moving data from one place to the other is slow and doing it all in in in inside a database is reasonable and it's not I don't think it's a lot of code it's just a pattern that we have to have and there isn't really a functional idempotence in SQL there's some research on

00:30:59 Why two-stage instead of bronze, silver, and gold

Eric Estabrooks: Yes.

Chris Bergh: It that's been going on but that doesn't really exist yet so if you read our blog you'll you'll learn more about the some of the techniques that we use and so You know, this is where we get to the two-stage thing. And so I just think because we're process centric and we're a bit lazy, having the sort of source and bronze and silver and gold and medallion, it just doesn't make sense, right? Because your customers almost always care about the final analytic output. Like they care about the facts and dimensions 99% of the time. And yeah, sometimes they care about the raw data or sometimes they care about intermediate products. You know and but the idea is our process is almost always two-stage right it goes from the immutable raw data which doesn't change to the final analytic output and if you're patching data or fixing data it ends up being in the process not in and which means it's versioned and you understand it and so when you're manually patching data you always forget and so here again the I got into LinkedIn flame war a few weeks ago with some guy saying this is two-stage makes no sense.

00:32:23 Nightly rebuilds with 45-minute patches

Chris Bergh: It takes up too much time. And I'm like yeah do you have any regression testing? How do you tell if you're right? And he goes oh I just look at it. And so the idea here is that we're what we're trying to do is get you to a a mindset of like optimize the ability of your AI and your team to iterate safely,

Eric Estabrooks: Yeah.

Chris Bergh: Right? And do production safely. And so once you do that, then you can optimize build speed and CPU speed. That's like it's almost becomes trivial to say, oh, I want to I don't want to do this full I want to pull a few nodes out and I want to do this as a patch fill. Fantastic. Like we we did that we have a a customer we do nightly I think it's nightly rebuilds

Eric Estabrooks: Right.

Chris Bergh: And then every 45 minutes now we do a patch and like it's great.

Eric Estabrooks: Yep.

Chris Bergh: It's fast, it's easy, we don't have to do it every time, but we've optimized for iteration speed of code, not optimized for server cost.

00:33:28 Don’t over-optimise cost before you deliver value

Chris Bergh: And why again? Because 90% of your cost is

Eric Estabrooks: Yeah. And on that a lot of most you know not all and everyone's situation

Chris Bergh: People.

Eric Estabrooks: Is different but most most of like your database storage and compute costs kind of scale with like the business right so if you've got certain levels of data you're typically talking about much larger organizations and so that's why again someone may be like oh my god we're spending a $100 on snowflake this year but like you know they're spending like three million in or more in like overhead on employees and stuff like that. And you know, what are you going to do? Save $10,000, $20,000 off the 100? It's like it's not $20,000. It's it's literally not even worth somebody thinking about it until you've del started delivering things of value to the customer and then go back and look at it, right? But unless you're delivering something, don't overoptimize early. Just get it out there, worry about it later.

00:34:32 Premature optimisation, and when two stages become three

Eric Estabrooks: So because usually it's like one or two most most

Chris Bergh: Yeah.

Eric Estabrooks: Processes that get kind of out of whack like that there's usually like one or two things you got to fix later instead of optimizing everything you fix those later after you've delivered value.

Chris Bergh: Yeah. Yeah.

Eric Estabrooks: So yeah.

Chris Bergh: It's premature optimization, right? It's just a it's a wrong way to optimize.

Eric Estabrooks: Yeah.

Chris Bergh: And so and you know you could turn this into you could take the two-stage and turn it into medallion. In fact a lot of times we have immutable raw data in S3. We have a set of raw tables which are have a little bit of fixes on then a final analytic output. So actually it is three stages and sometimes our final analytic has

Eric Estabrooks: Yeah. Just sorry just sorry Chris just for that point. Yeah. Always the what we recommend for people blob storage is the immutable raw

Chris Bergh: Four

00:35:18 Immutable raw data in blob storage

Eric Estabrooks: Data. Tons of industries you actually need that data if anyone ever wants to audit or go back and look at like the source right and then it gets once it's in your database like Chris said then we maybe start to do tiny little fixups and stuff but usually it's pretty close but if I ever need to restart you're just pulling from the raw again so sorry didn't mean to

Chris Bergh: Oh, that's that's great. Yeah.

Eric Estabrooks: Interrupt

Chris Bergh: And I think the the other anti-pattern here is that what happens in a lot of teams is they have a bronze team, a silver team, and a gold team,

Eric Estabrooks: Mhm.

Chris Bergh: Right? And they're they're at odds with each other. They're all fingerpointing on who's where the problem is. Thinking of it in in this way means that you've got an overarching process. So you could have intermediate drops of bronze, silver, gold, but the the overall process is the overall process. So it's easy to find the problem and easy to find where it is where it goes.

00:36:13 The anti-pattern of a bronze team, a silver team, and a gold team

Chris Bergh: And so thinking of it as a again as an overall functional idempotent unary process is a is such a treat. And if like if if you go into a world where you're like that, it just your mental complexity goes way up. And so having it this way, it decreases your mental complexity. And which again makes it available for agents. So the last thing is like we talked about being able to do things in multiple kitchens. We talked about being able to iterate quickly in those kitchens in functional idempotent. But the last thing is you got to check to see if it's right. Right. And then there there's kind of and there's two ways, right? One is is the development right and then is it actually going to work in production right and both of those things are tests right they are data they are tests on data and so you could think of them as kind of two ways like I'm writing I'm I'm dialoguing with Claude I'm trying to do a new data transform as part of that I've got data tests that I'm writing to see if it's right and so in that little piece that DAG I'm working on a couple of nodes in the DAG.

00:37:26 Two layers of testing: development and production

Chris Bergh: I'm iterating. I'm working. Now I've got those working. I've then got to be able to say, did that break anything downstream in the DAG or in in the steps. So this there's sort of two layers to this, right? And having people eye it up, having business people look at it, you're never going to get the productivity. You're going to write you're going to write you're going to write code fast and then it's going to take three weeks for someone to eye it up for you. And that just makes no sense. And so we've written and talked a lot about the idea of test coverage of being able to test all your tables automatically being able to both test in every step of the pipeline when the raw lands when the dimensions are made when the fact tables are made when the aggregates are made every step has got tests and then also being able to run those tests sort of shifting left in the pipeline but also shifting down to run tests in in development and be able to regress do an end-to-end regression and and they both fit this sort of 1-10-100 rule.

00:38:32 Find the problem as soon as the code is generated

Chris Bergh: The sooner you find a problem, the better you're off, right? If you find it in front of your customer, you're it's bad, right? You're running. It's crazy. You lose respect. So, you want to find problems as soon as the code is generated or as soon in production as the data drops and every minute that you wait after that is is cost. And so, what that means is having hundreds or thousands of tests running. And so we've I've got a couple of blogs and discussions about what adequate test coverage means, but it doesn't mean I've got a a single is it null check. And it doesn't mean just pulling the database like for freshness volume schema that that could also be a good thing, but it's it's being able to have coverage across. And so this is really an important part and it's I I can't say it again. How if I'm going to be able to create a environment to work in an agent multiple environments I'm going to be able to have I'll work on a git branch I'm going to be able to work in a functional idempotent way I'm trying lots of things and again the evaluation of that is an intern they don't know so you have to say to your AI run these tests to make sure it's right Eric

00:39:55 When the row counts matched but the numbers didn’t

Eric Estabrooks: Yeah and and sometimes I find I'll tell it what test I think it should run and then I'll ask it well what do you think you should run? However the big thing there is you can't always trust it. So good example is we deal with things that add up. So, dollars, doctor prescriptions, stuff like that, they all roll up to a number that you want to make sure the numbers match. So, early on, again, this is early on in the journey, right? Claude was, just rolling like doing row counts. Oh, the row counts match, right? But what did it do? It had changed the precision on one of the columns. So, all of a sudden, the numbers weren't matching kind of between the things I was testing. So, you know, when you ask it, you know, when you tell it what it's going to test or you ask it for suggestions, make sure you like have it kind of tell you what it's going to do.

00:40:55 Ask the agent how it plans to test

Eric Estabrooks: Even if you just told it what to do, what matters is how did it interpret it and how did it sort of save it to its memory or internalize what it's going to do. So, it's like, okay, how are you gonna do that? If it shows you, oh, I'm going to do row counts and all of a sudden you want sums on those columns, you know that you got to kind of redirect it. So, you gota got to keep an eye on that. That's a that's a gotcha that you learn sometimes after something slips out. So think of it early. So

Chris Bergh: I guess we think of it as as this is you want to have you can write 80% of the tests with AI but 20% you really kind of have to get some business people involved because there's knobs and numbers that are just they're not anywhere for you to get and so our belief is that tests are incredibly

Eric Estabrooks: Yeah.

Chris Bergh: Important resource right? And you're writing hundreds or thousands of tests.

00:41:50 TestGen writes 80% of the tests, business people supply 20%

Chris Bergh: So being able to get that 80% done very quickly and then being able to help you write those 20%. So what we've done is we've got this tool TestGen which is free and open source that generates thousands of data quality tests for you. And so it does it in a deterministic way. I mean it builds it learns your data. It builds a semantic data model, but it also since we've been doing this long so long, we know what tests matter. And so it does this data testing, but it also builds puts them in a database for you to manage. And then you can give them to your business customer, to your you can give them to your steward to be able to say, should this be 20% or should this be 30%. Is this an error or is this just a log? And so you can kind of manage your corpus of data quality tests with your business partners which I think is incredibly important. And then the other part we just released an MCP interface to TestGen.

00:42:45 The MCP interface, and getting the last 20%

Chris Bergh: And so this gets to the you know TestGen's goal is to get you 80% of the tests get you complete test coverage in and out across all your tables. It doesn't so how do you get that extra 20%. Well, one is we've got a nice UI to do it, but the other is like I I you want to be able to have like Eric said, I want to have Claude recommend some tests and add them in. And so our new MCP enables you to do that. Looks at tests, looks at test results, can evaluate them, and can write some custom tests for you. And so, I think this is a really interesting feature. We've got our first version of it. We're going to have a second version of it coming out here in a in a month or two. But this is the you know, the the the the idea of what make agents successful. I'm working in separate environments, DataOps. I'm working in a functional idempotent so I can idempotent way so I can iterate quickly in that environment.

00:43:41 The order to adopt all this in

Chris Bergh: And then lastly, I can tell if things work, and that's testing and test coverage. So, it's all about test coverage and being able to help your AI write tests by looking at the body of tests, looking at the test results, and make some recommendations. I think is it's a fantastic thing to do. And so, we're just released this and and we're learning more about how to do this. But I but hopefully people can try it out. It's available in the open source product. And so lastly, kind of finishing up, we're getting to the end. How do you do how do you do this? Well, I think this is this is in a lot of ways you start with testing. So to build test coverage and then take each piece of your ETL and try to find where you can make it functional and idempotent. And then third is try to work on environments being able to stand up kitchens and then introduce Claude Code. Like I think if you miss any of these steps, it's going to be harder, right?

00:44:39 Eric disagrees: put Claude Code first

Chris Bergh: You're going to run into more weird bugs and weird errors that you didn't see. Make sense, Eric?

Eric Estabrooks: This is the one where I have a slight disagreement. I've I've sequenced Claude at the beginning of at the beginning of everything

Chris Bergh: Okay.

Eric Estabrooks: Now. However, like I, you know, I've talked about some of the gotchas and stuff like that, like Claude Code and the other agent coding agents, they're not magic tools that you can just unleash on your data set and have it fix everything magically. They work best when you understand what's going on or you know how to explore to gain that understanding. Right? That at some level these coding agents they're only as going to be as good as the person who's operating it or as good as the person operating it plus context that maybe a more experienced or senior person has given it. So you know the way I've kind of converted our teams and some of our customers over is you know I spent a lot of time building the foundational context layer that now gets deployed with the repos.

00:45:49 A shared context layer, and when to test first instead

Eric Estabrooks: We kind of share it. It's constantly evolving. I just had a half a dozen merge requests from one of my engineers this morning on how they've improved the skills, right? That's because, you know, I I've got experience. I kind of know where the gotchas are. So, I I I put that first. So, it it a little bit depends on, you know, put Claude Code first if you feel confident and you kind of know how to explore or understand your codebase. If you don't, then yeah, co test first because test first keep it from getting to your customers. That's that's important. It's better to be, you know, you never want to be wrong with the data you release. So, but yeah. So, I'm gonna I'm gonna push for Claude Code first.

Chris Bergh: Yeah.

Eric Estabrooks: No.

Chris Bergh: Yeah. Yeah. Yeah. Okay, that makes sense. And so there's a whole discipline around using Claude Code,

00:46:29 The discipline: context files, plans, sessions, tests

Eric Estabrooks: So,

Chris Bergh: Right? Which is how to manage context, which are markdown files or descriptions of your tables and work. How about how to when to just write a when to just type something in or when to have it write create a plan and evaluate a plan. You know, how to make sure you've got tests for every change. How to manage sessions and backup, how to manage multiple sessions. And so these are, I think emerging. I don't know, Eric, if you want to talk a little bit about this. We've got, you know, we got a few more minutes here,

Eric Estabrooks: Yeah. This this this could be another session.

Chris Bergh: But

Eric Estabrooks: This is where though we're you know we're the data people but you know using this stuff. So a lot of people have used like the online chat windows like a ChatGPT or Anthropic's Claude or Gemini or whatever the and Chris said it early on the Claude Code and Cursor and Cortex, you know, the the the coding agents themselves that live on your machine can access your tools, can connect to databases, hook it up to Gmail, like they they change the game so dramatically.

00:47:55 Coding agents versus a chat window

Eric Estabrooks: It's It's almost like, you know, the the ChatGPT things are like they look like, you know, kindergarten. This is like, you know,

Chris Bergh: Professional.

Eric Estabrooks: Freshman in college.

Chris Bergh: You're professional.

Eric Estabrooks: Yeah. So, it's it's a huge thing,

Chris Bergh: Yeah.

Eric Estabrooks: But like I said earlier, it's not magic. There's a a bit of an I'm not even going to say an art. There's a lot of like well-known patterns, practices that get you the best performance out of this, keep you from making mistakes. And so the the biggest recommendation I can give in the time we have is if you start exploring Claude Code and to help you with your data engineering stay away from skills at first you're going to see everybody's like go to this repo and here's the top 50 skills or here's the 200 skills right like none of that's going to matter. All it's going to do is clutter up your thinking and keep you from learning how to use the tool which is Claude Code effectively.

00:48:53 Stay away from skills until you understand the tool

Eric Estabrooks: Once you've kind of learned the best practices, learned how to interact with it, get it to do stuff for you. Even if you have to repeat yourself a lot, you'll start to then understand, oh, I could use a skill for this specific thing I need to do. And then you go find it or make it yourself. And then you'll add another. But if you start with Claude Code and think skills are going to save you. You're going to end up in this position where you don't really know what's going on, you're going to lose control of it. And lose control of it is like, you know, those big high-profile articles you've seen, right? Somebody's Claude or whatever went and deleted their production database in their codebase, right? Like that stuff doesn't need to happen if you kind of work, you know, kind of educate yourself and learn how to use the tools on the way up before you start giving it control and start putting in skills that you don't understand.

00:49:45 Socratic dialogue, and plans as evidence of the work

Eric Estabrooks: So that's my biggest advice is start. Yes.

Chris Bergh: Right? You're going to ask them about this. Well, what about testing? What about deployment? How are you going to tell if things break downstream? You know, what data are you going to find data error? Like very simple things that you would talk to an intern. That's where that sort of Socratic dialogue comes in. And you can either do the Socratic dialogue in the code, which is slower, but it's faster to do it in a plan. And so, and also the plan becomes good evidence that you've done the work and can help you. You know, there's a a philosophy that you if you have enough plans, you should be able to like recreate your entire work from those plans. So, I think it's there's a bunch of stuff here. So I I think 10x data engineering is actually true if you have the architectural foundations and the techniques to build with Claude. And I think those techniques really are working in kitchens, working in functional idempotent chunks, running test suites, being able to deploy to production with regression tests, all those things.

00:51:31 Typing was never the bottleneck in data engineering

Chris Bergh: And it's not that Claude Code because the problem in data engineering has never been typing things on the keyboard with teams. You know, that's all Claude does. It types keyboard. 80% of the time on teams is wasted and so on waiting or redo or errors and so that's where we think the productivity comes from and so you have to think of it in a system in a process and so we've you know we've got a blog about this this is our webinar we since we had nobody show up and we may promote it some more and do it again sometime it went well we had another webinar about context which we had much more people we had 150 people show up so these are kind of paired both of them are sort of how to do data engineering with Claude Code and then how to use tools are our Genie code and then how to use Claude to actually query data and what that makes sense and so both these are kind of related to each other and so I'll share the slides and send this out but thank you for thank you for attending thanks everybody.

Eric Estabrooks: Thanks everybody.

Chris Bergh: Okay.