Handing data from one team to another looks like the simplest step in a pipeline. It is often the one that breaks most.
The problem
A large Fortune 500 healthcare company had a complex data landscape spread across many locations, and working across teams and data centers was hard.
One team in New Jersey ran an on-prem data warehouse. They shared data with an independent analytics team in California that worked in Azure and an Azure toolchain. The two teams were not coordinated, and that seemingly simple transfer of data produced a steady supply of problems.
Every time the New Jersey team changed their schema, they hoped it would not break anything downstream. The California team had no visibility into the change at all. Each time the two schemas diverged, the result was errors and delays. Trust went first, and finger-pointing followed.
Why not just centralize?
The obvious fix is to make both teams work the same way in the same place, and it is worth saying why that was not the answer here.
The California team had chosen Azure and an Azure toolchain for reasons that had nothing to do with New Jersey. The New Jersey team’s warehouse was on premises for reasons of its own. Consolidating either one is a migration project measured in quarters, and it would have been paid for out of the same budget as the analytics the business was waiting for. Meanwhile the actual defect was not the tool choice. It was that a schema change on one side was invisible on the other until something broke.
Fix the seam and you keep both teams productive. Consolidate and you stop both of them for two quarters to remove a problem that lives in ten minutes of schema comparison.
The solution
Neither team gave up its tools. That was the point.
One pipeline per team, in each team’s own tools
Each team built a DataKitchen Recipe, a pipeline that orchestrated its own local workflow and toolchain in its own environment. Both Recipes included automated tests that checked the tables before any data moved.
The on-prem team’s Recipe pulled data from a source into a data store, ran a transformation, and checked the data with Python. The Azure team’s Recipe received the data, moved it into storage, loaded it into SQL Server, transformed it, and tested it with SQL. Each team owned and maintained its own Recipe and its own tests, written in the language that team already worked in.
A schema check that runs every 10 minutes
The tests inside each Recipe covered each team’s own work. Neither could see the seam between them, so the teams added a test automation node that reads the full schemas from both tables and compares them every 10 minutes. When the schemas do not match, the test fails and alerts the right people, and the drift gets corrected before the next data transfer runs.
That is the change worth copying: the schema mismatch stopped being something California discovered after the fact and became something both teams saw within minutes.
Meta-orchestration across on-prem and cloud
Each team’s Recipe was saved as an Ingredient in the platform. An agent-based architecture reaches both the on-prem and the Azure cloud systems, so those Ingredients could be orchestrated by one centralized Recipe that calls the local ones. Two teams, two data centers, two toolchains, one workflow.
What the agent architecture bought
The reason one workflow could span both sides is that the platform reached into each environment through an agent rather than requiring a single network or a single cloud. That sounds like plumbing, and it decides whether this pattern is available to you at all: a central orchestrator that can only see the cloud side ends up triggering the on-premises work by convention and hope, which is where the schedule slips reappear.
With both sides reachable, the handoff has an owner. Somebody can point at one workflow and say what ran, in what order, and what it checked before the data moved.
Results
The teams collaborate across locations and data centers while still working in their own toolchains. Automated testing raised the quality of the analytics on both sides of the handoff.
The teams now get a warning when data or schemas drift, in time to fix it before an error reaches an end-user customer. They spend far less time finding and fixing surprise errors, and the level of trust between the two teams went up.
TIP
Enjoyed this case study? Download the PDF to share it with your team.
FAQ
What are the key points in this blog?
Two teams at a Fortune 500 healthcare company, one on premises in New Jersey and one on Azure in California, kept breaking each other with schema changes nobody could see. Each kept its own toolchain and pipeline, a schema comparison ran every 10 minutes across both tables, and one orchestrated workflow spanned both environments. Trust followed.
Why does a data handoff between teams break so often?
Because the seam has no owner. Each team tests its own work, and neither test looks at the boundary. When the upstream team changes a schema, the downstream team finds out from an error rather than a notification, which is why divergence turns into delay and then into finger-pointing.
How do you detect schema drift between two teams?
Compare the full schemas on both sides on a schedule and alert on mismatch. This team added a test automation node that reads both tables’ schemas every 10 minutes and fails when they do not match, so the drift is corrected before the next data transfer rather than discovered after it.
Should both teams standardise on the same tools?
Not necessarily, and here it would have cost more than the problem. Consolidating either side is a migration measured in quarters, paid from the same budget as the analytics the business is waiting for. The defect was not the tool choice; it was that a schema change on one side was invisible on the other.
How can one workflow span on-premises and cloud systems?
Through an agent in each environment rather than a single shared network. Each team’s pipeline was registered as a reusable component, and a central workflow called both. An orchestrator that can only reach the cloud side ends up triggering on-premises work by convention, which is where the schedule slips come back.
What changed for the two teams afterwards?
They kept their own tools, their own tests, and their own environments, and stopped surprising each other. Drift produces a warning in time to fix it before an error reaches a customer, far less time goes to hunting unexpected errors, and the level of trust between the two teams went up measurably.
