On-Demand Webinar · 48 min

Orchestrate Your Environment Pipelines for Reusability and Security

Part 3 of Orchestrating the Three Pipelines of DataOps. Chris Bergh on spinning up repeatable work environments, servers, software, and test data included, so the production and development pipelines they underpin can be rebuilt rather than hand-assembled. Recorded June 2020; updated August 2026.

Presented by Chris Bergh

What you'll learn 7 points
  • DataOps runs three pipeline orchestrations, not two. The Value Pipeline carries data to customers in production, the Innovation Pipeline moves changes from development to production, and an Environment Pipeline sits underneath both. Environments are the foundation the other two stand on.
  • Environment provisioning is a slow, manual, high-touch process, and weak environment management is a leading challenge for data and analytic organizations. The consequences are environments that cannot support testing, system outages and delivery disruption, infrastructure and license over-proliferation, and deployment errors caused by test data that differs from production data.
  • A development environment is more than a server. It is the code branch, the test data sets, the test result history, the tool set, the right hardware and software versions, and the network configuration, all matched to each other so a build-test-debug cycle can run fast.
  • The four environment types differ on every axis. An individual development environment is a data scientist on a laptop with development data running unit tests; a team environment adds engineers and analysts and functional tests; test and UAT runs against a copy of production data; production runs scheduled monitor tests with no development tool at all.
  • Test data management has three main challenges: distribution, meaning the time it takes to operationalize test data; quality, meaning high-fidelity data that reflects production; and security, meaning credit cards, medical records, and other private information that cannot simply be copied into a sandbox.
  • A top 5 US bank built a self-service data sandbox for more than 1,000 non-IT users to cut deployment time and self-service risk at once. Requests are approved, provisioned, monitored, and then decommissioned on a schedule that follows legal rules on data usage and lifetime. Dispatch takes about two minutes, provisioning about one minute, and decommissioning about one minute.
  • Four practices apply to every analytic environment: know what your environments look like, coordinate and communicate activity, automate the recurring tasks, and capture data for continual improvement. Keep everything under version control, bring the pain forward by doing painful things more often, and treat done as released to production.

Slides

63 slides

Questions from this session

What is an environment pipeline?

An environment pipeline is the orchestration that creates, configures, and tears down the environments in which data work happens: individual and team development sandboxes, test and UAT, and production. It sits underneath the Value Pipeline that runs production and the Innovation Pipeline that deploys change, because both of those depend on environments existing and matching each other. In the DataOps Manifesto the underlying principles are reproducibility and disposability.

Why is provisioning an analytic development environment so hard?

Because an environment is a matched set, not a machine. It needs the right code branch, the right tool versions, the right operating system libraries, network and hardware configuration, a usable test data set, and the test result history that tells you whether a change broke anything. Assembling that by hand is slow and high-touch, which is why development environments drift away from production and deployments fail.

What are the main challenges of test data management?

Three dominate. Data distribution is the time it takes to get test data operationalized. Data quality is the requirement that test data be high-fidelity enough to reflect production. Data security is minimizing risk from private information such as credit cards and medical records without slowing the team down. Storage cost, administrative effort, and the choice between masked real data and synthetic data follow from those three.

How does a self-service data sandbox work in a regulated organization?

A user requests an analytic development environment with the data sets and tools they need. An approver reviews it, an automated recipe provisions the environment, usage is monitored and tracked against the legal rules on data usage and lifetime, and a second recipe decommissions the database, tools, and workspace and archives the work when the term ends. In the bank example in this session, dispatch runs about two minutes and provisioning and decommissioning about one minute each.

How do you coordinate data work across two clouds?

Split pipelines create three questions: how the two halves coordinate, how each half can be changed independently, and where data quality testing and alerting happen across the boundary. The answer in this session is a single orchestration that spans both, so a job running Talend and Redshift on AWS and Python against BigQuery on Google Cloud is one process with tests on each part rather than two processes joined by hope.

How do you measure environment management?

Track the number of environments in use and the number of deployments between environments over time. Both should go up as environment creation gets automated: more environments means teams are no longer sharing one sandbox, and more deployments between them means change is moving instead of queuing. Analytic teams are rarely analytic about their own internal work, and these two numbers are a starting point.

Where to go next