On-Demand Webinar · 1 hr 21 min

DataOps in Practice, Swedish Style

Lars Albertsson, founder of Scling and previously of Spotify and Google, on Spotify's DataOps journey: how the company aligned its people, processes, and technology to build high-quality analytics quickly, what worked, what did not, and the lessons that carried. Recorded 2020; updated August 2026.

What you'll learn 8 points
  • Lars Albertsson of Scling traces DataOps back to Spotify, where Hadoop was installed in 2007 and by 2013 jobs were deployed as Debian packages with Puppet onto pet machines, redundant crons raced to run the same job, and a monitoring daemon sat at 100 percent for six months before anyone killed it.
  • The move out of that state is described as a set of shifts: stateful to stateless, pets to cattle, folklore to a golden path, testing in production to local tests and CI/CD. Learning to build a new pipeline went from weeks to under a day, and mending a bug from days to under an hour.
  • The functional pipeline principle: a raw source of truth feeding a refinement factory, immutable datasets and artifacts, and deterministic, idempotent, reproducible deployment and processing. Workflow orchestration is named the key success factor.
  • Offline batch systems make errors cheap. When a faulty job emits bad data you revert the serving datasets to the previous version, fix the bug, remove the faulty datasets, and let the orchestrator backfill automatically. QA can be reactive and the production environment is sufficient.
  • A production-critical upgrade runs dual datasets during the transition with parallel downstream pipelines, which is cheap, low risk, easy to roll back and testable end to end. On that model no separate dev and staging environments are needed.
  • Four data quality dimensions are used: timeliness, correctness, completeness and consistency. Correctness is measured with Spark or Hadoop counters bumped on odd code paths, and consistency with a dedicated quality assessment job that emits a tiny quality metadataset into standard graphing and alerting tools.
  • Autonomy has a cost. At Spotify it produced excessive heterogeneity, including 25 ways to store a timestamp, and end-to-end pipeline tests were culturally hard to adopt, which left pipelines difficult to change or retire.
  • Small teams move real volume: Bonnier News' data platform, built by four to five people, went from zero to a happy customer in three weeks and dozens of pipelines returning value in eight months, and a Scling retail client of one to three people ran 40 sources, 70 pipelines, 200 egress points and 3,400 datasets a day after one year, against a typical enterprise big data project of 6 to 24 months.

Slides

59 slides

Questions from this session

What are functional data pipelines?

A functional pipeline keeps a raw source of truth and refines it in a factory of downstream steps, using immutable datasets and artifacts and deterministic, idempotent processing, so the same input always yields the same output. Deployment is reproducible for the same reason. The key success factor is workflow orchestration, and the properties that matter in an orchestrator are being pure Python, simple, and free of features.

Why are batch data systems easier to operate than online services?

Because the blast radius is smaller. Upgrading an online service is a careful rollout that risks user impact and needs proactive QA, and a failure means data loss or a cascading outage. Offline batch upgrades roll out instantly with no user impact and reactive QA, a service failure only delays the pipeline without losing data, and a bug causes temporary data corruption that can be reprocessed away.

How do you recover from a bad batch job?

Four steps. Revert the serving datasets to the previous version so consumers are correct again, fix the bug, remove the faulty datasets, and let the orchestrator backfill automatically. Because the cost of an error is low, QA can be reactive rather than exhaustive, and the production environment alone is enough to work in.

Do you need separate dev and staging environments for data pipelines?

Not for a functional batch system. A production-critical upgrade can run dual datasets during the transition with parallel downstream pipelines, which is cheap, low in risk, easy to roll back and testable end to end against the real data. That removes the reason for a staging environment. What is not acceptable is the old habit of testing by temporarily changing the output path and running by hand.

What are the four data quality dimensions?

Timeliness, meaning the report was produced when expected. Correctness, meaning the numbers were calculated right. Completeness, meaning all customers and the whole time period are included. Consistency, meaning the summaries are all based on the same time period. Correctness is tracked with processing-tool counters bumped on odd code paths; consistency with a dedicated quality assessment job that writes a small quality metadataset.

How do you test a batch data pipeline?

A single job is tested with a standard test harness that generates input files, runs the job in local mode and verifies the output, which works from an IDE or in CI. A whole pipeline is tested one of two ways: a custom job that runs the sequence of jobs, or a customized setup of the workflow manager itself. End-to-end pipeline tests are technically straightforward and culturally the hard part.

Where to go next