Rumour-driven development
Somewhere in Spotify’s Stockholm offices in 2016, an engineer needed to spin up a new backend service. The instructions were somewhere — in a wiki nobody had updated, in a Slack message from someone who had since left, in the institutional memory of whichever colleague happened to be polite enough to answer. The engineering team had a name for this: “rumour-driven development.”
Spotify by 2016 was running more than 2,000 backend services, 300 websites, and 4,000 data pipelines across hundreds of engineering squads — all making their own tool choices, adopting their own deployment patterns, naming their systems after internal lore nobody had bothered to write down. It was the paradox at the heart of the microservices era: the architecture that was supposed to give each team autonomy had instead given each team a private island. Freedom, it turned out, produced a forest.
The answer came from a Hack Week. Eight senior engineers spent the week building a service catalog — a single place where any team could register a component, list its owners, link its documentation, and declare what it depended on. They called it System Z. The logic was blunt: if you could find a thing, you could understand it. System Z quietly became indispensable, then got absorbed into a broader internal tool called Backstage.
What distinguished Backstage from the configuration management databases that IT departments had been building and abandoning for decades was the addition of what the infrastructure team called “golden paths.” Coined by Gary Niemen and published in an August 2020 engineering post, a golden path was the opinionated and supported route for a common task: standing up a backend service, building a data pipeline, launching a website. Not the only route — the paved one. The platform team maintained the templates, tested them with real production services, and committed to keeping them current. Stray from the path and you were on your own; stay on it and you had a map. Netflix, independently, had arrived at the same idea and called it the “paved road.” Same metaphor, different continent.
Spotify open-sourced Backstage in March 2020. Six months later it entered the Cloud Native Computing Foundation’s Sandbox. On March 15, 2022, the CNCF Technical Oversight Committee voted to elevate it to Incubating status — by which point the project had more than 500 contributors, 15,000 GitHub stars, and formal adoption at Netflix, Expedia, and Roku, among a hundred others. The CNCF simultaneously formalized “platform engineering” as a named discipline: the working group that had been tracking this space formally renamed itself. A practice that had existed in back rooms for years acquired a title.
The problem Backstage addressed was older than Kubernetes. Every generation of software scale produces a version of it: the naming crisis, the orphaned service, the API reimplemented five times by five teams who never found each other. What changed in 2022 was the recognition that solving it was real engineering work — not a side project a team lead handled on a slow Friday, but a dedicated internal product with users, a roadmap, and on-call rotations. The platform team became a first-class citizen of the engineering organisation.
The jungle does not go away. You learn to build roads through it — which turns out to be a different profession altogether.
Sources
- How We Use Golden Paths to Solve Fragmentation in Our Software Ecosystem — Spotify Engineering — Origin of the “golden path” concept and “rumour-driven development,” Hack Week backstory, scale of Spotify’s microservices estate; published August 2020 by Gary Niemen.
- Backstage project joins the CNCF Incubator — CNCF — March 15, 2022 incubation milestone, adoption numbers, context from TOC member Erin Boyd on the SaaS-sprawl problem Backstage solves.