Tech blog notes: "Spotify’s New Experimentation Coordination Strategy"

Original post

What’s new

This post described a new experiment coordination strategy at Spotify, called “Bucket Reuse”. It buckets users into disjoint units and assign experiment treatments at bucket level. This is used to build a centralized experiment platform at Spotify: 1M buckets shared by all the experiments at any given time.
The post also advertised their paper that studies the statistical properties of sampling strategies to support experiment coordination.


Having a centralized list of buckets for experiment traffic allocation is an interesting choice. As described in this paper, Google uses a layered structure to coordinate experiments, so that experiment at different layers are mostly isolated, and experimenters don’t need to worry about non-related experiments. Maybe Spotify team chose the solution because their experiments often requires careful coordination.