Tag archive: experimentation
May 15, 2024
Fixed-Power Designs: It’s Not IF You Peek, It’s WHAT You Peek at
TL;DR Sometimes we cannot estimate the required sample size needed […]
Mar 5, 2024
Risk-Aware Product Decisions in A/B Tests with Multiple Metrics
TL;DR We summarize the findings in our recent paper, Schultzberg, […]
Sep 28, 2023
How to Accurately Test Significance with Difference in Difference Models
When we want to determine the causal effect of a […]
Aug 24, 2023
Encouragement Designs and Instrumental Variables for A/B Testing
At Spotify, we run a lot of A/B tests. Most […]
Aug 16, 2023
Experimentation at Spotify: Three Lessons for Maximizing Impact in Innovation
As companies mature, it’s easy to believe that the core […]
Aug 3, 2023
Coming Soon: Confidence — An Experimentation Platform from Spotify
TL;DR: Spotify is releasing a new commercial product for software […]
Jul 25, 2023
Bringing Sequential Testing to Experiments with Longitudinal Data (Part 2): Sequential Testing
In Part 1 of this series, we introduced the within-unit […]
Jul 18, 2023
Bringing Sequential Testing to Experiments with Longitudinal Data (Part 1): The Peeking Problem 2.0
Spotify’s approach to challenges in sequential testing with longitudinal data […]
Jun 28, 2023
Experimenting with Machine Learning to Target In-App Messaging
Messaging at Spotify At Spotify, we use messaging to communicate […]
Jun 15, 2023
Experimenting at Scale, the Spotify Home Way
In the fast-paced world of streaming, personalization plays a vital […]
Mar 21, 2023
Choosing a Sequential Testing Framework — Comparisons and Discussions
TL;DR Sequential tests are the bread and butter for any […]
Mar 23, 2022
Comparing Quantiles at Scale in Online A/B Testing
TL;DR: Using the properties of the Poisson bootstrap algorithm and […]
Feb 28, 2022
Search Journey Towards Better Experimentation Practices
At Spotify, we aim to build and improve our product […]
Mar 10, 2021
Spotify’s New Experimentation Coordination Strategy
At Spotify we run hundreds of experiments at any given […]
Nov 2, 2020
Spotify’s New Experimentation Platform (Part 2)
So you’ve read Part I of our two-part series about […]
Oct 29, 2020
Spotify’s New Experimentation Platform (Part 1)
At Spotify we try to be as scientific as possible […]