AI RESEARCH
Variance Reduction for Heavy-Tailed Monetization Metrics in Ranking Experiments via Post-Stratification
arXiv CS.LG
•
ArXi:2606.04110v1 Announce Type: new Online evaluation of ranking and retrieval systems often relies on downstream monetization metrics such as app revenue or creator earnings. These metrics are typically heavy-tailed, with a small fraction of users dominating both mean and variance, leading to low statistical power and unreliable We present a practical framework for variance reduction in online experiments by combining post-stratification with CUPED. Our approach leverages pre-experiment covariates to improve the sensitivity of monetization experiments without requiring additional traffic.