Experiments
Loading report...
Loading older data...

Experiments

Three tests, three reads — 80,000 assigned users
Page feels slow?
Browser extensions that scan input fields (password managers like Bitwarden, 1Password, LastPass, or autofill tools) can add significant overhead on dense reports — especially ones with many filters.
Fix: open the extension’s site settings and disable autofill for this domain. Or try an Incognito / Private window to confirm.

New Checkout (checkout_v2)

Variant
Converted

Starter Pack Pricing (starter_pack_price)

Price
Converted

Onboarding Tutorial (onboarding_v2)

Tutorial
Returned Day 1

Methodology

ℹ How to read this report — Winsor, CUPED, and 95% CI explained

This report shows the same A/B test through three lenses. The default view (CUPED) is the most trustworthy — the others are sanity checks.

The three modes

  • Raw — observed averages. Dominated by whale variance: a handful of high-spending users can swing the headline by tens of percent based on which variant they happened to land in. Useful as a backstop, not a headline.
  • Winsor p99 — caps each user's revenue at the 99th percentile of the pooled Control + Test population. Same cap on both arms (unlike per-variant "Exclude top X%" which biases the comparison). The whale is still counted as a payer; their contribution to the average is just capped.
  • CUPED (default) — subtracts each user's revenue baseline measured in the 14 days before the test started. Removes the predictable "this person was always going to spend a lot" component, leaving only the part the test could plausibly have caused. Typically 30–50% variance reduction on revenue metrics.

Reading the deltas

Every KPI row shows an observed delta (▲ +5.2%) and a 95% confidence interval (95% CI [-2.1, +12.3]) below it. The CI is the range of plausible true effects given the data:

CI looks likeInterpretationBar colour
[+2, +8] all positive Test is reliably better green
[-9, -1] all negative Test is reliably worse red
[-5, +12] crosses 0 Can't tell — treat as noise grey

A grey bar means the test might be moving the metric, but we don't yet have enough data to say so with confidence. Wide CIs come from heavy-tailed revenue + short test windows + small effect sizes — not from a bug. To halve a CI typically takes ~4× more data.

Two-stage decomposition

Conversion (payers / DAU) × ARPPU (revenue / payer) = ARPDAU. Showing them as separate rows makes it obvious whether a revenue lift comes from more payers, bigger baskets, or both.

See the A/B Testing section of the framework README for the full technique reference.