Run the test for 35 days
You need 20,622 paywall views per variant, 41,244 in total, to catch a 15% change in conversion from a baseline of 3.5% conversion.
- Decide the length before you start and stop on that day. Stopping the first time the result looks good makes false winners much more likely.
Bigger changes need fewer views
| Change to detect | Views per variant | Duration |
|---|---|---|
| 5% | 177,290 | 296 days |
| 10% | 45,362 | 76 days |
| 15% | 20,622 | 35 days |
| 20% | 11,858 | 20 days |
| 30% | 5,500 | 10 days |
Two-sided test at 95% confidence with 80% power, normal approximation. How this works