Run a two-sample A/B test in the simulator and predict — before you read it off the screen — the two-proportion z and its p-value for a textbook case, the false-positive rate when the two rates are truly equal, the per-arm sample size that buys 80% power, and what happens to the false-positive rate when you peek at the results early.
A two-sample A/B test compares two conversion rates, p_A and p_B. The two-proportion z-test asks whether the observed lift is larger than sampling noise; a small p-value means the data would be surprising if the rates were equal, not that the rates are equal by some probability. Two traps make a p-value lie: too little data (low power, so a real lift is missed) and peeking — testing repeatedly as data arrive and stopping at the first significant look. At a true lift of zero the rejection rate is exactly α; peeking pushes it far above α.
Open the simulator. It opens on its default inputs. Each step names the knob to move; leave the rest at these values. The readouts and panels print every number you predict; the false-positive and peeking bars come from a seeded simulation, so the same seed always gives the same estimate.
Sliders -> rate A / rate B / n per arm / alpha / tails / target power / looks / seed
Defaults -> 0.10 / 0.125 / 2000 / 0.05 / two / 0.80 / 5 / 42
Watch the z and p readouts and the significant/not verdict (a word, not just a colour), the power-vs-n curve and the required-n readout, and the false-positive bars labelled “single look” and “with peeking”. Press Run to draw the seeded simulation.
Leave the defaults: control rate p_A = 0.10 (200/2000) versus variant rate p_B = 0.125 (250/2000), n = 2000 per arm. The pooled two-proportion z divides the observed lift by its standard error under H₀. Predict z, then read the z readout.
The readout shows z ≈ 2.502. The lift of 0.025 is about 2.5 pooled standard errors above zero — well into the tail, which is why the next step's p-value is small.
Same inputs, two-tailed. The two-sided p-value is P(|Z| ≥ |z| given H₀) — the probability of a lift this extreme or more if the two rates were equal. Predict p and whether it clears α = 0.05, then read the p readout and the verdict.
The p-value is about 0.0124 — below 0.05, so the verdict reads “significant”. Read it correctly: it is P(data this extreme | H₀), not the probability that H₀ is true and not the probability the result was chance.
Set p_B equal to p_A = 0.10, so there is no real difference and H₀ is TRUE. The power curve at zero lift becomes the false-positive rate — the fraction of experiments that reject H₀ anyway. Predict it, then read the power-at-zero value.
It is exactly α = 0.05. A correctly sized test rejects a true H₀ exactly α of the time — that is the definition of the false-positive rate, and it is why α is the price you agree to pay.
Restore p_B = 0.125 and leave the target power at 0.80, two-tailed, α = 0.05. The required-n readout is the per-arm sample size that reaches that power at the textbook lift. Predict roughly how large it is, then read it.
It is about 2507 per arm — roughly 5000 users in total. The significant result in step 2 came from 2000 per arm; a real lift this small needs more data before 80% power is even on the table. Significance needs both a real lift and enough n.
Keep the rates equal (p_A = p_B = 0.10, so H₀ is true), leave looks at 5 and seed at 42, and press Run. The “single look” bar sits near α (about 0.0493). Predict the “with peeking” bar — the rejection rate when you test at 5 interim looks and stop at the first significant one — then read it.
The “with peeking” rate is about 0.1367 — nearly 3× the nominal 0.05, against a single honest look at 0.0493 ≈ α. Testing repeatedly and stopping at the first “win” manufactures significance out of pure noise; fix the sample size in advance, and read one final result.
Everything above is waiting in the simulator. Drag the two rates apart and watch z, p and the verdict move together; slide n along the power curve to see when a real lift becomes detectable; and change the number of looks to watch the peeking bar climb further above α.
- Read the pooled two-proportion z ≈ 2.502 for 200/2000 vs 250/2000
- Found its two-sided p ≈ 0.0124 < 0.05 — significant, and read as P(data | H₀)
- Saw the false-positive rate sit at α = 0.05 exactly when the two rates are equal
- Found that 80% power at this lift needs about 2507 per arm — more than the 2000 that reached significance
- Watched peeking (5 looks) inflate the false-positive rate to ≈ 0.1367, nearly 3× the 0.05 nominal