A/B testing is controlled experimentation on a live product: users are randomly split between the current version (A) and a variant (B) that differs in one deliberate way, and the variants’ performance on a pre-defined metric is compared with statistical rigour. Randomisation isolates causation — the difference in outcomes is attributable to the change, not to season, channel or audience mix. It is how modern product and growth organisations replace argument with evidence.
The craft is in the discipline around the test: a hypothesis written before the experiment, sample size and duration computed in advance (under-powered tests produce confident noise), one primary metric plus guardrail metrics (revenue up but churn rising is not a win), and honest handling of multiple comparisons — twenty simultaneous tests at 95% confidence will produce a false winner on average. Peeking early and stopping at significance is the classic self-deception.
Experimentation under data-protection law
A/B infrastructure randomises identified users and measures their behaviour — personal-data processing with a required lawful basis and honest disclosure under KVKK/GDPR; experiment assignment cookies fall under cookie consent rules for non-essential trackers. Boundaries firm up at sensitive contexts: experiments on pricing shown to different users edge into discrimination and consumer-fairness territory; experiments on emotional content earned platforms historic backlash; dark-pattern variants that win the metric can violate unfair-commercial-practice rules regardless of the lift. A short ethics-and-legality checklist gate on the experiment backlog — what data, which basis, any vulnerable groups, would we publish this design? — costs minutes and prevents the expensive category of winning tests.
If this is on your desk
Templates and checklists are free in the Founder Academy; for a specific situation, book a 30-minute intro call.
Founder AcademyBook an intro call