Experimentation in the AI Era: Signal, Not Volume
AI has made testing cheap - not the trust in the results. A team that runs 41 tests in a quarter but can barely name one that changed a real decision does not have a volume problem. It has a learning problem.
The expensive part stayed where it was. What became cheap is setting up and rolling out a test. What stayed expensive is knowing whether the result means anything - and the courage to stop a losing test before it burns budget.
Start with fewer bets. A model returns 200 ideas on request. 200 unsorted ideas are not a strategy. Every idea is scored against three questions before a live test:
- How large is the potential upside? Does the result move a business-relevant number or only a vanity metric?
- How confident are we from the start? Is there a reasoned hypothesis, or are we guessing?
- How high is the cost to run it? Time, traffic and budget that would do more elsewhere.
The discipline is killing a good-looking idea before it eats three weeks.
A clean test answers a question. It changes a single variable, measures against a real control group, and runs to a pre-registered sample size. Anyone who sets the sample size only after looking at the data is no longer measuring - they are searching for confirmation. AI may calculate the test duration, simulate outcomes and surface confounds. The choice of metric is always made by the human.