A/B test sample size, explained simply

1. Start with the decision, not a traffic target

Sample size is the number of eligible observations needed to give a test a reasonable chance of detecting a worthwhile difference. There is no universal minimum number of clicks or conversions. The answer depends on your baseline rate, the improvement you want to detect, the statistical method and how people enter the experiment.

Write the decision first: ‘Will replacing our product-led opening with a problem-led opening increase purchases per randomly assigned visitor?’ This defines the creative change, outcome and denominator. Every numerical scenario below is hypothetical.

Choose one primary metric. Click-through rate can answer whether a hook attracts attention, but not whether it improves purchase performance. A cheaper click can bring less qualified traffic.

• Specify the control, challenger, eligible audience and assignment unit.
• Name one primary outcome and a fixed measurement window.
• Set guardrails, such as refund rate or qualified-lead share, before launch.

2. Understand the four inputs that drive sample size

The baseline is your current primary-outcome rate in a comparable audience. If 50 of 1,000 eligible visitors purchase, the baseline is 5%. Use a representative window rather than an unusually strong promotion.

The minimum detectable effect, or MDE, is the improvement you design the test to detect. Moving from 5% to 6% is one percentage point, or a 20% relative lift. Entering ‘1%’ instead of ‘20%’ into a calculator using relative lift can radically change the answer.

The significance level controls the long-run false-positive rate under the method’s assumptions. Power is the chance of detecting the specified effect when it exists. A common planning choice is 5% significance with 80% power, but these are choices, not laws. Higher power or stricter significance generally requires more observations.

• Record whether the calculator expects absolute or relative uplift.
• Use a two-sided test unless a justified directional design was specified beforehand.
• Keep the chosen inputs with the test brief so others can reproduce the estimate.

3. Turn a sample estimate into a feasible schedule

Consider a hypothetical landing-page creative test with a 5% purchase baseline, a 6% target, equal allocation, 5% two-sided significance and 80% power. A conventional independent two-proportion calculation gives approximately 8,200 visitors per variant, or 16,400 total. Calculators can differ slightly because of their methods.

With 800 new eligible visitors daily across both groups, reaching that total takes roughly 21 days. That estimate excludes conversion maturation and assumes stable traffic. If only half your site visitors qualify, use eligible traffic, not total sessions.

Seeking a smaller change from 5% to 5.5% needs approximately 31,200 visitors per variant under the same assumptions. Small improvements can be commercially valuable, but proving them can be expensive. If the required duration exceeds your stable campaign window, redesign before spending.

• Check whether the result means per variant or total.
• Divide total required observations by daily eligible observations.
• Allow for outcome delays and normal trading cycles.
• Check that the creative, offer and inventory can remain stable throughout.

4. Count the right units, not the biggest number

The simple calculation above assumes independent observations and a binary outcome: purchase or no purchase. Ten impressions served to one person are not automatically ten independent chances to evaluate that person’s purchase behaviour. Repeated exposure, cross-device visits and shared accounts complicate counting.

For a hypothetical software trial page, assign visitors consistently and count whether each assigned visitor starts a trial within the chosen window. Do not switch between users, sessions and clicks because one denominator reaches the target sooner.

If assignment happens at store, region or company level, people within each group may behave similarly. A standard visitor-level calculator can understate the requirement. Revenue per visitor, average order value and CPA also need methods suited to their distributions, not a purchase-rate calculator with a different label.

• Match the analysis to the randomisation unit.
• Define repeat visits, duplicate outcomes and exclusions before launch.
• Seek statistical support for clustered designs or highly variable monetary outcomes.

5. Separate experiments from ordinary ad delivery

Two ads running together do not necessarily form a randomised A/B test. Delivery systems may favour different audiences, placements or times. Equal budgets do not establish equal exposure or comparable groups.

For a hypothetical restaurant booking campaign, comparing a lunch video shown mainly on weekdays with a dinner video shown mainly on weekends mixes creative and demand differences. A concurrent, appropriately randomised experiment provides stronger evidence about the creative change.

Google Ads provides experiments for testing campaign changes. Check the supported experiment type, allocation and reporting before launch. Availability and behaviour depend on the platform and campaign setup. Changing the headline, offer and audience together tests a package, not the headline alone.

• Use the provider’s supported experiment mechanism where appropriate.
• Keep non-tested settings consistent.
• Investigate unexpected allocation imbalance before interpreting outcomes.

6. Validate tracking and let outcomes mature

A large sample cannot rescue inconsistent measurement. Before launch, verify the primary event, deduplication, assignment records and attribution window. Record the reporting source explicitly.

Google distinguishes GA4 key events from Google Ads conversions. Do not treat similarly named figures as interchangeable without checking their configuration and reporting context.

For a hypothetical professional-services test, an enquiry may need several days to become a qualified lead. Comparing yesterday’s enquiries with last week’s qualified leads disadvantages the newer cohort. Give both variants the same follow-up window.

If tracking fails midway, document the affected period and assess whether assignment or outcome capture became biased. Quietly removing inconvenient days after seeing the result is not a defensible repair.

• Test the event journey before allocating meaningful spend.
• Keep outcome definitions unchanged during the experiment.
• Wait for the final included cohort’s measurement window to finish.

7. Set stopping rules before the first result

Repeatedly checking a conventional fixed-horizon test and stopping when significance appears increases false-positive risk. Monitor delivery, spend and tracking for safety, but make the performance decision at the planned endpoint.

If early decisions are essential, use a properly designed sequential method with its own stopping rules. A dashboard that refreshes daily is not automatically sequentially valid.

Testing six hooks and declaring the strongest of fifteen pairwise comparisons a winner creates another problem: more opportunities for chance findings. Preselect comparisons or use an appropriate multiple-testing adjustment. For limited traffic, screening concepts qualitatively before running one confirmatory comparison can be more practical.

• Specify sample target, minimum duration and maximum feasible duration.
• Document safety stops separately from success criteria.
• Do not restart the statistical clock simply because early results disappoint.

8. Read the interval, not just the winner badge

Suppose a hypothetical test estimates a 0.4 percentage-point lift, with a confidence interval from minus 0.3 to plus 1.1 points. That result remains compatible with harm and worthwhile improvement. It is inconclusive, not proof that both creatives perform equally.

Conversely, a statistically detectable improvement may be too small to cover production or fulfilment costs. Compare the estimated effect and interval with the commercial threshold you set beforehand. The planning MDE is not a rule that smaller real effects cannot exist.

When traffic is insufficient, retain the control, test a more distinct concept or gather evidence for a later experiment. Do not keep extending a fixed-horizon test until it turns positive. Report absolute rates, denominators, uncertainty and any departures from the plan.

• Distinguish evidence of improvement from evidence of worthwhile improvement.
• Record inconclusive outcomes as learning, not failed execution.
• Treat period comparisons as descriptive, not proof of causation.

9. Make test administration lighter without weakening evidence

Use one reusable brief containing the hypothesis, calculator inputs, allocation, tracking checks, owner and decision date. Measure team efficiency through preparation minutes, reporting minutes and rework per completed test. Compare similar work and retain quality checks; a faster report that omits uncertainty is not an improvement.

Its described RiseProjects workflow keeps tasks, owners and due dates together. RiseCampaigns keeps creative variants, spend and learnings together with explicit human approval controls. These descriptions do not establish a built-in sample-size calculator or statistical testing engine.

RiseInsights provides source-labelled reporting and period comparisons, which do not establish experimental causation. Connected workflows depend on permissions, provider capabilities and approvals; TikTok and LinkedIn approvals remain pending. Explore the relevant products and the solution overview below without assuming every planned connection is available.

• Time preparation and reporting separately from experiment runtime.
• Track corrections alongside minutes saved.
• Keep the final decision and supporting evidence with the task.