A/B Testing: When It Pays, What It Costs, and What You Need First

A/B testing is a controlled randomized experimentation methodology that splits live website traffic equally between an original control page and a single-variable treatment variant to measure statistically significant conversion rate uplift at a 95% confidence threshold and 80% statistical power before deploying permanent changes.

The mechanics are simple and widely explained. The part that decides whether experimentation pays back for your business is not the mechanics, it is whether your funnel generates enough sessions and conversions for the statistical signal to separate from day-to-day randomness.

Most articles about split testing are written by the vendors selling the software, so they skip the question of whether you should be testing at all. This guide starts there, because for a large share of B2B websites the honest answer is not yet.

The threshold is set by arithmetic, not enthusiasm. To detect a realistic improvement, say a 20% relative lift in conversions, you need roughly 100 conversions per variant before the result separates from chance. That is 200 conversions per test, and at a 2% baseline rate it means about 10,000 sessions on the specific page you are testing.

Below that volume, tests do not fail loudly. They produce a winner that is random noise, you ship it into production, and the promised lift never appears in the monthly revenue numbers. Two rounds of phantom winners and the team concludes experimentation does not work, when what failed was running split tests below the required sample size.

Every valid sample size calculation balances three statistical inputs: your baseline conversion rate, the Minimum Detectable Effect (MDE) worth acting on, and your error tolerances. Standard experimentation locks Type I error (alpha = 0.05, meaning a 95% confidence level and a 5% ceiling on false positives) alongside Type II error (beta = 0.20, meaning 80% statistical power and a 20% ceiling on false negatives). If your page already sits near a good conversion rate of 2.0% and you want to detect a 15% relative MDE (moving from 2.0% to 2.3%), you need roughly 17,000 sessions per variant. Evan Miller's sample size calculator lets you plug in your baseline rate and target MDE before writing a single line of code, which is far cheaper than waiting a month on an underpowered experiment.

Three things, and each takes hours rather than weeks. Missing any one produces results you cannot act on, which is more expensive than not testing, because the conclusions get quoted for a year afterwards.

Conversion tracking that fires once from every path, verified by hand across desktop and mobile browsers. A baseline of at least eight weeks so you know what normal weekly variation looks like before introducing a split. And a written hypothesis with a specific MDE target inside it: not "a shorter form will do better" but "cutting the form from nine fields to four will lift qualified submissions by at least 15% at 95% confidence". The written number is what stops a flat result from being reinterpreted as a small win.

Nobody has the traffic to test everything, so the order matters more than the list. The table below is what we start with by page type, drawn from audits rather than from theory, along with roughly the traffic each needs to give a clean answer within a month.

The pattern behind it: the first test on any page should be a removal, not an addition. Removals are faster to build, easier to interpret, and they win more often, because most pages carry more than they need rather than less.

Divide the sessions you need by your weekly sessions on that page, then round up to whole weeks. Round to weeks because traffic behaves differently on weekdays and weekends, and stopping mid-week tilts the result toward whichever days happened to be included.

Two rules keep tests honest once they are running. Do not peek at the result to call an early winner before the planned sample size completes, because repeated peeking inflates your false-positive rate from 5% to over 25%. And do not run two tests on the same funnel flow at once unless your experimentation platform isolates interaction effects, which most basic setups do not do by default.

Choosing between Frequentist and Bayesian statistical engines determines how you evaluate risk, while your technical split architecture determines whether the underlying data is trustworthy. Frequentist fixed-horizon testing enforces strict Type I (alpha = 0.05) and Type II (beta = 0.20) error boundaries across a predetermined sample size, making it the gold standard for high-traffic funnels where avoiding false positives is paramount.

Bayesian testing calculates the continuous posterior probability that Variant B beats Control alongside an expected loss metric. That flexibility makes Bayesian evaluation attractive for mid-sized B2B sites, provided the team uses uninformative or conservative priors and waits for at least two full weekly business cycles. However, neither statistical model can rescue an experiment suffering from Sample Ratio Mismatch (SRM). If a 50/50 split records 5,250 users in Control and 4,710 in Treatment, a chi-square test yielding p < 0.001 proves that client-side redirects, slow script execution, or browser ad-blockers dropped users asymmetrically. When SRM triggers, the test result must be discarded immediately.

Technical execution also governs whether your experiment harms user experience. Legacy client-side testing snippets modify the DOM after the browser downloads the original page, causing a visible Flash of Original Content (FOOC), pushing Interaction to Next Paint (INP) past 200 milliseconds, and inflating Cumulative Layout Shift (CLS). Modern engineering teams execute split routing at the network edge using Vercel Edge Middleware or server-side feature flags. Assigning the variant cookie at the CDN edge serves the final HTML on the first byte, eliminating flicker completely and protecting conversion performance across custom landing pages.

SEO split testing answers a different question and works differently in practice. Instead of splitting users, you split pages: half a set of similar templates gets the change, half does not, and you compare organic traffic between the groups over several weeks.

The reason for the difference is that a search engine sees one version of a URL, not two, so user-level splitting cannot measure a ranking effect. It also means the usual worry is misplaced. Google's own documentation on website testing states plainly that A/B testing does not carry a ranking penalty when implemented normally, and the cloaking concern applies only to serving different content by user agent.

SEO tests need more patience: four to six weeks minimum, and a set of at least twenty comparable pages before the comparison holds.

The examples that circulate hardest are the ones that photograph well, and button colour is the worst of them. It survives because a famous case study from 2009 showed a large lift, and because it is the easiest thing to change. In our audits it has never been the biggest lever on a page, and it is usually a test spent to learn nothing.

The rule that saves the most time: do not test anything you would ship regardless. If the current version is broken, unreadable, or contradicts the ad that brought the visitor, fix it immediately. A test on a broken page measures which broken version is less bad.

Why does testing micro-copy and button shading fail on low-traffic B2B sites? Because cosmetic tweaks typically move conversion rates by 1% to 3% relative, which requires more than 40,000 sessions per variant to detect at 95% confidence and 80% power. When your page receives 6,000 to 15,000 monthly sessions, only structural hypotheses such as restructuring the primary offer, removing form fields, or placing concrete case-study proof beside the call to action generate the 15% to 28% relative lifts needed to reach statistical significance within three to four weeks.

A test gives you three possible outcomes and most teams only plan for one. A clear winner is the rare case. A clear loser is genuinely useful, because it tells you the assumption behind the page is wrong and saves you from building more of it. And a flat result, which is the most common, means the change was too small to matter at your volume, not that the idea was wrong.

Write the interpretation rule before you start: what number counts as a win, what counts as flat, and what you will do in each case. Always pair the primary conversion rate with a commercial guardrail metric such as revenue per session or CRM-qualified pipeline rate so a jump in raw form submissions does not mask lower lead quality. Deciding afterwards is how a flat result becomes a small win in the retrospective.

Nielsen Norman Group's work on quantitative usability covers the sample size question in more depth, and it applies to testing as much as to research.

The failures we see repeat, and none of them are about statistics. They are about process, which is why buying a better tool rarely fixes them.

• Stopping early on a good day, which turns a coin flip into a decision
• Testing something so small that even a real effect would be undetectable at your volume
• Changing two things at once and learning which pair won rather than which change worked
• Never writing down the expected result, so every outcome gets reported as a success
• Running a test over a seasonal peak and applying the conclusion to the rest of the year

"The tests that teach us something are the ones where we wrote down what we expected and were wrong." - Vezert conversion lead

Tooling is the smallest line. A serviceable testing tool runs €0 to €200 a month at the volumes most B2B sites have, and the free tier of several is enough for one test at a time. The real cost is the build and the analysis.

Budget three to eight hours per test for implementation on a normal page, plus an hour to read the result properly and verify SRM and segment health. At agency rates that is €400 to €1,200 per test, and a programme that runs one test a month sits around €1,000 to €2,500 including the tool and the reporting. You can compare those ongoing experimentation budgets against fixed-scope page work on our pricing page. Below roughly 10,000 sessions a month on your key pages, that money returns more if it goes to traffic acquisition and fixing obvious funnel leaks instead.

One cost that never appears in the quote is the decision time. A test produces a result that somebody has to act on, and in most organisations that meeting is harder to schedule than the build. Programmes die from unread results far more often than from bad tooling.

If your key page clears 10,000 sessions a month, start with the removal test from the table and write the expected MDE number down first. If it does not, spend the same budget on qualified traffic and use a CRO audit to find the structural problems that do not need a split test to be obvious.

Most of what a first audit finds is not testable anyway. It is broken, and broken things get fixed rather than tested. Our conversion optimization work separates the two before anything gets built.