How do you A/B test a product demo?

Change one variable, decide a downstream metric like signups before starting, run for whole weeks, and fix the duration in advance. Most demo tests fail because the traffic is too low to detect a difference, so the result is noise that looks like a finding.

How much traffic do you need to test a demo?

Typically thousands of visitors per variant to detect a small difference, which most demos never reach in a sensible window. Below that, the drop-off curve and watching a handful of real people use it are more reliable than an underpowered test.

What is the highest-impact thing to test on a demo?

Gated versus ungated, by a wide margin. Length is second, and the opening step third -- it matters more than expected because it decides whether anyone reaches step two. Cosmetic changes are rarely worth the traffic.

← Back to blog

A/B Testing a Product Demo Without Fooling Yourself

August 18, 2026

Testing two demo variants sounds straightforward and is where a lot of confident wrong conclusions come from, because demo traffic is usually far below what a test needs.

This is about running one honestly, and about knowing when not to bother.

The sample size problem

Detecting a small difference in conversion needs a lot of traffic -- typically thousands of visitors per variant, not hundreds. Most demos on most sites will never see that in a reasonable window.

Running a test anyway produces a number that looks like a result and is noise. The most common failure is calling a winner after a week because one variant is ahead, when the lead would have flipped twice more given another fortnight.

What to do with low traffic instead

Change one big thing, not several small ones. Small differences need traffic you do not have; large ones sometimes show through.
Use the drop-off curve, which needs far fewer viewers to be informative than a conversion comparison does.
Watch five people use it. Qualitative observation beats an underpowered test, and five sessions surface most usability problems.
Ship the better-reasoned version and move on. Not every decision needs an experiment.

If you do have the traffic

Test one variable. Length, or the opening step, or gated versus ungated -- not a redesign.
Decide the metric before starting, and make it downstream: signups, not demo completions.
Run for whole weeks. Weekday and weekend traffic behave differently, and a part-week skews it.
Fix the duration in advance. Stopping when you like the number is how underpowered tests become confident conclusions.

Variables worth testing, in order

Gated versus ungated is by far the largest lever and the one most worth the traffic. Length is second. The opening step is third, and it matters more than people expect, because it decides whether anyone sees step two.

Cosmetic changes -- colours, button copy inside the demo -- are last, and are almost never worth a test at demo-level traffic.

Frequently asked questions

How do you A/B test a product demo?: Change one variable, decide a downstream metric like signups before starting, run for whole weeks, and fix the duration in advance. Most demo tests fail because the traffic is too low to detect a difference, so the result is noise that looks like a finding.
How much traffic do you need to test a demo?: Typically thousands of visitors per variant to detect a small difference, which most demos never reach in a sensible window. Below that, the drop-off curve and watching a handful of real people use it are more reliable than an underpowered test.
What is the highest-impact thing to test on a demo?: Gated versus ungated, by a wide margin. Length is second, and the opening step third -- it matters more than expected because it decides whether anyone reaches step two. Cosmetic changes are rarely worth the traffic.

Keep reading

This is what Demorta does

Click through it — the same kind of demo you can record of your own product.

Try it on your own product

Record your first interactive demo free.

Five demos on the free plan, forever. No credit card, no trial countdown, no sales call. Install the Chrome extension, click through your product once, and you have a shareable link in about ten minutes.

Start free →