Conversion Rate Optimization (CRO) From a UX Perspective: Process, Hypotheses, and Sample Size
Conversion rate optimization (CRO) is a cycle of hypotheses and tests to improve a key action. How to write the hypothesis and how much sample to use.
What is CRO from a UX perspective?
It is the part of product work devoted to raising conversions through tests. Design proposes changes to the flow, the content, or the visual hierarchy, and analytics shows whether those changes move the metric.
Abmatic describes the steps of a test as a sequence: hypothesis, setup, metric choice, sample size, running the test, significance analysis, rolling out the winner, and ongoing optimization (Abmatic, “How to use A/B testing for CRO”). The important point is that the sequence starts before anything on the screen is touched.
Why should a designer care?
Because conversion is the metric that causes the most arguments. A designer who knows how to measure and write hypotheses stops defending screens on taste and starts proposing changes with an expected result.
It also protects you. If a change “looks better” but lowers conversion, CRO tells you in time. And if it raises conversion, you have a solid argument for the next redesign.
How do you write the hypothesis?
A useful hypothesis links a change to a result and a reason. Abmatic asks for one that is specific, measurable, and based on data or research. A practical format:
If we do [change], we expect [metric] to rise, because [evidence or reason].
For example: “If we show the total price before the payment button, we expect more completed payments, because in interviews people left when they found extra costs.” The evidence can come from usability metrics, a funnel analysis, or a qualitative test. Without evidence, it is a hunch.
How much sample do you need?
This is where almost everyone goes wrong. The size depends on three things: the baseline conversion rate, the minimum detectable effect you care about, and statistical power, usually set at 80%. Evan Miller defines the minimum detectable effect as the smallest effect that will be detected with that probability (Evan Miller, “Sample size calculator”).
The rule that comes up again and again is simple: the smaller the effect you want to detect, the more sample you need. A large change on a low rate needs fewer visits than a small change on a high rate. That is why a fixed number floating around the internet is not useful: the figure depends on your data.
Calculate the sample before you start. If you have little traffic, test bigger changes or measure with other methods.
When do you stop the test?
Not when it is winning. Evan Miller shows the problem with a simulation: a change that does nothing (a new logo), a 50% baseline conversion rate, and a test that stops as soon as it reaches 5% significance or hits 150 observations. If you check the result after every observation, the real false-positive rate reaches 26.1%, more than five times what you would expect. That is a worst case, but even with 2 or 3 checks the real level already exceeds the nominal 5% (Evan Miller, “How not to run an A/B test”).
His advice is to fix the sample size in advance, wait until the test is over, and not use the dashboard to decide when to cut it. If you need to monitor continuously, sequential or Bayesian methods exist, but they require a different statistical design.
Example
Illustrative example: a store has a 3% checkout conversion rate. The team believes showing shipping costs before the last step will reduce abandonment. They calculate that detecting a half-point improvement with 80% power requires thousands of visits per variant. They let the test run for two full weeks, to cover weekends, and don’t look at the result until the sample is reached. In the end, the variant wins by 0.4 points, but the interval includes zero. The team doesn’t declare a winner. It treats the result as a lesson and moves on to test another hypothesis about trust at payment.
When NOT to do CRO
- With little traffic. Without volume, tests never reach a conclusion. Use qualitative research.
- When the problem is the product, not the screen. If the service doesn’t deliver what it promises, optimizing a button fixes nothing.
- As an excuse to skip research. Testing a hundred variants without understanding the problem is expensive and slow.
- With metrics that don’t reflect value. More clicks on a button don’t mean the customer is happier.
Improvement figures that circulate in the industry often come from vendors. Baymard, for example, publishes a checkout UX benchmark based on more than 44,000 usability scores: according to its article, 59% of desktop sites and 59% of mobile sites have a “mediocre” or worse checkout, and 17% of surveyed shoppers abandoned a purchase in the past three months because checkout was too long or complex. These figures describe the state of sites, not promised conversion gains, and the article itself does not quantify an uplift (Baymard, “Checkout UX”). Treat them as orientation.
Common mistakes
- Testing without a hypothesis: “let’s see what happens”.
- Changing several things at once and not knowing what worked.
- Stopping the test at the first favorable result.
- Measuring clicks instead of the final action.
- Declaring a winner with a difference that falls inside the margin of error.
How to start tomorrow
Pick the funnel step with the most abandonment. Write the hypothesis in one sentence using the format “if… we expect… because…”. Calculate the sample size with your baseline rate and the effect you actually care about. Write down the date you will look at the result, and don’t look before it. If the step is new to your team, start with the experimentation culture that makes it possible. And review the key metrics so you don’t optimize what doesn’t matter.