# A/B Testing for UX Designers

> Running controlled experiments to validate design changes—the difference between data-driven and guessing.

*Tags: ux, research, mid-level*

---



> [!info] Quick Definition
> Running controlled experiments to validate design changes—the difference between data-driven and guessing.


## What is A/B Testing?

A/B testing (also called split testing) shows version A to half your users and version B to the other half. You measure which version performs better. If version B has a higher conversion rate, version B wins. It's not about opinion; it's about data.

A/B testing isn't just for marketers. Designers use A/B tests to validate design decisions. Does a red button convert better than a blue button? A/B test it. Does a longer form reduce signups? A/B test it. Does highlighting the primary action increase clicks? A/B test it.

**One sentence punch:** A/B testing replaces opinions with evidence—the most powerful tool you have as a designer.**

## Why is it important?

- **Removes Opinions from Design:** Designers argue about button colors. A/B testing resolves it. Data wins. This stops endless debates.
- **Validates Assumptions:** You assume users want X. Test it. Sometimes reality surprises you. A/B tests reveal truth faster than debates.
- **Optimizes Conversion:** Small changes compound. A 2% improvement in signup conversion is small. Across millions of users, it's significant. A/B testing finds these wins.
- **Prevents Bad Launches:** You're uncertain about a redesign. A/B test it before full launch. If it performs worse, you've avoided damaging live users.

## How to Run an A/B Test

1. **Define your hypothesis** — "Changing the button from blue to red will increase clicks." Specific hypothesis, not vague.
2. **Define success metric** — What are you measuring? Clicks? Conversions? Time on page? Be specific.
3. **Determine sample size** — How many users do you need for statistical significance? Tools like Statsig calculate this.
4. **Create variants** — Version A (control): the current design. Version B (variant): your change. One change only. Multiple changes muddy results.
5. **Run the test** — Send traffic to both versions. 50/50 split. Run until you reach statistical significance (usually 95% confidence).
6. **Analyze results** — Did B perform better? By how much? Is the difference statistically significant or random variation?
7. **Implement winner** — If B is statistically better, launch B to all users. If A is better, stick with A. If results are tied, keep A (simpler is better).

## A/B Test Best Practices

- **One variable at a time** — Change the button color, not the button color and text. Multiple changes confound results.
- **Run until statistical significance** — Don't stop early if B looks better. Random variation can fool you. Wait for confidence threshold (95%).
- **Avoid peeking** — Checking results daily introduces bias. Let the test run to completion.
- **Test the right metric** — Users clicking a button is good, but do they convert? Clicks aren't conversions. Measure what matters.
- **Run tests sequentially, not parallel** — Multiple simultaneous tests can interfere with each other. Run one, learn, run the next.

## A/B Test Ideas for Designers

- Button color, size, or text
- Form field lengths (long form vs short form)
- Hero image vs hero video
- Call-to-action placement
- Navigation style
- Heading copy
- Page layout
- Empty states

## Mentor Tips

- **First tip: Statistical significance isn't guaranteed.** You might run a test, and A and B are statistically equivalent. Both are equally good. This is data, too. It simplifies decisions.
- **Avoid the novelty effect.** Sometimes B wins not because it's better, but because it's new. Run tests long enough for novelty to wear off.
- **Share surprising results with teams.** When data contradicts designer opinions, share it. "My hypothesis was wrong, but data shows..." builds credibility.
- **Keep winning variants.** After a test, lock in the winner. Don't revert. Iterating on winners accelerates improvement.

## Resources and Tools

- **Books:** "Trustworthy Online Controlled Experiments" by Kohavi, Tang, and Xu, "Testing and Optimizing Online Ads" by Jeevan Yeatts
- **Tools:** VWO, Optimizely, [[Figma]] for variant design, Statsig or SplitBee for analysis
- **Articles:** A/B testing guides on Conversion Rate Experts, experimentation articles on [[UX Collective]]

---

Source: https://www.fernandoux.com/en/wiki/research/ab-testing/