Bias in AI Models: Where It Comes From and What Design Can Do
Bias in an AI model is a tendency that gives worse results to certain groups. Real cases, where it comes from, and how to detect and reduce it through design.
What is bias in an AI model?
A language model has no intentions, but it learns patterns from text and data produced by people, and those materials contain prejudice and gaps. NIST, the US standards agency, published guidance on AI bias in 2022 (SP 1270). It starts from an uncomfortable idea: bias is neither new nor unique to AI, and it is not possible to achieve zero risk of bias in an AI system. So the job is not to “have a clean model” but to measure bias, reduce it and decide what level is acceptable.
An early example with embeddings (the numeric representation of words, see embeddings): in 2016, Bolukbasi and colleagues showed that embeddings trained on Google News articles exhibited gender stereotypes to a disturbing extent, in the authors’ own words. The paper’s title is the analogy itself: “Man is to Computer Programmer as Woman is to Homemaker?”. The authors proposed a method to remove those stereotypes while keeping useful associations, such as “queen” with the feminine.
Where does bias come from?
NIST sorts bias into three categories that mix inside a single product:
- Systemic. It comes from the procedures and practices of institutions that favor some groups and disadvantage others, without conscious prejudice. It is present in the data used for training.
- Statistical and computational. It appears when the sample does not represent the population. It happens, for example, when an algorithm is trained on one kind of data and cannot generalize to others.
- Human. These are systematic errors of judgment by the people who design, evaluate or use the system, including when they interpret its results.
The cases in the next section show how these appear in practice.
What documented cases exist?
| Case | What happened | What it illustrates |
|---|---|---|
| US health algorithm (Obermeyer et al., Science, 2019) | A commercial algorithm used healthcare cost as an indicator of need. At the same risk score, Black patients were considerably sicker. Fixing it would raise their share among those receiving extra help from 17.7% to 46.5%. | A bad proxy: cost was measured, not illness |
| Gender Shades (Buolamwini and Gebru, 2018) | Three commercial gender classification systems had error rates of up to 34.7% for darker-skinned women, against a maximum of 0.8% for lighter-skinned men. Two benchmarks in use at the time were 79.6% and 86.2% lighter-skinned subjects. | Evaluation data that is not diverse |
| Amazon recruiting (Reuters, 2018) | An experimental resume engine, started in 2014, penalized resumes containing the word “women’s” and downgraded graduates of two all-women’s colleges. It was trained on ten years of resumes, mostly from men. Recruiters looked at its recommendations but never relied solely on them. | Unbalanced historical data |
| LLM resume ranking (University of Washington, 2024) | Three language models ranking otherwise identical resumes that differed only in the name preferred white-associated names 85% of the time and Black-associated names 9%. Black male-associated names were never favored over white male-associated names. | A language model used to rank people |
The UW figures come from more than three million comparisons using 120 names, over 550 resumes and over 500 job listings across nine occupations. It is a study of names, not of a specific product in real use, so read it as a risk signal, not as your product’s error rate.
Why does it matter to a designer?
Because product decisions can amplify or reduce bias, and because the design team often decides what gets measured:
- Which indicator is the target. The health case shows the risk of picking a convenient number (cost) instead of the one that matters (need). Always ask what the indicator really measures.
- Who you test with. Gender Shades showed that an average can hide a group with a 34.7% error rate against 0.8% in another. Test with diverse cases and review results by group.
- Default examples. If a tool always shows the same kind of people or voices, it defines who counts as “normal” for the user.
- A reporting channel. If users cannot flag an unfair answer, the bias stays invisible to the team.
- Where AI decides and where a person does. In hiring, health or credit, keep a real human review, see human-in-the-loop.
How do you detect and reduce it?
These practices are design recommendations; NIST organizes the challenges into data, testing and evaluation, and human factors.
- Test with minimal pairs. Repeat the same input changing only one detail (the name, the gender, the language) and compare. This is the logic of the UW study.
- Compare results across groups, not only the average.
- Review outputs with people from the affected groups when possible.
- Provide a visible way to report unfair answers, and use those reports as a source of tests.
- Document what data was used, what was measured and what limits are known.
- Repeat the tests when the model or the prompt changes: a version change can move behavior.
When is bias not a problem?
Not every bias is harmful. A recipe assistant that prefers ingredients from one cuisine may be doing its job well if the product is built for that cuisine. The useful test is whether the bias causes harm or unjust inequality for someone, not whether the system has preferences. Nor should you promise neutrality: NIST itself rules out zero risk.
Example
Illustrative example: a team designs a feature that ranks candidates for a job. Before launch, it takes a single resume and submits it several times changing only the name, using names common in different groups. It compares the position each version receives. If the order changes systematically, the feature does not ship as it is. The team redesigns it so the system does not see the name (and repeats the test, because hiding it is not enough if other data act as a proxy), shows the recruiter the criteria used to rank, and leaves the final decision to the recruiter. It also adds a button to report a ranking that looks unfair.
Common mistakes
- Believing a model is neutral because it is “mathematical”.
- Measuring only the average quality.
- Testing with a single kind of user.
- Using a convenient indicator as if it were the real goal, such as cost instead of need.
- Ignoring bias in use: a system that is correct in one context can be unfair in another.
- Hiding error reporting in a support form.
Sources
- NIST, “Towards a Standard for Identifying and Managing Bias in Artificial Intelligence” (SP 1270, 2022): https://www.nist.gov/publications/towards-standard-identifying-and-managing-bias-artificial-intelligence
- Bolukbasi et al., “Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings” (2016): https://arxiv.org/abs/1607.06520
- Obermeyer et al., “Dissecting racial bias in an algorithm used to manage the health of populations”, Science, 2019 (PubMed record): https://pubmed.ncbi.nlm.nih.gov/31649194/
- Buolamwini and Gebru, “Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification” (2018): https://proceedings.mlr.press/v81/buolamwini18a.html
- Euronews, “Amazon scraps secret AI recruiting tool that showed bias against women” (based on Reuters, 2018): https://www.euronews.com/business/2018/10/10/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women
- University of Washington, “AI tools show biases in ranking job applicants’ names according to perceived race and gender” (2024): https://www.washington.edu/news/2024/10/31/ai-bias-resume-screening-race-gender/