What Is an Example of Bias and Discrimination in AI? 7 Real Cases

AI bias isn't abstract — it's happening in production systems right now. These aren't edge cases or hypotheticals. They're real AI systems, built by well-resourced teams, that produced discriminatory outcomes at scale. Under Colorado SB 24-205, every one of these scenarios would trigger enforcement action if it involved a Colorado consumer. Understanding what went wrong in each case is the first step to ensuring your AI doesn't repeat these mistakes.

Related: bias audit guide · algorithmic bias explained · what is an AI bias audit

**What happened:** In 2018, Amazon scrapped an AI recruiting tool after discovering it systematically downgraded resumes from women. The system was trained on 10 years of historical hiring data — a decade in which Amazon's tech workforce was predominantly male. **The bias:** The AI learned that male-associated patterns predicted "success" (because historically, men were hired more). It penalized resumes containing the word "women's" (as in "women's chess club captain") and downgraded graduates of all-women's colleges. **Root cause:** Historical training data bias. The AI replicated the very patterns the company was trying to move past. **SB 24-205 relevance:** Hiring is a "consequential decision." Any Colorado business using AI in recruiting would need to audit for exactly this kind of gender bias — and document the results.

**What happened:** In 2019, Apple Card (issued by Goldman Sachs) was found to offer women significantly lower credit limits than men — even when the women had higher credit scores and identical financial profiles. Steve Wozniak publicly confirmed his wife received a credit limit 10x lower than his despite shared assets. **The bias:** The underwriting algorithm used features that correlated with gender without using gender directly — classic proxy discrimination. **Root cause:** Proxy variables and lack of bias testing across protected classes before launch. **SB 24-205 relevance:** Lending decisions are explicitly "consequential" under the law. Colorado lenders must audit AI credit decisions for gender, race, and other protected class disparities.

**What happened:** A 2019 Science journal study revealed that a healthcare algorithm used by hospitals nationwide to identify patients needing extra care was systematically biased against Black patients. At a given risk score, Black patients were significantly sicker than white patients with the same score. **The bias:** The algorithm used healthcare spending as a proxy for health needs. Because Black patients historically had less access to healthcare (and therefore lower spending), the AI concluded they were "healthier" — a dangerous inversion. **Root cause:** Measurement bias. Healthcare spending ≠ health needs. The metric measured access, not sickness. **SB 24-205 relevance:** Healthcare AI making treatment or triage decisions is a consequential decision. This exact scenario — AI deprioritizing care for protected groups — is what the law was designed to prevent.

**What happened:** ProPublica's 2016 investigation found that the COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) algorithm was twice as likely to falsely label Black defendants as high-risk compared to white defendants, while also being twice as likely to falsely label white defendants as low-risk. **The bias:** Different error rates by race — false positive rates were dramatically different across racial groups. **Root cause:** Training data reflected historical criminal justice disparities. The algorithm encoded structural inequality as "risk." **SB 24-205 relevance:** While criminal justice AI has specific carve-outs, any AI used in legal or government decisions affecting individuals falls under the law's scope.

**What happened:** MIT researcher Joy Buolamwini's landmark 2018 "Gender Shades" study found that commercial facial recognition systems from IBM, Microsoft, and Face++ had error rates of 0.8% for light-skinned males but up to 34.7% for dark-skinned females — a 43x accuracy gap. **The bias:** Dramatically different performance across intersections of race and gender. **Root cause:** Training data overwhelmingly featured light-skinned male faces. The AI simply never learned to recognize darker-skinned faces with the same accuracy. **SB 24-205 relevance:** Any AI system used for identity verification, access control, or security that touches Colorado consumers must demonstrate equal performance across demographics.

**What happened:** Multiple AI-powered tenant screening tools have been found to disproportionately reject applicants of color by weighing factors like credit history, eviction records, and criminal background — all of which correlate strongly with race due to systemic inequality. **The bias:** Facially neutral criteria that produce racially disparate outcomes at scale. **Root cause:** Proxy discrimination through features correlated with protected classes. **SB 24-205 relevance:** Housing is an explicitly listed consequential decision. Colorado property managers using AI screening must audit for disparate impact across race, ethnicity, and other protected classes.

**What happened:** Consumer Reports and ProPublica investigations found that auto insurance pricing algorithms charged drivers in predominantly minority neighborhoods higher premiums than drivers in majority-white neighborhoods — even after controlling for driving records, claims history, and all other actuarial factors. **The bias:** Geographic and socioeconomic proxy variables producing racial disparities in pricing. **Root cause:** Using zip code, credit score, and other proxies that correlate with race. **SB 24-205 relevance:** Insurance underwriting and pricing using AI is a consequential decision. Colorado insurers must audit pricing algorithms for discriminatory proxy effects.

Every one of these examples shares common patterns: 1. **The organizations involved weren't trying to discriminate.** Bias was an emergent property of the system. 2. **The AI was technically accurate** — it optimized for the metric it was given. The metric was the problem. 3. **Bias was discoverable** through proper testing. None of these cases were undetectable. 4. **The cost of not testing far exceeded the cost of testing.** Reputational damage, lawsuits, regulatory action, and lost trust. CO-AIMS exists so your business doesn't become the next case study. Automated bias audits, continuous monitoring, and documented evidence trails — the entire compliance stack, starting at $199/month.