Skip to main content

Bias in Medical AI

Medical AI systems inherit biases from their training data, their developers, and the healthcare systems they’re built to serve. The result: algorithms that work well for some patients and poorly for others, in ways that track existing health disparities.

finished certainty: likely importance: 8

In 2019, Ziad Obermeyer and colleagues published what became the most-cited paper on algorithmic bias in healthcare. They examined a commercial risk prediction algorithm used by major health systems to identify patients for enrollment in “high-risk care management” programs — intensive, proactive clinical support. The algorithm affected the care of roughly 200 million Americans annually. And it was systematically biased against Black patients.1

The algorithm hadn’t been designed to discriminate. It didn’t use race as an input. Its engineers had, by most accounts, followed standard machine learning best practices. The bias emerged from a seemingly reasonable design choice: using healthcare costs as a proxy for health needs. But costs reflect utilization, not need — and utilization is shaped by access, insurance, trust, and structural racism. The algorithm faithfully learned these patterns and reproduced them at scale.

This is the central problem of bias in medical AI. It’s rarely about malice or even carelessness. It’s about the gap between what we measure and what we care about — Goodhart’s Law applied to healthcare.

Sources of Bias

Bias in medical AI is not a single phenomenon. It enters at every stage of the machine learning pipeline, and mitigating it requires understanding each entry point:

Training data bias is the most commonly discussed and probably the most damaging. Medical datasets overwhelmingly represent white, male, relatively affluent populations treated at academic medical centers in wealthy countries. The UK Biobank is 94% white. MIMIC-III, the most widely used critical care dataset for ML research, comes from a single hospital in Boston. ImageNet, the foundation of most transfer learning in medical imaging, has well-documented demographic skew.2

Label bias is what Obermeyer found. When the outcome variable itself is a biased measurement of the thing we actually care about, no amount of model sophistication can fix it. Other examples: using ICD codes as ground truth for disease diagnosis (coding practices vary by institution and insurance status), using radiologist reads as ground truth for imaging AI (radiologist error rates differ across demographics), and using survival as a proxy for disease severity (survival reflects treatment quality, not just disease biology).3

Feature selection bias occurs when the input features themselves encode demographic information. ZIP code, insurance type, and medication history all correlate with race and socioeconomic status. Even after removing explicit demographic variables, models can reconstruct them from these proxies — a phenomenon called redundant encoding.4

Evaluation bias is the failure to measure performance separately across demographic groups. A model with 95% overall accuracy might achieve 98% accuracy for white patients and 85% for Black patients — and the aggregate metric hides the disparity. The FDA has been slow to require demographic subgroup analysis for AI/ML-based medical devices, though recent guidance is beginning to address this.5

Case Studies in Harm

The Obermeyer paper is the most famous example, but bias has been documented across medical AI:

  • Dermatology AI: Skin lesion classifiers trained predominantly on light-skinned patients perform significantly worse on darker skin tones. Daneshjou et al. (2022) found that leading dermatology AI models had up to 30% lower sensitivity for skin conditions on dark skin. Given that skin cancer is already diagnosed later in Black patients, AI that reinforces this disparity is actively dangerous.

  • Pulse oximetry: Not strictly an AI problem, but illustrative. Sjoding et al. (2020) showed in the New England Journal of Medicine that pulse oximeters overestimate oxygen saturation in patients with darker skin pigmentation. This hardware bias flows downstream into any AI system that uses SpO2 as an input feature — including sepsis prediction models and ventilator management algorithms.

  • Clinical NLP: Zhang et al. (2020) showed that clinical NLP models trained on electronic health records encoded racial biases present in clinical notes — including the well-documented tendency of physicians to use more negative language when describing Black patients. Models trained on these notes inherit and amplify these linguistic biases.

The Fairness-Accuracy Tradeoff (And Why It’s Overstated)

A common objection to bias mitigation is that it necessarily reduces overall accuracy — that there’s an inherent tradeoff between fairness and performance. This is technically true in some formulations. Chouldechova (2017) proved that certain fairness criteria are mathematically incompatible when base rates differ across groups.

But the practical relevance of this impossibility result is often overstated. In most real-world medical AI applications, the binding constraint is not a fundamental tradeoff between fairness and accuracy — it’s that nobody bothered to optimize for both. When Obermeyer et al. corrected the label definition in their risk prediction algorithm, they reduced racial bias by 84% while maintaining the same overall predictive performance. The bias wasn’t buying anything. It was just waste.

Moreover, the “accuracy” that biased models achieve is itself misleading. A model that works well for 80% of patients and poorly for 20% is not a 95%-accurate model — it’s a model with a hidden failure mode that disproportionately harms a specific population. Reporting only aggregate metrics is a form of statistical gerrymandering.

The more productive framing is: what fairness criteria are appropriate for this specific clinical application, and how can we optimize for them alongside (not instead of) accuracy? This is a design decision, not a mathematical inevitability. Rajkomar et al. (2018) provide a useful framework for thinking about fairness in clinical ML.

Mitigation Strategies

No single intervention eliminates bias. Effective mitigation requires action at every stage:

Data collection: Build more representative datasets. The All of Us research program, MIMIC-IV, and efforts like the Diverse Dermatology Images dataset are steps in the right direction. But diverse data alone is insufficient — it addresses sampling bias but not label bias or feature bias.

Algorithm design: Fairness-aware training methods — including adversarial debiasing, calibration constraints, and equalized odds regularization — can reduce disparities during model training. Chen et al. (2023) review these methods in the clinical context.6

Evaluation: Report performance stratified by demographics. Always. Mitchell et al. (2019) proposed “model cards” — standardized documentation that includes disaggregated evaluation metrics. This is the minimum viable transparency standard.7

Deployment monitoring: Bias can emerge or shift after deployment as patient populations change or clinical workflows evolve. Continuous monitoring for performance disparities — not just overall accuracy — is essential. Vokinger et al. (2021) argue for ongoing post-market surveillance of AI devices, analogous to pharmacovigilance for drugs.

The Structural Problem

The deepest challenge is that medical AI bias is downstream of healthcare bias, which is downstream of social inequality. An algorithm trained on data from an inequitable healthcare system will, by default, reproduce that inequity — regardless of how technically sophisticated it is. You cannot machine-learn your way out of structural racism.8

The most dangerous scenario is not that medical AI will be obviously racist. It’s that it will be subtly and systematically biased in ways that are invisible to aggregate performance metrics, approved by regulators who don’t require subgroup analysis, deployed by clinicians who trust the algorithm, and experienced by patients who have no idea the algorithm exists. This is not a hypothetical. It’s the default trajectory.

Further Reading