The Larger World ArchiveClaims · Sources · Evidence
Internet, media and modern lifeLive debateMachine learning · 2016 – present

Algorithmic Fairness

In 2016 three papers proved, within months of each other, that several intuitive definitions of fairness cannot hold together. The field has been arguing about what to do with that ever since.

A decision flowchart. With unequal base rates, calibration and equal error rates cannot bo
A decision flowchart. With unequal base rates, calibration and equal error rates cannot both hold. Image: Kevin Scasny · Public domain — Wikimedia Commons.

The impossibility results

Kleinberg, Mullainathan and Raghavan, and independently Chouldechova, showed that when groups have different base rates, a predictor cannot simultaneously satisfy calibration within groups, equal false positive rates and equal false negative rates.

The three properties are each individually compelling. Calibration says a score of 0.7 means a 70 per cent chance for everyone. Equal false positive rates say innocent people are wrongly flagged at the same rate in each group. Equal false negative rates say the same for missed cases. Pick two.

Hardt, Price and Srebro's equalised odds framework made the trade-off explicit and computable: you can adjust a classifier's thresholds post hoc to equalise error rates, at a cost in overall accuracy.

The COMPAS case

ProPublica's 2016 analysis of the COMPAS recidivism tool found that Black defendants were more likely than white defendants to be falsely flagged as high risk, and white defendants more likely to be falsely cleared.

Northpointe, the vendor, replied that the tool was calibrated: among defendants with the same score, the reoffending rate was similar across groups. Both analyses were correct, which is exactly the theorem.

The case is the canonical illustration because it shows that the dispute was never about arithmetic. It was about which error is worse, and that is a normative question the data cannot answer.

What the field now agrees on

The impossibility is proven under stated assumptions, and it is not a limitation of current methods — it follows from the statistics.

Choosing a definition is a policy decision, and it should be made by the institution accountable for the harm, not left to the vendor's default.

Measurement matters: the label a model predicts is often a proxy — arrests rather than crimes — and the proxy's bias is upstream of any fairness constraint.

No definition is neutral. Demographic parity ignores base rates; calibration can entrench them. Each encodes a view about what equal treatment means.

The archive labels this live debate because the mathematics is settled and the ethics and law are not.

Why it is in the archive

Because it is a case where a rigorous result killed a naive hope — that there is a technical fix for a political problem — and the field has mostly absorbed that honestly.

Related: open versus closed models, echo chambers.

Sources for this page

  • Wikipedia, “Fairness (machine learning)” — the definitions and the impossibility resultsen.wikipedia.org
  • Kleinberg, Mullainathan & Raghavan (2016), “Inherent Trade-Offs in the Fair Determination of Risk Scores”arxiv.org
  • Chouldechova (2017), “Fair prediction with disparate impact”doi.org
  • ProPublica (2016), “Machine Bias” — the COMPAS investigationwww.propublica.org