Discrimination does not enter an automated system at one point. It enters at four, in order — and each one needs a different fix, which is why “make the data better” so often changes nothing.
The data, the label, the test and the use — four entry points and four different fixes.
Interactive diagram
Explore the labelled diagram, charts and maps for this topic in full study mode.
Free preview
This is the free notes preview
You're reading the free notes. Aimnova Pro unlocks the full study experience — and you can try it with your first topic free to keep:
- FlashcardsLock in vocabulary and key terms with spaced repetition.
- Practice questionsAnswer exam-style questions and get instant AI marking.
- Mock exams & past-paper vaultSit full mocks and see exactly how examiners award marks.
- Personalised study planA daily plan built around your exam date and weak areas.
The data — who is barely in it
A group with few examples is handled worst, and that is knowable before training begins by counting.
The label — what it predicts
A system trained to predict who was hired learns who was hired before, including patterns nobody chose to keep.
The test — how fairness was checked
Testing one characteristic at a time passes systems that fail badly at the intersection of two.
The fourth entry point is the use: A score meant as advice becomes a decision when the person reading it has thirty seconds and has to justify disagreeing. No change to the model reaches that one.
Never wonder what to study next
Get a personalized daily plan based on your exam date, progress, and weak areas. We'll tell you exactly what to review each day.
Real-world examples you can name
COMPAS recidivism scoring — ProPublica investigation May 2016
A commercial risk-scoring tool used in some US courts to estimate how likely a defendant was to reoffend. Journalists found that among people who did not go on to reoffend, Black defendants were roughly twice as likely as white defendants to have been labelled high risk. The company disputed the measure of fairness used, and the argument that followed showed that competing definitions of a fair algorithm cannot all be satisfied at once.
Who it affected: Defendants whose bail and sentencing decisions were informed by the score.
Amazon's scrapped CV-screening tool — reported October 2018
An experimental hiring tool was trained on a decade of CVs submitted to the company, most of them from men. It learned to downgrade CVs containing the word 'women's' and graduates of two women's colleges. The project was abandoned.
Who it affected: Women applying for technical roles, and every organization that assumed historical data was neutral.
San Francisco's facial recognition ban — May 2019
The first major US city to bar its own agencies, including the police, from using facial recognition — a rule about government use, not about the technology existing. Several other cities followed, and a few later relaxed theirs.
Who it affected: Residents, and police forces that had to find other methods.
| Entry point | The fix | What it does not fix |
|---|---|---|
| The data | Collect more, deliberately | A label that encodes past behaviour |
| The label | Predict a different thing | A test that averages groups away |
| The test | Report accuracy by combination | How the output is used |
| The use | Give the reader time and a real option to disagree | Anything in the model itself |
How this is tested — HL asks for an intervention against algorithmic discrimination, and the four entry points make that answer precise. It comes up two ways:
Paper 1 Section B — extended response
- Extended response on rights and automated systems
- Concepts — identity, power — named explicitly
Paper 3 — the intervention paper
- Recommend a measure against discrimination in a described system
- Justify which entry point it reaches
The trap: “fix the training data”: More data fixes the first entry point and none of the other three. Say which one you are addressing and why that one.
Explain one reason why improving the training data may not remove discrimination from a system.
Model answer plan
See the mark-by-mark plan — for / against / judgement, with marking guidance — in study mode.
Recommend one measure to reduce discrimination by automated hiring systems, and justify it.
Model answer plan
See the mark-by-mark plan — for / against / judgement, with marking guidance — in study mode.