Key Idea: A model learns whatever is in the data — including its faults, and no fault raises an error. Most of the work in machine learning happens before training: fixing what is wrong, choosing what the model sees, and cutting what it does not need.
Paper 1
- Define a method; compute a measure.
- Identify which approach a scenario needs.
Paper 2
- Evaluate a model against an alternative.
- Reason about a reward or fitness function.
Both
- Name what is given up, not only what is gained.
🧼 Cleaning
| Fault | What it does to the model |
|---|---|
| Missing values | A 0 meaning "not recorded" is learned as a real, impossible value |
| Duplicates | That record's characteristics carry double weight |
| Inconsistent formats | "UK" and "United Kingdom" become two different countries |
| Outliers | An error should go; a genuine extreme may be the target |
Important: If income is missing more often for low earners, deleting those rows makes the data richer than reality. Ask WHY a value is missing before deciding what to do with it.
🎯 Feature selection
What to drop, and why
- Identifier — unique per row, so the model memorises and generalises nothing
- Leaking — contains the answer, or is unknown at prediction time. Test accuracy looks superb and then collapses
- Redundant — height in cm and in inches say the same thing twice
- Constant — no information at all
- Removing a sensitive attribute does not remove it: postcode and school are proxies
Would we actually know this at the moment the prediction has to be made? If not, it is leakage, however well it scores in testing.
📉 Dimensionality reduction
| Selection | Extraction (PCA) | |
|---|---|---|
| Keeps | Original columns | New ones built from combinations |
| Interpretable | Yes | No — a component is a mixture |
| Reduces further | Less | More — correlated features merge |
| Prefer when | A decision must be explained | Only predictive power matters |
Important: Each extra feature grows the space exponentially, so the same records become sparse inside it. Distances stop meaning anything, and exponentially more data is needed to see a pattern at all.
📝 Exam-style questions
A recruitment model is trained on ten years of hiring decisions and scores 96% on held-out data. The company removes the gender field before training. Explain why this does not make the model fair, and what the 96% may be hiding.
🔒 Model answer plan
See the mark-by-mark plan — for / against / judgement, with marking guidance — in study mode.
A dataset has 3,000 features and 500 records. Describe two problems this creates and one way to address them.
🔒 Model answer plan
See the mark-by-mark plan — for / against / judgement, with marking guidance — in study mode.
✅ Quick check
Cover the answers.
Why is a recorded 0 dangerous? It may mean not recorded. The model learns from an impossible value, and nothing reports it.
What is data leakage? A feature containing the answer, or unavailable at prediction time. Test accuracy looks superb, then collapses.
Does dropping a sensitive attribute remove bias? No — other fields proxy for it and the model finds those instead.
What is the curse of dimensionality? More features means an exponentially larger space, so data becomes sparse and distances stop meaning anything.
Selection or extraction — which keeps meaning? Selection. Extraction builds mixtures that cannot be explained to the person a decision was made about.
Exam tips
- Name a SPECIFIC fault and trace it to a specific effect on the model.
- Ask of every feature: would we know this at prediction time?
- Suspiciously high accuracy is a leakage symptom, not a success.
- Proxies survive the removal of a sensitive attribute. Name them.
- Reduction is lossy — say what is given up, not just what is gained.