The big idea: Dimensionality reduction means describing the same data with fewer features, keeping as much of the variation as possible.
Not just dropping columns — building a smaller set of new ones that carry most of the information.
The curse of dimensionality: As features are added, the space the data lives in grows exponentially, so the same records become sparser and sparser inside it.
Everything ends up roughly equidistant from everything else, distances stop meaning anything, and the model needs exponentially more data to see a pattern.
Free preview
This is the free notes preview
You're reading the free notes. Aimnova Pro unlocks the full study experience — and you can try it with your first topic free to keep:
- FlashcardsLock in vocabulary and key terms with spaced repetition.
- Practice questionsAnswer exam-style questions and get instant AI marking.
- Mock exams & past-paper vaultSit full mocks and see exactly how examiners award marks.
- Personalised study planA daily plan built around your exam date and weak areas.
| Feature selection | Feature extraction | |
|---|---|---|
| Does | Keeps a subset of the original columns | Builds new columns from combinations |
| Result | Original features, fewer of them | New features that did not exist before |
| Interpretable | Yes — they are the original meanings | Often not — a component is a mixture |
| Example | Drop shoe size | PCA combining height and weight into one axis |
PCA in one sentence: Principal component analysis finds the directions along which the data varies most, and uses those as the new axes — so the first few components carry most of the variation, and the rest can be dropped.
You need the idea, not the mathematics.
The cost is interpretability: A selected feature still means "income". A principal component means "0.4 × income + 0.3 × age − 0.2 × distance" — mathematically useful and impossible to explain to the person it was used on.
Where a decision must be justified, that cost can be decisive.
Know your predicted grade
Take timed mock exams and get detailed feedback on every answer. See exactly where you're losing marks.
Practical
- Faster training and inference
- Less memory, which can be what makes a model fit on a phone
- Less storage for the dataset itself
Statistical
- Less overfitting — fewer dimensions to fit noise in
- Removes correlated features that were saying the same thing twice
- Distances regain meaning, so clustering works again
Human
- Data with 2 or 3 dimensions can be plotted
- Which is often how a pattern is first noticed at all
- But a component cannot be named in plain words
Say what is lost: Reduction is lossy by construction. A full-mark answer names the benefit and then the cost: some variation is discarded, and the new features are usually not interpretable.
How this is tested — you must explain why many features is a problem in itself, and weigh reduction against interpretability. It comes up two ways:
Paper 1 Section A
- Describe the importance of dimensionality reduction, 3-4 marks
- State what the curse of dimensionality means
- Distinguish selection from extraction
Paper 1 Section B — case study
- Justify reduction for a scenario
- Explain the trade-off against explainability
The classic trap: Treating it as the same thing as feature selection. Selection keeps some original columns; extraction creates new ones from combinations — and only the second sacrifices interpretability.
Projecting onto the direction the data actually varies along.
Interactive diagram
Explore the labelled diagram, charts and maps for this topic in full study mode.
A medical imaging model has 4,000 features per scan and only 800 training scans. Describe why dimensionality reduction is important here, and one reason to be cautious.
Model answer plan
See the mark-by-mark plan — for / against / judgement, with marking guidance — in study mode.