The big idea: Dimensionality reduction means describing the same data with fewer features, keeping as much of the variation as possible.
Not just dropping columns — building a smaller set of new ones that carry most of the information.
The curse of dimensionality: As features are added, the space the data lives in grows exponentially, so the same records become sparser and sparser inside it.
Everything ends up roughly equidistant from everything else, distances stop meaning anything, and the model needs exponentially more data to see a pattern.
The curse of dimensionality, point by point
| Problem | What goes wrong as features are added |
|---|---|
| Overfitting | More features give the model more chances to fit noise in the training data |
| Computational complexity | Training and prediction take longer with every extra feature |
| Data sparsity | The same records are spread thinly over a much bigger space |
| Distance metrics | Points end up almost equally far apart, so methods like k-NN stop working |
| Data visualisation | Humans can plot 2 or 3 dimensions, not 50, so patterns cannot be seen |
| Sample size | Far more records are needed to cover the space well |
| Memory usage | Each extra column costs storage for every record |
Dimensionality reduction cuts the number of variables while preserving the relevant aspects of the data, e.g. keeping the 10 measurements that carry most of the information out of 200.
Free preview
This is the free notes preview
You're reading the free notes. Aimnova Pro unlocks the full study experience — and you can try it with your first topic free to keep:
- FlashcardsLock in vocabulary and key terms with spaced repetition.
- Practice questionsAnswer exam-style questions and get instant AI marking.
- Mock exams & past-paper vaultSit full mocks and see exactly how examiners award marks.
- Personalised study planA daily plan built around your exam date and weak areas.
| Feature selection | Feature extraction | |
|---|---|---|
| Does | Keeps a subset of the original columns | Builds new columns from combinations |
| Result | Original features, fewer of them | New features that did not exist before |
| Interpretable | Yes — they are the original meanings | Often not — a component is a mixture |
| Example | Drop shoe size | Combining height and weight into one ‘body size’ feature |
The cost is interpretability: A selected feature still means "income". An extracted feature means "0.4 × income + 0.3 × age − 0.2 × distance" — mathematically useful and impossible to explain to the person it was used on.
Where a decision must be justified, that cost can be decisive.
Know your predicted grade
Take timed mock exams and get detailed feedback on every answer. See exactly where you're losing marks.
Practical
- Faster training and inference
- Less memory, which can be what makes a model fit on a phone
- Less storage for the dataset itself
Statistical
- Less overfitting — fewer dimensions to fit noise in
- Removes correlated features that were saying the same thing twice
- Distances regain meaning, so clustering works again
Human
- Data with 2 or 3 dimensions can be plotted
- Which is often how a pattern is first noticed at all
- But a component cannot be named in plain words
Say what is lost: Reduction is lossy by construction. A full-mark answer names the benefit and then the cost: some variation is discarded, and the new features are usually not interpretable.
How this is tested — you must explain why many features is a problem in itself, and weigh reduction against interpretability. It comes up two ways:
Paper 1 Section A
- Describe the importance of dimensionality reduction, 3-4 marks
- State what the curse of dimensionality means
- Distinguish selection from extraction
Paper 1 Section B — case study
- Justify reduction for a scenario
- Explain the trade-off against explainability
The classic trap: Treating it as the same thing as feature selection. Selection keeps some original columns; extraction creates new ones from combinations — and only the second sacrifices interpretability.
Projecting onto the direction the data actually varies along.
Interactive diagram
Explore the labelled diagram, charts and maps for this topic in full study mode.
A medical imaging model has 4,000 features per scan and only 800 training scans. Describe why dimensionality reduction is important here, and one reason to be cautious.
Model answer plan
See the answer broken into clear points, with guidance on how each point earns marks.