The big idea: A feature is one input column. Feature selection is deciding which ones the model is given.
More is not better: an irrelevant feature is noise the model can still fit, and a leaking feature is a disaster.
Why fewer features often wins
- Less overfitting — fewer chances to fit noise that will not recur
- Faster to train and to run, on less data
- Explainable — a model with six inputs can be reasoned about; one with six hundred cannot
- Cheaper to collect — every feature is data someone has to gather and store
- Fewer places for bias to enter through a proxy
Free preview
This is the free notes preview
You're reading the free notes. Aimnova Pro unlocks the full study experience — and you can try it with your first topic free to keep:
- FlashcardsLock in vocabulary and key terms with spaced repetition.
- Practice questionsAnswer exam-style questions and get instant AI marking.
- Mock exams & past-paper vaultSit full mocks and see exactly how examiners award marks.
- Personalised study planA daily plan built around your exam date and weak areas.
| Kind | Problem | Example |
|---|---|---|
| Irrelevant | No real relationship to the target | Shoe size when predicting exam grades |
| Redundant | Duplicates another feature | Height in cm and height in inches |
| Constant | Same value throughout — no information | A country column when all records are from one country |
| Identifier | Unique per row, so it memorises | Customer ID |
| Leaking | Contains the answer | "Refund issued" when predicting returns |
Data leakage is the one that fools you: A leaking feature makes accuracy look superb in testing and then collapse in production, because the feature was not available at the moment the prediction has to be made.
Ask of every feature: would we actually know this at prediction time?
An ID memorises rather than learns: A customer ID is unique to one row, so a model can map each ID to its answer and score perfectly on training data while learning nothing generalisable.
It is overfitting in its purest form.
Practice with exam-style questions
Answer exam-style questions and get AI feedback that shows you exactly what examiners want to see in a full-marks response.
Domain knowledge
- A person who understands the problem removes what cannot matter
- Catches leakage that statistics will not
- The cheapest and most reliable filter
Statistical
- Correlation with the target — weak ones are candidates to drop
- Correlation between features finds redundancy
- But correlation is not causation, and misses combinations
Model-based
- Train, read the feature importances, drop the lowest, retrain
- Measures what actually helps rather than what looks like it should
- Costs many training runs
The three strategies the guide names
| Strategy | How it picks features | Trade-off |
|---|---|---|
| Filter method | Scores each feature on its own with a statistic, such as its correlation with the target, before any model is trained; keeps the best-scoring | Fast and simple, but ignores how features work together |
| Wrapper method | Trains the model on different subsets of features and keeps the subset that gives the best results | Finds the most informative set for that model, but training many times is slow |
| Embedded method | Selection happens during training itself, e.g. a decision tree splits only on the most useful features, or penalties shrink useless weights to zero | Faster than wrappers; tied to one kind of model |
Removing a sensitive attribute is not selection for fairness: Dropping ethnicity does not remove ethnicity from the model — postcode, school and employment history correlate with it and act as proxies.
Say this explicitly: it is examined under ethics as well as here.
How this is tested — you must judge whether a feature belongs, and justify removal by its effect on the model. It comes up two ways:
Paper 1 Section A
- Describe the role of feature selection, 3-4 marks
- Identify a feature that should be removed
- State one benefit of using fewer features
Paper 1 Section B — case study
- Explain data leakage in a scenario
- Justify keeping or dropping a specific feature
The classic trap: Assuming more data is always better. An identifier lets the model memorise, and a leaking feature gives near-perfect test accuracy that vanishes in production.
Columns struck out one reason at a time — irrelevant, identifier, duplicate.
Interactive diagram
Explore the labelled diagram, charts and maps for this topic in full study mode.
A bank predicts loan default from: customer ID, income, age, postcode, number of previous defaults, and whether debt collection was started. Describe which features should be removed and why.
Model answer plan
See the answer broken into clear points, with guidance on how each point earns marks.