The big idea: A parameter is learned from the data during training. A hyperparameter is a setting chosen before training, which controls how the learning happens.
The model cannot learn its own hyperparameters, because they decide what learning even means.
| Parameter | Hyperparameter | |
|---|---|---|
| Set by | Training | You, in advance |
| Examples | Weights, the gradient of a line | Learning rate, tree depth, k in k-NN |
| Changes during training? | Constantly | No — fixed for the run |
| Found by | Minimising error on the data | Searching and comparing results |
Free preview
This is the free notes preview
You're reading the free notes. Aimnova Pro unlocks the full study experience — and you can try it with your first topic free to keep:
- FlashcardsLock in vocabulary and key terms with spaced repetition.
- Practice questionsAnswer exam-style questions and get instant AI marking.
- Mock exams & past-paper vaultSit full mocks and see exactly how examiners award marks.
- Personalised study planA daily plan built around your exam date and weak areas.
What each controls
- Learning rate — how big a step each update takes. Too high overshoots the minimum; too low never arrives
- Number of epochs — how many passes over the data. Too few underfits; too many overfits
- Tree depth — how many questions a decision tree may ask. Deeper fits more, including noise
- k in k-NN — how many neighbours vote. Small k is sensitive to noise; large k blurs the boundaries
- Batch size — how many examples per update. Affects both speed and stability
Too high and too low both fail, differently: It is rarely "bigger is better". A learning rate that is too high bounces past the minimum and never settles; too low and training is still creeping towards it when time runs out.
An answer that names a failure at both ends shows the parameter is understood.
Memorize terms 3x faster
Smart flashcards show you cards right before you forget them. Perfect for definitions and key concepts.
Grid search
- Try every combination from a list per hyperparameter
- Thorough, and exponentially expensive
- 3 settings × 4 hyperparameters is 81 training runs
Random search
- Sample combinations at random within ranges
- Usually finds something good far sooner
- Because only a few hyperparameters actually matter much
Cross-validation
- Split into k folds, train on k−1, test on the held-out one, rotate
- Every record is used for testing exactly once
- Far more reliable than one arbitrary split
Three sets, not two: Tuning against the test set leaks it: you end up choosing settings that suit that particular data, and the reported score is optimistic.
Training to learn parameters · validation to choose hyperparameters · test, used once, to report. Keeping the test set untouched is what makes the final number honest.
How this is tested — you must distinguish a hyperparameter from a parameter, and explain why tuning needs its own data split. It comes up two ways:
Paper 1 Section A
- Explain the role of hyperparameter tuning, 3-4 marks
- Identify a hyperparameter from a list
- State the effect of a learning rate that is too high
Paper 1 Section B — case study
- Explain why a test set must not be used for tuning
- Justify cross-validation over a single split
The classic trap: Calling a learned weight a hyperparameter. The test is simple: was it chosen before training started? If the model worked it out, it is a parameter.
Where the validation curve bottoms out, and why a third set exists.
Interactive diagram
Explore the labelled diagram, charts and maps for this topic in full study mode.
A team tunes a model by trying settings and keeping whichever scores best on the test set, reporting 94%. Explain what is wrong and what they should have done.
Model answer plan
See the mark-by-mark plan — for / against / judgement, with marking guidance — in study mode.