Knowledge Maxxing← Back to topic library
ML MaxxingBeginner60 sec
Test set
The test set is a portion of data held out entirely from training and tuning, used only once for a final, honest estimate of model performance.
Prerequisite: Validation set
What matters
Because it is never used to make any modeling decision, the test set is the only data that can honestly estimate how the model will perform on truly new data.
Example
After all tuning decisions are finalized using the validation set, a model is run once on the test set to report its expected real-world performance.
Common misconception
The test set should not be used repeatedly to guide decisions; doing so lets information leak in and makes its performance estimate optimistic and unreliable.
Recall question
Why can't you use the test set to tune hyperparameters?
Keywords and related concepts
test setholdoutfinal evaluationvalidation set