Knowledge MaxxingCalibration
Calibration is a fairness property where a model's predicted probability actually matches the real observed outcome rate for every group, not just overall.
Prerequisite: Equalised odds
What matters
A model can be well calibrated overall while being poorly calibrated for a specific subgroup, meaning its stated confidence is misleading for exactly that group.
Example
If a model predicts '70% likely to default' for a group, calibration means roughly 70% of that specific group's borrowers actually do default.
Common misconception
Good overall calibration does not guarantee good calibration for every subgroup individually; a model can look well calibrated in aggregate while being off for one particular group.
Recall question
Why can a model be well calibrated overall but still poorly calibrated for one subgroup?