Knowledge MaxxingPreference tuning
Preference tuning adjusts a model using human judgments about which of several candidate responses is better, to steer its output toward preferred styles and behaviors.
Prerequisite: Instruction tuning
What matters
This step is what pushes a model toward answers people actually find helpful, honest, and safe, beyond simply being able to follow instructions correctly.
Example
Given two possible answers to the same question, human raters mark which one is more helpful, and the model is adjusted to favor generating responses like the preferred one.
Common misconception
Preference tuning does not guarantee a factually correct answer; it optimizes for what raters judged as preferable, which is not the same as verified accuracy.
Recall question
What kind of signal does preference tuning use to adjust a model's behavior?