What does “Reward Model” mean?
A model that scores outputs based on preference signals, often used in RLHF pipelines.
How Reward Model Works
Prompt + candidate answer
Reward model
Score: how much people would prefer it
Used to guide training.
Practical Example
A reward model trained on thousands of human rankings can give two draft replies different scores. During tuning, the chatbot is nudged toward replies that score higher.
Why It Matters
It stands in for human judgment at scale. If the reward model is flawed, the tuned model can learn to exploit its flaws.
Common Questions About Reward Model
Q: How is it trained?
On examples where people chose the better of two or more answers.
Q: Reward model vs human preference model?
They are closely related. A human preference model is one trained to predict what people prefer.
Q: What is the risk?
Over-optimizing against it can produce answers that score highly but are not actually good.