AI ART Discover · Learn · Create
Plain English Definition

What does “Reward Model” mean?

Core Definition

A model that scores outputs based on preference signals, often used in RLHF pipelines.

Operational Mechanism

How Reward Model Works

1

Prompt + candidate answer

2

Reward model

3

Score: how much people would prefer it

4

Used to guide training.

Real-World Scenario

Practical Example

A reward model trained on thousands of human rankings can give two draft replies different scores. During tuning, the chatbot is nudged toward replies that score higher.

Core Significance

Why It Matters

It stands in for human judgment at scale. If the reward model is flawed, the tuned model can learn to exploit its flaws.

Expert Q&A

Common Questions About Reward Model

Q: How is it trained?

On examples where people chose the better of two or more answers.

Q: Reward model vs human preference model?

They are closely related. A human preference model is one trained to predict what people prefer.

Q: What is the risk?

Over-optimizing against it can produce answers that score highly but are not actually good.