What does “HumanEval” mean?
A benchmark of Python programming problems used to measure code-generation correctness via unit tests.
How HumanEval Works
Function signature and description
Model writes the code
Run hidden unit tests
Pass or fail per problem.
Practical Example
A problem gives a function name and a docstring such as 'return the sum of even numbers in a list'. The model writes the body, and tests check it against several inputs.
Why It Matters
HumanEval offered a simple way to measure whether generated code works. Its 164 problems are small, so strong models have largely saturated it.
Common Questions About HumanEval
Q: Who introduced it?
OpenAI researchers in 2021, alongside the Codex model.
Q: How many problems?
164 hand-written Python programming problems.
Q: Limits?
The problems are short and well known, so it says little about real software projects.