AI ART Discover · Learn · Create
H AI Glossary Term • Term 100 of 224
Plain English Definition

What does “HumanEval” mean?

Core Definition

A benchmark of Python programming problems used to measure code-generation correctness via unit tests.

Operational Mechanism

How HumanEval Works

1

Function signature and description

2

Model writes the code

3

Run hidden unit tests

4

Pass or fail per problem.

Real-World Scenario

Practical Example

A problem gives a function name and a docstring such as 'return the sum of even numbers in a list'. The model writes the body, and tests check it against several inputs.

Core Significance

Why It Matters

HumanEval offered a simple way to measure whether generated code works. Its 164 problems are small, so strong models have largely saturated it.

Expert Q&A

Common Questions About HumanEval

Q: Who introduced it?

OpenAI researchers in 2021, alongside the Codex model.

Q: How many problems?

164 hand-written Python programming problems.

Q: Limits?

The problems are short and well known, so it says little about real software projects.