HumanEval — LLM 评估与基准测试 Dataset

OpenAI's hand-crafted Python coding benchmark: 164 programming problems with function signatures, docstrings, and unit tests. Models are evaluated by Pass@k — the probability of generating at least one correct solution in k tries. The definitive benchmark for code generation ability.

Dataset Details

from datasets import load_dataset
ds = load_dataset("openai/openai_humaneval")

相关数据集

• SWE-bench — Benchmarking real-world software engineering - never train on it
• MMLU (Massive Multitask Language Understanding) — Benchmarking general knowledge - never train on it
• Winogrande — Benchmarking commonsense reasoning
• MT-Bench — Judging multi-turn chat quality with LLM-as-judge

常见问题

← 全部数据集 | Fine-Tuning Guide