MMLU (Massive Multitask Language Understanding) — LLM 评估与基准测试 Dataset
The gold-standard multi-subject benchmark: 16k multiple-choice questions spanning 57 subjects from STEM and humanities to law, medicine, and social sciences. Every major LLM is evaluated on MMLU. A model scoring >80% is considered strong; GPT-4 scores ~87%, Llama 3 70B ~82%.
Dataset Details
| Provider | cais |
| Category | 评估与基准测试 |
| Size | 16k Questions |
| License | MIT |
| Downloads | 12M |
| Tags | Benchmark, Multiple-Choice, 57-Subjects, STEM, Reasoning |
from datasets import load_dataset
ds = load_dataset("cais/mmlu")
相关数据集
- MATH (Hendrycks) — Benchmarking competition math ability
- Winogrande — Benchmarking commonsense reasoning
- HumanEval — Benchmarking Python code generation - never train on it
- MT-Bench — Judging multi-turn chat quality with LLM-as-judge
常见问题
MMLU (Massive Multitask Language Understanding) 可以商用吗？
可以——MMLU (Massive Multitask Language Understanding) 采用 MIT 宽松许可证，允许商业使用，包括训练用于产品的模型。发布前请查看数据集卡片中的署名要求。
MMLU (Massive Multitask Language Understanding) 有多少数据？需要全部使用吗？
MMLU (Massive Multitask Language Understanding) 包含 16k Questions。它是评估基准，用于完整地衡量模型——切勿混入训练数据，否则基准分数将失去意义。
MMLU (Massive Multitask Language Understanding) 最适合做什么？
Benchmarking general knowledge - never train on it。它属于数据集中心的「评估与基准测试」板块，那里有替代和互补的数据集。
← 全部数据集 | Fine-Tuning Guide