LLM Training Dataset Hub

Written by Jakub Rusinowski · Last updated June 15, 2026

47 curated open datasets for fine-tuning and training LLMs — instruction tuning, reasoning traces, function calling, preference data, and more. Every entry includes the exact license, size, and a load_dataset() snippet.

Start here: Fine-Tune Your First LLM in 1 Hour · Fine-Tuning Reference (SFT, LoRA & DPO) · Check if your GPU can fine-tune

Compare all 47 datasets

Click a column header to sort. Click a dataset name for the full card with license details, sample data, and code.

How to choose a dataset

Match the dataset to the task, not the other way around. Teaching a model to follow instructions or adopt your tone? Use an instruction/SFT set — start with No Robots (human-written) or Dolly 15k (commercial-safe). Improving how a model ranks and phrases answers? You need a preference set for DPO: UltraFeedback is the default choice, HelpSteer2 adds fine-grained quality ratings.

Building a math or coding "thinking" model? Fine-tune on reasoning traces: OpenThoughts3 for breadth, NuminaMath for math, OpenCodeReasoning for code. Making an agent that calls APIs? Function calling is its own skill — Glaive v2 for the basics, xLAM when precision matters. Non-English users? Aya for instructions, FineWeb 2 for continued pretraining.

Two rules save the most time: check the license before you train (CC BY-NC sets can't ship in a commercial product), and start smaller than you think — 1,000 good examples beat 100,000 noisy ones. When in doubt, follow the one-hour walkthrough in Fine-Tune Your First LLM, which uses these exact datasets.

Instruction / SFT (18)

Instruction datasets (SFT sets) are prompt–response pairs used for supervised fine-tuning — the first and most important step in teaching a base model to follow instructions, answer in your format, or adopt a specific tone. They range from 1k hand-written examples (LIMA) to multi-million-sample synthetic mixtures (Infinity-Instruct).

• Python-Edu — Continued pretraining for Python code understanding (400 GB · MIT)
• OpenHermes 2.5 — The default general-purpose SFT mix for 7B-13B fine-tunes (1M Rows · Apache 2.0)
• Databricks Dolly 15k — Commercially safe instruction tuning - fully human-written (15k Rows · CC-BY-SA 3.0)
• Stanford Alpaca — Learning the classic instruction format; research only (CC BY-NC) (52k Rows · CC-BY-NC 4.0)
• ShareGPT 52K — Teaching natural multi-turn dialogue (non-commercial) (52k Conversations · CC-BY-NC 4.0)
• WizardLM Evol Instruct 70k — Boosting complex-instruction handling with Evol-Instruct data (70k Rows · Apache 2.0)
• LIMA: Less Is More for Alignment — Style and format alignment with a tiny curated set - quality over quantity (1k Rows · CC-BY-NC-SA 4.0)
• Smoltalk — General SFT for small models (the SmolLM2 recipe) (1M Rows · Apache 2.0)
• Open-Platypus — A quick STEM and logic boost on a small budget (non-commercial) (25K Questions · CC BY NC 4.0)
• OpenOrca — Explanation-style SFT at scale (the Orca recipe) (4.2M Samples · MIT)
• Infinity-Instruct — Large-scale general SFT when you need millions of samples (7.5M Instructions · Apache 2.0)
• Magpie-Align — Fresh synthetic SFT data without seed data or scraping (3M Pairs · Apache 2.0)
• Tulu 3 SFT Mix — Reproducing a state-of-the-art fully open post-training recipe (939K Samples · ODC-BY)
• WebInstruct — Web-scale instruction mining for domain skills (220M Pairs · CC BY-SA 4.0)
• LMSYS-Chat-1M — Studying real-world usage patterns across 25 models (non-commercial) (1M Conversations · CC BY-NC 4.0)
• WildChat-1M — Training on real user conversations; safety and robustness research (1M Chats · AI2 ImpACT)
• No Robots — Your first fine-tune — small, clean, 100% human-written SFT data (non-commercial license) (10k Rows · CC BY-NC 4.0)
• SmolTalk 2 — Reproducing a complete modern post-training pipeline (mid-training → SFT → preference) for small models (3 Subsets (Mid 4.8M Rows) · Apache 2.0)

Reasoning (6)

Reasoning datasets store full chains of thought — the step-by-step "thinking" that leads to an answer — rather than just final responses. Fine-tuning on traces distilled from strong reasoning models (DeepSeek R1, QwQ) is how small local models learn to work through math, code, and science problems before answering.

• MetaMathQA — Boosting GSM8K/MATH-style math skills in 7B models (395K Pairs · MIT)
• Orca Math Word Problems — Grade-school math word problems for small models (200K Problems · MIT)
• NuminaMath — Chain-of-thought math fine-tuning up to olympiad level (860K Problems · Apache 2.0)
• OpenThoughts3-1.2M — Distilling strong math/code/science reasoning into 7B–32B models (1.2M Rows · Apache 2.0)
• s1K-1.1 — Cheap, fast reasoning fine-tunes — 1k samples means minutes of training, not days (1k Rows · Apache 2.0)
• OpenCodeReasoning — Fine-tuning coding models that show their reasoning before writing the solution (735k Samples · CC BY 4.0)

Agentic & Function Calling (3)

Agentic datasets teach a model to call functions and tools: read a JSON schema, decide when a call is needed, emit valid arguments, and use the result. This is the core skill behind local AI agents — and it must be trained explicitly, because general chat data won't produce reliable tool calls.

• Glaive Function Calling v2 — Teaching a model basic function-calling: when to call, what JSON to emit, how to use results (113k Rows · Apache 2.0)
• xLAM Function Calling 60k — High-precision tool calling — every sample was verified by actually executing the API call (60k Rows · CC BY 4.0)
• Hermes Function Calling v1 — Models that must return valid structured JSON — for agents, extraction, and tool pipelines (12k Samples · Apache 2.0)

Preference (RLHF / DPO) (5)

Preference datasets contain a prompt plus ranked or scored responses — a chosen answer and a rejected one. They power RLHF, reward models, and DPO: the post-SFT step that teaches a model which of two plausible answers humans actually prefer. Run these after an instruction-tuning pass, not instead of it.

• UltraFeedback — The default DPO preference set to run after any SFT pass (64k Rows · MIT)
• Anthropic HH-RLHF — Safety-focused preference training (helpfulness and harmlessness) (170k Pairs · MIT)
• DPO Mix 7K — A small, balanced DPO starter set (7k Pairs · Apache 2.0)
• HelpSteer2 — Training reward models with fine-grained quality ratings (21K Samples · CC BY 4.0)
• Nectar — Reward-model training with 7-way ranked responses (183K Prompts · Apache 2.0)

Pretraining (4)

Pretraining corpora are massive raw-text collections — trillions of tokens of filtered web text, textbooks, and code — used to train base models from scratch or continue pretraining on a new language or domain. You won't fine-tune on these with a consumer GPU, but they define what open base models know.

• FineWeb — Pretraining or continued pretraining on high-quality English web text (15 TB · ODC-By 1.0)
• Cosmopedia — Synthetic-textbook pretraining for small models (the SmolLM recipe) (25B Tokens · Apache 2.0)
• SlimPajama — Efficient English pretraining on a heavily deduplicated corpus (627B Tokens · Apache 2.0)
• FineWeb 2 — Pretraining or continued pretraining in languages other than English (8TB Compressed · ODC-By 1.0)

Multilingual (1)

Multilingual datasets bring instruction-following and pretraining quality to languages beyond English — from human-annotated instruction pairs in 65 languages (Aya) to quality-filtered web text in 1,000+ languages (FineWeb 2). Essential if your users don't prompt in English.

• Aya Dataset — Instruction-tuning in non-English languages with real human-written data (204k Rows · Apache 2.0)

Code (3)

Code datasets range from massive permissively-licensed source corpora (The Stack v2) to instruction pairs synthesized from real open-source code. Use them to specialize a model for your stack, improve completion quality, or train a local coding assistant.

• The Stack v2 — Pretraining code models across 600+ programming languages (900B Tokens · Various (per-file))
• Magicoder-OSS-Instruct-75K — Code instruction tuning grounded in real open-source snippets (75k Rows · MIT)
• CodeFeedback-Filtered-Instruction — Training coding models that respond to execution feedback (74k Rows · Apache 2.0)

Vision (1)

Vision (multimodal) datasets pair images with text instructions so language models learn to see. LLaVA-style instruction sets are the standard recipe for adding image understanding to an open LLM.

• LLaVA Instruct — Adding image understanding to an open LLM (the LLaVA recipe) (158k Samples · CC-BY-NC 4.0)

Evaluation & Benchmarks (6)

Evaluation sets are benchmarks — MMLU, HumanEval, SWE-bench — used to measure models, not to train them. Never mix them into training data: benchmark contamination inflates scores and is the first thing careful reviewers check for.

• MMLU (Massive Multitask Language Understanding) — Benchmarking general knowledge - never train on it (16k Questions · MIT)
• HumanEval — Benchmarking Python code generation - never train on it (164 Problems · MIT)
• Winogrande — Benchmarking commonsense reasoning (44k Problems · Apache 2.0)
• MT-Bench — Judging multi-turn chat quality with LLM-as-judge (80 Questions · Apache 2.0)
• SWE-bench — Benchmarking real-world software engineering - never train on it (2,294 Tasks · MIT)
• MATH (Hendrycks) — Benchmarking competition math ability (12,500 Problems · MIT)

Frequently asked questions