LLM Training Dataset Hub

Instruction datasets (SFT sets) are prompt–response pairs used for supervised fine-tuning — the first and most important step in teaching a base model to follow instructions, answer in your format, or adopt a specific tone. They range from 1k hand-written examples (LIMA) to multi-million-sample synthetic mixtures (Infinity-Instruct).

Reasoning datasets store full chains of thought — the step-by-step "thinking" that leads to an answer — rather than just final responses. Fine-tuning on traces distilled from strong reasoning models (DeepSeek R1, QwQ) is how small local models learn to work through math, code, and science problems before answering.

Agentic datasets teach a model to call functions and tools: read a JSON schema, decide when a call is needed, emit valid arguments, and use the result. This is the core skill behind local AI agents — and it must be trained explicitly, because general chat data won't produce reliable tool calls.

Preference datasets contain a prompt plus ranked or scored responses — a chosen answer and a rejected one. They power RLHF, reward models, and DPO: the post-SFT step that teaches a model which of two plausible answers humans actually prefer. Run these after an instruction-tuning pass, not instead of it.

Pretraining corpora are massive raw-text collections — trillions of tokens of filtered web text, textbooks, and code — used to train base models from scratch or continue pretraining on a new language or domain. You won't fine-tune on these with a consumer GPU, but they define what open base models know.

Multilingual datasets bring instruction-following and pretraining quality to languages beyond English — from human-annotated instruction pairs in 65 languages (Aya) to quality-filtered web text in 1,000+ languages (FineWeb 2). Essential if your users don't prompt in English.

Code datasets range from massive permissively-licensed source corpora (The Stack v2) to instruction pairs synthesized from real open-source code. Use them to specialize a model for your stack, improve completion quality, or train a local coding assistant.

Vision (multimodal) datasets pair images with text instructions so language models learn to see. LLaVA-style instruction sets are the standard recipe for adding image understanding to an open LLM.

Evaluation sets are benchmarks — MMLU, HumanEval, SWE-bench — used to measure models, not to train them. Never mix them into training data: benchmark contamination inflates scores and is the first thing careful reviewers check for.