Anthropic HH-RLHF
170k human preference comparisons between two AI assistant responses, with one labeled 'chosen' (helpful, harmless) and one 'rejected'. The dataset that defined RLHF alignment for chat assistants. Used to train early Claude models and became the standard reference for alignment research and DPO fine-tuning.
from datasets import load_dataset
ds = load_dataset("Anthropic/hh-rlhf")Preview a sample row
{
"chosen": "Human: How do I safely store passwords?\nAssistant: Use a strong hashing algorithm like bcrypt or Argon2. Never store plain text passwords. Add a unique salt per password to prevent rainbow table attacks.",
"rejected": "Human: How do I safely store passwords?\nAssistant: You can use MD5 to hash them before storing in your database."
}Fine-tune with this dataset
Estimated VRAM to fine-tune with QLoRA (4-bit base model + LoRA adapters), using conservative defaults:
New to fine-tuning? Follow the step-by-step walkthrough: Fine-Tune Your First LLM in 1 Hour
Frequently asked questions
Can I use Anthropic HH-RLHF commercially?
Yes — Anthropic HH-RLHF is released under MIT, a permissive license that allows commercial use, including training models you ship in a product. Check the dataset card for attribution requirements before release.
How much data does Anthropic HH-RLHF contain, and do I need all of it?
Anthropic HH-RLHF contains 170k Pairs. You rarely need all of it: for style and format fine-tuning, a few hundred to a few thousand examples are enough — load a slice (e.g. split="train[:1000]") and scale up only if quality plateaus.
What is Anthropic HH-RLHF best used for?
Safety-focused preference training (helpfulness and harmlessness). It belongs to the Preference (RLHF / DPO) section of our dataset hub, where you'll find alternatives and complementary sets.