The EU AI Act, White House voluntary commitments, and NIST AI RMF all reference adversarial testing as a prerequisite for responsible deployment. Every serious lab has a safety team building or buying adversarial data pipelines. The question is no longer whether to red-team — it's how to collect data that's actually useful for training, and how to structure it so it doesn't introduce more noise than signal.
1. What Red-Teaming Actually Is (and Isn't)
Three distinct activities travel under the "red-teaming" label. Confusing them produces bad data pipelines.
(a) Automated red-teaming
GAN-style attacks, LLM-vs-LLM adversarial generation, gradient-based suffix attacks (GCG, AutoDAN). Fast and scalable. The failure mode: automated systems generate variations on known attack surfaces — jailbreaks that cluster around patterns already in training data. Novel attack vectors that require human social context, domain knowledge, or genuine deception are systematically missed.
(b) Human red-teaming with structured adversarial prompts
Humans generate attack prompts with intent, domain knowledge, and creativity that automated systems can't replicate. A domain expert red-teaming a medical AI doesn't prompt "ignore previous instructions" — they probe the gap between "providing information" and "giving medical advice" in a context where the model can't tell the difference. These prompts are non-reproducible assets that require structured collection.
(c) Hybrid pipelines
Automated expansion of human-generated seed prompts. Humans create the novel attack vectors; automated systems generate paraphrases, translations, and variations. This is the right architecture for most production red-team datasets — combining the coverage advantages of (a) with the novelty advantages of (b).
Human red-teamers find the attacks that feel human because they ARE human. The rest of this post focuses on (b) and (c) — the components that require structured annotation and produce the training data that actually matters.
2. The Two Failure Modes of LLM Red-Teaming
Most red-team datasets fail in one of two ways before the first annotation is collected.
(a) Coverage failure
Your red-team prompt set looks like a red-team prompt set from 2023. DAN variants, role-play jailbreaks, hypothetical framing — all of which your model has almost certainly seen in training data already. High volume, low coverage: you're testing failure modes the model knows about, not novel vectors from real-world misuse.
The fix is systematic prompt taxonomy (section 3) and intentional sampling from categories that are underrepresented relative to real-world attack frequency. A dataset that's 70% DAN variants because DAN variants are easy to generate is a coverage failure waiting to ship.
(b) Annotation failure
"Harmful" is not a label — it's a rubric problem. Without defined harm categories, severity scales, and explicit behavioral anchors, two expert annotators looking at the same adversarial prompt will produce systematically different labels. You're not measuring harm; you're measuring annotator disagreement.
The IAA target for safety classification is κ ≥ 0.85 — higher than the κ ≥ 0.70 minimum for preference annotation tasks, because the downstream cost of a miscalibrated safety classifier is higher. The IAA/Kappa methodology post covers how to measure and improve inter-annotator agreement before you scale. Running IAA checks before production is not optional for safety data.
3. Adversarial Prompt Taxonomy
Your red-team dataset needs explicit coverage targets across attack categories. Here are the five categories production safety teams use, with notes on what distinguishes each.
Direct jailbreaks
Role-play personas ("act as DAN, you have no restrictions"), developer mode prompts, hypothetical framing ("in a story where..."), base64 encoding, token manipulation. Highest-volume category in most existing datasets and the most likely to already be in training data. Coverage is important; novelty is lower here than in other categories.
Indirect injections
Multi-turn escalation (benign context → gradual escalation across turns), context manipulation (embedding harmful requests in legitimate context), goal hijacking in long conversations. These require multi-turn data collection infrastructure — a single-turn annotation pipeline won't capture them. Structuring these for training requires conversation-level annotations.
Domain-specific attacks
Medical misinformation (drug interaction advice that's plausible but wrong), legal advice (specific guidance that crosses the information/advice line), financial manipulation (pump-and-dump advice framed as market analysis). These can only be generated by domain experts who know what "plausible but harmful" looks like. A generalist doesn't know the drug interaction claim is wrong — they see "sounds like a reasonable medical answer" and label it non-harmful.
Cross-lingual attacks
Models trained primarily in English exhibit alignment gaps when attacked in Mandarin, Arabic, Hindi, or other languages. The safety training distribution doesn't cover these languages at the same density. Cross-lingual red-teaming requires bilingual domain experts — not just translators, but people who understand the harm categories well enough to evaluate whether a translated response is actually harmful in context.
Agentic attack surfaces
Tool-call injection (malicious instructions embedded in tool outputs that redirect model behavior), memory poisoning for long-context models, indirect prompt injection through retrieved documents. As models gain tool access and persistent memory, the attack surface expands significantly. These attacks require testers who understand the agentic architecture — a general red-teamer without tool-use knowledge won't generate meaningful attacks here.
Sampling proportionally to real-world attack frequency is important. Audit your prompt set composition before annotation begins — if direct jailbreaks are over-represented relative to their actual prevalence, your safety classifier will be over-fit to the easy cases.
4. Structuring Adversarial Data for Training
The output format you choose determines what you can train on. This decision must be made before annotation begins — retrofitting annotations from one format to another is expensive and usually requires re-annotation.
(a) Binary harmful / not-harmful labels
Simplest to collect, lowest signal. Binary labels collapse harm severity, harm category, and harm probability into a single bit. Useful for training a coarse safety filter, not useful for a nuanced safety classifier that needs to distinguish "potentially sensitive but benign" from "clearly harmful at high severity." If your safety training objective requires any gradient, binary labels will underserve it.
(b) Harm category + severity rating
Per-category annotations across dimensions — physical harm, financial harm, privacy violation, misinformation, manipulation — each rated on a 5-point Likert scale. The right format for training a multi-dimensional safety classifier. IAA requirements are higher: you need κ ≥ 0.85 per dimension. Low IAA on a single dimension (often "manipulation" — hardest to calibrate) doesn't invalidate other dimensions, but it means that category's labels can't be used for training until the rubric is fixed.
(c) Preference pairs — DPO-ready
Adversarial prompt → refused response vs. harmful response, annotated as a preference pair where the refused response is preferred. This format plugs directly into DPO and Constitutional AI RL-CAI pipelines. It directly trains the model's refusal behavior, not just a separate classifier. The preference dataset construction guide covers structural requirements. The RLHF vs. DPO framework covers how this format fits into preference optimization pipelines.
Format decision rule: Binary labels for high-volume coarse filtering. Per-category severity for multi-dimensional safety classifiers. Preference pairs for direct policy training. Many production safety pipelines use all three at different stages. Pick your primary format before writing the annotation brief.
5. Domain Expertise Requirements
Crowdsourced annotators fail at red-team data in three specific ways. Understanding these failure modes tells you exactly what the right annotator profile looks like.
Failure mode 1: Can't recognize domain-specific harm
A medical expert sees "plausible but wrong" in drug interaction advice — the claim uses correct terminology and is confidently stated. The generalist annotator sees "sounds like a reasonable answer" and marks it non-harmful. The label is wrong. The safety classifier trained on that label learns to allow exactly the kind of confident-but-incorrect medical advice that's most dangerous. No rubric can fix a knowledge gap.
Failure mode 2: Annotation fatigue causes κ drift
Red-team annotation is psychologically demanding. Annotators spend extended time with content designed to elicit harm — explicit violence, manipulation tactics, disturbing requests. Fatigue causes systematic drift: harm calibration shifts over a session, and IAA between the first 50 and last 50 annotations in a session diverges measurably. Production red-team pipelines need session length limits, regular recalibration, and IAA monitoring over time — not just at project kickoff.
Failure mode 3: Over-labeling benign prompts as harmful
Generalist annotators, particularly those not calibrated on the deployment context, systematically over-label benign-but-sensitive prompts. A question about medication dosage appropriate for a medical professional is not a jailbreak. A prompt about historical violence for educational purposes is not harmful. Over-labeling inflates the false positive rate in your safety classifier, degrading model helpfulness without improving safety. Expert annotators who know what DOESN'T count as a jailbreak are essential for false-positive control.
Right annotator profile: domain expert (can recognize domain-specific harm) + adversarial mindset (can distinguish harmful from sensitive) + calibration training (completed rubric calibration with documented IAA before production). The domain experts vs. crowdsourcing analysis covers the general case; for red-team data, the expertise requirement is higher than for standard annotation tasks.
6. Red-Team Data Collection Pipeline
A 5-step process that applies whether you're building 500 examples for a pilot or 50,000 for a production safety dataset.
Step 1 — Define the harm taxonomy tied to deployment context
A general-purpose assistant has a broad harm taxonomy. A medical chatbot has a narrower but deeper taxonomy — physical harm is primary, with sub-categories (dosage errors, contraindication advice, diagnostic overconfidence) that a general taxonomy wouldn't include. Define the taxonomy before generating prompts. The taxonomy determines what you're collecting; collecting first and defining later produces a dataset misaligned with your actual safety objective.
Step 2 — Generate seed prompts
Mix three sources: (a) known jailbreaks from public datasets and prior red-team rounds — establishes baseline coverage; (b) domain expert brainstorming — generates novel attack vectors requiring subject matter knowledge; (c) automated expansion — LLM-generated paraphrases, translations, and variations of seed prompts. The ratio matters: aim for at least 30–40% of seed prompts from domain expert brainstorming. If (a) dominates, you have a high-coverage but low-novelty dataset.
Step 3 — Annotate with per-category severity + rationale
Rationale is the critical element most teams skip. Each annotation should include a sentence explaining the annotator's reasoning, not just the severity rating. Rationale serves two functions: (1) it forces explicit rubric engagement — annotators who write rationale are less prone to drift; (2) it enables rubric refinement — when annotators disagree, comparing rationales reveals whether the disagreement is about the rubric (fixable) or genuinely ambiguous content (requires adjudication). Without rationale, disagreement is opaque.
Step 4 — IAA check before scaling
Run IAA on the calibration batch before scaling to production. Two layers: (a) overall κ — below 0.70 means fundamental rubric problems; scaling produces noise, not signal; (b) per-category κ — even if overall κ is acceptable, a single category with κ < 0.70 means that category's labels shouldn't be used for training until the rubric is fixed.
Never skip this step. κ < 0.70 on safety data means your safety signal is noise. Scaling that noise to 50,000 examples amplifies it, it doesn't improve it. See the IAA methodology guide for calculation details.
Step 5 — Iterate: low IAA → fix rubric → re-annotate
Low-IAA categories are rubric ambiguity signals, not annotator quality signals. When κ is below target on a category, diagnose the rubric: review disagreements, compare rationales, identify the specific edge cases that produce split decisions. Fix the behavioral anchors with explicit examples for ambiguous boundary cases. Re-run the calibration batch on the updated rubric. Repeat until stable.
The annotation pilot checklist covers running a calibration batch from scratch. The Constitutional AI vs. RLHF post covers how this pipeline fits into alignment approaches that go beyond standard RLHF.
7. Where Human Consensus AI Fits (and Doesn't)
Honest fit assessment for red-team data projects.
Right for:
- Domain-specific red-team annotation (medical, legal, financial, code) — expert annotators who can evaluate domain-specific harm that generalists can't. Taxonomy calibrated to deployment context; annotations include per-category severity ratings and rationale.
- Structured harm taxonomy annotation with per-category severity ratings — multi-dimensional annotation across physical harm, financial harm, privacy violation, misinformation, and manipulation with documented IAA targets per dimension.
- DPO-ready preference pair annotation for safety-tuned models — adversarial prompt → refused vs. harmful response pairs, annotated by domain experts with per-dimension IAA reporting. Plugs directly into DPO and RL-CAI pipelines. See the Starter Pack product page for details.
- IAA-verified safety datasets where κ targets are contractually specified — for compliance, audit, or governance requirements. Per-category kappa reports included. See the Enterprise Bundle product page for structured safety dataset contracts.
Not right for:
- High-volume automated jailbreak generation — if you need 100,000 paraphrases of known jailbreaks, that's an automated pipeline problem.
- Commodity safety filtering at 100K+ scale — if you need binary harmful/not-harmful labels on high-volume content at crowdsourcing prices, domain expert time isn't the right input.
- Fully automated red-teaming pipelines with no human in the loop — if your pipeline is GCG + automated scoring + no human review, there's no annotation task here.