RLHF for Code Generation Models: How to Build Training Data for Coding AI

Why Code Generation RLHF Is Different from NLP RLHF

In natural language RLHF, preference is almost entirely subjective. Ask two expert annotators which response better explains a historical event — they might disagree on emphasis, depth, or framing, but there's no ground-truth correct answer to anchor the disagreement. The preference signal is genuinely the thing you're training toward.

Code has a different structure. There is an objective layer: does the function return the correct output? Does it pass the test suite? Does it handle edge cases without crashing? This layer can, in principle, be checked mechanically. But mechanical verification of correctness is not the same as evaluating code quality — and conflating the two is the most expensive mistake teams make when building code RLHF pipelines.

Consider GitHub Copilot or DeepSeek-Coder generating a Python function to parse JSON from an API response. A test suite might verify that the function returns the right data structure on a set of controlled inputs. It will not catch that the function uses a global variable that creates a race condition under concurrent requests, that it swallows all exceptions silently, or that it hardcodes a retry delay in a way that will cause production timeouts at scale. The code passes the tests and fails in production.

This is the fundamental tension in code RLHF: automated test suites address functional correctness, but they leave two of the three dimensions that determine code quality entirely unmeasured. Those dimensions require human judgment — specifically, the judgment of engineers who write production code in the language and domain the model is being trained on.

For context on why automated metrics generally fail as a training signal proxy, see LLM evaluation benchmarks vs. human evaluation. The code case makes that argument concrete.

The Three Dimensions of Code Quality Annotators Must Evaluate

Code preference annotation requires evaluating three distinct dimensions. Each has a different relationship to automated checking — and each requires a different annotator profile to evaluate reliably.

The critical insight: automated test suites catch dimension (a) for the specific inputs they test. They catch almost nothing in (b) and essentially nothing in (c). A code RLHF pipeline that relies on test pass rates as the preference signal is training a reward model on one-third of what determines code quality — and ignoring the two dimensions that are most dangerous to get wrong.

Why Automated Metrics Alone Don't Work for Code RLHF

HumanEval and MBPP are the standard code generation benchmarks. Both measure pass@k: given k generated solutions, what fraction solve the problem on the provided test suite? This is a useful signal for capability assessment. It is not a useful signal for RLHF preference training.

The problem is that a fixed test suite is not the same as "good code." Three concrete examples of code that passes all HumanEval tests and fails on real quality dimensions:

The broader argument — that benchmark scores are a poor proxy for the quality signal you actually want to optimize — applies throughout AI evaluation. See LLM evaluation benchmarks vs. human evaluation for the full case. In code, the gap between "passes HumanEval" and "good production code" is wide enough to drive a security incident through.

Who Should Annotate Code Training Data

The annotator quality problem is more acute in code RLHF than almost anywhere else in the RLHF pipeline. The reason is structural: evaluating code quality requires professional coding experience in the specific language and domain being annotated. A non-programmer cannot evaluate whether a function is idiomatic Python. A generalist developer cannot evaluate whether a Rust function has sound memory management. A backend engineer cannot reliably evaluate whether a React component uses appropriate patterns for the ecosystem.

The minimum bar: 2+ years of professional coding experience in the target language. The ideal: senior engineers who write production code in the language and domain the model is being trained on. If you're fine-tuning a model for ML engineering tasks, you want Python annotators who work on ML systems daily — not generalist Python developers. If you're training a model for systems programming, you want annotators who understand memory safety implications in context, not engineers whose primary language is TypeScript.

Note that even domain-matched senior engineers achieve κ ≈ 0.55–0.65 at best on style and quality dimensions — not because one annotator is wrong, but because senior engineers often have legitimate disagreements about idiomatic style. Two experienced Python engineers can reasonably disagree about whether a list comprehension or an explicit loop is more readable in a given context. This is an argument for calibration rounds (to surface and resolve systematic disagreements in the rubric) rather than for crowdworkers (who disagree for the wrong reasons: they don't understand the code well enough to have an informed opinion).

For the full case on why domain expertise determines annotation quality across RLHF tasks, see domain-expert annotators vs. crowdsourcing for AI training data.

What a High-Quality Code Preference Dataset Looks Like

The minimum viable structure for a code preference pair is more complex than a standard NLP preference pair. A standard preference pair needs: prompt + response A + response B + preference label. A code preference pair needs to capture the three evaluation dimensions separately, because a reward model trained on aggregate preference conflates correctness, quality, and security into a single score — making it impossible to know what the RM actually learned to optimize.

Minimum structure per preference pair:

• Problem statement (task description + acceptance criteria)
• Target language + relevant constraints (performance, memory, style guide)
• Candidate A — the preferred solution
• Candidate B — a realistic failure mode (not a straw man: code that compiles and works on simple cases but fails on the dimensions being evaluated)
• Annotator judgment at the dimension level: correctness verdict + rationale, quality verdict + rationale, security verdict + rationale
• Overall preference label with annotator ID for IAA computation

The InstructGPT approach of requiring annotators to write rationales — not just pick A or B — is especially important for code preference data. A rationale forces the annotator to articulate why one solution is preferred at each dimension. This serves two functions: it makes the annotation auditable (you can verify the rationale reflects genuine expert judgment) and it surfaces rubric gaps (when annotators struggle to articulate a rationale, the rubric is under-specified). See how to build a preference dataset for RLHF for the broader structure.

Language-specific considerations

def get_user_scores(user_list):
    result = []
    for i in range(len(user_list)):
        u = user_list[i]
        score = 0
        for j in range(len(u["events"])):
            score = score + u["events"][j]["points"]
        result.append({"id": u["id"], "total": score})
    return result

def get_user_scores(users: list[dict]) -> list[dict]:
    return [
        {"id": user["id"], "total": sum(e["points"] for e in user["events"])}
        for user in users
    ]

Language-specific rubric considerations:

• Python: PEP 8 compliance, type hints, generator expressions vs. list comprehensions by context, memory efficiency for large data structures
• JavaScript/TypeScript: async/await patterns vs. raw Promise chains, type safety and generic appropriateness, bundle size awareness (tree-shaking, side-effect-free imports)
• Systems languages (Rust, C++): memory safety (ownership, borrowing, lifetimes), allocation patterns, zero-cost abstraction usage, unsafe block justification

Each of these requires annotators who work in the language daily — not engineers who have passing familiarity with it.

Cold Start: How to Bootstrap a Code RLHF Dataset

The cold start problem in code RLHF is acute: you need preference pairs that accurately represent the quality dimension you want to optimize, but generating high-quality pairs requires either a capable base model (to generate plausible candidates) or expensive expert time (to write them from scratch). Two approaches address this with different cost and signal tradeoffs:

Practical recommendation: start with approach (b) for the first 500 preference pairs (cold start signal). Expert demonstration traces at this volume provide enough high-quality signal to train an initial reward model that can guide further data collection. Then shift to approach (a) at scale — model-generated pairs with expert correction are 3–5× cheaper per pair and maintain quality once the base model is capable enough to generate plausible candidates. The cold start investment in (b) makes (a) viable.

For the foundational RLHF dataset mechanics this builds on, see how to build an RLHF dataset from scratch. The code case applies the same principles with additional structure for the three quality dimensions.

Getting Started: Minimum Viable Code RLHF Dataset

Minimum viable code RLHF dataset for a production-quality code generation model:

For guidance on reward model quality once you have the preference data, see how to evaluate RLHF reward models. The evaluation methodology applies to code reward models — with the additional consideration that reward hacking in code RLHF often manifests as length bias (longer code with more comments scores higher regardless of quality) or format exploitation (the RM learns to prefer code with docstrings regardless of whether the docstring is accurate).