What an LLM eval harness actually contains

An eval harness is the test suite for an LLM system. It answers one question for any change to a prompt, model, retrieval setting or tool: is the system better or worse than before, and on which cases.

It is not a vendor dashboard or a leaderboard score. It is code and data in your repository that your team can run and read. Here is what goes in it.

1. A versioned dataset of real cases

Start with 50 to 200 representative inputs, drawn from real traffic where possible. Include the easy common cases, the known hard ones, and every production failure anyone has reported. Each case carries the expected output, or the criteria a good output must meet.

Version the dataset like code. When a case changes, the history should show who changed it and why. Split off a held-out set nobody tunes against, so you can catch overfitting to the suite.

2. Deterministic checks

Anything that can be checked with code should be. Does the output parse against the schema? Does it include required fields? Did the agent call the right tool with valid arguments? Is the cited document actually in the retrieved set? Is the answer under the length limit?

These checks are fast, cheap and never disagree with themselves. They should cover as much of the quality definition as possible before any model judge is involved.

3. Model judges, calibrated against humans

Some qualities can't be checked with code: whether an answer is faithful to its sources, whether a summary missed the key point, whether the tone fits. A model judge scores these against a written rubric.

A judge is only useful if it agrees with your experts. Label a sample by hand, run the judge on the same sample, and measure agreement. If agreement is weak, rewrite the rubric, split it into narrower questions, or keep a human in that loop. Re-check agreement whenever the judge model changes.

4. A runner and a report

A script that runs every case against the current system, applies the scorers, and writes a report: overall score, score per category, and a diff of which cases flipped from pass to fail since the last run. The diff is the most useful part. An aggregate score can stay flat while ten important cases break and ten trivial ones improve.

Record cost and latency per case alongside quality. A change that improves accuracy and triples cost is a product decision, and the report should make that visible.

5. CI integration

Run a fast subset on every pull request that touches prompts, models, retrieval or tools, and the full suite nightly or before release. Set thresholds that block a merge when a category drops. Without this, the harness exists but nobody runs it.

6. A loop from production back to the suite

Trace production requests with inputs, retrieved context, model calls and outputs. Sample them, review failures, and add the interesting ones to the dataset. Over time the suite stops reflecting what the team imagined users would do and starts reflecting what they actually do.

Add adversarial cases too: prompt injection in retrieved documents, attempts to extract system prompts, requests that should be refused. Red team findings belong in the suite so they stay fixed.

What it takes to build one

A first useful harness takes two to four weeks for an existing system, most of it spent with domain experts defining what good means and labelling cases. The code is the easy part. The definitions are the work, and they are worth having written down regardless.