Build LLM Evaluation Sets by Sampling Production Logs, Not by Inventing Cases

Why does a hand-written case set measure your imagination?

If you write evaluation cases from memory, you are testing the failures you can imagine, and the failures that reach production are the ones nobody imagined. I have not yet seen a hand-written suite catch an incident that the team had not already thought of, which is the definition of the problem.

The mechanism is arithmetic rather than discipline. Take a workflow handling 12,000 runs in a quarter and an edge case that occurs in 3 percent of them. That is 360 real occurrences in the logs. A case set written by four engineers in an afternoon will contain, at most, one case for that path, and often none, because nobody remembered it. When it finally fails in production, the suite is green and the team concludes that testing does not work. What failed was the sampling frame, not the idea of a regression suite.

There is a second, quieter cost. A hand-written case encodes the workflow as it was understood on the day it was written, including the fields the author happened to know about. Production rows encode the workflow as it actually receives input: the missing reference number, the supplier name that exists twice, the currency with no rate on file. You cannot invent those, because inventing them requires already knowing them.

The best evaluation set is forty real historical cases with known-good outcomes, because it encodes the distribution you actually serve instead of the one you imagined, and it is extracted from records you are already storing rather than authored.

That sentence is easy to agree with and slightly annoying to implement, because it moves the work from writing cases to querying run records. That is the whole point: a query is auditable, repeatable and cheap, and it can be re-run next quarter against a larger table.

Which rows in the run table are worth keeping?

Four buckets, and every one of them is defined by a column you already write. This matters more than the numbers: when a bucket is a function of recorded data, the selection rule survives the person who invented it.

with recent as (
  select r.id                     as run_id,
         r.input_hash,
         r.workflow,
         r.value_at_stake,
         r.outcome,
         (o.run_id is not null)   as has_override,
         count(*) over (partition by r.input_hash) as duplicate_count
  from ai_run r
  left join human_override o on o.run_id = r.id
  where r.completed_at >= now() - interval '90 days'
),
unique_runs as (
  select * from recent where duplicate_count = 1
),
bucketed as (
  select *,
         case
           when outcome = 'failed'      then 'known_failure'
           when value_at_stake >= 50000 then 'high_value'
           when has_override            then 'corrected'
           else 'routine'
         end as bucket,
         row_number() over (
           partition by case
             when outcome = 'failed'      then 'known_failure'
             when value_at_stake >= 50000 then 'high_value'
             when has_override            then 'corrected'
             else 'routine'
           end
           order by md5(run_id)
         ) as pick
  from unique_runs
)
select run_id, input_hash, bucket, value_at_stake
from bucketed
where pick <= 10
order by bucket, value_at_stake desc;

Three details in that query are load-bearing. The join to human_override is where ground truth comes from, and it is free: a reviewer already corrected the output, so the corrected value is a documented correct answer rather than an opinion you have to collect later. The duplicate_count = 1 filter stops a retry storm from filling the sample with forty copies of one input. And the ordering is md5(run_id) rather than random(), so the query is deterministic — run it twice and you get the same forty rows, which is what makes the suite reproducible a year later when you want to know when a case entered it.

The forty rows this returns for a 12,000-run quarter are not a statistical sample of anything. They are a coverage set: ten cases where being wrong costs money, ten that already failed, ten that a person corrected, ten from the boring middle that proves you did not break the happy path while fixing the edge. Almost every regression I have traced back to a "quality improvement" landed in the routine bucket.

What does a case have to store to be replayable?

A case is an input, an expectation, and an environment, and the third is the one teams omit. Without a pinned environment you are comparing two different experiments and calling the difference a regression.

{
  "id": "quote-2026-01-14-8f41",
  "bucket": "high_value",
  "input": { "accountId": "acct_88231", "quantity": 4800, "currency": "CNY" },
  "environment": {
    "promptVersion": "quote-v14",
    "modelVersion": "claude-sonnet-4-5-20250929",
    "toolSchemaHash": "sha256:9f2c1d4a"
  },
  "expectation": {
    "kind": "assert",
    "check": "marginWithinBand",
    "args": { "minPct": 18, "maxPct": 22 }
  },
  "provenance": {
    "runId": "run_01HZX4K8",
    "humanOverrideId": "ovr_4471",
    "selectedOn": "2026-03-02"
  }
}

The modelVersion string is whatever the provider returned in the recorded run, not what your config file says you asked for — the two diverge the first time an alias moves, and that divergence is worth an incident of its own. provenance is not documentation for its own sake: when someone proposes deleting a case, the override id is the argument for keeping it, and the selectedOn date is how you find the case that has quietly outlived its rate card.

The expectation is one sentence of intent, expressed as a check with arguments. marginWithinBand between 18 and 22 percent is a tolerance band, and the band lives in the case file where a reviewer can argue about it, not in a reviewer's head where it cannot be diffed.

How do you grade forty cases without a judge model?

Grade the decision the workflow produced, in system language, against a rule. Grade the prose and you get a score no build can fail, because a rubric with five dimensions moves a little in every direction and the average holds.

export type CaseResult =
  | { id: string; status: "pass" }
  | { id: string; status: "fail"; detail: string }
  | { id: string; status: "skipped"; reason: string };

type Args = Record<string, number | string>;
type Assertion = (output: WorkflowOutput, args: Args) => boolean;

export const checks: Record<string, Assertion> = {
  marginWithinBand: (o, a) => {
    const pct = Number(o.marginPct);
    return Number.isFinite(pct) && pct >= Number(a.minPct) && pct <= Number(a.maxPct);
  },
  citesActiveClause: (o, a) =>
    o.citations.some(
      (cit) => cit.clauseId === a.clauseId && cit.effectiveOn <= o.asOf,
    ),
  routesToApprover: (o, a) => o.approvalLevel === a.expectedLevel,
};

export function grade(kase: EvalCase, output: WorkflowOutput): CaseResult {
  if (kase.expectation.kind === "human") {
    return { id: kase.id, status: "skipped", reason: kase.expectation.note };
  }
  const { check, args } = kase.expectation;
  const assertion = checks[check];
  if (!assertion) throw new Error(`case ${kase.id} references unknown check "${check}"`);
  return assertion(output, args)
    ? { id: kase.id, status: "pass" }
    : { id: kase.id, status: "fail", detail: check };
}

The unknown-check throw is deliberate. A typo in a check name that silently grades nothing is worse than a crash, because it turns a case into decoration while the report stays green. The human branch returns skipped rather than pass, so the count of ungraded cases is visible in the output instead of being absorbed into the success rate.

Roughly three quarters of the forty cases I extract land on a code check with no judgement involved. The rest either need a tolerance band or are genuinely contested, where two reviewers at the client would disagree about the correct output. Those contested cases are the most valuable ones in the set, because they mark the boundary where the workflow should route to a person instead of deciding.

Which failures does a log-mined suite actually catch?

The distribution below is my own observation from the quote and pricing workflows I have instrumented, not a published measurement, and the shares are meant to show shape rather than to be quoted.

The rows sum to 1,920 runs out of 12,000, so roughly 16 percent of traffic produced something a human flagged. Two of those classes are not suite material. The 480 phrasing complaints cannot be asserted against at all, because no field is wrong; they belong in a review queue where a prompt edit is a judgement call, not a build failure. The 60 duplicate-supplier cases have ground truth, but it lives in the supplier master record rather than next to the run, so the extraction query needs the second join before those rows are gradeable — which is worth doing, because entity mismatch is the failure most likely to reach a customer unnoticed.

What does the suite cost to keep running?

Cheap enough that the argument is never about money, which is why the argument is usually about ownership.

Those unit prices are illustrative; substitute your provider's current rates and the arithmetic holds. Pass two is the only line that grows with model pricing, and it is the line that catches a provider moving a model behind an unchanged alias.

The gate is two commands, and only the first one runs on every commit:

# pass one: replay stored model responses, no network, deterministic
pnpm tsx evals/run.ts --suite evals/cases --source recorded --report evals/report.json

# pass two: live calls on the same 40 cases, before release only
pnpm tsx evals/run.ts --suite evals/cases --source live --budget-usd 2.00

# the gate reads pass one: a case that passed last release and fails now stops the merge
node -e "const r=require('./evals/report.json');if(r.regressions.length){console.error(r.regressions);process.exit(1)}"

--budget-usd is not decoration. Without a ceiling, pass two is the command someone eventually runs in a loop while debugging a flaky check, and the invoice for a forty-case suite stops being the cost of one hour of traffic.

When is mining production logs the wrong approach?

Three situations, and in each one the query is not the answer.

Low-volume workflows have no distribution to sample. If a workflow runs forty times a quarter, the population is the case set, and the right move is to keep all of it and stop sampling. The same applies at the other end when the volume is high but the inputs are near-identical — a classification step over ten thousand rows with three real variants has three buckets, not four thousand cases, and a stratified sample will keep handing you the same row.

Contested quality with no recorded outcome is the second. If nobody corrected the output and no downstream system recorded a final value, the logs contain inputs and no answers, and a suite built on them grades nothing. In that state the first deliverable is the override path: make reviewers record what they changed and why, wait a month, and mine afterwards. This is also the case where the whole approach depends on a gate rather than the sample, which is the argument in what a replay gate over forty archived cases actually buys you: the sample without a gate is a report, and reports do not fail builds.

Third, regulated data changes the storage question rather than the method. Run records that contain customer identifiers need a retention rule and a per-subject index before they become a test fixture, and a case file that quotes a corrected price may itself be regulated. Hash the identifiers, keep the case under the same retention as the run it came from, and route anything that cannot be de-identified through a synthetic case derived from the same shape rather than the original row.

There is a fourth failure mode that is not about legality. A suite built from historical rows will keep testing last quarter's business, and every case passed by a tolerance band encodes a policy you may have since changed. Refresh the extraction each quarter and archive retired cases with a one-line note, or you will end up with a suite that is fully green while the override rate climbs — which I have watched happen, in a workflow whose cases had frozen six months before anyone looked.

The first query to run

Pick the workflow with the largest financial exposure and the fewest steps, and run the extraction query against ninety days of its run records before you write a single case by hand. If it returns ten rows rather than forty, that is the finding: either the workflow is too new to have a distribution, or nobody has been recording outcomes, and the outcome-recording path is what to build first. Then put pass one in the pipeline so a case that passed at the last release and fails now stops the merge while the change is still cheap to reverse. The suite will be wrong about your error rate from the first day and useful from the first regression, and that trade is the one worth taking.