Evaluating AI Output at Scale: Building a Simple LLM-as-Judge Pipeline

Manually reading through AI-generated output to judge whether it was actually good worked fine for the first fifty samples, got tedious by the hundredth, and stopped being honest around the two-hundredth — by then, fatigue was doing more to shape the judgments than the actual quality differences between samples were. The fix was building a simple LLM-as-judge pipeline: a second model call that scores the first model's output against explicit criteria. The version that shipped worked well. The version that shipped first did not, and the gap between them is the actually useful part of this post.

What an LLM-as-judge pipeline actually is

The core idea is narrow: instead of a human reading every generated output to score it, a separate prompt — often to the same model, sometimes to a stronger one — is given the original input, the generated output, and explicit scoring criteria, and asked to return a structured judgment. It's not a replacement for human review entirely; it's a way to make human review sample-based and targeted instead of exhaustive, by using the automated judge to flag the outputs most likely to need a closer human look.

You are evaluating an AI-generated product description for accuracy and tone.

Original product data: {product_json}
Generated description: {generated_text}

Score each criterion 1-5 and give a one-sentence reason:
1. Factual accuracy (no claims not supported by the product data)
2. Tone match (matches a confident, plain-language brand voice)
3. Completeness (covers price, key features, and primary use case)

Return as JSON: {"accuracy": n, "accuracy_reason": "...", "tone": n, ...}

The first version's core mistake: no calibration against human judgment

The first pipeline was built, run against 300 generated descriptions, and the scores looked reasonable at a glance — mostly 4s and 5s, a scattering of 2s and 3s on outputs that seemed genuinely weaker when spot-checked. What wasn't done, and should have been done first, was checking the judge's scores against a smaller set of the same outputs scored independently by a human, to see whether the judge's numbers actually tracked human judgment or just looked plausible in isolation. When that comparison finally happened — 40 outputs scored by both a human and the judge — the correlation was weak: the judge was giving high accuracy scores to descriptions containing a specific, recurring error (restating a product's price in the wrong currency symbol when the source data used a non-dollar currency) that a human reviewer caught immediately and the judge consistently missed.

Why the judge was missing an error a human caught easily

The scoring prompt asked the judge to check "factual accuracy" as a single, broad criterion, without ever explicitly directing attention to numeric or currency details specifically — and a general-purpose accuracy check, it turned out, weighted narrative coherence and plausible-sounding phrasing more heavily than a line-by-line check of every number against the source. The fix wasn't a smarter model; it was a more specific prompt, breaking "accuracy" into explicit sub-checks including one dedicated specifically to numbers and units: "Does every number, price, and unit in the generated text exactly match the source data, including currency symbol?" Asking for that check explicitly, rather than trusting a general accuracy instruction to catch it implicitly, is what actually surfaced the currency errors in a second evaluation pass.

The second mistake: not testing the judge against known-bad examples

A more reliable calibration method, adopted after the currency-error discovery, was building a small set of deliberately flawed examples — outputs with an injected factual error, an off-brand tone, or a missing required detail — and confirming the judge actually caught each one before trusting it on real output. This caught a second gap: the judge consistently rated outputs with a slightly off-brand, overly enthusiastic tone as acceptable, scoring tone 4 or 5 even on examples a human had deliberately written to violate the brand voice guideline. The scoring prompt's tone criterion had been described only as "matches a confident, plain-language brand voice" — vague enough that the judge's own interpretation of "confident" drifted toward "enthusiastic," which wasn't the same thing. Adding two or three concrete positive and negative examples directly into the tone-scoring prompt closed that gap far more effectively than further rewording the abstract description alone did.

What the calibration process looks like in practice

1. Build a small human-scored reference set — 30-50 real outputs, scored independently by a person against the same criteria the judge will use, kept separate from any set used to write or tune the judge prompt.
2. Run the judge against that same set and compare scores directly, looking specifically for outputs where the judge and human disagree by more than one point on any criterion, not just for an overall average that looks close.
3. Investigate every meaningful disagreement individually rather than treating a handful of outliers as noise — the currency-symbol gap above only became visible by reading the specific outputs where scores diverged, not by looking at aggregate correlation alone.
4. Inject known-bad examples covering the specific failure modes that matter for the use case, and confirm the judge scores them appropriately low before trusting it on real, unlabeled output.
5. Re-calibrate whenever the underlying generation model or prompt changes — a judge tuned against one model's typical output patterns and failure modes isn't guaranteed to transfer cleanly to a different model's different tendencies.

What the pipeline looks like once it's actually trustworthy

In production, the calibrated judge now runs against every generated description automatically, and anything scoring below a 4 on any single criterion — not just the average — gets routed into a human review queue rather than published automatically; anything scoring below a 3 on accuracy specifically gets blocked outright pending review, since a low tone score is a polish problem and a low accuracy score is a trust problem, and those two failure modes deserve different handling rather than being averaged together into one number that obscures which kind of failure actually occurred. That queue now receives roughly 8% of generated outputs, down from reviewing 100% manually before the pipeline existed, and spot-checks of a random sample from the 92% the judge approves continue on a smaller, ongoing basis specifically to catch any new failure mode the judge hasn't been calibrated against yet.

Where this connects to prompting the generation step itself

Several of the accuracy failures the calibrated judge now catches trace back to generation-prompt issues covered in five prompt engineering patterns that actually improve output quality — being explicit about required fields and giving the generation prompt concrete positive and negative examples reduced the raw error rate before the judge ever saw the output, which mattered more for overall quality than the judge catching errors after the fact. The judge is a safety net for what generation-side prompting doesn't fully prevent, not a substitute for getting the generation prompt right in the first place, and treating it as the latter is a mistake worth naming directly.

The honest limitation still worth stating

An LLM judge, however well-calibrated, is still a model making probabilistic judgments, not a deterministic accuracy check — for anything with real financial, legal, medical, or safety consequences riding on the accuracy of generated content, an automated judge should reduce how often a human needs to look, not eliminate human review of the highest-stakes outputs entirely. What changed here was the volume a small team could actually keep quality control over, from roughly 300 reviewable outputs a week done manually and exhaustedly, to several thousand a week reviewed by a calibrated judge with a focused human queue behind it — a real, measurable improvement in coverage, not a claim that human judgment is no longer needed anywhere in the process.