How often does your AI judge agree with a person?

Why the % agreement can mislead

An AI judge is a model that checks another model's results and says pass or fail. When most results work, agreeing with people is easy. If 90% of your support bot's answers work, a judge that says "pass" every time agrees with people 90% of the time and catches nothing. So the percentage looks best in the apps where failures are rare and hardest to spot. Load the example above: its judge agrees with people 85% of the time, and a judge that always said "worked" would score 90%.

Catching failures is the judge's real job

Two numbers tell you whether it does that job:

• Caught the failures: of the results people said didn't work, the share the judge also failed. Low means bad answers, wrong invoice fields or off-target translations reach users unnoticed.
• Right when it said didn't work: of the results the judge failed, the share people agreed with. Low means you retry, block or investigate results that were fine.

In one app, Storybirdie, the judge had 0.28 precision against human labels. That's a single app, not a benchmark, but it shows how a judge can look reasonable overall while most of its flags are wrong.

What Cohen's kappa adds

The statistics section reports Cohen's kappa, which asks how much better the judge does than a judge guessing with the same mix of verdicts. 0 means no better than chance and 1 means perfect agreement. As a rule of thumb, 0.6 or higher is usable for automatic decisions, 0.4 to 0.6 is fine for watching trends, and below 0.4 the judge mostly isn't telling you anything. With a few dozen labels the confidence interval is wide, so read it too.

How many labels, and which ones

Label at least 30 results; 50 or more is better. A random sample of recent results matches real use, but if most results work it will hold only a handful of failures, and the failure numbers will be guesses. Add failures on purpose until you have at least ten. That doesn't change how many the judge catches, but it flatters how often its failures are right, so read that as a best case.

The most honest labels come from what users did next: asked again, asked for a person, fixed a field, hit “try again”, or moved on. If you label by hand, hide the judge's verdict while you do it.

How to fix a weak judge

Read the disagreements first. Results that worked but the judge failed usually point to a vague or over-strict rubric. Results it passed that didn't work point to a missing condition. Then:

• Tighten the rubric. Replace words like "helpful" with conditions you can check, such as "quotes the refund window from the policy page".
• Add real examples. Two or three results that worked and two or three that didn't, written out in full.
• Make it binary. Ask for pass or fail against each condition instead of a 1–10 score you threshold later.

The judge prompt builder writes a prompt in this shape. These numbers hold for one model, one prompt and one judge prompt, so re-check after any change: run the new judge on your labelled set and paste the verdicts here again.