Write a judge that checks your AI's results
Pick what your AI step does, such as answer a question or pull fields from a document, and get a pass/fail judge prompt with starter conditions and JSON output.
Free. Runs in your browser. No sign-up.
The AI step to check
A judge is a second AI call that checks the first one's result and says pass or fail. Each kind of step fails in its own way, so each gets its own judge. A support bot, for example, finds help articles, answers, then decides whether to hand over to a person.
- Worked first time, for Answer
- It correctly answered the question the user asked, so they could act on it without asking again.
- Failed, as your app sees it
- The user asked again, corrected it, or asked for a human.
Who uses it, what they ask for, and what a result that worked gives them.
Pass conditions
The result passes only if every condition holds. Start from these, then write one for each reason this step's results fail in your app, specific enough that two people would agree.
Real examples
Worth more than extra rules. Paste real results from your app, written out in full. Empty slots show in the prompt as [brackets] to fill later; remove any you won't use.
Results that worked
It correctly answered the question the user asked, so they could act on it without asking again. Marked pass in the prompt. Up to 3.
Worked example 1
Worked example 2
Results that failed
The user asked again, corrected it, or asked for a human. Pick ones that look fine at a glance: those are the ones a judge misses. Marked fail in the prompt. Up to 3.
Failed example 1
Failed example 2
The judge's answer
One or two sentences. It costs a few tokens per call and makes the judge check the conditions before it decides.
Your judge prompt
Updates as you edit. Your code fills in {{input}} and {{result}} for each result it checks.
Still in [brackets]: the feature description, 2 slots for results that worked and 2 slots for results that failed. Fill them in here, or in the copied text before you use it.
You are a judge: a second AI call that checks one result from an AI app. The step that made it replies to a user's question that has a right answer, such as a support reply or a policy question. You will see the input the step received and the result it produced, and decide whether the result passes or fails.
Pass means the result worked first time: it correctly answered the question the user asked, so they could act on it without asking again. Fail means it didn't: in the live app, the user asked again, corrected it, or asked for a human.
# The feature
[Describe the feature: who uses it, what they ask for, and what a result that worked gives them.]
# Pass conditions
The result passes only if every one of these conditions holds:
1. Every claim is correct and supported by the provided context. Nothing is made up, including features, prices, policies and links.
2. It answers the question the user actually asked, not a nearby or more general one.
3. If the context doesn't contain the answer, it says so and offers a person or a next step instead of guessing.
4. It gives the user what they need to act, such as the steps or the setting to change, not just a pointer to the docs.
The verdict is "fail" if any condition fails. Check each condition on its own: doing well on one doesn't make up for failing another. Judge only these conditions, so length, tone and style don't count unless a condition says they do. If you can't tell whether a condition holds, ask which is more likely in the live app: that this result worked first time, or that it failed.
# Examples
These are real results from this step. The ones marked pass worked first time. The ones marked fail didn't.
<example>
<input>
[A real customer message, with the context the step was given]
</input>
<result>
[The reply, in full, from a real case that worked first time]
</result>
<verdict>pass</verdict>
</example>
<example>
<input>
[A real customer message, with the context the step was given]
</input>
<result>
[The reply, in full, from a real case where the user asked again, corrected it, or asked for a human]
</result>
<verdict>fail</verdict>
</example>
<example>
<input>
[A real customer message, with the context the step was given]
</input>
<result>
[The reply, in full, from a real case that worked first time]
</result>
<verdict>pass</verdict>
</example>
<example>
<input>
[A real customer message, with the context the step was given]
</input>
<result>
[The reply, in full, from a real case where the user asked again, corrected it, or asked for a human]
</result>
<verdict>fail</verdict>
</example>
# The result to judge
The input is the user's message, the conversation so far and the context the step was given, such as help-centre articles. The result is the reply.
<input>
{{input}}
</input>
<result>
{{result}}
</result>
# Your answer
Reply with only this JSON object, with no other text before or after it:
{"reason": "…", "failed_conditions": […], "verdict": "pass" | "fail"}
- "reason": one or two short sentences, written before you decide. Name each failed condition by its number and what you saw, or say why every condition holds.
- "failed_conditions": the numbers of the conditions that failed, or [] if none did.
- "verdict": "fail" if any condition failed, otherwise "pass".Check this judge against 50+ human verdicts before trusting it. The judge agreement checker shows how often it agrees with people beyond chance, and whether it catches the results that failed.
What a judge is
A judge is a second AI call that checks the first one's result. You give it the input, the result and a short list of written conditions, and it answers pass or fail. You can't read every reply your support bot sends or every invoice your app reads, but a judge can check all of them. That tells you how often each step works first time, and which results failed.
Start from what the step does
Different kinds of step fail in different ways. A support reply fails when it makes up a policy or answers a nearby question. An invoice reader fails when one digit of the total is wrong. A lead score fails when it's confident on thin evidence. Pick the kind of step above and you get starter conditions for its usual failures, plus the signal your app already has for them: a customer asking again, someone fixing a field, a moderator overruling a label.
Pass or fail beats a 1–10 score
Scores drift. A 7 from one run is an 8 from the next, and you still have to pick a threshold before you can act. A pass or fail tied to written conditions is easier to check against people and easier to act on: retry the result or don't. It also matches what users do with a result, which is use it or try again.
Write conditions from real failures
Open the results that failed and ask why each one did. Each reason becomes a condition, such as "the refund window matches the policy article" or "the VAT amount matches the invoice", not "high quality". A good condition is one that two people would agree on. Generic conditions like "helpful" or "accurate" pass almost everything.
Fill the slots with real results
Two or three results that worked and two or three that failed, written out in full, show the judge where the line is better than more rules do. Choose failures that look fine at a glance, because those are the ones a judge misses. Don't invent examples: made-up failures are more obvious than real ones, so the judge only learns the easy cases.
Keep the reasoning short, and before the verdict
A reason written first makes the judge look at the conditions before it commits; one written after only justifies the verdict. Keep it to a sentence or two that names the failed conditions. If cost matters most, turn it off and keep the list of failed conditions.
One judge per AI step
A support bot that finds help articles, writes a reply and decides whether to hand over to a person has three steps that fail in three different ways. A single judge for the whole bot ends up checking nothing in particular. Build one prompt per step.
Check the judge against people
A judge prompt is a guess until you compare it with people. Take 50 or more results, including some that failed, run the judge on them, and paste its verdicts and yours into the judge agreement checker. Look at how many failures it catches, not just how often it agrees. Check again whenever you change the judge, the model it checks or that model's prompt.
Questions
What is a judge?
A second AI call that checks the first one's result against conditions you wrote, and answers pass or fail. You can't read every reply or extraction your app makes, but a judge can check them all.
Why pass or fail instead of a 1–10 score?
Scores drift and are hard to compare between runs and judges. A pass or fail tied to written conditions is easier to check against people and easier to act on.
Should the judge be a bigger model than the one it checks?
Not necessarily. Clear conditions and a few real examples matter more than model size. Check any judge against people's verdicts before you trust it.
Where do the examples come from?
From your own app. Results that worked first time are passes. Results where the user asked again, undid or fixed something, or overruled the AI are fails. Two or three of each, written out in full, beat a long list of rules.