Workflow·AI & Agents

Write Judge Prompt

Design LLM-as-Judge evaluators for subjective criteria that code-based checks cannot handle.

You say
Buy it · $12 Read it before you buy $12 Written by hamelsmu · unverified publisher
Context cost
1.9k tokensestimated from the bundle, loaded when it triggers
Bundle
1 file · 7.7 kBtext throughout, nothing executable
Licence
MITpaid listing
Last change
no release on file
Servers it uses
Noneruns standalone

What it does

Design LLM-as-Judge evaluators for subjective criteria that code-based checks cannot handle. Use when a failure mode requires interpretation (tone, faithfulness, relevance, completeness). Do NOT use when the failure mode can be checked with code (regex, schema validation, execution tests). Do NOT use when you need to validate or calibrate the judge — use validate-evaluator instead.

Installed, it changes the agent in these ways.

What this skill changes about the agent is not written down here yet. The listing was collected from its source, and the description is in its own SKILL.md.

Workflow

Runs a procedure end to end.

evalsllm-as-judgeprompt-engineering
Filed under

AI & Agents

The skill itself

This is the whole product. A skill is instructions the model reads, so there is nothing behind the listing you cannot see first — the front matter loads with every session, and the body below it loads when the skill triggers.

SKILL.md7.7 kB · 145 lines
--- name: write-judge-prompt description: > Design LLM-as-Judge evaluators for subjective criteria that code-based checks cannot handle. Use when a failure mode requires interpretation (tone, faithfulness, relevance, completeness). Do NOT use when the failure mode can be checked with code (regex, schema validation, execution tests). Do NOT use when you need to validate or calibrate the judge — use validate-evaluator instead. ---
11# Write LLM-as-Judge Prompt
12
13Design a binary Pass/Fail LLM-as-Judge evaluator for one specific failure mode. Each judge checks exactly one thing.
14
15## Prerequisites
16
17- Error analysis is complete. The failure mode is identified.
18- You have human-labeled traces for this failure mode (at least 20 Pass and 20 Fail examples).
19- A code-based evaluator cannot check this failure mode. Exhaust code-based options before reaching for a judge — many failure modes that seem subjective reduce to keyword checks, regex, or API calls when you understand the domain. Example: detecting whether an AI interviewing coach suggests "general" questions (asking about typical behavior instead of a specific past event) seems to require semantic understanding, but in practice a keyword check for words like "usually," "typical," and "normally" could work quite well.
20
21## The Four Components
22
23Every judge prompt requires exactly four components:
24
25### 1. Task and Evaluation Criterion
26
27State what the judge evaluates. One failure mode per judge.
28
29```
30You are an evaluator assessing whether a real estate assistant's email
31uses the appropriate tone for the client's persona.
32```
33
34Not: "Evaluate whether the email is good" or "Rate the email quality from 1-5."
35
36### 2. Pass/Fail Definitions
37
38Outcomes are strictly binary: Pass or Fail. No Likert scales, no letter grades, no partial credit. Define exactly what constitutes Pass and Fail. These definitions come from your error analysis failure mode descriptions.
39
40```
41## Definitions
42
43PASS: The email matches the expected communication style for the client persona:
44- Luxury Buyers: formal language, emphasis on exclusive features, premium
45 market positioning, no casual slang
46- First-Time Homebuyers: warm and encouraging tone, educational explanations,
47 avoids jargon, patient and supportive
48- Investors: data-driven language, ROI-focused, market analytics, concise
49 and professional
50
51FAIL: The email uses a tone mismatched to the client persona. Examples:
52- Using casual slang ("hey, check out this pad!") for a luxury buyer
53- Using heavy financial jargon for a first-time homebuyer
54- Using overly emotional language for an investor
55```
56
57### 3. Few-Shot Examples
58
59Include labeled Pass and Fail examples from your human-labeled data.
60
61```
62## Examples
63
64### Example 1: PASS
65Client Persona: Luxury Buyer
66Email: "Dear Mr. Harrington, I am pleased to present an exclusive listing
67at 1200 Pacific Heights Drive. This distinguished property features..."
68Critique: The email opens with a formal salutation and uses language
69consistent with luxury positioning — "exclusive listing," "distinguished
70property." No casual slang or informal phrasing. The tone matches the
71luxury buyer persona throughout.
72Result: Pass
73
74### Example 2: FAIL
75Client Persona: Luxury Buyer
76Email: "Hey! Just found this awesome place you might like. It's got a
77pool and stuff, super cool neighborhood..."
78Critique: The greeting "Hey!" is informal. Phrases like "awesome place,"
79"got a pool and stuff," and "super cool" are casual slang inappropriate
80for a luxury buyer. The email reads like a text message, not a
81professional communication for a high-end client.
82Result: Fail
83
84### Example 3: PASS (borderline)
85Client Persona: First-Time Homebuyer
86Email: "Hi Sarah, I found a property that might be a great fit for your
87first home. The neighborhood has good schools nearby, and the monthly
88payment would be similar to what you're currently paying in rent..."
89Critique: The greeting is warm but not overly casual. The email explains
90the property in relatable terms — comparing mortgage to rent, mentioning
91schools — which is educational without being condescending. It avoids
92jargon like "amortization" or "LTV ratio." While not deeply technical,
93this matches the supportive tone expected for a first-time buyer.
94Result: Pass
95```
96
97**Rules for selecting examples:**
98- Include at least one clear Pass, one clear Fail, and one borderline case. Borderline examples are the most valuable — they teach nuance.
99- Draw examples from the training split (10-20% of labeled data set aside for this purpose).
100- Any example used in the judge prompt must be excluded from dev and test sets. Using dev/test examples is data leakage.
101- 2-4 examples is typical. Performance plateaus after 4-8.
102
103### 4. Structured Output Format
104
105Enforce structured output using your LLM provider's schema enforcement (e.g., response_format in OpenAI, tool definitions in Anthropic) or a library like Instructor or Outlines. If the provider doesn't support schema enforcement, specify the JSON schema in the prompt.
106
107The output must include a critique before the verdict. Placing the critique first forces the judge to articulate its assessment before committing to a decision.
108
109```json
110{
111 "critique": "string — detailed assessment of the output against the criterion",
112 "result": "Pass or Fail"
113}
114```
115
116Critiques must be detailed, not terse. A good critique explains what specifically was correct or incorrect and references concrete evidence from the output. The critiques in your few-shot examples set the bar for the level of detail the judge will produce.
117
118## Choosing What to Pass to the Judge
119
120Feed only what the judge needs for an accurate decision:
121
122| Failure Mode | What the Judge Needs |
123|-------------|---------------------|
124| Tone mismatch | Client persona + generated email |
125| Answer faithfulness | Retrieved context + generated answer |
126| SQL correctness | User query + generated SQL + schema |
127| Instruction following | System prompt rules + generated response |
128| Tool call justification | Conversation history + tool call + tool result |
129
130For long documents, feed only the relevant snippet, not the entire document.
131
132## Model Selection
133
134Start with the most capable model available. The same model used for the main task works as judge (the judge performs a different, narrower task). Optimize for cost later once alignment is confirmed.
135
136## Anti-Patterns
137
138- **Vague criteria like "is this helpful?"** Target a specific, observable failure mode from error analysis.
139- **Holistic judge for the entire trace.** A single judge covering multiple dimensions produces unactionable verdicts.
140- **No few-shot examples.** Without examples, the model won't know what counts as a failure in your application.
141- **Dev/test examples used as few-shot.** This is data leakage. Use only the training split.
142- **Likert scales (1-5, letter grades, etc.).** Binary pass/fail only. Likert scales produce scores that sound precise but can't be calibrated: annotators disagree on the difference between a 3 and a 4, and the judge inherits that noise. Binary forces you to define a clear decision boundary upfront, which makes inter-annotator agreement measurable and the judge's errors actionable. If you need to capture severity, use multiple binary judges (e.g., "factually wrong" and "dangerously wrong") rather than one ordinal scale.
143- **Skipping validation.** Measure alignment with human labels using validate-evaluator before trusting the judge.
144- **Judges for specification failures without fixing the prompt first.** If the prompt never asked for the behavior, add the instruction before building an evaluator. For critical requirements, a judge can still serve as a regression guard.
145
In the file
SKILL.md1,173 words
Files1
LicenceMIT
Why you can read it

Nothing in a skill executes. The client loads the text and the model follows it, so a skill can be audited the way a runbook is — by reading it.

What it costs in context

Skills are not billed by the call. They are paid for in context: every token the instructions occupy is a token your code, your diff and your conversation cannot use. Here is what this one takes and when it takes it.

≈110
always loaded
The name and description, so the model knows the skill exists and when to reach for it.
1,815
on trigger
The instruction body, read only when the skill fires.
0.96%
of a 200k window
Ten skills this size would take about 10% of the window before you open a file.
050k100k150k200k context window

1.9k tokens, estimated from the bundle at four bytes to the token, held for the rest of the session once it triggers. Middling. Fine to keep on in a project where you use it weekly, worth unloading in one where you never do.

Servers bill, skills cost

A server charges by the month. A skill charges once per session, in context, and then keeps charging it for as long as the session lives.

Before and after

The same question, put to the same model twice: once as it comes, and once with these instructions loaded.

No worked example has been published for this skill yet.

Adoption
Installsnone yet
Ratingno reviews yet

The procedure it runs

The procedure has not been published here. It is in the skill’s own SKILL.md, which its author has not sent to the marketplace yet.

Prose, not code

These steps are written for a model to follow, not executed by a runtime. It can still be told to skip one, and it will say so when it does.

Servers it uses

None. This skill calls no MCP servers at all.

Everything it needs is in the instructions, so it works in a project with nothing connected — the model reads the file and changes how it works with what it can already reach.

It writes no files and reaches no network. All it changes is how the model reasons and writes.

What it asks for
Writes filesno
Network accessno

Read from the allowed-tools line of this skill’s own SKILL.md. A skill grants no permissions of its own — it can only ask for tools your client already has.

What it will not do

Every skill is narrow, and the useful ones say where they stop. These are the jobs this one is the wrong tool for.

What this skill is not for has not been published here. Nothing is implied by that: it is a section the author has not filled in.

What is in the bundle

1 file, 7.7 kB on disk. A bundle is text throughout: the instructions the model reads, plus the templates it fills in.

  • SKILL.md7.7 kB
What is not in it

No dependencies and nothing executable: a skill is text the agent reads, so the bundle is 1 file you can review in full before installing. The MIT licence covers the templates and examples as well as the instructions.

Install

Installing copies the bundle into your project. Nothing runs at install time — the files sit on disk until the model reads them.

$12 once
Write Judge Prompt · MIT · hamelsmu
one-time
Price$12 once
LicenceMIT — the author’s, unchanged by this purchase
Paid throughStripe, once, on the card you add at the checkout
Keeps workingfor good — the files are yours once they are on disk
Updatesevery update its author ships, delivered through this account

You can read the whole bundle before paying — the SKILL.md above is the product, not a preview of it. What the money buys is the delivery: the folder packaged and handed to your machine by key, every update its author ships, and our support if it does not do what this listing says. The terms of use are MIT, set by the author and unchanged by buying it here.

Payment runs through Stripe, on a page like this one rather than a redirect. Once there is an account it joins the same mcprush invoice as everything else you run, so there is never a second card to enter.

Which clients pick it up on their own

A skill is a folder of text. A client with a skills folder reads it without being told; everywhere else the same text works, it is just handed to the model rather than found.

Claude Code.claude/skills/
Claude Desktop
ChatGPT
Cursor.cursor/skills/
VS Code.github/skills/
Codex CLI.agents/skills/
Gemini CLI.gemini/skills/
Grok.grok/skills/
Zed.agents/skills/
Windsurf.windsurf/skills/
Agent SDK.claude/skills/
HTTP / API
This release
Versionnot versioned
Publishedno release date on file
Price$12
Referencehamelsmu/write-judge-prompt

Versions

Its author publishes no version number, so there is nothing here to pin to: what you install is the folder as it stands today. Instructions change more often than APIs do — a skill can be rewritten entirely without anything it depends on moving.

v
  • No earlier releases have been published to the marketplace.
Pinning

Nothing to pin to: this skill carries no version number of its own. What you install is what the folder holds on the day you install it.

Reviews

no reviews yet · no installs yet

Nobody has reviewed this skill yet. The rating is the mean of the reviews written here, so there is none until somebody writes the first.

Who can post

Only accounts that have had the skill installed for fourteen days, so a review is written after living with it rather than after reading it. Publishers may reply once.

Publisher
Servers0