Publish Sign in
Workflow·AI & Agents

Validate Evaluator

Calibrate an LLM judge against human labels using data splits, TPR/TNR, and bias correction.

You say
Buy it · $49 Read it before you buy $49 Written by hamelsmu · unverified publisher
Context cost
2.2k tokensestimated from the bundle, loaded when it triggers
Bundle
1 file · 8.9 kBtext throughout, nothing executable
Licence
MITpaid listing
Last change
—no release on file
Servers it uses
Noneruns standalone

What it does

Calibrate an LLM judge against human labels using data splits, TPR/TNR, and bias correction. Use after writing a judge prompt (write-judge-prompt) when you need to verify alignment before trusting its outputs. Do NOT use for code-based evaluators (those are deterministic; test with standard unit tests).

Installed, it changes the agent in these ways.

What this skill changes about the agent is not written down here yet. The listing was collected from its source, and the description is in its own SKILL.md.

Workflow

Runs a procedure end to end.

evalsllm-as-judgecalibration
Filed under

AI & Agents

The skill itself

This is the whole product. A skill is instructions the model reads, so there is nothing behind the listing you cannot see first — the front matter loads with every session, and the body below it loads when the skill triggers.

SKILL.md8.9 kB · 216 lines
--- name: validate-evaluator description: > Calibrate an LLM judge against human labels using data splits, TPR/TNR, and bias correction. Use after writing a judge prompt (write-judge-prompt) when you need to verify alignment before trusting its outputs. Do NOT use for code-based evaluators (those are deterministic; test with standard unit tests). ---
10# Validate Evaluator
11
12Calibrate an LLM judge against human judgment.
13
14## Overview
15
161. Split human-labeled data into train (10-20%), dev (40-45%), test (40-45%)
172. Run judge on dev set and measure TPR/TNR
183. Iterate on the judge until TPR and TNR > 90% on dev set
194. Run once on held-out test set for final TPR/TNR
205. Apply bias correction formula to production data
21
22## Prerequisites
23
24- A built LLM judge prompt (from write-judge-prompt)
25- Human-labeled data: ~100 traces with binary Pass/Fail labels per failure mode
26 - Aim for ~50 Pass and ~50 Fail (balanced, even if real distribution is skewed)
27 - Labels must come from a domain expert, not outsourced annotators
28- Candidate few-shot examples from your labeled data
29
30## Core Instructions
31
32### Step 1: Create Data Splits
33
34Split human-labeled data into three disjoint sets:
35
36| Split | Size | Purpose | Rules |
37|-------|------|---------|-------|
38| **Training** | 10-20% (~10-20 examples) | Source of few-shot examples for the judge prompt | Only clear-cut Pass and Fail cases. Used directly in the prompt. |
39| **Dev** | 40-45% (~40-45 examples) | Iterative evaluator refinement | Never include in the prompt. Evaluate against repeatedly. |
40| **Test** | 40-45% (~40-45 examples) | Final unbiased accuracy measurement | Do NOT look at during development. Used once at the end. |
41
42Target: 30-50 examples of each class (Pass and Fail) across dev and test combined. Use balanced splits even if real-world prevalence is skewed — you need enough Fail examples to measure TNR reliably.
43
44```python
45from sklearn.model_selection import train_test_split
46
47# First split: separate test set
48train_dev, test = train_test_split(
49 labeled_data, test_size=0.4, stratify=labeled_data['label'], random_state=42
50)
51# Second split: separate training examples from dev set
52train, dev = train_test_split(
53 train_dev, test_size=0.75, stratify=train_dev['label'], random_state=42
54)
55# Result: ~15% train, ~45% dev, ~40% test
56```
57
58### Step 2: Run Evaluator on Dev Set
59
60Run the judge on every example in the dev set. Compare predictions to human labels.
61
62### Step 3: Measure TPR and TNR
63
64**TPR (True Positive Rate):** When a human says Pass, how often does the judge also say Pass?
65
66```
67TPR = (judge says Pass AND human says Pass) / (human says Pass)
68```
69
70**TNR (True Negative Rate):** When a human says Fail, how often does the judge also say Fail?
71
72```
73TNR = (judge says Fail AND human says Fail) / (human says Fail)
74```
75
76```python
77from sklearn.metrics import confusion_matrix
78
79tn, fp, fn, tp = confusion_matrix(human_labels, evaluator_labels,
80 labels=['Fail', 'Pass']).ravel()
81tpr = tp / (tp + fn)
82tnr = tn / (tn + fp)
83```
84
85Use TPR/TNR, not Precision/Recall or raw accuracy. These two metrics directly map to the bias correction formula. Use Cohen's Kappa only for measuring agreement between two human annotators, not for judge-vs-ground-truth.
86
87### Step 4: Inspect Disagreements
88
89Examine every case where the judge disagrees with human labels:
90
91| Disagreement Type | Judge | Human | Fix |
92|-------------------|-------|-------|-----|
93| **False Pass** | Pass | Fail | Judge is too lenient. Strengthen Fail definitions or add edge-case examples. |
94| **False Fail** | Fail | Pass | Judge is too strict. Clarify Pass definitions or adjust examples. |
95
96For each disagreement, determine whether to:
97- Clarify wording in the judge prompt
98- Swap or add few-shot examples from the training set
99- Add explicit rules for the edge case
100- Split the criterion into more specific sub-checks
101
102### Step 5: Iterate
103
104Refine the judge prompt and re-run on the dev set. Repeat until TPR and TNR stabilize.
105
106**Stopping criteria:**
107- **Target:** TPR > 90% AND TNR > 90%
108- **Minimum acceptable:** TPR > 80% AND TNR > 80%
109
110**If alignment stalls:**
111
112| Problem | Solution |
113|---------|---------|
114| TPR and TNR both low | Use a more capable LLM for the judge |
115| One metric low, one acceptable | Inspect disagreements for the low metric specifically |
116| Both plateau below target | Decompose the criterion into smaller, more atomic checks |
117| Consistently wrong on certain input types | Add targeted few-shot examples from training set |
118| Labels themselves seem inconsistent | Re-examine human labels; the rubric may need refinement |
119
120### Step 6: Final Measurement on Test Set
121
122Run the judge **exactly once** on the held-out test set. Record final TPR and TNR.
123
124Do not iterate after seeing test set results. Go back to step 4 with new dev data if needed.
125
126### Step 7 (Optional): Estimate True Success Rate (Rogan-Gladen Correction)
127
128Raw judge scores on unlabeled production data are biased. If you need an accurate aggregate pass rate, correct for known judge errors:
129
130```
131theta_hat = (p_obs + TNR - 1) / (TPR + TNR - 1)
132```
133
134Where:
135- p_obs = fraction of unlabeled traces the judge scored as Pass
136- TPR, TNR = from test set measurement
137- theta_hat = corrected estimate of true success rate
138
139Clip to [0, 1]. Invalid when TPR + TNR - 1 is near 0 (judge is no better than random).
140
141**Example:**
142- Judge TPR = 0.92, TNR = 0.88
143- 500 production traces: 400 scored Pass -> p_obs = 0.80
144- theta_hat = (0.80 + 0.88 - 1) / (0.92 + 0.88 - 1) = 0.68 / 0.80 = **0.85**
145- True success rate is ~85%, not the raw 80%
146
147### Step 8: Confidence Interval
148
149Compute a bootstrap confidence interval. A point estimate alone is not enough.
150
151```python
152import numpy as np
153
154def bootstrap_ci(human_labels, eval_labels, p_obs, n_bootstrap=2000):
155 """Bootstrap 95% CI for corrected success rate."""
156 n = len(human_labels)
157 estimates = []
158 for _ in range(n_bootstrap):
159 idx = np.random.choice(n, size=n, replace=True)
160 h = np.array(human_labels)[idx]
161 e = np.array(eval_labels)[idx]
162
163 tp = ((h == 'Pass') & (e == 'Pass')).sum()
164 fn = ((h == 'Pass') & (e == 'Fail')).sum()
165 tn = ((h == 'Fail') & (e == 'Fail')).sum()
166 fp = ((h == 'Fail') & (e == 'Pass')).sum()
167
168 tpr_b = tp / (tp + fn) if (tp + fn) > 0 else 0
169 tnr_b = tn / (tn + fp) if (tn + fp) > 0 else 0
170 denom = tpr_b + tnr_b - 1
171
172 if abs(denom) < 1e-6:
173 continue
174 theta = (p_obs + tnr_b - 1) / denom
175 estimates.append(np.clip(theta, 0, 1))
176
177 return np.percentile(estimates, 2.5), np.percentile(estimates, 97.5)
178
179lower, upper = bootstrap_ci(test_human, test_eval, p_obs=0.80)
180print(f"95% CI: [{lower:.2f}, {upper:.2f}]")
181```
182
183Or use judgy (pip install judgy):
184
185```python
186from judgy import estimate_success_rate
187
188# judgy expects 0/1 integer labels (1 = Pass, 0 = Fail)
189test_labels = [1 if l == 'Pass' else 0 for l in test_human_labels]
190test_preds = [1 if l == 'Pass' else 0 for l in test_eval_labels]
191unlabeled_preds = [1 if l == 'Pass' else 0 for l in prod_eval_labels]
192
193theta_hat, lower, upper = estimate_success_rate(
194 test_labels, test_preds, unlabeled_preds
195)
196print(f"Corrected rate: {theta_hat:.2f}")
197print(f"95% CI: [{lower:.2f}, {upper:.2f}]")
198```
199
200## Practical Guidance
201
202- **Pin exact model versions** for LLM judges (a dated snapshot id like <model>-<YYYY-MM-DD>, not a floating alias). Providers update models without notice, causing silent drift.
203- **Re-validate** after changing the judge prompt, switching models, or when production confidence intervals widen unexpectedly.
204- Use ~100 labeled examples (50 Pass, 50 Fail). Below 60, confidence intervals become wide.
205- **One trusted domain expert** is the most efficient labeling path. If not feasible, have two annotators label 20-50 traces independently and resolve disagreements before proceeding.
206- **Improving TPR narrows the confidence interval more than improving TNR.** The correction divides by (TPR + TNR - 1), so a low TPR shrinks the denominator and amplifies estimation errors into wide CIs.
207
208## Anti-Patterns
209
210- **Assuming judges "just work" without validation.** A judge may consistently miss failures or flag passing traces.
211- **Using raw accuracy or percent agreement.** Use TPR and TNR. With class imbalance, raw accuracy is misleading.
212- **Dev/test examples as few-shot examples.** This is data leakage.
213- **Reporting dev set performance as final accuracy.** Dev numbers are optimistic. The test set gives the unbiased estimate.
214- **Raw judge scores without bias correction.** If you report an aggregate pass rate, apply the Rogan-Gladen formula (Step 7).
215- **Point estimates without confidence intervals.** A corrected rate of 85% could easily be 78-92% with small test sets. Report the range so stakeholders know how much to trust the number.
216
In the file
SKILL.md1,400 words
Files1
LicenceMIT
Why you can read it

Nothing in a skill executes. The client loads the text and the model follows it, so a skill can be audited the way a runbook is — by reading it.

What it costs in context

Skills are not billed by the call. They are paid for in context: every token the instructions occupy is a token your code, your diff and your conversation cannot use. Here is what this one takes and when it takes it.

≈90
always loaded
The name and description, so the model knows the skill exists and when to reach for it.
2,135
on trigger
The instruction body, read only when the skill fires.
1.1%
of a 200k window
Ten skills this size would take about 11% of the window before you open a file.
050k100k150k200k context window

2.2k tokens, estimated from the bundle at four bytes to the token, held for the rest of the session once it triggers. Middling. Fine to keep on in a project where you use it weekly, worth unloading in one where you never do.

Servers bill, skills cost

A server charges by the month. A skill charges once per session, in context, and then keeps charging it for as long as the session lives.

Before and after

The same question, put to the same model twice: once as it comes, and once with these instructions loaded.

No worked example has been published for this skill yet.

Adoption
Installsnone yet
Ratingno reviews yet

The procedure it runs

The procedure has not been published here. It is in the skill’s own SKILL.md, which its author has not sent to the marketplace yet.

Prose, not code

These steps are written for a model to follow, not executed by a runtime. It can still be told to skip one, and it will say so when it does.

Servers it uses

None. This skill calls no MCP servers at all.

Everything it needs is in the instructions, so it works in a project with nothing connected — the model reads the file and changes how it works with what it can already reach.

It writes no files and reaches no network. All it changes is how the model reasons and writes.

What it asks for
Writes filesno
Network accessno

Read from the allowed-tools line of this skill’s own SKILL.md. A skill grants no permissions of its own — it can only ask for tools your client already has.

What it will not do

Every skill is narrow, and the useful ones say where they stop. These are the jobs this one is the wrong tool for.

What this skill is not for has not been published here. Nothing is implied by that: it is a section the author has not filled in.

What is in the bundle

1 file, 8.9 kB on disk. A bundle is text throughout: the instructions the model reads, plus the templates it fills in.

  • SKILL.md8.9 kB
What is not in it

No dependencies and nothing executable: a skill is text the agent reads, so the bundle is 1 file you can review in full before installing. The MIT licence covers the templates and examples as well as the instructions.

Install

Installing copies the bundle into your project. Nothing runs at install time — the files sit on disk until the model reads them.

$49 once
Validate Evaluator · MIT · hamelsmu
one-time
Price$49 once
LicenceMIT — the author’s, unchanged by this purchase
Paid throughStripe, once, on the card you add at the checkout
Keeps workingfor good — the files are yours once they are on disk
Updatesevery update its author ships, delivered through this account

You can read the whole bundle before paying — the SKILL.md above is the product, not a preview of it. What the money buys is the delivery: the folder packaged and handed to your machine by key, every update its author ships, and our support if it does not do what this listing says. The terms of use are MIT, set by the author and unchanged by buying it here.

Sign in to buyCreate an account

Payment runs through Stripe, on a page like this one rather than a redirect. Once there is an account it joins the same mcprush invoice as everything else you run, so there is never a second card to enter.

Which clients pick it up on their own

A skill is a folder of text. A client with a skills folder reads it without being told; everywhere else the same text works, it is just handed to the model rather than found.

Claude Code.claude/skills/
Claude Desktop
ChatGPT
Cursor.cursor/skills/
VS Code.github/skills/
Codex CLI.agents/skills/
Gemini CLI.gemini/skills/
Grok.grok/skills/
Zed.agents/skills/
Windsurf.windsurf/skills/
Agent SDK.claude/skills/
HTTP / API
This release
Versionnot versioned
Publishedno release date on file
Price$49
Referencehamelsmu/validate-evaluator

Versions

Its author publishes no version number, so there is nothing here to pin to: what you install is the folder as it stands today. Instructions change more often than APIs do — a skill can be rewritten entirely without anything it depends on moving.

v
  • No earlier releases have been published to the marketplace.
Pinning

Nothing to pin to: this skill carries no version number of its own. What you install is what the folder holds on the day you install it.

Reviews

—no reviews yet · no installs yet

Nobody has reviewed this skill yet. The rating is the mean of the reviews written here, so there is none until somebody writes the first.

Who can post

Only accounts that have had the skill installed for fourteen days, so a review is written after living with it rather than after reading it. Publishers may reply once.

Who wrote it

HA
hamelsmu

Publishes on mcprush.

0 servers listed4 skills listednot claimed
Profile
Publisher
Servers0