Eval Harness

Formal evaluation framework for Claude Code sessions implementing eval-driven development (EDD) principles.

You say
Buy it · $89 Read it before you buy $89 Written by affaan-m · unverified publisher
Context cost
1.6k tokensestimated from the bundle, loaded when it triggers
Bundle
1 file · 6.6 kBtext throughout, nothing executable
Licence
MITpaid listing
Last change
no release on file
Servers it uses
Noneruns standalone

What it does

Formal evaluation framework for Claude Code sessions implementing eval-driven development (EDD) principles. Use when a Claude Code workflow needs a formal eval before it is trusted or changed.

Installed, it changes the agent in these ways.

What this skill changes about the agent is not written down here yet. The listing was collected from its source, and the description is in its own SKILL.md.

Expertise

Domain judgement the base model does not have.

securitytesting

The skill itself

This is the whole product. A skill is instructions the model reads, so there is nothing behind the listing you cannot see first — the front matter loads with every session, and the body below it loads when the skill triggers.

SKILL.md6.6 kB · 272 lines
--- name: eval-harness description: Formal evaluation framework for Claude Code sessions implementing eval-driven development (EDD) principles. Use when a Claude Code workflow needs a formal eval before it is trusted or changed. metadata: origin: ECC tools: Read, Write, Edit, Bash, Grep, Glob ---
9# Eval Harness Skill
10
11A formal evaluation framework for Claude Code sessions, implementing eval-driven development (EDD) principles.
12
13## When to Activate
14
15- Setting up eval-driven development (EDD) for AI-assisted workflows
16- Defining pass/fail criteria for Claude Code task completion
17- Measuring agent reliability with pass@k metrics
18- Creating regression test suites for prompt or agent changes
19- Benchmarking agent performance across model versions
20
21## Philosophy
22
23Eval-Driven Development treats evals as the "unit tests of AI development":
24- Define expected behavior BEFORE implementation
25- Run evals continuously during development
26- Track regressions with each change
27- Use pass@k metrics for reliability measurement
28
29## Eval Types
30
31### Capability Evals
32Test if Claude can do something it couldn't before:
33```markdown
34[CAPABILITY EVAL: feature-name]
35Task: Description of what Claude should accomplish
36Success Criteria:
37 - [ ] Criterion 1
38 - [ ] Criterion 2
39 - [ ] Criterion 3
40Expected Output: Description of expected result
41```
42
43### Regression Evals
44Ensure changes don't break existing functionality:
45```markdown
46[REGRESSION EVAL: feature-name]
47Baseline: SHA or checkpoint name
48Tests:
49 - existing-test-1: PASS/FAIL
50 - existing-test-2: PASS/FAIL
51 - existing-test-3: PASS/FAIL
52Result: X/Y passed (previously Y/Y)
53```
54
55## Grader Types
56
57### 1. Code-Based Grader
58Deterministic checks using code:
59```bash
60# Check if file contains expected pattern
61grep -q "export function handleAuth" src/auth.ts && echo "PASS" || echo "FAIL"
62
63# Check if tests pass
64npm test -- --testPathPattern="auth" && echo "PASS" || echo "FAIL"
65
66# Check if build succeeds
67npm run build && echo "PASS" || echo "FAIL"
68```
69
70### 2. Model-Based Grader
71Use Claude to evaluate open-ended outputs:
72```markdown
73[MODEL GRADER PROMPT]
74Evaluate the following code change:
751. Does it solve the stated problem?
762. Is it well-structured?
773. Are edge cases handled?
784. Is error handling appropriate?
79
80Score: 1-5 (1=poor, 5=excellent)
81Reasoning: [explanation]
82```
83
84### 3. Human Grader
85Flag for manual review:
86```markdown
87[HUMAN REVIEW REQUIRED]
88Change: Description of what changed
89Reason: Why human review is needed
90Risk Level: LOW/MEDIUM/HIGH
91```
92
93## Metrics
94
95### pass@k
96"At least one success in k attempts"
97- pass@1: First attempt success rate
98- pass@3: Success within 3 attempts
99- Typical target: pass@3 > 90%
100
101### pass^k
102"All k trials succeed"
103- Higher bar for reliability
104- pass^3: 3 consecutive successes
105- Use for critical paths
106
107## Eval Workflow
108
109### 1. Define (Before Coding)
110```markdown
111## EVAL DEFINITION: feature-xyz
112
113### Capability Evals
1141. Can create new user account
1152. Can validate email format
1163. Can hash password securely
117
118### Regression Evals
1191. Existing login still works
1202. Session management unchanged
1213. Logout flow intact
122
123### Success Metrics
124- pass@3 > 90% for capability evals
125- pass^3 = 100% for regression evals
126```
127
128### 2. Implement
129Write code to pass the defined evals.
130
131### 3. Evaluate
132```bash
133# Run capability evals
134[Run each capability eval, record PASS/FAIL]
135
136# Run regression evals
137npm test -- --testPathPattern="existing"
138
139# Generate report
140```
141
142### 4. Report
143```markdown
144EVAL REPORT: feature-xyz
145========================
146
147Capability Evals:
148 create-user: PASS (pass@1)
149 validate-email: PASS (pass@2)
150 hash-password: PASS (pass@1)
151 Overall: 3/3 passed
152
153Regression Evals:
154 login-flow: PASS
155 session-mgmt: PASS
156 logout-flow: PASS
157 Overall: 3/3 passed
158
159Metrics:
160 pass@1: 67% (2/3)
161 pass@3: 100% (3/3)
162
163Status: READY FOR REVIEW
164```
165
166## Integration Patterns
167
168### Pre-Implementation
169```
170/eval define feature-name
171```
172Creates eval definition file at .claude/evals/feature-name.md
173
174### During Implementation
175```
176/eval check feature-name
177```
178Runs current evals and reports status
179
180### Post-Implementation
181```
182/eval report feature-name
183```
184Generates full eval report
185
186## Eval Storage
187
188Store evals in project:
189```
190.claude/
191 evals/
192 feature-xyz.md # Eval definition
193 feature-xyz.log # Eval run history
194 baseline.json # Regression baselines
195```
196
197## Best Practices
198
1991. **Define evals BEFORE coding** - Forces clear thinking about success criteria
2002. **Run evals frequently** - Catch regressions early
2013. **Track pass@k over time** - Monitor reliability trends
2024. **Use code graders when possible** - Deterministic > probabilistic
2035. **Human review for security** - Never fully automate security checks
2046. **Keep evals fast** - Slow evals don't get run
2057. **Version evals with code** - Evals are first-class artifacts
206
207## Example: Adding Authentication
208
209```markdown
210## EVAL: add-authentication
211
212### Phase 1: Define (10 min)
213Capability Evals:
214- [ ] User can register with email/password
215- [ ] User can login with valid credentials
216- [ ] Invalid credentials rejected with proper error
217- [ ] Sessions persist across page reloads
218- [ ] Logout clears session
219
220Regression Evals:
221- [ ] Public routes still accessible
222- [ ] API responses unchanged
223- [ ] Database schema compatible
224
225### Phase 2: Implement (varies)
226[Write code]
227
228### Phase 3: Evaluate
229Run: /eval check add-authentication
230
231### Phase 4: Report
232EVAL REPORT: add-authentication
233==============================
234Capability: 5/5 passed (pass@3: 100%)
235Regression: 3/3 passed (pass^3: 100%)
236Status: SHIP IT
237```
238
239## Product Evals (v1.8)
240
241Use product evals when behavior quality cannot be captured by unit tests alone.
242
243### Grader Types
244
2451. Code grader (deterministic assertions)
2462. Rule grader (regex/schema constraints)
2473. Model grader (LLM-as-judge rubric)
2484. Human grader (manual adjudication for ambiguous outputs)
249
250### pass@k Guidance
251
252- pass@1: direct reliability
253- pass@3: practical reliability under controlled retries
254- pass^3: stability test (all 3 runs must pass)
255
256Recommended thresholds:
257- Capability evals: pass@3 >= 0.90
258- Regression evals: pass^3 = 1.00 for release-critical paths
259
260### Eval Anti-Patterns
261
262- Overfitting prompts to known eval examples
263- Measuring only happy-path outputs
264- Ignoring cost and latency drift while chasing pass rates
265- Allowing flaky graders in release gates
266
267### Minimal Eval Artifact Layout
268
269- .claude/evals/<feature>.md definition
270- .claude/evals/<feature>.log run history
271- docs/releases/<version>/eval-summary.md release snapshot
272
In the file
SKILL.md943 words
Files1
LicenceMIT
Why you can read it

Nothing in a skill executes. The client loads the text and the model follows it, so a skill can be audited the way a runbook is — by reading it.

What it costs in context

Skills are not billed by the call. They are paid for in context: every token the instructions occupy is a token your code, your diff and your conversation cannot use. Here is what this one takes and when it takes it.

≈70
always loaded
The name and description, so the model knows the skill exists and when to reach for it.
1,580
on trigger
The instruction body, read only when the skill fires.
0.83%
of a 200k window
Ten skills this size would take about 8% of the window before you open a file.
050k100k150k200k context window

1.6k tokens, estimated from the bundle at four bytes to the token, held for the rest of the session once it triggers. Middling. Fine to keep on in a project where you use it weekly, worth unloading in one where you never do.

Servers bill, skills cost

A server charges by the month. A skill charges once per session, in context, and then keeps charging it for as long as the session lives.

Before and after

The same question, put to the same model twice: once as it comes, and once with these instructions loaded.

No worked example has been published for this skill yet.

Adoption
Installsnone yet
Ratingno reviews yet

The procedure it runs

The procedure has not been published here. It is in the skill’s own SKILL.md, which its author has not sent to the marketplace yet.

Prose, not code

These steps are written for a model to follow, not executed by a runtime. It can still be told to skip one, and it will say so when it does.

Servers it uses

None. This skill calls no MCP servers at all.

Everything it needs is in the instructions, so it works in a project with nothing connected — the model reads the file and changes how it works with what it can already reach.

It asks the agent to write files, using whatever file access your client already has. It never touches the network.

What it asks for
Writes filesyes
Network accessno

Read from the allowed-tools line of this skill’s own SKILL.md. A skill grants no permissions of its own — it can only ask for tools your client already has.

What it will not do

Every skill is narrow, and the useful ones say where they stop. These are the jobs this one is the wrong tool for.

What this skill is not for has not been published here. Nothing is implied by that: it is a section the author has not filled in.

What is in the bundle

1 file, 6.6 kB on disk. A bundle is text throughout: the instructions the model reads, plus the templates it fills in.

  • SKILL.md6.6 kB
What is not in it

No dependencies and nothing executable: a skill is text the agent reads, so the bundle is 1 file you can review in full before installing. The MIT licence covers the templates and examples as well as the instructions.

Install

Installing copies the bundle into your project. Nothing runs at install time — the files sit on disk until the model reads them.

$89 once
Eval Harness · MIT · affaan-m
one-time
Price$89 once
LicenceMIT — the author’s, unchanged by this purchase
Paid throughStripe, once, on the card you add at the checkout
Keeps workingfor good — the files are yours once they are on disk
Updatesevery update its author ships, delivered through this account

You can read the whole bundle before paying — the SKILL.md above is the product, not a preview of it. What the money buys is the delivery: the folder packaged and handed to your machine by key, every update its author ships, and our support if it does not do what this listing says. The terms of use are MIT, set by the author and unchanged by buying it here.

Payment runs through Stripe, on a page like this one rather than a redirect. Once there is an account it joins the same mcprush invoice as everything else you run, so there is never a second card to enter.

Which clients pick it up on their own

A skill is a folder of text. A client with a skills folder reads it without being told; everywhere else the same text works, it is just handed to the model rather than found.

Claude Code.claude/skills/
Claude Desktop
ChatGPT
Cursor.cursor/skills/
VS Code.github/skills/
Codex CLI.agents/skills/
Gemini CLI.gemini/skills/
Grok.grok/skills/
Zed.agents/skills/
Windsurf.windsurf/skills/
Agent SDK.claude/skills/
HTTP / API
This release
Versionnot versioned
Publishedno release date on file
Price$89
Referenceaffaan-m/eval-harness

Versions

Its author publishes no version number, so there is nothing here to pin to: what you install is the folder as it stands today. Instructions change more often than APIs do — a skill can be rewritten entirely without anything it depends on moving.

v
  • No earlier releases have been published to the marketplace.
Pinning

Nothing to pin to: this skill carries no version number of its own. What you install is what the folder holds on the day you install it.

Reviews

no reviews yet · no installs yet

Nobody has reviewed this skill yet. The rating is the mean of the reviews written here, so there is none until somebody writes the first.

Who can post

Only accounts that have had the skill installed for fourteen days, so a review is written after living with it rather than after reading it. Publishers may reply once.

Publisher
Servers0