Publish Sign in
Workflow·AI & Agents

Error Analysis

Help the user systematically identify and categorize failure modes in an LLM pipeline by reading traces.

You say
Buy it · $59 Read it before you buy $59 Written by hamelsmu · unverified publisher
Context cost
2.1k tokensestimated from the bundle, loaded when it triggers
Bundle
1 file · 8.3 kBtext throughout, nothing executable
Licence
MITpaid listing
Last change
—no release on file
Servers it uses
Noneruns standalone

What it does

Help the user systematically identify and categorize failure modes in an LLM pipeline by reading traces. Use when starting a new eval project, after significant pipeline changes (new features, model switches, prompt rewrites), when production metrics drop, or after incidents.

Installed, it changes the agent in these ways.

What this skill changes about the agent is not written down here yet. The listing was collected from its source, and the description is in its own SKILL.md.

Workflow

Runs a procedure end to end.

evalsllmtracesfailure-analysis
Filed under

AI & Agents

The skill itself

This is the whole product. A skill is instructions the model reads, so there is nothing behind the listing you cannot see first — the front matter loads with every session, and the body below it loads when the skill triggers.

SKILL.md8.3 kB · 165 lines
--- name: error-analysis description: > Help the user systematically identify and categorize failure modes in an LLM pipeline by reading traces. Use when starting a new eval project, after significant pipeline changes (new features, model switches, prompt rewrites), when production metrics drop, or after incidents. ---
10# Error Analysis
11
12Guide the user through reading LLM pipeline traces and building a catalog of how the system fails.
13
14## Overview
15
161. Collect ~100 representative traces
172. Read each trace, judge pass/fail, and note what went wrong
183. Group similar failures into categories
194. Label every trace against those categories
205. Compute failure rates to prioritize what to fix
21
22## Core Process
23
24### Step 1: Collect Traces
25
26Capture the full trace: input, all intermediate LLM calls, tool uses, retrieved documents, reasoning steps, and final output.
27
28**Target: ~100 traces.** This is roughly where new traces stop revealing new kinds of failures. The number depends on system complexity.
29
30**From real user data (preferred):**
31- Small volume: random sample
32- Large volume: sample across key dimensions (query type, user segment, feature area)
33- Use embedding clustering (K-means) to ensure diversity
34
35**From synthetic data (when real data is sparse):**
36- Use the generate-synthetic-data skill
37- Run synthetic queries through the full pipeline and capture complete traces
38
39### Step 2: Read Traces and Take Notes
40
41Present each trace to the user. For each one, ask: **did the system produce a good result?** Pass or Fail.
42
43For failures, note what went wrong. Focus on the **first thing that went wrong** in the trace — errors cascade, so downstream symptoms disappear when the root cause is fixed. Don't chase every issue in a single trace.
44
45Write observations, not explanations. "SQL missed the budget constraint" not "The model probably didn't understand the budget."
46
47**Template:**
48
49```
50| Trace ID | Trace | What went wrong | Pass/Fail |
51|----------|-------|-----------------|-----------|
52| 001 | [full trace] | Missing filter: pet-friendly requirement ignored in SQL | Fail |
53| 002 | [full trace] | Proposed unavailable times despite calendar conflicts | Fail |
54| 003 | [full trace] | Used casual tone for luxury client; wrong property type | Fail |
55| 004 | [full trace] | - | Pass |
56```
57
58**Heuristics:**
59- Do NOT start with a pre-defined failure list. Let categories emerge from what the user actually sees.
60- If the user is stuck articulating what feels wrong, prompt with common failure types: made-up facts, malformed output, ignored user requirements, wrong tone, tool misuse.
61
62### Step 3: Group Failures into Categories
63
64After reviewing 30-50 traces, start grouping similar notes into categories. Don't wait until all 100 are done — grouping early helps sharpen what to look for in the remaining traces. The categories will evolve. The goal is names that are specific and actionable, not perfect.
65
661. Read through all the failure notes
672. Group similar ones together
683. Split notes that look alike but have different root causes
694. Give each category a clear name and one-sentence definition
70
71**When to split vs. group:**
72
73Split these (different root causes):
74- "Made up property features (solar panels)" vs. "Made up client activity (scheduled a tour never requested)" — one fabricates external facts, the other fabricates user intent.
75
76Group these (same root cause):
77- "Missing bedroom count filter" + "Missing pet-friendly filter" + "Missing price range filter" → **Missing Query Constraints**
78
79**LLM-assisted clustering** (use only after the user has reviewed 30-50 traces):
80
81```
82Here are failure annotations from reviewing LLM pipeline traces.
83Group similar failures into 5-10 distinct categories.
84For each category, provide:
85- A clear name
86- A one-sentence definition
87- Which annotations belong to it
88
89Annotations:
90[paste annotations]
91```
92
93Always review LLM-suggested groupings with the user. LLMs cluster by surface similarity (e.g., grouping "app crashes" and "login is slow" because both mention login).
94
95**Aim for 5-10 categories** that are:
96- Distinct (each failure belongs to one category)
97- Clear enough that someone else could apply them consistently
98- Actionable (each points toward a specific fix)
99
100### Step 4: Label Every Trace
101
102Go back through all traces and apply binary labels (pass/fail) for each failure category. Each trace gets a column per category. Use whatever tool the user prefers — spreadsheet, annotation app (see build-review-interface), or a simple script.
103
104### Step 5: Compute Failure Rates
105
106```python
107failure_rates = labeled_df[failure_columns].sum() / len(labeled_df)
108failure_rates.sort_values(ascending=False)
109```
110
111The most frequent failure category is where to focus first.
112
113### Step 6: Decide What to Do About Each Failure
114
115Work through each category with the user in this order:
116
117**Can we just fix it?** Many failures have obvious fixes that don't need an evaluator at all:
118- The prompt never mentioned the requirement. Example: the LLM never includes photo links in emails because the prompt never asked for them. Add the instruction.
119- A tool is missing or misconfigured. Example: the user wants to reschedule but there's no rescheduling tool exposed to the LLM. Add the tool.
120- An engineering bug in retrieval, parsing, or integration. Fix the code.
121
122If a clear fix resolves the failure, do that first. Only consider an evaluator for failures that persist after fixing.
123
124**Is an evaluator worth the effort?** Not every remaining failure needs one. Building and maintaining evaluators has real cost. Ask the user:
125- Does this failure happen frequently enough to matter?
126- What's the business impact when it does happen? A rare failure that causes revenue loss may outrank a frequent failure that's merely annoying.
127- Will this evaluator actually get used to iterate on the system, or is it checkbox work?
128
129Reserve evaluators for failures the user will iterate on repeatedly. Start with the highest-frequency, highest-impact category.
130
131**For failures that warrant an evaluator:** prefer code-based checks (regex, parsing, schema validation) for anything objective. Use write-judge-prompt only for failures that require judgment. Critical requirements (safety, compliance) may warrant an evaluator even after fixing the prompt, as a guardrail.
132
133### Step 7: Iterate
134
135Expect 2-3 rounds of reviewing and refining categories. After each round:
136- Merge categories that overlap
137- Split categories that are too broad
138- Clarify definitions where the user would hesitate
139- Re-label traces with the refined categories
140
141## Stopping Criteria
142
143Stop reviewing when new traces aren't revealing new kinds of failures. Roughly: ~100 traces reviewed with no new failure types appearing in the last 20. The exact number depends on system complexity.
144
145## Trace Sampling Strategies
146
147When production volume is high, use a mix:
148
149| Strategy | When to Use | Method |
150|----------|------------|--------|
151| **Random** | Default starting point | Sample uniformly from recent traces |
152| **Outlier** | Surface unusual behavior | Sort by response length, latency, tool call count; review extremes |
153| **Failure-driven** | After guardrail violations or user complaints | Prioritize flagged traces |
154| **Uncertainty** | When automated judges exist | Focus on traces where judges disagree or have low confidence |
155| **Stratified** | Ensure coverage across user segments | Sample within each dimension |
156
157## Anti-Patterns
158
159- **Brainstorming failure categories before reading traces.** Read first, categorize what you find.
160- **Starting with pre-defined categories.** A fixed list causes confirmation bias. Let categories emerge.
161- **Skipping the user for initial review.** The user must review the first 30-50 traces to ground categories in domain knowledge.
162- **Using generic scores as categories.** "Hallucination score," "helpfulness score," "coherence score" are not grounded in the application's actual failure modes.
163- **Building evaluators before fixing obvious problems.** Fix prompt gaps, missing tools, and engineering bugs first.
164- **Treating this as a one-time activity.** Re-run after every significant change: new features, prompt rewrites, model switches, production incidents.
165
In the file
SKILL.md1,262 words
Files1
LicenceMIT
Why you can read it

Nothing in a skill executes. The client loads the text and the model follows it, so a skill can be audited the way a runbook is — by reading it.

What it costs in context

Skills are not billed by the call. They are paid for in context: every token the instructions occupy is a token your code, your diff and your conversation cannot use. Here is what this one takes and when it takes it.

≈80
always loaded
The name and description, so the model knows the skill exists and when to reach for it.
1,995
on trigger
The instruction body, read only when the skill fires.
1.0%
of a 200k window
Ten skills this size would take about 10% of the window before you open a file.
050k100k150k200k context window

2.1k tokens, estimated from the bundle at four bytes to the token, held for the rest of the session once it triggers. Middling. Fine to keep on in a project where you use it weekly, worth unloading in one where you never do.

Servers bill, skills cost

A server charges by the month. A skill charges once per session, in context, and then keeps charging it for as long as the session lives.

Before and after

The same question, put to the same model twice: once as it comes, and once with these instructions loaded.

No worked example has been published for this skill yet.

Adoption
Installsnone yet
Ratingno reviews yet

The procedure it runs

The procedure has not been published here. It is in the skill’s own SKILL.md, which its author has not sent to the marketplace yet.

Prose, not code

These steps are written for a model to follow, not executed by a runtime. It can still be told to skip one, and it will say so when it does.

Servers it uses

None. This skill calls no MCP servers at all.

Everything it needs is in the instructions, so it works in a project with nothing connected — the model reads the file and changes how it works with what it can already reach.

It writes no files and reaches no network. All it changes is how the model reasons and writes.

What it asks for
Writes filesno
Network accessno

Read from the allowed-tools line of this skill’s own SKILL.md. A skill grants no permissions of its own — it can only ask for tools your client already has.

What it will not do

Every skill is narrow, and the useful ones say where they stop. These are the jobs this one is the wrong tool for.

What this skill is not for has not been published here. Nothing is implied by that: it is a section the author has not filled in.

What is in the bundle

1 file, 8.3 kB on disk. A bundle is text throughout: the instructions the model reads, plus the templates it fills in.

  • SKILL.md8.3 kB
What is not in it

No dependencies and nothing executable: a skill is text the agent reads, so the bundle is 1 file you can review in full before installing. The MIT licence covers the templates and examples as well as the instructions.

Install

Installing copies the bundle into your project. Nothing runs at install time — the files sit on disk until the model reads them.

$59 once
Error Analysis · MIT · hamelsmu
one-time
Price$59 once
LicenceMIT — the author’s, unchanged by this purchase
Paid throughStripe, once, on the card you add at the checkout
Keeps workingfor good — the files are yours once they are on disk
Updatesevery update its author ships, delivered through this account

You can read the whole bundle before paying — the SKILL.md above is the product, not a preview of it. What the money buys is the delivery: the folder packaged and handed to your machine by key, every update its author ships, and our support if it does not do what this listing says. The terms of use are MIT, set by the author and unchanged by buying it here.

Sign in to buyCreate an account

Payment runs through Stripe, on a page like this one rather than a redirect. Once there is an account it joins the same mcprush invoice as everything else you run, so there is never a second card to enter.

Which clients pick it up on their own

A skill is a folder of text. A client with a skills folder reads it without being told; everywhere else the same text works, it is just handed to the model rather than found.

Claude Code.claude/skills/
Claude Desktop
ChatGPT
Cursor.cursor/skills/
VS Code.github/skills/
Codex CLI.agents/skills/
Gemini CLI.gemini/skills/
Grok.grok/skills/
Zed.agents/skills/
Windsurf.windsurf/skills/
Agent SDK.claude/skills/
HTTP / API
This release
Versionnot versioned
Publishedno release date on file
Price$59
Referencehamelsmu/error-analysis

Versions

Its author publishes no version number, so there is nothing here to pin to: what you install is the folder as it stands today. Instructions change more often than APIs do — a skill can be rewritten entirely without anything it depends on moving.

v
  • No earlier releases have been published to the marketplace.
Pinning

Nothing to pin to: this skill carries no version number of its own. What you install is what the folder holds on the day you install it.

Reviews

—no reviews yet · no installs yet

Nobody has reviewed this skill yet. The rating is the mean of the reviews written here, so there is none until somebody writes the first.

Who can post

Only accounts that have had the skill installed for fourteen days, so a review is written after living with it rather than after reading it. Publishers may reply once.

Who wrote it

HA
hamelsmu

Publishes on mcprush.

0 servers listed4 skills listednot claimed
Profile
Publisher
Servers0