Workflow·AI & Agents

Evaluate RAG

Name: Evaluate RAG
Author: hamelsmu

Guides evaluation of RAG pipeline retrieval and generation quality.

You say

Buy it · $29 Read it before you buy $29 Written by hamelsmu · unverified publisher

Context cost: 1.9k tokensestimated from the bundle, loaded when it triggers
Bundle: 1 file · 7.6 kBtext throughout, nothing executable
Licence: MITpaid listing
Last change: —no release on file
Servers it uses: Noneruns standalone

What it does

Guides evaluation of RAG pipeline retrieval and generation quality. Use when evaluating a retrieval-augmented generation system, measuring retrieval quality, assessing generation faithfulness or relevance, generating synthetic QA pairs for retrieval testing, or optimizing chunking strategies.

Installed, it changes the agent in these ways.

What this skill changes about the agent is not written down here yet. The listing was collected from its source, and the description is in its own SKILL.md.

Workflow

Runs a procedure end to end.

ragevalsretrieval

Filed under

AI & Agents

The skill itself

This is the whole product. A skill is instructions the model reads, so there is nothing behind the listing you cannot see first — the front matter loads with every session, and the body below it loads when the skill triggers.

SKILL.md7.6 kB · 178 lines

--- name: evaluate-rag description: > Guides evaluation of RAG pipeline retrieval and generation quality. Use when evaluating a retrieval-augmented generation system, measuring retrieval quality, assessing generation faithfulness or relevance, generating synthetic QA pairs for retrieval testing, or optimizing chunking strategies. ---

10# Evaluate RAG

12## Overview

141. Do error analysis on end-to-end traces first. Determine whether failures come from retrieval, generation, or both.

152. Build a retrieval evaluation dataset: queries paired with relevant document chunks.

163. Measure retrieval quality with Recall@k (most important for first-pass retrieval).

174. Evaluate generation separately: faithfulness (grounded in context?) and relevance (answers the query?).

185. If retrieval is the bottleneck, optimize chunking via grid search before tuning generation.

20## Prerequisites

22Complete error analysis on RAG pipeline traces before selecting metrics. Inspect what was retrieved vs. what the model needed. Determine whether the problem is retrieval, generation, or both. Fix retrieval first.

24## Core Instructions

26### Evaluate Retrieval and Generation Separately

28Measure each component independently. Use the appropriate metric for each retrieval stage:

30- **First-pass retrieval:** Optimize for Recall@k. Include all relevant documents, even at the cost of noise.

31- **Reranking:** Optimize for Precision@k, MRR, or NDCG@k. Rank the most relevant documents first.

33### Building a Retrieval Evaluation Dataset

35You need queries paired with ground-truth relevant document chunks.

37**Manual curation (highest quality):** Write realistic questions and map each to the exact chunk(s) containing the answer.

39**Synthetic QA generation (scalable):** For each document chunk, prompt an LLM to extract a fact and generate a question answerable only from that fact.

41Synthetic QA prompt template:

43```

44Given a chunk of text, extract a specific, self-contained fact from it.

45Then write a question that is directly and unambiguously answered

46by that fact alone.

48Return output in JSON format:

49{ "fact": "...", "question": "..." }

51Chunk: "{text_chunk}"

52```

54**Adversarial question generation:** Create harder queries that resemble content in multiple chunks but are only answered by one.

56Process:

571. Select target chunk A containing a clear fact.

582. Find similar chunks B, C using embedding search (chunks that share terminology but lack the answer).

593. Prompt the LLM to write a question using terminology from B and C that only chunk A answers.

61Example:

62- Chunk A: "In April 2020, the company reported a 17% drop in quarterly revenue, its largest decline since 2008."

63- Chunk B: "The company experienced significant losses in 2008 during the financial crisis."

64- Generated question: "When did the company experience its largest revenue decline since the 2008 financial crisis?"

66Only chunk A contains the answer. Chunk B is a plausible distractor.

68**Filtering synthetic questions:** Rate synthetic queries for realism using few-shot LLM scoring. Keep only those rated realistic (4-5 on a 1-5 scale). Likert scoring is appropriate here, since the goal is fuzzy ranking for dataset curation, not measuring failure rates.

70### Retrieval Metrics

72**Recall@k:** Fraction of relevant documents found in the top k results.

74```

75Recall@k = (relevant docs in top k) / (total relevant docs for query)

76```

78Prioritize recall for first-pass retrieval. LLMs can ignore irrelevant content but cannot generate from missing content.

80**Precision@k:** Fraction of top k results that are relevant.

82```

83Precision@k = (relevant docs in top k) / k

84```

86Use for reranking evaluation.

88**Mean Reciprocal Rank (MRR):** How early the first relevant document appears.

90```

91MRR = (1/N) * sum(1/rank_of_first_relevant_doc)

92```

94Best for single-fact lookups where only one key chunk is needed.

96**NDCG@k (Normalized Discounted Cumulative Gain):** For graded relevance where documents have varying utility. Rewards placing more relevant items higher.

98```

99DCG@k = sum over i=1..k of: rel_i / log2(i+1)

100IDCG@k = DCG@k with documents sorted by decreasing relevance

101NDCG@k = DCG@k / IDCG@k

102```

103

104Caveat: Optimal ranking of weakly relevant documents can outscore a highly relevant document ranked lower. Supplement with Recall@k.

105

106**Choosing k:** k varies by query type. A factual lookup uses k=1-2. A synthesis query ("summarize market trends") uses k=5-10.

107

108#### Metric Selection

109

110| Query Type | Primary Metric |

111|---|---|

112| Single-fact lookups | MRR |

113| Broad coverage needed | Recall@k |

114| Ranked quality matters | NDCG@k or Precision@k |

115| Multi-hop reasoning | Two-hop Recall@k |

116

117### Evaluating and Optimizing Chunking

118

119Treat chunking as a tunable hyperparameter. Even with the same retriever, metrics vary based on chunking alone.

120

121**Grid search for fixed-size chunking:** Test combinations of chunk size and overlap. Re-index the corpus for each configuration. Measure retrieval metrics on your evaluation dataset.

122

123Example search grid:

124

126|-----------|---------|----------|--------|

127| 128 tokens | 0 | 0.82 | 0.69 |

128| 128 tokens | 64 | 0.88 | 0.75 |

129| 256 tokens | 0 | 0.86 | 0.74 |

130| 256 tokens | 128 | 0.89 | 0.77 |

131| 512 tokens | 0 | 0.80 | 0.72 |

132| 512 tokens | 256 | 0.83 | 0.74 |

133

134**Content-aware chunking:** When fixed-size chunks split related information:

135- Use natural document boundaries (sections, paragraphs, steps).

136- Augment chunks with context: prepend document title and section headings to each chunk before embedding.

137

138### Evaluating Generation Quality

139

140After confirming retrieval works, evaluate what the LLM does with the retrieved context along two dimensions:

141

142**Answer faithfulness:** Does the output accurately reflect the retrieved context? Check for:

143- **Hallucinations:** Information absent from source documents. In RAG, even correct facts from the LLM's own knowledge count as hallucinations.

144- **Omissions:** Relevant information from the context ignored in the output.

145- **Misinterpretations:** Context information represented inaccurately.

146

147**Answer relevance:** Does the output address the original query? An answer can be faithful to the context but fail to answer what the user asked.

148

149Use error analysis to discover specific manifestations in your pipeline. Identify what kind of information gets hallucinated and which constraints get omitted.

150

151#### Diagnosing Failures by Metric Pattern

152

154|---|---|---|---|

156| High | Low | -- | Hallucination or misinterpretation of retrieved content |

157| Low | -- | -- | Retrieval problem. Fix chunking, embeddings, or query preprocessing |

158

159### Multi-Hop Retrieval Evaluation

160

161For queries requiring information from multiple chunks:

162

163**Two-hop Recall@k:** Fraction of 2-hop queries where both ground-truth chunks appear in the top k results.

164

165```

166TwoHopRecall@k = (1/N) * sum(1 if {Chunk1, Chunk2} ⊆ top_k_results)

167```

168

169Diagnose failures by classifying: hop 1 miss, hop 2 miss, or rank-out-of-top-k.

170

171## Anti-Patterns

172

173- Using a single end-to-end correctness metric without separating retrieval and generation measurement.

174- Jumping directly to metrics without reading traces first.

175- Overfitting to synthetic evaluation data. Validate against real user queries regularly.

176- Using similarity metrics (ROUGE, BERTScore, cosine similarity) as primary generation evaluation. Use binary evaluators driven by error analysis.

177- Evaluating generation without checking context grounding.

178

In the file

SKILL.md1,133 words

Files1

LicenceMIT

Why you can read it

Nothing in a skill executes. The client loads the text and the model follows it, so a skill can be audited the way a runbook is — by reading it.

What it costs in context

Skills are not billed by the call. They are paid for in context: every token the instructions occupy is a token your code, your diff and your conversation cannot use. Here is what this one takes and when it takes it.

≈90

always loaded

The name and description, so the model knows the skill exists and when to reach for it.

1,810

on trigger

The instruction body, read only when the skill fires.

0.95%

of a 200k window

Ten skills this size would take about 10% of the window before you open a file.

050k100k150k200k context window

1.9k tokens, estimated from the bundle at four bytes to the token, held for the rest of the session once it triggers. Middling. Fine to keep on in a project where you use it weekly, worth unloading in one where you never do.

Servers bill, skills cost

A server charges by the month. A skill charges once per session, in context, and then keeps charging it for as long as the session lives.

Before and after

The same question, put to the same model twice: once as it comes, and once with these instructions loaded.

No worked example has been published for this skill yet.

Adoption

Installsnone yet

Ratingno reviews yet

The procedure it runs

The procedure has not been published here. It is in the skill’s own SKILL.md, which its author has not sent to the marketplace yet.

Prose, not code

These steps are written for a model to follow, not executed by a runtime. It can still be told to skip one, and it will say so when it does.

Servers it uses

None. This skill calls no MCP servers at all.

Everything it needs is in the instructions, so it works in a project with nothing connected — the model reads the file and changes how it works with what it can already reach.

It writes no files and reaches no network. All it changes is how the model reasons and writes.

What it asks for

Writes filesno

Network accessno

Read from the allowed-tools line of this skill’s own SKILL.md. A skill grants no permissions of its own — it can only ask for tools your client already has.

What it will not do

Every skill is narrow, and the useful ones say where they stop. These are the jobs this one is the wrong tool for.

What this skill is not for has not been published here. Nothing is implied by that: it is a section the author has not filled in.

If this is not it

14 other skills in AI & Agents

What is in the bundle

1 file, 7.6 kB on disk. A bundle is text throughout: the instructions the model reads, plus the templates it fills in.

SKILL.md7.6 kB

What is not in it

No dependencies and nothing executable: a skill is text the agent reads, so the bundle is 1 file you can review in full before installing. The MIT licence covers the templates and examples as well as the instructions.

Install

Installing copies the bundle into your project. Nothing runs at install time — the files sit on disk until the model reads them.

$29 once

Evaluate RAG · MIT · hamelsmu

one-time

Price$29 once

LicenceMIT — the author’s, unchanged by this purchase

Paid throughStripe, once, on the card you add at the checkout

Keeps workingfor good — the files are yours once they are on disk

Updatesevery update its author ships, delivered through this account

You can read the whole bundle before paying — the SKILL.md above is the product, not a preview of it. What the money buys is the delivery: the folder packaged and handed to your machine by key, every update its author ships, and our support if it does not do what this listing says. The terms of use are MIT, set by the author and unchanged by buying it here.

Payment runs through Stripe, on a page like this one rather than a redirect. Once there is an account it joins the same mcprush invoice as everything else you run, so there is never a second card to enter.

Which clients pick it up on their own

A skill is a folder of text. A client with a skills folder reads it without being told; everywhere else the same text works, it is just handed to the model rather than found.

Claude Code.claude/skills/

Claude Desktop

ChatGPT

Cursor.cursor/skills/

VS Code.github/skills/

Codex CLI.agents/skills/

Gemini CLI.gemini/skills/

Grok.grok/skills/

Zed.agents/skills/

Windsurf.windsurf/skills/

Agent SDK.claude/skills/

HTTP / API

This release

Versionnot versioned

Publishedno release date on file

Price$29

Referencehamelsmu/evaluate-rag

Versions

Its author publishes no version number, so there is nothing here to pin to: what you install is the folder as it stands today. Instructions change more often than APIs do — a skill can be rewritten entirely without anything it depends on moving.

No earlier releases have been published to the marketplace.

Pinning

Nothing to pin to: this skill carries no version number of its own. What you install is what the folder holds on the day you install it.

Reviews

—no reviews yet · no installs yet

Nobody has reviewed this skill yet. The rating is the mean of the reviews written here, so there is none until somebody writes the first.

Who can post

Only accounts that have had the skill installed for fourteen days, so a review is written after living with it rather than after reading it. Publishers may reply once.

Who wrote it

hamelsmu

Publishes on mcprush.

0 servers listed4 skills listednot claimed

Profile

Publisher

Servers0