MCP Server for Z.AI - A Model Context Protocol server that provides AI capabilities
Nothing to run from here
Evalsmith is installed from its publisher's own source and answers where it runs, so this marketplace is not in the path of a single call. There is no address here to send one to, and a panel that pretended otherwise would be showing you an answer we made up. Install it and call it from your own client — the Installation tab has the entry for each one.
99/100
trust score
10
downloads a day
local
runs on your machine
468
downloads in 30d
What it does
Evaluate RAG retrieval, agent tool calls and prompt regressions with deterministic metrics.
Quickstart
# 1 — add it to Claude Code
claude mcp add --scope user evalsmith-mcp '--' npx -y evalsmith-mcp
# 2 — ask your agent for something these tools do
# check_tool_calls, compare_configs, detect_prompt_regression and 2 more
Evalsmith costs nothing on mcprush: there is no plan to choose here, no cap to set and nothing mcprush can bill you for.
Publisher
Installation
Where are you running it?
Every route below installs the same thing and ends at the same approval screen. This one runs on your machine: your client starts Evalsmith as a process under your own user, with your files and your network, so the tool surface below is what it can do to your computer rather than to a server somewhere else. Its scan result is on the Security tab — read the surface before you approve it.
Claude Codedeclared by this listing
This is a public server: you run it yourself and this marketplace is not in the path. You need Node.js — the current LTS (24) is the safe choice; npx comes with it. Claude Code registers it in one command; --scope user makes it available in every project. The '--' is in quotes so that PowerShell passes it on.
claude mcp add --scope user evalsmith-mcp '--' npx -y evalsmith-mcpReconnect, or start a new session, and the tools appear in the model’s tool list.
Free on mcprush, and it runs from its publisher’s own package: this marketplace is not in the path, so there is no key of ours to hold and no card to add.
What it asks of you
Nothing is required, but it can take EVALSMITH_JUDGE_BASE_URL (OpenAI-compatible endpoint for the optional LLM judge (default: Ollama on localhost)), EVALSMITH_JUDGE_MODEL (Model used by the optional LLM judge), EVALSMITH_JUDGE_API_KEY (API key for the judge endpoint, if it needs one), LOG_LEVEL (pino log level; logs go to stderr) if you want to set them.
What it writes
One config entry your client uses to start the process locally. A local server runs with your file system and your network, which is why it is priced without metering.
What you approve
5 tools, with what each one reads, writes and reaches shown before you agree — the same list on every route above. Read the tool surface.
Tool surface
What the model actually sees. Descriptions are diffed on every release — see version history.
Read from its source code. A server that builds its tool list when it starts can list more tools than this.
Compare an agent's tool-call trace (evalsmith, Anthropic Messages or OpenAI format, as logged) with the expected calls: right tool, right order (strict or lax) and arguments that satisfy exact / regex / subset / any matchers. Use it to test an agent change against a known-good trace. Returns a…
| Parameter | Type | Required | Notes |
|---|---|---|---|
| trace | yes | — | |
| expected | array | yes | — |
| order | no | — |
Compare 2-3 retrieval configurations on the same qrels. Ranks them on a primary metric, attaches seeded paired-bootstrap confidence intervals, and says whether the winner's lead over the runner-up is significant (interval of the gap excludes zero). Use it before switching retriever, chunking or…
| Parameter | Type | Required | Notes |
|---|---|---|---|
| qrels | yes | — | |
| configs | array | yes | Two or three runs over the same queries. |
| metric | string | no | Primary metric that decides the winner, e.g. ndcg@10, recall@5, mrr. |
| k | no | — | |
| bootstrap | no | — |
Run a golden set's deterministic checks (exact, regex, contains, JSON Schema, length) against baseline and candidate outputs. A regression is a case that passes on baseline and fails on candidate. An LLM judge can be enabled per call for a second opinion; the verdict only depends on the checks.…
| Parameter | Type | Required | Notes |
|---|---|---|---|
| golden | array | yes | Golden cases with deterministic checks. |
| baseline | yes | Outputs of the current prompt/model, keyed by case id. | |
| candidate | yes | Outputs of the proposed prompt/model, keyed by case id. | |
| judge | object | no | — |
Score a ranked retrieval run against relevance judgements (qrels). Returns recall@k, precision@k, nDCG@k (graded, linear gain), hit rate@k and MRR, averaged over judged queries, plus per-query values. Use it to check whether one retriever is good enough; to choose between several, use…
| Parameter | Type | Required | Notes |
|---|---|---|---|
| qrels | yes | — | |
| run | yes | — | |
| k | no | — |
Turn the results of the other evalsmith tools into a Markdown summary, a JSON document or JUnit XML for CI dashboards. Use it last, passing the structuredContent of each tool you ran. Optional gates (metric -> minimum) decide the overall pass/fail; returns the rendered report and the pass flag.
| Parameter | Type | Required | Notes |
|---|---|---|---|
| format | string | no | — |
| name | string | no | Suite name shown in the report title. |
| retrieval | no | structuredContent of evaluate_retrieval. | |
| comparison | no | structuredContent of compare_configs. | |
| toolCalls | no | structuredContent of check_tool_calls. | |
| regression | no | — | |
| gates | object | no | Minimum values, e.g. { "recall@5": 0.8, "tool_calls": 1 }. |
Free
$0 forever
- Every tool, no call limit
- No card, and mcprush charges nothing for it
- Source published under MIT
- Runs on your machine — nothing of it reaches our gateway
- Nothing to cap, because mcprush bills nothing
What counts against your monthly calls
| Tool | Unit | Calls used | Out of the allowance |
|---|
Nothing here is billable: mcprush charges nothing to install Evalsmith or to call it, at any volume.
Call allowance · this installenforced at the gateway
Monthly allowance, in calls
—
—
Alert at 80% · then the tool returns a normal MCP error
Used this month: —
ATrust score 99 / 100
Release 0.1.0 · scanned 30 Oct 2026
Two independent axes, because powerful and malicious are different questions. The grade is threat only. The capability level is blast radius, and it is never a penalty on the grade — it is priced as one subtract-only term in the score, where you can see it.
Threat score100
Client score99
Findings0 threat · 0 capability
Capabilityminimal
Coveragesource
Verificationrepo
Surface digest80fabf2008295ac32662b1f74fcb1b43840764f079f266e838974ecfc57bbfc0
Score vectorthreat 100 → client 99
| Term | Level | What it prices | Points |
|---|---|---|---|
| capability-exposure | minimal | capability blast radius (minimal) — client exposure if the model is manipulated | −0 |
| verification-discount | repo | publisher verification (public source) — no provenance, but the source is public and inspectable | −1 |
| coverage-honesty | source | inspection depth (source) — how much of the target the scan could see | −0 |
score = clamp(round(threatScore − E_cap − E_ver − E_cov), 0, 100). No model, no opinion, no network call at scoring time: the same methodology version against the same bytes always produces this number, and every line above is a field of the report rather than a summary of it.
What the scan could actually read
A grade is only as meaningful as its coverage, so the scanner publishes its own depth before it publishes its result.
Tool surfaceread
Implementation sourceread
Package metadataread
Live transportnot available
Tools were statically extracted from the published source (5 recovered), not enumerated from a running server. Tool-poisoning, Unicode-smuggling, capability and toxic-flow analysis ran on this inferred surface, but a mis-parsed registration could be missed or mis-attributed, so tool-derived findings are capped below “confirmed”. To grade the real runtime surface, scan the running server: --command "npx -y <package>".
Capability — what it could do if the model were manipulated
Tags derived from each tool’s schema and the implementation, not from what the tool calls itself. minimal is the level these add up to.
| Tool | Capability tags | Why the tag was assigned |
|---|---|---|
| check_tool_calls | no tags | |
| compare_configs | no tags | |
| detect_prompt_regression | no tags | |
| evaluate_retrieval | no tags | |
| generate_report | no tags |
Toxic-flow graph
The lethal trifecta, checked as a graph rather than as a checklist: untrusted input, a sensitive source and an external sink have to meet before there is a path worth worrying about.
The public result for this release does not print the flow graph, so there is nothing to show here. That is not the same as "no paths were found": what the scan did read is above, under coverage.
Supply chain and provenance
Integrityfirst-seen
Surface digest80fabf2008295ac3
Scanned2026-10-30
This is the first scan of this surface here, so there is nothing yet to compare it against.
Every result on this tab comes from one deterministic pass over the published package — offline, rule by rule, and auditable line by line above. Same methodology version, same bytes, same score.
Release history
Always latest — the install command names no version, so it installs the newest release when it runs; the current release on file is 0.1.0. Pin a release below and the command asks for exactly that one.
0.1.0
undated
latestNo release note was published with this version.
—
no reviews yet
Who can review
One review per account, from a signed-in account that does not publish this listing. Nothing else is asked — you can say what you think before you install it. Publishers can reply once per review, and a review can be edited or deleted by whoever wrote it.
Sign in to write one
Any signed-in account that does not publish Evalsmith can review it, once — before installing it or after. A review from an account that has installed it is marked as one.
No reviews yet
Nobody has reviewed this listing. The rating on the card is the mean of the reviews written here and nothing else, so there is no rating until somebody writes the first — which takes a signed-in account that does not publish it, and nothing else.