Name: IA-QA Eval & RAG — 38 tools
Author: JcJamet

IA-QA Eval & RAG — 38 tools MCP

LLM & RAG evals: hallucination, eval contracts, semantic tests, tokens & model info. Free, no key.

Tool-list grade B Remote · Streamable HTTP No auth needed 38 tools v1.0.1 MIT

No reviews yet by JcJametGitHub The publisher’s own endpoint — this marketplace is not in the path

84/100

tool-list score

tools declared

remote

at its publisher

—

calls last 7d

What it does

LLM & RAG evals: hallucination, eval contracts, semantic tests, tokens & model info. Free, no key.

Quickstart

# 1 — add it to Claude Code
claude mcp add --scope user --transport http ia-qa-eval-rag-38-tools-mcp https://www.ia-qa.com/mcp/eval

# 2 — ask your agent for something these tools do
#   get_testing_guidelines, find_tool, hallucination_check and 35 more

IA-QA Eval & RAG — 38 tools costs nothing on mcprush: there is no plan to choose here, no cap to set and nothing mcprush can bill you for.

Publishernot claimed

JcJamet4 servers

3 categories

All 4 servers

Collected from a public index. Nobody has claimed this account, so nothing here was written by its author — claim it if it is yours.

Tool surface

What the model actually sees. Descriptions are diffed on every release — see version history.

Only tools that write or reach the network

Query the IA-QA methodology knowledge base. Returns structured testing guidelines, assertion strategies, thresholds, best practices, and relevant MCP tools for a given topic. Call without a topic to list all available topics. Topics: llm-unit-testing, rag-pipeline, prompt-stability, prompt-ab-testing, embedding-quality, eval-framework, semantic-testing, auto-testing, security, api-testing, ci-cd,

Parameter	Type	Required	Notes
topic	string	no	The testing topic to retrieve guidelines for. Omit to get the start-here map and the full list of available topics.

Search available MCP tools by keyword or category before calling them. Returns matching tool names, descriptions, and optionally their inputSchemas. Call this when you are unsure which tool to use or want to explore the catalogue. Categories: data, encoding, text, llm, qa, rag, dev, security, web.

Parameter	Type	Required	Notes
query	string	yes	Keyword(s) to search in tool name and description (e.g. "cors", "token", "vector", "json")
category	string	no	Optional: filter by category — data \| encoding \| text \| llm \| qa \| rag \| dev \| security \| web
with_schema	boolean	no	Set true to include inputSchema in results (default: false)
max_results	number	no	Maximum tools to return (default 10, max 50). Results are ranked by IDF-weighted relevance, so common words like "test" do not inflate the list.

Lexical hallucination check: verifies an LLM answer's words, numbers and polarity against the provided source/context. Fast, deterministic, no API key needed. Each answer sentence is aligned to its best-matching source sentence, so a number only counts as support when it sits on the SAME statement ("founded in 1998" is not grounded by "sold 1998 units"), and a negation or antonym flip against that

Parameter	Type	Required	Notes
answer	string	yes	The LLM-generated answer to verify
context	string	yes	The source/reference text that should ground the answer
strict	boolean	no	If true, every sentence in the answer must be supported (default: false)

Parse a .ia-eval.yaml LLM test suite, call the specified LLM model for each scenario, run all configured scorers, and return a structured JSON report with per-scenario Pass/Fail verdicts and a Markdown summary. Use list_local_tests to discover available test files.

Parameter	Type	Required	Notes
contract_path	string	no	Absolute or relative path to a .ia-eval.yaml file (required unless inline_contract is provided)
inline_contract	object	no	Raw contract object (alternative to contract_path). Must contain top-level "metadata" ({name, version, model?, provider?}), "expectations" ({min_score?}), and "scenarios" ([{id, input, ground_truth?}]
api_keys	object	no	API keys to use for LLM generation (all optional — falls back to server env vars)
overrides	object	no	Override contract defaults
validate_only	boolean	no	Check the contract against the IA-EVAL schema and stop: no model call, no key, deterministic. Returns valid + issues instead of throwing. Gate a PR that edits a contract on valid.

Semantic assertion primitive: compare actual vs expected text pairs using cosine similarity + ROUGE-L. Two modes: tfidf (default, free, no API key) or embeddings (OpenAI text-embedding-3-small, BYOK, true semantic similarity). Returns per-case PASS/FAIL verdicts and an overall verdict. CI-ready: pipe the JSON verdict field to gate a build.

Parameter	Type	Required	Notes
cases	array	yes	Array of (actual, expected) pairs to evaluate.
thresholds	object	no	Pass/fail thresholds (defaults: cosine 0.75, rouge_l 0.5).
require_all	boolean	no	If true (default), all cases must pass for overall PASS. If false, at least one case passing returns PASS.
mode	string	no	tfidf (default): fast, free, lexical. embeddings: OpenAI text-embedding-3-small, true semantic similarity, requires api_key.
api_key	string	no	OpenAI API key — required only when mode is embeddings.

Generate a ready-to-commit GitHub Actions workflow that gates a build on IA-QA. Two gate types, combinable: "eval_contract" runs a .ia-eval.yaml through ia-qa-com/eval-action@v1 (LLM quality gate, needs a provider API key as a repo secret), and "cli_checks" runs deterministic primitives via npx @ia-qa/cli (secret scan, prompt-injection scan, security headers…) whose exit code fails the build. Dete

Parameter	Type	Required	Notes
gate	string	no	Which gate to emit. eval_contract = LLM eval via the action (default). cli_checks = deterministic CLI assertions. selector_drift = an E2E selector-drift gate via @ia-qa/self-healing (boots the app, ca
contract_path	string	no	Path to the .ia-eval.yaml contract, relative to the repo root (default: evals/smoke.ia-eval.yaml). Only used when the gate includes eval_contract.
provider	string	no	LLM provider the contract runs against — decides which repository secret the workflow wires (default: groq).
cli_tools	array	no	IA-QA tool names to run as deterministic gates, e.g. ["secret_scan","prompt_injection_scan"]. Tools with no known CI recipe get a --stdin step flagged in notes.
triggers	array	no	Workflow triggers (default: push + pull_request).
cron	string	no	Cron expression when triggers include 'schedule' (default: '0 6 * * 1' — Mondays 06:00 UTC).
min_score	number	no	Override the contract min_score (0-100). Omit to use the value in the contract.
fail_on_fail	boolean	no	Fail the build on a FAIL/PARTIAL verdict (default: true). Set false to report without gating.
workflow_name	string	no	Workflow display name (default: "IA-QA Quality Gate").
node_version	string	no	Node version for the CLI steps (default: "20").
app_start_command	string	no	Command that boots the app for the selector_drift gate, e.g. "npm run start:ci". Cannot be guessed — omitted, the step carries an explicit TODO and the note says so, because a plausible default would
app_base_url	string	no	URL the drift gate waits for before capturing (default: http://127.0.0.1:3000). Must match config.baseUrl in .ia-qa/config.json.

Generate a "needle in a haystack" test: embeds a target fact into a large block of filler text at a specified position. Use this to test LLM context window retrieval accuracy. Returns the full haystack, the question to ask, and metadata. No API key needed.

Parameter	Type	Required	Notes
needle	string	yes	The fact to hide (e.g. "The secret code is ALPHA-42")
question	string	yes	The question to ask the LLM (e.g. "What is the secret code?")
tokens	integer	no	Target haystack size in tokens (default: 5000, max: 100000)
position	string	no	Where to insert the needle: "start", "middle", "end", "random" (default: "middle")

Generate a complete .ia-eval.yaml evaluation contract from a plain-language description of what your LLM should do. Uses Groq openai/gpt-oss-20b (server-side, no API key needed). Returns ready-to-run YAML for the LLM Test Runner (run_eval_contract). Picks appropriate evaluators (cosine_similarity, contains_check, hallucination_check, etc.) based on the task type.

Parameter	Type	Required	Notes
description	string	yes	Plain-language description of what the LLM under test should do. Be specific: describe inputs, expected behaviour, and constraints.
system_prompt	string	no	Optional system prompt of the LLM under test. Helps generate more accurate test cases.
scenario_count	number	no	Number of scenarios to generate (default: 5). Covers happy path + edge cases + adversarial.
task_type	string	no	Optional task type hint to guide evaluator selection.

Reference-free stability primitive: instead of comparing an answer to a ground truth, it checks that an assistant's answer stays INVARIANT when the QUESTION is transformed (typo, casing, paraphrase, reordering, translation). Catches the failure class no reference answer can expose — an assistant that handles one phrasing well and a trivial variant of it badly. You bring the outputs (no model is ca

Parameter	Type	Required	Notes
base	object	yes	The reference run: the original question and the answer your system produced for it.
variants	array	yes	Answers produced for transformed versions of the same question, each tagged with the relation that was applied.
mode	string	no	tfidf (default): free, lexical, deterministic — but a genuine paraphrase rarely reaches 0.80, so gate on case/typo and treat paraphrase as a trend. embeddings: OpenAI text-embedding-3-small, true sema
api_key	string	no	OpenAI API key — required only when mode is embeddings.
thresholds	object	no	Per-relation threshold overrides. Calibrate on your own corpus before gating — the defaults are starting points, not measurements.
baseline_guard	object	no	Correctness floor applied to the BASE answer before anything is scored. Failing it returns INVALID, not FAIL.
require_all	boolean	no	If true (default), every gated variant must pass. KEEP THE DEFAULT for any run you gate on. Setting it false is not a tolerance dial but an off switch: relations have asymmetric pass rates (a typo var

Test a SKILL.md in the open Agent Skills format (https://agentskills.io/specification — read by Claude Code, Codex, GitHub Copilot, Cursor, Gemini CLI, OpenCode and others) in two stages. (1) lint: the spec's frontmatter rules checked without a model (name 1-64 chars, lowercase/digits/single hyphens, matches dir_name when given; description 1-1024 chars; compatibility ≤500; metadata string map) —

Parameter	Type	Required	Notes
skill_md	string	yes	Full content of the SKILL.md file to test, YAML frontmatter included.
dir_name	string	no	Name of the folder holding the SKILL.md. The spec requires name to match it; checked only when given.
scenarios	array	no	Fixed user requests for a reproducible gate: [{"prompt": "...", "should_trigger": true}]. Include near misses with should_trigger false. Max 20. When omitted, scenario_count requests are generated.
other_skills	array	no	The other skills installed next to this one, [{"name", "description"}], so triggering is tested against the real neighbours. Default: six built-in generic skills.
scenario_count	number	no	Number of requests to generate when scenarios is omitted: half should-load, half near misses. Default: 6.
lint_only	boolean	no	Run the format check only: no model call, no key, deterministic. The CI gate for a SKILL.md — gate on lint.valid.
model	string	no	LLM model ID to use for both scenario generation and testing (e.g. gpt-4o-mini, claude-3-5-haiku-20241022). Defaults to openai/gpt-oss-20b (Groq, server key).
api_key	string	no	API key for the chosen model provider. Not required when using the default Groq model.

Validate an LLM response against QA criteria: format checks (JSON, code, markdown), content rules (must-include, must-not-include), length constraints, language detection, and safety patterns. Essential for QA testing LLM-powered features.

Parameter	Type	Required	Notes
output	string	yes	The LLM output text to validate
expected_format	string	no	Expected output format
must_include	string	no	Comma-separated strings that MUST appear in the output
must_not_include	string	no	Comma-separated strings that must NOT appear (e.g. "TODO, FIXME, undefined, NaN")
max_length	number	no	Maximum character length for the output
min_length	number	no	Minimum character length for the output
expected_language	string	no	Expected language of the output (en, fr, es, de…). Checks for common words.
check_safety	boolean	no	Check for PII patterns (emails, phones, SSN), profanity signals, and prompt leakage
check_json_schema	string	no	If expected_format is JSON, provide required keys as comma-separated list to validate the structure

Compare two ALREADY-PRODUCED outputs (e.g. model A vs model B on the same task) side by side. Returns deterministic metrics (token cosine, ROUGE-L, Jaccard, length/structure deltas, JSON diff) and a verdict. If a reference (ground truth) is given, scores each output against it and picks the closer one. If model + api_key are given, an LLM judge also picks a qualitative winner for the task. N

Parameter	Type	Required	Notes
response_a	string	yes	First output (e.g. model A's answer)
response_b	string	yes	Second output (e.g. model B's answer)
label_a	string	no	Label for output A (e.g. "GPT-4o", "v1.0")
label_b	string	no	Label for output B (e.g. "GPT-5-nano", "v1.1")
task	string	no	The task/prompt both outputs were answering — used by the LLM judge for context
reference	string	no	Optional ground-truth / expected answer. If set, each output is scored against it and the closer one wins (deterministic).
check_json	boolean	no	Try to parse as JSON and compare structurally (keys, types, values)
model	string	no	Optional judge model id (BYOK). When set with api_key, an LLM judge picks a qualitative winner.
api_key	string	no	Optional API key for the judge model (BYOK). Used only for the judge call; never stored.

Semantically analyze N already-produced model outputs for the SAME task (the MCP counterpart to the LLM Sandbox). Without a reference: computes consensus — pairwise cosine agreement, the most-representative output, and the outlier. With a reference (ground truth): also ranks every output by closeness (token cosine + ROUGE-L composite) and names the closest. No LLM, no key, runs locally. The refe

Parameter	Type	Required	Notes
responses	array	yes	The outputs to analyze (same task, N models/prompts/versions). Each item is a plain string or { "label": "GPT-4o", "text": "..." }. At least 2 required.
reference	string	no	Optional ground-truth answer. If set, each output is also ranked by closeness to it and the closest one is named.

Define a test suite for a prompt: provide the system prompt, user prompt, and expected output criteria. Returns a test plan with scored rubric — use this as input for manual or automated LLM evaluation.

Parameter	Type	Required	Notes
system_prompt	string	yes	The system prompt under test
user_prompt	string	yes	The user prompt to send
expected_behavior	string	no	Description of what the LLM should do (free text)
expected_format	string	no	Expected output format
must_include	string	no	Required content (comma-separated)
must_not_include	string	no	Forbidden content (comma-separated)
max_tokens	number	no	Max token budget for the test
temperature	number	no	Temperature to use
check_safety	boolean	no	Include safety/PII checks in the rubric
adversarial_prompts	boolean	no	Auto-generate adversarial test variants (jailbreak, injection, edge cases)

Validate that an LLM output matches an expected format: JSON, Markdown, code block, bullet list, numbered list, table, YAML, XML, or custom regex. Essential for structured output testing.

Parameter	Type	Required	Notes
output	string	yes	The LLM output to validate
expected_format	string	yes	Expected format
regex_pattern	string	no	Custom regex pattern (only when expected_format is "regex")

Compare multiple LLM responses to the same prompt and detect inconsistencies using Jaccard word-overlap similarity and fact drift (number comparison). Fast, deterministic, no API key needed. Limitations: relies on surface-level word matching — "Paris is the capital of France" vs "Paris is the French capital" may score low despite semantic equivalence. For true semantic consistency, use run_semanti

Parameter	Type	Required	Notes
responses	array	yes	Array of 2+ LLM responses to compare (same prompt, different runs)
check_facts	boolean	no	Check for contradictory numbers/facts across responses (default: true)

Validate that an LLM JSON output matches a JSON Schema definition. Tests required fields, types, enums, nested objects, and arrays. Critical for function-calling and structured output testing.

Parameter	Type	Required	Notes
output	string	yes	The LLM JSON output (raw string, will be parsed)
schema	object	yes	JSON Schema (draft-07 subset) to validate against

Score an LLM response against the criteria you pass: coverage of expected_keywords and compliance with max_length. Returns a 0-100 score over the criteria actually measured — or total_score null with the reason when none is given, because relevance and correctness cannot be read off word overlap (a correct one-word answer shares no words with its question). For meaning against a reference answer,

Parameter	Type	Required	Notes
question	string	yes	The original question/prompt
response	string	yes	The LLM response to score
expected_keywords	array	no	Keywords a correct answer must contain (case-insensitive). Scored as coverage.
max_length	number	no	Maximum character length. Scored as a proportional penalty beyond it.

Analyze a multi-turn conversation for context retention, topic drift, instruction following, and repetition. Accepts messages array [{role, content}]. Essential for chatbot QA.

Parameter	Type	Required	Notes
messages	array	yes	Conversation messages in order

Analyze a Chain-of-Thought (CoT) or reasoning trace from an LLM. Detects step count, logical flow, conclusion presence, backtracking, and estimates reasoning depth. Useful for o1/o3/DeepSeek-R1 evaluation.

Parameter	Type	Required	Notes
reasoning	string	yes	The CoT / reasoning trace text (e.g. from <think> tags or step-by-step output)
expected_conclusion	string	no	Expected final answer to check against (optional)

Generate an A/B test report comparing two prompts or model configurations. Accepts arrays of scores and returns mean, median, std deviation, improvement percentage, and a Welch t-test at 95 %: winner names a variant only when the difference is significant — otherwise "no_significant_difference", or "insufficient_samples" under 2 scores per variant.

Parameter	Type	Required	Notes
variant_a	object	yes	First variant configuration with name and score array
variant_b	object	yes	Second variant configuration with name and score array

Compute text similarity between reference and hypothesis using multiple metrics: Cosine (BoW, TF-IDF), Jaccard, ROUGE-1, ROUGE-2, ROUGE-L, and BLEU. No API key needed. Ideal for LLM eval (expected vs actual), RAG quality checks, and NLG benchmarking. Supports batch mode.

Parameter	Type	Required	Notes
reference	string	no	Reference / expected text (ground truth)
hypothesis	string	no	Hypothesis / actual text (LLM output)
metrics	array	no	Metrics to compute (default: all). Options: "cosine_bow", "cosine_tfidf", "jaccard", "rouge1", "rouge2", "rougeL", "bleu"
batch	array	no	Batch mode: array of {reference, hypothesis} pairs.
threshold	number	no	Optional pass/fail threshold (0-1). Applies to ROUGE-L F1 score.

Run a test suite against a Vision-Language Model (VLM) — send an image (URL or base64) + N test cases (each with a question + assertion) to GPT-4o, Claude 3.5, or Gemini. Returns per-case PASS/FAIL verdicts, a pass rate, an overall PASS/WARNING/FAIL verdict (customizable threshold), and latency stats. Assertion types: contains, not_contains, json_format, min_length, max_length, semantic_contains (

Parameter	Type	Required	Notes
image_url	string	no	Public URL of the image to evaluate (required unless image_base64 is provided).
image_base64	string	no	Base64-encoded image data (required unless image_url is provided).
image_mime_type	string	no	MIME type of the image if using image_base64 (default: image/jpeg).
system_prompt	string	no	Optional system prompt sent to the VLM.
threshold	number	no	Pass rate threshold for overall verdict (default: 80, 0–100).
model	string	yes	VLM model to use.
api_key	string	yes	API key for the model provider (OpenAI sk-, Anthropic sk-ant-, or Google AIzaSy...).
test_cases	array	yes	Array of test cases to run.

Compute text similarity using local algorithms (Bag of Words, TF-IDF, Character N-grams). No API key needed — runs entirely in-process. NOT real embeddings: for true semantic similarity with vector embeddings, use run_semantic_tests with mode="embeddings" and your OpenAI API key. Supports single pair or batch mode with pipe-separated pairs. Useful for RAG retrieval testing, semantic search evaluat

Parameter	Type	Required	Notes
text_a	string	no	First text to compare (single-pair mode)
text_b	string	no	Second text to compare (single-pair mode)
batch	array	no	Batch mode: array of { text_a, text_b } pairs. Overrides text_a/text_b if provided.
methods	array	no	Algorithms to use (default: all three). Options: "bow", "tfidf", "ngram"

Rank an array of text chunks by relevance to a query using TF-IDF scoring. Simulates retrieval ranking for RAG testing without needing embeddings or an API.

Parameter	Type	Required	Notes
query	string	yes	The user query
chunks	array	yes	Array of text chunks to rank
top_k	number	no	Return top K results (default: all)

Compute BM25 relevance score between a query and one or more documents. BM25 is the industry-standard keyword-based ranking algorithm used in Elasticsearch, OpenSearch, and Weaviate hybrid search. Returns ranked results with normalized scores.

Parameter	Type	Required	Notes
query	string	yes	The search query
documents	array	yes	Array of documents to rank
k1	number	no	Term frequency saturation (default: 1.5)
b	number	no	Length normalization factor (default: 0.75)
top_k	number	no	Return top K results (default: all)

Assemble a complete RAG (Retrieval-Augmented Generation) prompt from retrieved context chunks and a user query. Handles token budgeting, citation numbering, system instruction injection, and source attribution.

Parameter	Type	Required	Notes
query	string	yes	The user question to answer
chunks	array	yes	Retrieved context chunks with .text (required), .source (optional), .score (optional)
system_instruction	string	no	Custom system instruction (default: standard RAG grounding instruction)
max_context_tokens	number	no	Max tokens for context section (default: 2000)
cite_sources	boolean	no	Add [1], [2] citation numbers (default: true)
language	string	no	Response language instruction (e.g. "French", "Spanish")

Evaluate RAG retrieval quality: rank passages against a query and compute Precision@k / Recall@k plus a PASS/FAIL CI verdict from ground-truth relevance labels. Three modes, all keyless except the last. (1) BYO scores — give each passage the score your own reranker produced (Cohere, Jina, a self-hosted NIM, a cross-encoder): deterministic, offline, and it evaluates YOUR reranker rather than some

Parameter	Type	Required	Notes
query	string	yes	The search query or question to rank against
passages	array	yes	Array of passage objects to rank (min 2, max 20)
top_k	integer	no	k for Precision@k evaluation (default 3)
threshold	number	no	Minimum Precision@k to PASS (0-1, default 0.5)
api_key	string	no	Your NVIDIA API key (BYOK), used only when no passage carries a score. Transits RAM for the single call, never stored.

Estimate the token count of a text string using the cl100k_base approximation (~4 chars/token). Call this BEFORE sending any text to an LLM API to check if it fits within the model context window and to estimate cost. Returns token estimate, character count, and word count.

Parameter	Type	Required	Notes
input	string	yes	Text to count tokens for

Extract the first valid JSON object or array embedded in chaotic LLM output (surrounded by markdown fences, prose, or explanatory text). Handles ```json blocks and inline JSON. Call this whenever an LLM returns structured data mixed with explanation text instead of raw JSON.

Parameter	Type	Required	Notes
input	string	yes	Raw text (e.g., LLM output) that may contain a JSON object or array

Estimate the API cost in USD for a given model and token counts. Supports all major 2024–2026 models: GPT-4o, GPT-4.1, o3, o4-mini, Claude Opus 4, Claude Sonnet 4/4.5, Gemini 2.5 Pro/Flash, DeepSeek V3/R1, Grok 3, and legacy models.

Parameter	Type	Required	Notes
model	string	yes	Model name, e.g. "gpt-4o", "claude-3.5-sonnet", "deepseek-v3"
input_tokens	number	yes	Number of input/prompt tokens
output_tokens	number	no	Number of output/completion tokens (default: 0)

Plan token allocation across system prompt, user input, context/RAG chunks, and expected output. Warns if budget exceeds model context window. Supports 25+ models.

Parameter	Type	Required	Notes
model	string	yes	Model name (e.g. gpt-4o, claude-3.5-sonnet, gemini-3.5-flash)
system_prompt_tokens	number	no	Token count for system prompt
user_input_tokens	number	no	Token count for user message
context_tokens	number	no	Token count for RAG context / documents
expected_output_tokens	number	no	Expected max output tokens
system_prompt	string	no	Actual system prompt text (will estimate tokens)
user_input	string	no	Actual user input text (will estimate tokens)
context	string	no	Actual context text (will estimate tokens)

Given an array of message objects [{role, content}], estimate total token usage and check if it fits in the target model's context window. Warns about truncation risk.

Parameter	Type	Required	Notes
messages	array	yes	Array of messages (system/user/assistant)
model	string	yes	Target model name (e.g. gpt-4o, claude-3.5-sonnet)
max_output_tokens	number	no	Reserved tokens for output (default: 4096)

Get detailed specs for an AI model: context window, pricing per 1K tokens, knowledge cutoff, provider, multimodal support, reasoning capabilities, and feature list. Covers 30+ models from OpenAI, Anthropic, Google, DeepSeek, Meta, Mistral, Cohere, xAI.

Parameter	Type	Required	Notes
model	string	yes	Model name (e.g. "gpt-4o", "claude-3.5-sonnet", "gemini-3.1-pro-preview")

Compare 2-5 AI models side by side: context window, pricing, multimodal, reasoning capabilities, and provider. Returns a comparison table with a recommendation based on your use case.

Parameter	Type	Required	Notes
models	array	yes	Array of 2-5 model names (e.g. ["gpt-4o","claude-3.5-sonnet","gemini-3.5-flash"])
use_case	string	no	Optimize recommendation for this criterion

List all LLM models available on ia-qa.com with their provider, API endpoint, and capabilities. Filter by provider name (e.g. "Groq", "HuggingFace", "OpenAI") or return the full catalog. Use this to discover which models are available before calling an LLM API, or to compare providers.

Parameter	Type	Required	Notes
provider	string	no	Filter by provider name (case-insensitive). E.g. "Groq", "HuggingFace", "OpenAI", "Anthropic", "Google", "DeepSeek", "xAI", "Ollama". Omit for full catalog.

Generate text using open-source LLM models hosted on Groq (ultra-fast) or HuggingFace Inference (serverless). No API key required — the server provides its own keys. Supported models: Qwen3 32B, Gemma 4 27B, Gemma 3 27B, Llama 3.3 70B, Llama 4 Scout, DeepSeek R1, Mistral Small 24B, and more. Use list_llm_models to see the full catalog. Rate-limited to prevent abuse.

Parameter	Type	Required	Notes
prompt	string	yes	The user prompt / instruction to send to the model
model	string	no	Model ID (default: "openai/gpt-oss-20b"). Server-keyed whitelist only — Groq: openai/gpt-oss-20b, openai/gpt-oss-120b; HuggingFace: Qwen/Qwen3-32B, meta-llama/Llama-3.3-70B-Instruct, deepseek-ai/DeepS
system	string	no	Optional system prompt to set context or persona
temperature	number	no	Sampling temperature 0.0–1.5 (default: 0.7)
max_tokens	number	no	Maximum tokens to generate (default: 2048, max: 4096)

Give honest usage feedback on an IA-QA MCP tool. Provide a score (1-5) and a comment. Rate low (1-2) if the tool was wrong, irrelevant, or a poor fit; rate high (4-5) only if it genuinely solved your need. Ratings are read by the IA-QA maintainers and are not published, one vote per agent and tool a month. Comments are screened before storing: one carrying a credential is not stored and personal d

Parameter	Type	Required	Notes
tool_name	string	yes	Name of the MCP tool to rate (e.g. "format_json", "shield_analyze")
score	number	yes	Rating from 1 (poor) to 5 (excellent)
comment	string	no	Strongly encouraged — explain what you were trying to do and whether the tool got you there. Be specific about what was missing, wrong, or a poor fit. This is the most valuable part of the rating. Up

Free

$0 forever

Every tool — any call limit is its publisher’s, not ours
No card, and mcprush charges nothing for it
Runs at its publisher’s own address — this marketplace is not in the path
Nothing to cap, because mcprush bills nothing

Nothing here is billable: mcprush charges nothing to install IA-QA Eval & RAG — 38 tools or to call it, at any volume.

BTool-list grade · score 84 / 100

Release 1.0.1 · scanned 28 Oct 2026

Graded from the tools this server lists, not from its source code.

Two independent axes, because powerful and malicious are different questions. The grade is threat only. The capability level is blast radius, and it is never a penalty on the grade — it is priced as one subtract-only term in the score, where you can see it.

Threat score94

Client score84

Findings1 threat · 4 capability

Capabilityhigh

Coveragemanifest

Verificationnot in this report

Surface digest3546e31ab6922f6b12c08e9064aeb9efff737544333275438da7a9a1c8c571b3

Score vectorthreat 94 → client 84

Term	Level	What it prices	Points
capability-exposure	high	capability blast radius (high) — client exposure if the model is manipulated	−6
coverage-honesty	manifest	inspection depth (manifest) — how much of the target the scan could see	−4

score = clamp(round(threatScore − E_cap − E_ver − E_cov), 0, 100). No model, no opinion, no network call at scoring time: the same methodology version against the same bytes always produces this number, and every line above is a field of the report rather than a summary of it.

What the scan could actually read

A grade is only as meaningful as its coverage, so the scanner publishes its own depth before it publishes its result.

The scan read each tool’s name, description, input schema and annotations exactly as the server listed them the last time it answered us; no code was reviewed. We ask the server again about once a month, and a list that changed is graded again.

Tool surfaceread

Implementation sourcenot available

Package metadatanot available

Live transportnot available

The implementation source was not read — this reflects the declared tool metadata only.

Capability — what it could do if the model were manipulated

Tags derived from each tool’s schema and the implementation, not from what the tool calls itself. high is the level these add up to.

Tool	Capability tags	Why the tag was assigned
get_testing_guidelines	no tags
find_tool	no tags
hallucination_check	no tags
run_eval_contract	untrusted-input	untrusted-input: declares openWorldHint: true (interacts with entities outside the server)
run_semantic_tests	no tags
generate_ci_workflow	code-exec	code-exec: parameter "command"
needle_haystack_generate	no tags
generate_eval_yaml	untrusted-input	untrusted-input: declares openWorldHint: true (interacts with entities outside the server)
metamorphic_check	no tags
test_skill	untrusted-input	untrusted-input: declares openWorldHint: true (interacts with entities outside the server)
llm_output_validator	no tags
compare_responses	no tags
analyze_responses	no tags
prompt_test_suite	no tags
llm_format_check	no tags
consistency_check	no tags
llm_json_schema_check	no tags
response_quality_score	no tags
conversation_analyze	no tags
cot_analyzer	no tags
ab_test_report	no tags
similarity_score	no tags
run_vlm_test_suite	untrusted-input	untrusted-input: declares openWorldHint: true (interacts with entities outside the server)
embedding_similarity	no tags
rag_relevance_rank	no tags
bm25_score	no tags
build_rag_prompt	no tags
rerank_evaluate	untrusted-input	untrusted-input: declares openWorldHint: true (interacts with entities outside the server)
count_tokens	no tags
extract_json_from_text	no tags
estimate_llm_cost	no tags
token_budget_calculator	no tags
context_window_check	no tags
model_info	no tags
compare_models	no tags
list_llm_models	no tags
llm_generate	untrusted-input	untrusted-input: declares openWorldHint: true (interacts with entities outside the server)
rate_tool	state-write	state-write: declares readOnlyHint: false

Toxic-flow graph

The lethal trifecta, checked as a graph rather than as a checklist: untrusted input, a sensitive source and an external sink have to meet before there is a path worth worrying about.

flow-1

Untrusted input can drive an external action even though no sensitive source is exposed.

Supply chain and provenance

Integrityfirst-seen

Surface digest3546e31ab6922f6b

Scanned2026-10-28

This is the first scan of this surface here, so there is nothing yet to compare it against.

Every result on this tab comes from one deterministic pass over the published package — offline, rule by rule, and auditable line by line above. Same methodology version, same bytes, same score.

Where are you running it?

Tool surface

Release history

IA-QA Eval & RAG — 38 tools MCP

What it does

Quickstart

Where are you running it?

Tool surface

What counts against your monthly calls

What the scan could actually read

Capability — what it could do if the model were manipulated

Toxic-flow graph

Supply chain and provenance

Release history

See also