Hugging Face Community Evals
Run evaluations for Hugging Face Hub models using inspect-ai and lighteval on local hardware.
- Context cost
- 7.5k tokensestimated from the bundle, loaded when it triggers
- Bundle
- 6 files · 30.2 kB3 scripts among them — read before you run
- Licence
- Apache-2.0paid listing
- Last change
- —no release on file
- Servers it uses
- Noneruns standalone
What it does
Run evaluations for Hugging Face Hub models using inspect-ai and lighteval on local hardware. Use for backend selection, local GPU evals, and choosing between vLLM / Transformers / accelerate. Not for HF Jobs orchestration, model-card PRs, .eval_results publication, or community-evals automation.
Installed, it changes the agent in these ways.
What this skill changes about the agent is not written down here yet. The listing was collected from its source, and the description is in its own SKILL.md.
And these are the jobs people reach for it for.
Runs a procedure end to end.
The skill itself
This is the whole product. A skill is instructions the model reads, so there is nothing behind the listing you cannot see first — the front matter loads with every session, and the body below it loads when the skill triggers.
Nothing in a skill executes. The client loads the text and the model follows it, so a skill can be audited the way a runbook is — by reading it.
What it costs in context
Skills are not billed by the call. They are paid for in context: every token the instructions occupy is a token your code, your diff and your conversation cannot use. Here is what this one takes and when it takes it.
7.5k tokens, estimated from the bundle at four bytes to the token, held for the rest of the session once it triggers. Heavy. Teams tend to install this one per project rather than globally, and load it only when the job comes up.
A server charges by the month. A skill charges once per session, in context, and then keeps charging it for as long as the session lives.
Before and after
The same question, put to the same model twice: once as it comes, and once with these instructions loaded.
No worked example has been published for this skill yet.
The procedure it runs
The procedure has not been published here. It is in the skill’s own SKILL.md, which its author has not sent to the marketplace yet.
These steps are written for a model to follow, not executed by a runtime. It can still be told to skip one, and it will say so when it does.
Servers it uses
None. This skill calls no MCP servers at all.
Everything it needs is in the instructions, so it works in a project with nothing connected — the model reads the file and changes how it works with what it can already reach.
It writes no files and reaches no network. All it changes is how the model reasons and writes.
Read from the allowed-tools line of this skill’s own SKILL.md. A skill grants no permissions of its own — it can only ask for tools your client already has.
What it will not do
Every skill is narrow, and the useful ones say where they stop. These are the jobs this one is the wrong tool for.
What this skill is not for has not been published here. Nothing is implied by that: it is a section the author has not filled in.
What is in the bundle
6 files, 30.2 kB on disk. Mostly text — the instructions the model reads — with 3 scripts in it that your client would run only if the instructions tell it to.
- SKILL.md6.6 kB
- examples/.env.example0.2 kB
- examples/USAGE_EXAMPLES.md2.1 kB
- scripts/inspect_eval_uv.py3.0 kB
- scripts/inspect_vllm_uv.py9.1 kB
- scripts/lighteval_vllm_uv.py9.2 kB
A skill installs nothing and depends on nothing: it is a folder your client reads. This one carries 3 scripts beside the text, so the bundle is 6 files you can review in full before installing. The Apache-2.0 licence covers the templates and examples as well as the instructions.
Install
Installing copies the bundle into your project. Nothing runs at install time — the files sit on disk until the model reads them.
You can read the whole bundle before paying — the SKILL.md above is the product, not a preview of it. What the money buys is the delivery: the folder packaged and handed to your machine by key, every update its author ships, and our support if it does not do what this listing says. The terms of use are Apache-2.0, set by the author and unchanged by buying it here.
Payment runs through Stripe, on a page like this one rather than a redirect. Once there is an account it joins the same mcprush invoice as everything else you run, so there is never a second card to enter.
# Hugging Face Community Evals · 7.5k tokens when loaded
npx mcprush@latest skill add huggingface/hugging-face-community-evalsWrites to .claude/skills/hugging-face-community-evals/ in the current project. Add --global to put it in your home directory instead, for every project.
Which clients pick it up on their own
A skill is a folder of text. A client with a skills folder reads it without being told; everywhere else the same text works, it is just handed to the model rather than found.
Versions
Its author publishes no version number, so there is nothing here to pin to: what you install is the folder as it stands today. Instructions change more often than APIs do — a skill can be rewritten entirely without anything it depends on moving.
- No earlier releases have been published to the marketplace.
Nothing to pin to: this skill carries no version number of its own. What you install is what the folder holds on the day you install it.
Reviews
Nobody has reviewed this skill yet. The rating is the mean of the reviews written here, so there is none until somebody writes the first.
Only accounts that have had the skill installed for fourteen days, so a review is written after living with it rather than after reading it. Publishers may reply once.