28<!-- Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. Licensed under the Apache License, Version 2.0; see http://www.apache.org/licenses/LICENSE-2.0 -->
29
30# tao-finetune-huggingface-model
31
32Local NVIDIA GPU fine-tuning for HuggingFace models, grounded in live-fetched
33documentation with curated references as a fallback safety net. One NGC container,
34a few focused scripts, one push to HF Hub. Follow the rules in this file; don't
35improvise.
36
37**Order of authority (highest first):**
38
391. **User input** — explicit model_id, dataset_id, training_method, config.yaml overrides.
402. **Live research** — model card, HF repo example, author finetune script, HF task docs, paper; always fetched (Step 3 + references/research-priorities.md).
413. **Curated references** (references/*.md) — fallback when live research is silent/ambiguous.
424. **Your training-data memory** — last resort; suspect, cross-check against (2)/(3).
43
44Conflict resolution between (2) and (3) and the source-line discrepancy note are
45in references/research-priorities.md.
46
47---
48
49## Inputs
50
51**Required:**
52- model_id — HuggingFace model ID, e.g. google/vit-base-patch16-224
53
54**Conditional credentials (read from the session environment, exported before launching when present):**
55- HF_TOKEN — only when the model/dataset is **gated** (read) or push_to_hub is on (write); public + public + push_to_hub: false needs none. Value never read — presence-only via [ -n "$HF_TOKEN" ].
56- WANDB_API_KEY, WANDB_PROJECT — only when WandB is enabled; WANDB_MODE=disabled opts out.
57
58**Dataset — exactly one:**
59- dataset_id — HuggingFace dataset ID *(source: hf)*
60- local_dataset_path — local folder or file *(source: local)*; optional
61 local_dataset_format ∈ {auto, imagefolder, coco, voc, jsonl, arrow, parquet,
62 csv} (default: auto-detect).
63- *(omit)* — agent recommends popular datasets *(source: recommend)*
64
65**Optional (have defaults):**
66- task_type — auto-detected from config + model card
67- n_train=10000, n_eval=1000, n_epochs=3, lora_r=16
68- output_dir=./output/<model_short_name>
69- hf_model_repo — push target; if unset and HF_TOKEN has write access,
70 auto-derived as <whoami>/<model_short_name>-finetuned.
71- push_to_hub=True — set to False to skip
72- skip_baseline=False — skip zero-shot baseline eval
73
74**Optional deliverables (off by default):**
75```yaml
76emit_progress_log: false # output_dir/PROGRESS.md (per-step journal)
77emit_report: false # reports/report.{pdf,html} with curves & samples
78emit_unit_tests: false # tests/ with fake-data heterogeneous-batch tests
79```
80
81All values live in output_dir/config.yaml. Never hardcode in Python.
82
83---
84
85## Execution platform
86
87This skill orchestrates *what* to run; the platform skills own *how* to run it on
88a GPU host — read them first.
89
90| Concern | Authoritative skill |
91|---|---|
92| GPU host runtime (driver 580, CUDA Toolkit 13.0, NVIDIA Container Toolkit 1.19.0) | [tao-skill-bank:tao-setup-nvidia-gpu-host](../../platform/tao-setup-nvidia-gpu-host/SKILL.md) |
93| docker run flags, NGC auth, mounts, env passthrough | [tao-skill-bank:tao-run-on-docker](../../platform/tao-run-on-docker/SKILL.md) |
94| Local Docker job preflight (daemon, GPU smoke) | [tao-skill-bank:tao-run-on-local-docker](../../platform/tao-run-on-local-docker/SKILL.md) |
95
96**Default platform:** local-docker — build a one-off image (run-<short>:latest)
97and run it on the local Docker daemon. Ask only when the user explicitly needs a
98different backend (Brev remote GPU, SLURM/Kubernetes); then run that platform's
99Preflight first and route the Steps 4–5 docker run commands through it. The
100GPU-runtime and presence-only credential preflights (values never read), the
101canonical docker run flag set, the list_tao_platforms.py selection command, and
102the workflow-specific flags (--entrypoint /bin/bash -lc, PYTORCH_CUDA_ALLOC_CONF,
103--name hft_train) are in references/workflow-intake-preflight.md.
104
105---
106
107## References — fallback safety net
108
109Consulted **only** when live research is silent, ambiguous, or unavailable; live
110docs always win for the specific model and current API. Each step links the
111references it needs; full catalog in references/detailed-workflow.md.
112
113Always-on: core-rules.md, error-playbook.md, compat-workarounds.md,
114model-discovery.md, dataset-recommendations.md, dataset-sources.md,
115dataset-patterns.md, hardware-container.md, research-priorities.md,
116cv-scripts.md, vlm-scripts.md, docker-runs.md, hub-push.md,
117pipeline-skill-template.md, deliverables.md. Opt-in (when their flag/need
118applies): progress-tracking.md, testing.md, reporting.md,
119workflow-intake-preflight.md, workflow-generate-train.md, workflow-push-rerun.md.
120
121**Rule:** before falling back, log the live source you tried and why it was
122insufficient (config.yaml notes:, and PROGRESS.md if enabled). [FETCH LIVE]
123markers in cv-scripts.md / vlm-scripts.md are a research checklist, not code to
124inline — refetch the listed URL if a block has no Step 3 finding.
125
126---
127
128## Core rules
129
130Non-negotiable behaviors. **Short version** (full enumeration —
131hallucinated-imports list, never-without-approval list, full error-recovery and
132hardware-sizing tables — in references/core-rules.md, consult before any
133training-time decision):
134
135- **Your HF-library knowledge is outdated.** Fetch live docs (model card, HF
136 repo example, task doc) before writing any ML code — don't generate trainer
137 args / collator / transforms from memory (Step 3).
138- **Smoke-test on real data with --max_steps 1** before any full run; no batch
139 launches without a verified smoke.
140- **Never silently substitute** model_id, dataset_id, or training_method — if
141 what the user asked for doesn't load, stop and ask.
142- **Error recovery is minimal-change.** OOM → halve batch, double grad_accum,
143 enable gradient checkpointing (no LoRA switch without approval); NaN → reduce
144 LR 10×; flat loss → inspect collator; same error 3× → stop and ask. Don't loop.
145- **Dataset columns verified BEFORE the collator** — rename in prepare_data.py;
146 restructuring needed → stop and ask.
147- **Hardware-sizing thumb (bf16):** ≤3B → 24 GB, 7–13B → 80 GB, 30B+ → multi-GPU
148 or LoRA on 1× 80 GB, 70B+ → 8× 80 GB or LoRA. Full finetune won't fit and no
149 LoRA requested → ask before switching.
150
151---
152
153## Workflow — 6 steps
154
155Single pass, sequential; each step has a clear gate before the next begins.
156
157### Step 1 — Inspect & qualify
158
159**Goal:** decide whether to proceed. Probe model + dataset, apply accept/reject,
160register applicable compat fixes, write the initial config.yaml.
161
162Prerequisites: MODEL_ID, optional DATASET_ID / local_dataset_path,
163optional HF_TOKEN, OUTPUT_DIR (default ./output/<model_short_name>). Probes
164run in a CPU-only python:3.12-slim Docker container (bind-mounted .probe/
165scratch) so the host needs no virtualenv — Docker must exist first. Docker-presence
166guard, container env, full probe invocation, and the model/dataset probe scripts
167are in references/workflow-intake-preflight.md, references/model-discovery.md,
168and references/dataset-sources.md.
169
170Probe requirements:
171
172- Model: load AutoConfig, read model-card tags, detect task from
173 architectures + tags + card examples (fallback logging in model-discovery.md).
174- Dataset: for recommended datasets, first present 3-5 choices from
175 dataset-recommendations.md; for local data, bind-mount read-only and use
176 dataset-sources.md format detection.
177- Reject early if the model config fails, the task is out of scope, no recipe
178 source exists, or the dataset cannot load / match the task schema.
179- Evaluate compat-workarounds.md against the model/task; defer hardware-dependent
180 rules to Step 2.
181
182Write the initial config.yaml (model_id, task, dataset_id or
183local_dataset_path, research_sources: [] filled in Step 3,
184applicable_workarounds: from Step 1, notes: [] for reference fallbacks,
185push_to_hub: true default — annotated template in
186references/workflow-intake-preflight.md). Optionally rm -rf "$OUTPUT_DIR/.probe"
187once the gate is met.
188
189**Gate:** config.yaml exists with model, dataset, task, applicable_workarounds;
190do not proceed if any field is missing.
191
192---
193
194### Step 2 — Hardware audit & NGC image
195
196**Goal:** verify Docker + GPU + disk, pick the NGC PyTorch image live, finalize
197hardware-dependent compat rules.
198
199**2a. Audit (hard gate)** — three checks (commands in
200references/workflow-intake-preflight.md):
2011. GPU host runtime — tao-setup-nvidia-gpu-host's
202 setup-nvidia-gpu-host.sh --backend docker --check-only; on fail, ask approval
203 then re-run with --install --yes.
2042. Free-disk soft-warn — override via MIN_DISK_GB (default 100 GB); recommend
205 ≥ 100 GB for NGC base (~20 GB) + HF cache + checkpoints + data.
2063. Conditional credential presence (from the session environment, values never
207 read) — HF_TOKEN only when gated or push_to_hub is on; WANDB_* only when
208 WandB is on.
209
210**Do not proceed to Step 4 on a hard-fail** — Step 4's docker build pulls a
21120+ GB NGC base, and a missing nvidia-container-toolkit only surfaces later as
212could not select device driver "" with capabilities: [[gpu]]. Record gpu_count,
213gpu_name, driver_major, vram_gb_per_gpu in config.yaml.
214
215**2b. Pick NGC image (live):** from the NVIDIA Deep Learning Frameworks support
216matrix (<https://docs.nvidia.com/deeplearning/frameworks/support-matrix/index.html>),
217PyTorch NGC container section, pick the highest-versioned image where
218Min driver ≤ detected driver_major and container CUDA ≤ host CUDA Toolkit
219(match closely so cuDNN / TensorRT line up). Do **not** reject an image for an
220aN/bN/rcN PyTorch tag — NGC validates the full image; pick the newest
221CUDA-aligned one and let compat-workarounds.md handle per-version issues. If the
222matrix is unreachable, use the fallbacks in references/hardware-container.md;
223default nvcr.io/nvidia/pytorch:24.09-py3 (driver ≥ 545; SDPA+GQA bug — if
224num_key_value_heads < num_attention_heads, set attn_implementation: "eager").
225Record ngc_image in config.yaml.
226
227**2c. Re-evaluate hardware-dependent compat rules:** re-run the
228compat-workarounds.md walk for entries whose detect needs hw; update
229applicable_workarounds: in place.
230
231**2d. Model-fit check:** estimate param_bytes ≈ 2×param_count (bf16); if
232> 60% of vram_gb_per_gpu × 1e9, recommend LoRA in the user-facing summary.
233
234**Gate:** config.yaml has ngc_image, gpu_count, gpu_name, driver_major,
235vram_gb_per_gpu; hardware-dependent compat fixes recorded.
236
237---
238
239### Step 3 — Research the recipe
240
241**Goal:** fetch the live recipe — training-data knowledge of
242transformers/trl/peft is suspect, so Step 3 is non-negotiable. Walk
243references/research-priorities.md in priority order (Priority 1 → 6); stop once
244you have, for the detected task:
245
246- AutoModel / processor class
247- Train + eval transforms
248- Collator
249- compute_metrics
250- Hyperparameter hints (LR, batch size, epochs, scheduler)
251
252Record findings in meta/recipe.md, append source URLs to
253config.yaml: research_sources:. A slot with no live finding falls back to the
254matching scaffold (cv-scripts.md / vlm-scripts.md), logged as "fallback to
255scaffold — no live source for <slot>" under notes:. Conflict-resolution rules
256are in references/research-priorities.md.
257
258**Gate:** every required slot filled, with a source URL or scaffold-fallback note.
259
260---
261
262### Step 4 — Generate project & smoke-test
263
264**Goal:** write all scripts, build the image, prepare data, run a 1-step smoke on
265real data (one docker build, two docker runs).
266
267**4a. Generate project files** in output_dir/: config.yaml, Dockerfile,
268requirements.txt, prepare_data.py, train.py, run_eval.py, infer.py,
269optional merge_lora.py, optional tests/, .gitignore. Live Step 3 research is
270authority; cv-scripts.md / vlm-scripts.md give scaffold shape only. Apply every
271applicable_workarounds entry as a Dockerfile block, requirement pin, config
272override, or runtime env var. Hard rules: run_eval.py keeps that exact filename
273(avoids colliding with the HF evaluate package); every generated .py starts
274with the NVIDIA Apache-2.0 copyright header and any emitter fails when it is
275missing; emit_unit_tests: true generates and runs tests per
276references/testing.md. Script bodies, Dockerfile shape, and the emitter contract
277are in references/workflow-generate-train.md.
278
279**4b. Build, prepare, smoke** — docker build -t run-<short>:latest ., then
280prepare_data and the --smoke --max_steps 1 run (references/docker-runs.md
281§1-3). Smoke pass criteria (in logs/smoke.log):
282- No exception
283- Loss is finite (not 0.0, not NaN)
284- grad_norm > 0 at step 1
285
286If emit_unit_tests: true, also run pytest tests/ in the container. Any failure → STOP.
287
288**4c. Preflight summary** — before full training, print and verify: reference URL,
289dataset columns, Hub target, monitoring target, NGC image, hardware, smoke loss/grad norm.
290
291**Gate:** project files written, image built, smoke PASSED, preflight has no
292blank fields.
293
294---
295
296### Step 5 — Train, evaluate, infer
297
298**Goal:** baseline eval, full training, post-train eval, optional LoRA merge, 5
299inference samples (all commands: references/docker-runs.md §4-8).
300
301| Sub-step | docker-runs.md | Skip if |
302|---|---|---|
303| 5a. Baseline eval (zero-shot) | §4 | skip_baseline: true |
304| 5b. Full training (detached) | §5 | — |
305| 5c. LoRA merge | §6 | not VLM+LoRA |
306| 5d. Post-train eval | §7 | — |
307| 5e. Inference (5 samples) | §8 | — |
308
309Multi-GPU: prepend torchrun --nproc_per_node=$gpu_count to python train.py.
310
311While training streams, watch docker logs -f hft_train: loss should drop within
31210-20 steps; flat loss (collator/label-masking bug), NaN (LR too high), and OOM
313all stop the run — recovery in references/core-rules.md. If emit_report: true,
314run report.py after Step 5e per references/reporting.md.
315
316**Gate:** all of:
317- checkpoints/final/ (or checkpoints/merged/ for LoRA) exists
318- reports/eval_results.json has a numeric primary metric
319- reports/baseline_results.json exists (unless skipped)
320- reports/inference_samples/ has 5 samples
321- wandb URL shows descending loss
322
323---
324
325### Step 6 — Push & emit rerun skill
326
327**Goal:** publish the run and make it reproducible without re-research.
328
329Push per references/hub-push.md (weights, model card, eval/baseline JSONs,
330config.yaml, Dockerfile, requirements.txt, inference samples, reports when
331emitted) unless push_to_hub: false is explicit. Emit
332<output_dir>/skills/run-<short>/SKILL.md from
333references/pipeline-skill-template.md — substitute every placeholder, include
334full YAML metadata + the NVIDIA copyright HTML comment, and make any emitter fail
335if those are missing.
336
337**Gate (Done criteria):** all of:
338- Step 5 gate met
339- HF Hub repo exists at the resolved URL with weights + card + results/
340 (unless push_to_hub: false)
341- <output_dir>/skills/run-<short>/SKILL.md exists, no <placeholder> left,
342 with metadata + copyright HTML comment per pipeline-skill-template.md
343
344Final message: wandb URL, HF Hub URL, baseline -> fine-tuned primary metric,
345reports/inference_samples/, and the rerun skill path.
346
347---
348
349## Error playbook
350
351On a known runtime error, consult the symptom → minimal-fix table in
352references/error-playbook.md (NGC entrypoint, PyTorch/Transformers regressions,
353numpy ABI, Albumentations bbox, PEFT/checkpointing, LoRA target breadth, CV
354augmentation gaps, OOM at step 0) before redesigning anything. When a row there
355fires twice across runs, lift it into compat-workarounds.md with a detect rule
356— auto-applied in Step 1 before the error can fire.
357
358---
359
360## Communication style
361
362- Terse. No filler, no restating the request; one-word answers when appropriate.
363- Always include direct Hub and wandb URLs when referencing artifacts.
364- On error: state what went wrong, why, what you changed — no menus.
365- Never present "Option A/B/C" for a request with a clear answer. Act.
366
367## Example pipelines
368
369- [tao-rerun-convnext-cifar10](references/tao-rerun-convnext-cifar10.md)
370- [tao-rerun-detr-cppe5](references/tao-rerun-detr-cppe5.md)
371- [tao-rerun-segformer-foodseg103](references/tao-rerun-segformer-foodseg103.md)
372- [tao-rerun-smolvlm-vqav2](references/tao-rerun-smolvlm-vqav2.md)
373