Publish Sign in
Workflow·Data Science & ML·v1.0.0

TRL Training

Train and fine-tune transformer language models using TRL (Transformers Reinforcement Learning), with SFT, DPO, GRPO, KTO, RLOO and…

You say
Buy it · $35 Read it before you buy $35 Written by huggingface · unverified publisher
Context cost
2.2k tokensestimated from the bundle, loaded when it triggers
Bundle
1 file · 8.8 kBtext throughout, nothing executable
Licence
Apache-2.0paid listing
Last change
—v1.0.0
Servers it uses
Noneruns standalone

What it does

Train and fine-tune transformer language models using TRL (Transformers Reinforcement Learning). Supports SFT, DPO, GRPO, KTO, RLOO and Reward Model training via CLI commands.

Installed, it changes the agent in these ways.

What this skill changes about the agent is not written down here yet. The listing was collected from its source, and the description is in its own SKILL.md.

Workflow

Runs a procedure end to end.

trlrlhffine-tuningtransformers
Filed under

Data Science & ML

The skill itself

This is the whole product. A skill is instructions the model reads, so there is nothing behind the listing you cannot see first — the front matter loads with every session, and the body below it loads when the skill triggers.

SKILL.md8.8 kB · 319 lines
--- name: trl-training description: Train and fine-tune transformer language models using TRL (Transformers Reinforcement Learning). Supports SFT, DPO, GRPO, KTO, RLOO and Reward Model training via CLI commands. license: Apache-2.0 metadata: version: "1.0.0" author: huggingface commands: trl sft, trl dpo, trl grpo, trl kto, trl rloo, trl reward categories: machine-learning, llm-training, reinforcement-learning tags: rlhf, supervised-fine-tuning, dpo, grpo, huggingface, transformers documentation: https://huggingface.co/docs/trl/en/clis ---
14# TRL Training Skill
15
16You are an expert at using the TRL (Transformers Reinforcement Learning) library to train and fine-tune large language models.
17
18## Overview
19
20TRL provides CLI commands for post-training foundation models using state-of-the-art techniques:
21
22- **SFT** (Supervised Fine-Tuning): Fine-tune models on instruction-following or conversational datasets
23- **DPO** (Direct Preference Optimization): Align models using preference data
24- **GRPO** (Group Relative Policy Optimization): Train models by ranking multiple sampled outputs relative to each other and optimizing based on their comparative rewards.
25- **RLOO** (Reinforce Leave One Out): Online RL training with generation-based rewards
26- **Reward Model Training**: Train reward models for RLHF
27
28TRL is built on top of Hugging Face Transformers and Accelerate, providing seamless integration with the Hugging Face ecosystem.
29
30## Core Commands
31
32### trl sft - Supervised Fine-Tuning
33
34Fine-tune language models on instruction-following or conversational datasets.
35
36**Full training:**
37
38```bash
39trl sft \
40 --model_name_or_path Qwen/Qwen2-0.5B \
41 --dataset_name trl-lib/Capybara \
42 --learning_rate 2.0e-5 \
43 --num_train_epochs 1 \
44 --packing \
45 --per_device_train_batch_size 2 \
46 --gradient_accumulation_steps 8 \
47 --eos_token '<|im_end|>' \
48 --eval_strategy steps \
49 --eval_steps 100 \
50 --output_dir Qwen2-0.5B-SFT \
51 --push_to_hub
52```
53
54**Train with LoRA adapters:**
55
56```bash
57trl sft \
58 --model_name_or_path Qwen/Qwen2-0.5B \
59 --dataset_name trl-lib/Capybara \
60 --learning_rate 2.0e-4 \
61 --num_train_epochs 1 \
62 --packing \
63 --per_device_train_batch_size 2 \
64 --gradient_accumulation_steps 8 \
65 --eos_token '<|im_end|>' \
66 --eval_strategy steps \
67 --eval_steps 100 \
68 --use_peft \
69 --lora_r 32 \
70 --lora_alpha 16 \
71 --output_dir Qwen2-0.5B-SFT \
72 --push_to_hub
73```
74
75### trl dpo - Direct Preference Optimization
76
77Align models using preference data (chosen/rejected pairs).
78
79**Full training:**
80
81```bash
82trl dpo \
83 --dataset_name trl-lib/ultrafeedback_binarized \
84 --model_name_or_path Qwen/Qwen2-0.5B-Instruct \
85 --learning_rate 5.0e-7 \
86 --num_train_epochs 1 \
87 --per_device_train_batch_size 2 \
88 --max_steps 1000 \
89 --gradient_accumulation_steps 8 \
90 --eval_strategy steps \
91 --eval_steps 50 \
92 --output_dir Qwen2-0.5B-DPO \
93 --no_remove_unused_columns
94```
95
96**Train with LoRA adapters:**
97
98```bash
99trl dpo \
100 --dataset_name trl-lib/ultrafeedback_binarized \
101 --model_name_or_path Qwen/Qwen2-0.5B-Instruct \
102 --learning_rate 5.0e-6 \
103 --num_train_epochs 1 \
104 --per_device_train_batch_size 2 \
105 --max_steps 1000 \
106 --gradient_accumulation_steps 8 \
107 --eval_strategy steps \
108 --eval_steps 50 \
109 --output_dir Qwen2-0.5B-DPO \
110 --no_remove_unused_columns \
111 --use_peft \
112 --lora_r 32 \
113 --lora_alpha 16
114```
115
116### trl grpo - Group Relative Policy Optimization
117
118Train models using reward functions or LLM-as-a-judge for evaluating generations and providing rewards.
119
120**Basic usage:**
121
122```bash
123trl grpo \
124 --model_name_or_path Qwen/Qwen2.5-0.5B \
125 --dataset_name trl-lib/gsm8k \
126 --reward_funcs accuracy_reward \
127 --output_dir Qwen2-0.5B-GRPO \
128 --push_to_hub
129```
130
131### trl rloo - Reinforce Leave One Out
132
133Online RL training where the model generates text and receives rewards based on custom criteria.
134
135**Basic usage:**
136
137```bash
138trl rloo \
139 --model_name_or_path Qwen/Qwen2.5-0.5B \
140 --dataset_name trl-lib/tldr \
141 --reward_model_name_or_path sentiment-analysis:nlptown/bert-base-multilingual-uncased-sentiment \
142 --output_dir Qwen2-0.5B-RLOO \
143 --push_to_hub
144```
145
146### trl reward - Reward Model Training
147
148Train a reward model to score text quality for RLHF.
149
150**Full training:**
151
152```bash
153trl reward \
154 --model_name_or_path Qwen/Qwen2-0.5B-Instruct \
155 --dataset_name trl-lib/ultrafeedback_binarized \
156 --output_dir Qwen2-0.5B-Reward \
157 --per_device_train_batch_size 8 \
158 --num_train_epochs 1 \
159 --learning_rate 1.0e-5 \
160 --eval_strategy steps \
161 --eval_steps 50 \
162 --max_length 2048
163```
164
165**Train with LoRA adapters:**
166
167```bash
168trl reward \
169 --model_name_or_path Qwen/Qwen2-0.5B-Instruct \
170 --dataset_name trl-lib/ultrafeedback_binarized \
171 --output_dir Qwen2-0.5B-Reward-LoRA \
172 --per_device_train_batch_size 8 \
173 --num_train_epochs 1 \
174 --learning_rate 1.0e-4 \
175 --eval_strategy steps \
176 --eval_steps 50 \
177 --max_length 2048 \
178 --use_peft \
179 --lora_task_type SEQ_CLS \
180 --lora_r 32 \
181 --lora_alpha 16
182```
183
184## Configuration Files
185
186TRL supports YAML configuration files for reproducible training. All CLI arguments can be specified in a config file.
187
188**Example config (sft_config.yaml):**
189
190```yaml
191model_name_or_path: Qwen/Qwen2.5-0.5B
192dataset_name: trl-lib/Capybara
193learning_rate: 2.0e-5
194num_train_epochs: 1
195per_device_train_batch_size: 8
196gradient_accumulation_steps: 2
197output_dir: ./sft_output
198use_peft: true
199lora_r: 16
200lora_alpha: 16
201report_to: trackio
202```
203
204**Launch with config:**
205
206```bash
207trl sft --config sft_config.yaml
208```
209
210**Override config values:**
211
212```bash
213trl sft --config sft_config.yaml --learning_rate 1.0e-5
214```
215
216## Distributed Training
217
218TRL integrates with Accelerate for multi-GPU and multi-node training.
219
220**Multi-GPU training:**
221
222```bash
223trl sft \
224 --config sft_config.yaml \
225 --num_processes 4
226```
227
228**Use predefined Accelerate configs:**
229
230TRL provides predefined configs: single_gpu, multi_gpu, fsdp1, fsdp2, zero1, zero2, zero3
231
232```bash
233trl sft \
234 --config sft_config.yaml \
235 --accelerate_config zero2
236```
237
238**Custom Accelerate config:**
239
240```bash
241# Generate custom config
242accelerate config
243
244# Use custom config
245trl sft --config sft_config.yaml --config_file ~/.cache/huggingface/accelerate/default_config.yaml
246```
247
248**Fully Sharded Data Parallel (FSDP):**
249
250```bash
251trl sft --config sft_config.yaml --accelerate_config fsdp2
252```
253
254**DeepSpeed ZeRO:**
255
256```bash
257trl sft --config sft_config.yaml --accelerate_config zero3
258```
259
260## Troubleshooting
261
262### CUDA Out of Memory
263
264- Reduce --per_device_train_batch_size and increase --gradient_accumulation_steps
265- Enable --use_peft for LoRA training
266- Use --gradient_checkpointing to save memory
267- Try smaller model or longer sequence truncation
268
269### Dataset Loading Issues
270
271- Verify dataset exists: check Hugging Face Hub or local path
272- Check dataset format matches expected columns
273- Use --dataset_config for multi-config datasets
274- Inspect dataset: from datasets import load_dataset; ds = load_dataset(name)
275
276### Model Loading Issues
277
278- Verify model exists on Hugging Face Hub
279- Check if gated model requires authentication: hf auth login
280- For local models, provide absolute path
281- Ensure sufficient disk space and memory
282
283### Slow Training
284
285- Enable dataset --packing for short sequences
286- Use larger --per_device_train_batch_size if memory allows
287- Enable --tf32 for faster computation on Ampere GPUs
288- Use --bf16 on supported hardware
289- Consider multi-GPU training with --num_processes
290
291### Generation Issues (GRPO/RLOO)
292
293- Check prompt format in dataset
294- Adjust --temperature and --top_p for generation
295- Verify the reward function (for GRPO/RLOO)
296
297## Additional Resources
298
299- **Documentation**: https://huggingface.co/docs/trl
300- **GitHub**: https://github.com/huggingface/trl
301- **Examples**: https://github.com/huggingface/trl/tree/main/examples
302
303## Best Practices
304
3051. **Start with SFT**: Always fine-tune base models with SFT before preference alignment
3062. **Use LoRA for efficiency**: Enable --use_peft for faster training and lower memory
3073. **Monitor training**: Use --report_to trackio (or --report_to wandb or --report_to tensorboard) for tracking
3084. **Save checkpoints**: TRL automatically saves checkpoints in --output_dir
3095. **Test on small datasets first**: Verify pipeline works before full training
3106. **Use configuration files**: Create YAML configs for reproducibility
3117. **Leverage Accelerate**: Use multi-GPU training for faster iteration
312
313When helping users with TRL:
314- Always check which training method is appropriate for their use case
315- Verify dataset format matches the expected schema
316- Recommend starting with smaller models for testing
317- Suggest LoRA for resource-constrained environments
318- Point to specific documentation sections for advanced features
319
In the file
SKILL.md1,042 words
Files1
LicenceApache-2.0
Why you can read it

Nothing in a skill executes. The client loads the text and the model follows it, so a skill can be audited the way a runbook is — by reading it.

What it costs in context

Skills are not billed by the call. They are paid for in context: every token the instructions occupy is a token your code, your diff and your conversation cannot use. Here is what this one takes and when it takes it.

≈140
always loaded
The name and description, so the model knows the skill exists and when to reach for it.
2,060
on trigger
The instruction body, read only when the skill fires.
1.1%
of a 200k window
Ten skills this size would take about 11% of the window before you open a file.
050k100k150k200k context window

2.2k tokens, estimated from the bundle at four bytes to the token, held for the rest of the session once it triggers. Middling. Fine to keep on in a project where you use it weekly, worth unloading in one where you never do.

Servers bill, skills cost

A server charges by the month. A skill charges once per session, in context, and then keeps charging it for as long as the session lives.

Before and after

The same question, put to the same model twice: once as it comes, and once with these instructions loaded.

No worked example has been published for this skill yet.

Adoption
Installsnone yet
Ratingno reviews yet

The procedure it runs

The procedure has not been published here. It is in the skill’s own SKILL.md, which its author has not sent to the marketplace yet.

Prose, not code

These steps are written for a model to follow, not executed by a runtime. It can still be told to skip one, and it will say so when it does.

Servers it uses

None. This skill calls no MCP servers at all.

Everything it needs is in the instructions, so it works in a project with nothing connected — the model reads the file and changes how it works with what it can already reach.

It writes no files and reaches no network. All it changes is how the model reasons and writes.

What it asks for
Writes filesno
Network accessno

Read from the allowed-tools line of this skill’s own SKILL.md. A skill grants no permissions of its own — it can only ask for tools your client already has.

What it will not do

Every skill is narrow, and the useful ones say where they stop. These are the jobs this one is the wrong tool for.

What this skill is not for has not been published here. Nothing is implied by that: it is a section the author has not filled in.

What is in the bundle

1 file, 8.8 kB on disk. A bundle is text throughout: the instructions the model reads, plus the templates it fills in.

  • SKILL.md8.8 kB
What is not in it

No dependencies and nothing executable: a skill is text the agent reads, so the bundle is 1 file you can review in full before installing. The Apache-2.0 licence covers the templates and examples as well as the instructions.

Install

Installing copies the bundle into your project. Nothing runs at install time — the files sit on disk until the model reads them.

$35 once
TRL Training · Apache-2.0 · huggingface
one-time
Price$35 once
LicenceApache-2.0 — the author’s, unchanged by this purchase
Paid throughStripe, once, on the card you add at the checkout
Keeps workingfor good — the files are yours once they are on disk
Updatesevery release of 1.x through this account

You can read the whole bundle before paying — the SKILL.md above is the product, not a preview of it. What the money buys is the delivery: the folder packaged and handed to your machine by key, every update its author ships, and our support if it does not do what this listing says. The terms of use are Apache-2.0, set by the author and unchanged by buying it here.

Sign in to buyCreate an account

Payment runs through Stripe, on a page like this one rather than a redirect. Once there is an account it joins the same mcprush invoice as everything else you run, so there is never a second card to enter.

Which clients pick it up on their own

A skill is a folder of text. A client with a skills folder reads it without being told; everywhere else the same text works, it is just handed to the model rather than found.

Claude Code.claude/skills/
Claude Desktop
ChatGPT
Cursor.cursor/skills/
VS Code.github/skills/
Codex CLI.agents/skills/
Gemini CLI.gemini/skills/
Grok.grok/skills/
Zed.agents/skills/
Windsurf.windsurf/skills/
Agent SDK.claude/skills/
HTTP / API
This release
Version1.0.0
Publishedno release date on file
Price$35
Referencehuggingface/trl-training

Versions

v1.0.0 is what is on the shelf; no release here carries a date. Instructions change more often than APIs do — a skill can be rewritten entirely without anything it depends on moving.

v1.0.0
  • No earlier releases have been published to the marketplace.
Pinning

Put huggingface/trl-training@1.0.0 in the install command to hold this exact version. Without the suffix you get whatever is current the day you install, and nothing moves under you afterwards.

Reviews

—no reviews yet · no installs yet

Nobody has reviewed this skill yet. The rating is the mean of the reviews written here, so there is none until somebody writes the first.

Who can post

Only accounts that have had the skill installed for fourteen days, so a review is written after living with it rather than after reading it. Publishers may reply once.

Who wrote it

HU
huggingface

Publishes on mcprush.

0 servers listed5 skills listednot claimed
Profile
Publisher
Servers0