yqi19/hf-lerobot
Updated
yu
yqi19
AI & ML interests
None yet
Recent Activity
updated
a model
9 days ago
yqi19/hf-lerobot
published
a model
9 days ago
yqi19/hf-lerobot
liked
a dataset
15 days ago
daixianjie/rlbench_lerobot_train
Organizations
None yet
updated
a
model
9 days ago
published
a
model
9 days ago
liked
a
dataset
15 days ago
updated
a
model
26 days ago
published
a
model
26 days ago
upvoted
an
article
about 1 month ago
Article
Ļ0 and Ļ0-FAST: Vision-Language-Action Models for General Robot Control
- +2
ā¢
187
reacted to
anakin87's
post with š§
2 months ago
Post
2881
LLMs can leak their post-training data (RL included) š§
New interesting paper on this topic from Google DeepMind: Extracting alignment data in open models (2510.18554)
It's known that Language Models memorize data that can be extracted via prompting.
In this paper, the authors investigate this aspect:
- using open models, where prompting can be fully customized by the user, including special tokens.
- focusing on open-source models like Olmo, where full training data is available.
š¤ How do they extract data?
During post-training (like SFT), new tokens such as <|user|> are introduced.
The authors hypothesize prompting the model with these tokens can make it output its alignment data (remember Magpie?).
For example, for SFT, their extraction prompt is <|endoftext|><|user|>.
š Evaluating memorization
The authors compare each sampled example with the original data using vector search with embedding similarity.
They find that many outputs are semantically very similar to the original data, even if the exact words differ.
Traditional string-matching algorithms underestimate memorization by 10x.
š What about RL?
Surprisingly, the same technique works to extract data from Reinforcement Learning (PPO/GRPO) phases.
This is counter-intuitive because the RL objective is not designed to increase sequence likelihoods (unlike SFT).
Practical limitation: in this case, extraction relies on using the initial part of the training prompt, which is not generally public.
š Are the extracted data effective for post-training?
Both in SFT and RL, the extracted data can be used to fine-tune models to similar performance to the originals.
The authors suggest that model distillation, where a stronger model is used to drive the training of a weaker one, may be a form of indirect training on the original dataset.
New interesting paper on this topic from Google DeepMind: Extracting alignment data in open models (2510.18554)
It's known that Language Models memorize data that can be extracted via prompting.
In this paper, the authors investigate this aspect:
- using open models, where prompting can be fully customized by the user, including special tokens.
- focusing on open-source models like Olmo, where full training data is available.
š¤ How do they extract data?
During post-training (like SFT), new tokens such as <|user|> are introduced.
The authors hypothesize prompting the model with these tokens can make it output its alignment data (remember Magpie?).
For example, for SFT, their extraction prompt is <|endoftext|><|user|>.
š Evaluating memorization
The authors compare each sampled example with the original data using vector search with embedding similarity.
They find that many outputs are semantically very similar to the original data, even if the exact words differ.
Traditional string-matching algorithms underestimate memorization by 10x.
š What about RL?
Surprisingly, the same technique works to extract data from Reinforcement Learning (PPO/GRPO) phases.
This is counter-intuitive because the RL objective is not designed to increase sequence likelihoods (unlike SFT).
Practical limitation: in this case, extraction relies on using the initial part of the training prompt, which is not generally public.
š Are the extracted data effective for post-training?
Both in SFT and RL, the extracted data can be used to fine-tune models to similar performance to the originals.
The authors suggest that model distillation, where a stronger model is used to drive the training of a weaker one, may be a form of indirect training on the original dataset.
authored
a
paper
3 months ago
commented
a
paper
3 months ago
upvoted
2
papers
3 months ago
upvoted
a
collection
3 months ago
upvoted
5
papers
3 months ago
RLinf-VLA: A Unified and Efficient Framework for VLA+RL Training
Paper
ā¢
2510.06710
ā¢
Published
ā¢
40
VeriGUI: Verifiable Long-Chain GUI Dataset
Paper
ā¢
2508.04026
ā¢
Published
ā¢
161
SSRL: Self-Search Reinforcement Learning
Paper
ā¢
2508.10874
ā¢
Published
ā¢
97
The Landscape of Agentic Reinforcement Learning for LLMs: A Survey
Paper
ā¢
2509.02547
ā¢
Published
ā¢
228
ThinkGrasp: A Vision-Language System for Strategic Part Grasping in Clutter
Paper
ā¢
2407.11298
ā¢
Published
ā¢
6
commented
a
paper
3 months ago
upvoted
2
papers
3 months ago