← Monday, October 5, 2026Models & trainingPaper

Distillation study finds KL direction and learning rate outweigh rollout policy

huggingface.co by Julianna Piskorz et al. Editor score 8/10

In controlled distillation experiments across Llama 3 and Qwen2.5, KL direction more clearly affects performance and output coverage, while learning rate governs forgetting and update sparsity. These results challenge assumptions about when on-policy training is necessary.

Read on huggingface.co →

Headline, summary, brief and diagram are written by a model from the source's own text; the link, byline and numbers come from the source. Read the original before relying on a claim.