← Friday, October 2, 2026Models & trainingPaper

Distilled T5Gemma-2 embeddings create a more connected diffusion space

huggingface.co by Zekai Zhang et al. Editor score 7/10

The work distills teacher token probabilities into an encoder that pulls plausible alternative-word embeddings closer together, aiming to reduce invalid outputs from imperfect sampling. Embedding geometry is a practical design choice for teams building continuous diffusion language models.

What: Scaling embeddings from T5 to T5Gemma-1 to T5Gemma-2 improves continuous diffusion language model generation, but raw T5Gemma-2 embeddings remain too discriminative.
Method: A student encoder learns the teacher's decoded probabilities as soft labels, pulling plausible alternative-word embeddings closer together.
Why it matters: The distilled space improves diffusibility; a medium-sized DLM reaches Gen. PPL 17.8 at real-text entropy on OpenWebText, outperforming GPT-2-M.

The student encoder learns from probabilities decoded by the T5Gemma-2 teacher. Sketched from the article's own text.

Read on huggingface.co → Code · 2 stars

Discussion

Discussion is not switched on yet. Use Note above to keep your own thoughts on this story.

Headline, summary, brief and diagram are written by a model from the source's own text; the link, byline and numbers come from the source. Read the original before relying on a claim.