← Friday, October 2, 2026Performance & OSDeep dive

TLX Jagged Flash Attention beats FA4 by 13% forward and 50% backward on Blackwell

pytorch.org 15 min read Editor score 8/10

The TLX kernel uses about 3.2K lines of Triton-level code and outperforms the May 2026 FA4 version by roughly 13% forward and 50% backward on GEM’s jagged shapes. It shows how hardware-aware Triton extensions can improve performance without a large hand-written CUDA kernel.

What

TLX adds explicit, hardware-aware controls to Triton for the Jagged Flash Attention kernel on Blackwell.

How

It uses explicit memory allocation, warp specialization, barriers, and asynchronous TMA and MMA to pipeline attention over packed Q/K/V and offsets.

Why it matters

On GEM’s jagged shapes, the TLX kernel beats FA4 by about 13% forward and 50% backward, while remaining easier to extend.

Read on pytorch.org →

Headline, summary, brief and diagram are written by a model from the source's own text; the link, byline and numbers come from the source. Read the original before relying on a claim.