FLAMIA announced for fast and causally safe training on Blackwell
FLAMIA has emerged as a family of attention kernels that reduce Transformer training costs and leverage NVIDIA Blackwell GPUs, supporting MXFP8 and mixed NVFP4/FP8 paths.
In Transformer training, attention remains a substantial computational cost, particularly at long context lengths. While NVIDIA Blackwell GPUs provide high-throughput low-precision matrix multiplication, exploiting this in attention requires addressing quantization overhead, causal information leakage through shared scales, and numerical inconsistencies between the forward and backward passes.
In response, FLAMIA was presented. It is a family of attention kernels for Blackwell with MXFP8 and mixed NVFP4/FP8 training paths. The paper received 12 upvotes on Hugging Face Daily Papers.
This item is a summary. The full, checkable version is at the original source:
Read the original at Hugging Face Papers ↗