Research paper on 'Muir', an improved method for Linear Attention, released

ResearchPublished: Source: Hugging Face Papers

#Hugging Face

Key point

Muir is proposed, using sensitivity-allocated gating and matched-filter readout to address Linear Attention challenges.

Linear Attention compresses context into a fixed-size key-value state to enable linear-time training and constant-memory inference. Gated DeltaNet-2 (GDN-2) generalizes this by decaying the state and erasing its prediction for an incoming key before writing a new value, utilizing channel-wise decay and decoupled, full-rank erase and write gates.

The research shows that state updates are far less sensitive to erasure than to writes. Because key activation and unit normalization concentrate key energy onto a sparse subset of channels, while value writes scale all channels, the erase gate can be parameterized at a lower rank. This paper received 4 upvotes on Hugging Face Daily Papers.

This item is a summary. The full, checkable version is at the original source:

Read the original at Hugging Face Papers ↗