Research paper on 'Muir', an improved method for Linear Attention, released
Muir is proposed, using sensitivity-allocated gating and matched-filter readout to address Linear Attention challenges.
Linear Attention compresses context into a fixed-size key-value state to enable linear-time training and constant-memory inference. Gated DeltaNet-2 (GDN-2) generalizes this by decaying the state and erasing its prediction for an incoming key before writing a new value, utilizing channel-wise decay and decoupled, full-rank erase and write gates.
The research shows that state updates are far less sensitive to erasure than to writes. Because key activation and unit normalization concentrate key energy onto a sparse subset of channels, while value writes scale all channels, the erase gate can be parameterized at a lower rank. This paper received 4 upvotes on Hugging Face Daily Papers.
This item is a summary. The full, checkable version is at the original source:
Read the original at Hugging Face Papers ↗