Research on increasing weight bits during uncertain decoding steps
A study on quantization simulation was announced, which re-decodes at 8-bit only when output uncertainty is high using Qwen3-0.6B.
A study regarding weight bit allocation in small local models was published on Hugging Face Daily Papers. The method involves storing 8-bit weights and decoding with their top 4 bits, but redoing a step at 8 bits only when the low-precision output distribution appears uncertain, such as when entropy is above a threshold or there is a small top-two margin.
Using Qwen3-0.6B, the researchers simulated quantization with the GSM8K benchmark. Cost was measured in weight bytes read per token. In 36,724 replayed steps, the entropy of the low-pass ranked argmax flips with an AUC of 0.93. At a 15% escalation budget, the method caught 78% of flips and 94% via the margin gate, compared to the 15% expected for a random gate of an equal rate.
This item is a summary. The full, checkable version is at the original source:
Read the original at Hugging Face Papers ↗