vLLM v0.34.0rc1 released
Key point
vLLM v0.34.0rc1, a high-throughput and memory-efficient inference and serving engine for LLMs, has been released.
vLLM v0.34.0rc1, a high-throughput and memory-efficient inference and serving engine for LLMs, has been released.
This release includes bugfixes for Spec Decode and XPU. Specifically, it addresses greedy_argmax to treat NaN as -inf.
This item is a summary. The full, checkable version is at the original source:
Read the original at vLLM ↗