vLLM v0.34.0rc1 released

Local & Open Source✍ TextPublished: Source: vLLM

#Ollama

Key point

vLLM v0.34.0rc1, a high-throughput and memory-efficient inference and serving engine for LLMs, has been released.

vLLM v0.34.0rc1, a high-throughput and memory-efficient inference and serving engine for LLMs, has been released.

This release includes bugfixes for Spec Decode and XPU. Specifically, it addresses greedy_argmax to treat NaN as -inf.

This item is a summary. The full, checkable version is at the original source:

Read the original at vLLM ↗