Hugging Face Transformers v5.23.0 Released with New Model Additions including Inkling2

Hugging Face✍ TextPublished: Source: Transformers

#Hugging Face

Key point

Hugging Face released Transformers v5.23.0, which includes new model additions such as Inkling2 Small, a multimodal model from Thinking Machines Lab, and the JEV decision head.

Hugging Face released Transformers v5.23.0. New model additions include Inkling2 Small, a multimodal model from Thinking Machines Lab that accepts text and image inputs to generate text outputs. It is intended for use in English and other languages, as well as across multiple coding languages. Its language backbone features 48 mixture-of-experts decoder layers, with each layer routing tokens to 8 out of 384 experts, resulting in 594B total and 17B active parameters. It retains the Inkling decoder and adds block attention residuals, gated attention with a learned sink, and dynamic key/value convolutions. Images are encoded by a Perception Encoder-based vision transformer and projected into the language model embeddings, with one embedding per 40x40 pixel patch.

Additionally, JEV, a decision head, has been added. It scores single-token labels with a linear layer at the last token of the prompt, soft-capped like the logits of a language model head. JevForDecision is the model of the checkpoint and its head, reading the head files head.safetensors and judge_config.json shipped with the weights. The same head is also used by GEV and reads the 255 answer codes of pplx-decider and d3, and it can be run through the experimental decision pipeline or directly with decide.

This item is a summary. The full, checkable version is at the original source:

Read the original at Transformers ↗