Running LFM3-1.7B on 6GB VRAM: All 3 quantizations on GPU, Q4_K_M and QAD version Q4_0 are approx. 152 tok/s
Results of measuring three GGUFs of LFM3-1.7B (Q4_K_M, QAD version Q4_0, Q8_0) on a GPU with 6GB of VRAM used alongside screen display. Using llama.cpp (b12005) llama-server, with thinking off and context setting 4096, all three loaded all layers onto the GPU. Generation was approx. 152 tok/s for both Q4_K_M and QAD version Q4_0, and approx. 107 tok/s for Q8_0.
Testing was conducted on three GGUF versions of LFM3-1.7B (Q4_K_M, QAD version Q4_0, and Q8_0) using a GPU with 6GB of VRAM that was also being used for screen display. The test used llama.cpp (b12005) llama-server with thinking turned off and a context setting of 4096. As a result, all three versions loaded all layers onto the GPU.
Regarding generation speeds, Q4_K_M and the QAD version Q4_0 reached approximately 152 tok/s, while Q8_0 reached approximately 107 tok/s. Detailed results, including tables for each issue, VRAM transition graphs, and how all questions were answered incorrectly, are available in the detailed version of the blog.
This item is a summary. The full, checkable version is at the original source:
Read the original at Zenn (LLM) ↗