Qwen3.5-4B Performance Benchmark on a 16GB RTX 5060 Ti: Comparing Llama.cpp, Ollama, and vLLM
A detailed performance benchmark evaluates the Qwen3.5-4B Q4_K_M model running on an NVIDIA GeForce RTX 5060 Ti 16GB GPU using Llama.cpp, Ollama, and vLLM. It h
Detailed cross-benchmarks have been published evaluating the small language model Qwen3.5-4B on an NVIDIA GeForce RTX 5060 Ti graphics card equipped with 16GB of video memory. The study provides an in-depth performance profile comparing model execution speed and resource utilization across three popular local inference tools: Llama.cpp, Ollama, and vLLM.

Image source: https://dev.to/elcances/qwen35-4b-on-a-16-gb-rtx-5060-ti-823-tokens-per-second-for-16-users-6i
As a compact model with 4 billion parameters, Qwen3.5-4B runs efficiently on mobile devices, edge computing nodes, and personal desktops, making it an attractive choice for practical everyday use. This analysis measures the performance limits and efficiency of each execution engine across various scenarios, ranging from short prompt-and-response tasks to long-context inputs stretching up to 260,000 tokens.
Single-User Execution Tool Comparison
When running the model locally for a single user, clear differences emerged in start-up times and token generation speeds depending on each tool's architecture and default configuration.
- Llama.cpp: Proves to be the most straightforward and efficient choice for a single user on a local machine. It recorded the fastest start-up time of 1.03 seconds and matched Ollama's peak generation speed of 114 tokens per second (tok/s) on short prompts. It requires no complex server setup.
- Ollama: Widely appreciated for its user-friendly interface and management features, Ollama required 4.7 seconds to start up and exhibited a slightly longer Time To First Token (TTFT) of 165 ms on short prompts compared to Llama.cpp and vLLM. However, its raw text generation speed matched Llama.cpp at 114 tok/s.
- vLLM: Engineered as a high-performance serving engine for multi-user environments, vLLM takes 48 seconds to initialize, making it heavier for single-user workflows, though it excels in concurrent handling and power efficiency.
Context Scaling and KV Cache Bottlenecks
The most critical insight from the benchmark centers on how performance degrades and memory consumption shifts as the context length increases.
At a short prompt length (512-token context), Llama.cpp generates text at 114 tok/s with a snappy 101 ms TTFT. However, expanding the context to 32K tokens drops generation speed to 88 tok/s, and reaching the model's ceiling at a full 262K-token context causes generation speed to plummet to 34 tok/s.
The primary bottleneck behind this slowdown is the KV Cache (Key-Value Cache) consuming vast amounts of GPU memory. Processing the massive 262K context occupies 12GB of the RTX 5060 Ti's 16GB memory pool exclusively for the KV cache. Consequently, the card spends substantial time (around 135 seconds) preprocessing the huge input prompt before text generation can even begin.
Multi-User Concurrency and Hardware Benchmark Setup
Unlike single-user tools, configuring a vLLM server demonstrated that a single budget-class GPU can effectively back a team or small product workload.
- vLLM Concurrent Serving: When handling 16 concurrent users simultaneously, the vLLM server delivered an aggregate throughput of 823 tok/s. Each individual user received roughly 51 tok/s while experiencing a stable time-to-first-token delay of about 1,210 ms. Furthermore, vLLM achieved the highest energy efficiency at 0.52 kWh per million generated tokens.
- Test Bench Hardware: All tests ran on a standardized Linux system featuring an Intel Core Ultra 7 265K CPU, an ASUS ROG MAXIMUS Z890 APEX motherboard, 64GB of DDR5-5600 RAM, and an MSI GeForce RTX 5060 Ti 16GB (GDDR7) GPU. Because every model executed on identical hardware, all performance variations stem strictly from the software stack and model behavior.
In summary, for an individual user on an RTX 5060 Ti, Llama.cpp stands out as the optimal pick due to its instant start-up and lack of server overhead. For shared environments where multiple users submit simultaneous requests, vLLM is the better fit. Nonetheless, operators must account for memory ceilings and heavy preprocessing delays when feeding ultra-long context documents near the model's limit.
Sources
- DEV Community (Ahmet Özel): Qwen3.5-4b on a 16 GB RTX 5060 Ti: 823 tokens per second for 16 users