Local Inference Stack Runs Qwen3.8-27B at Up to 381 t/s on a Single RTX 3090

An open-source inference optimization stack (syv-ai/qwen38-27b-rtx3090) has demonstrated generating up to 381 tokens/sec with the Qwen3.8-27B model on a single NVIDIA GeForce RTX 3090 GPU (24GB VRAM), scaling up from an initial baseline of approximately 82 tokens/sec. Without upgrading GPUs or adding secondary hardware, systematic pipeline improvements across speculative decoding, prompt lookup, and caching achieved a roughly 4.6x speedup.

Image source: @Oluwaphilemon1 via X / syv-ai via GitHub

The project highlights that local LLM inference throughput depends heavily on pipeline architecture rather than raw memory bandwidth alone. By combining speculative decoding via DFlash2, input context lookup, and prefix caching, the stack bypasses traditional sequential generation bottlenecks on consumer-grade hardware.

Step-by-Step Optimization Benchmarks

According to the benchmark progression shared by the developer, generation throughput increased incrementally as each optimization layer was introduced:

• Initial baseline setup: ~82 tokens/sec
• Optimized Multi-Token Prediction (MTP): ~114 tokens/sec
• DFlash2 speculative decoding + context lookup: ~138 tokens/sec
• Optimized lookup verification: ~381 tokens/sec

The final configuration achieved roughly 4.6x the initial baseline generation speed. On a document reproduction and extraction benchmark involving approximately 25K tokens, an earlier DFlash2 setup delivered around 260 tokens/sec, while the optimized context lookup setup surged to roughly 382 tokens/sec.

Speculative Decoding with DFlash2 and 16-Token Context Lookup

Standard autoregressive language models generate tokens sequentially, one at a time, where each step requires a full forward pass dependent on previous outputs. This setup creates substantial latency and memory bandwidth pressure. The optimization stack resolves this with two interlocking techniques:

First, DFlash2 speculative decoding uses a lightweight draft mechanism to propose multiple candidate tokens simultaneously. The full 27B parameter model then verifies these proposals in parallel in a single forward pass, skipping multiple sequential decoding cycles whenever proposals are accepted.

Second, the system incorporates Context Lookup. When the target output reproduces or extracts text already present in the prompt, the system searches the input context to propose matching candidate sequences directly.

• Expanded proposal blocks: Instead of verifying typical 7-token proposals, the system proposes and verifies candidate blocks of up to 16 tokens.
• High acceptance rate: The developer reports acceptance rates of roughly 15 out of 16 proposed tokens per verification step on overlapping text, minimizing sequential autoregressive steps.

For workloads with strong prompt overlap, the primary model spends most of its computation verifying pre-matched text blocks rather than computing brand-new tokens from scratch.

40x TTFT Reduction via Prefix Caching and Workload Constraints

For multi-turn queries over large documents, Prefix Caching introduces another major efficiency boost.

When issuing a follow-up query against the same 25K-token document, the runtime reuses the KV cache computations from the shared prompt prefix. The reported Time to First Token (TTFT) dropped from 22.4 seconds to 0.56 seconds—an approximately 40x speedup for repeated-prefix queries.

However, real-world deployments must account for clear workload characteristics:

• Workload specificity: The headline 381 tokens/sec figure was measured on a document reproduction and extraction task (~25K tokens) where the generated output closely overlaps with prompt text.
• Conversational chat speeds: For standard open-ended conversational tasks requiring original reasoning or creative prose, throughput is reported at approximately 133 tokens/sec.
• Ideal use cases: Retrieval-augmented generation (RAG), long-document extraction and structured summarization, and code refactoring workflows where significant text is repurposed from the context.

The complete optimization stack and configuration files are available open-source on GitHub at syv-ai/qwen38-27b-rtx3090.

Sources

• GitHub Repository: syv-ai/qwen38-27b-rtx3090
• @Oluwaphilemon1 on X: Qwen3.8-27B Inference Optimization Breakdown