Qwen3.8-Flash-Next Preview Runs on a Single 32GB GPU at Over 80 tok/s

Image source: https://unsloth.ai/docs/models/qwen3.8-next

On October 3, 2026, AgentionAI released a specialized quantization preview designed to run Qwen's next-generation MoE architecture model, Qwen3.8-Flash-Next, on a single 32GB GPU. In initial benchmarks on a single Radeon R9700 PRO with 32GB VRAM, the build reportedly achieved generation speeds exceeding 80 tokens per second (tok/s).

Qwen3.8-Flash-Next combines a 125B-parameter main model (activating 6B parameters per token) with a 51B-parameter N-gram embedding table. Due to its massive footprint, running the architecture on consumer or workstation-grade GPUs previously posed severe memory bottlenecks. AgentionAI's custom quantization and memory layout restructure the model's weight placement to deliver usable local inference speeds within a 32GB VRAM ceiling.

1. N-Gram Host Offloading and Rotor-Based MoE Quantization

Rather than applying uniform low-bit compression across the entire model, AgentionAI's 32GB quantization preview employs an asymmetric memory and quantization strategy tailored to each architectural component.

• Host RAM Offloading for N-Gram Embeddings: The 51B N-gram embedding table, which accounts for a substantial portion of total model size, is offloaded to host system memory rather than occupying GPU VRAM.
• Rotor-Based Low-Bit Quantization for Routed Experts: A specialized low-bit rotor quantization format is applied to the active routed expert layers, drastically reducing the memory footprint while mitigating degradation to core reasoning capabilities.
• Optimized VRAM Footprint: Under a 64K context window, the model reportedly occupies approximately 28.8 GiB of GPU VRAM, allowing it to operate comfortably within a 32GB card without triggering out-of-memory errors.

2. Qwen3.8-Flash-Next Architecture and Inference Throughput

Qwen3.8-Flash-Next serves as an architectural preview from Alibaba's Qwen team, previewing the core design principles scheduled for the upcoming Qwen4 series.

• Hybrid Attention and Sparse Indexing: Pairing Gated DeltaNet (GDN) with Qwen Sparse Attention (QSA) prevents indexing latency from exploding across long context windows.
• Selective MoE Activation: Activating only 6B parameters per token out of the 125B total allows the model to maintain low computational latency while delivering flagship-tier reasoning and coding performance.
• Single-GPU 80+ tok/s Throughput: AgentionAI reports sustained generation speeds above 80 tok/s on its 32GB GPU testbed, addressing the throughput collapse historically associated with running massive MoE models on single-card local hardware.

3. Hardware Variances in Preview and Upcoming Roadmap

AgentionAI emphasized that the current release is an early preview aimed at broader community testing and hardware validation.

• Hardware and Environment Dependencies: Reported speeds of 80+ tok/s represent specific test configurations rather than a guaranteed baseline across all systems. Performance can fluctuate based on GPU architecture (NVIDIA/AMD), driver revisions, context length, inference backends, and speculative decoding (MTP) configurations.
• Multi-Tier Memory Roadmap: Following the initial 32GB preview, AgentionAI plans to roll out dedicated optimization builds for 48GB and 64GB VRAM environments alongside further inference engine enhancements.

Sources

• AgentionAI Model Page: Qwen3.8-Flash-Next
• Oluwaphilemon1 X Post (@Oluwaphilemon1): Qwen3.8-Flash-Next 32GB GPU 80+ tok/s Preview Announcement