TAU-HOME.COM
LOADING

Recursive Releases 9GB 1.8-Bit MoE Model 'Boole-32B-A2.5B-Preview', Reaching 565 tok/s on a Single RTX 5090

The Recursive team has released Boole-32B-A2.5B-Preview on r/LocalLLaMA, applying 1.8-bit mixed-precision quantization and expert pruning to fit a 32B MoE model

tau · November 14, 2026

#Quantization #LocalLLM #MoE #RTX5090 #HuggingFace

Recursive Releases 9GB 1.8-Bit MoE Model 'Boole-32B-A2.5B-Preview', Reaching 565 tok/s on a Single RTX 5090

On November 13, 2026, the Recursive team unveiled 'Boole-32B-A2.5B-Preview', an early 1.8-bit quantized preview build based on its flint model, via the local language model community r/LocalLLaMA on Reddit. By combining mixed-precision quantization, expert pruning, and custom compute kernels, the team compressed the model footprint down to 9GB while achieving a generation speed of 565 tokens per second (565 tok/s) on a single NVIDIA RTX 5090 graphics card.

Conceptual technical illustration of a 9GB 1.8-bit Mixture of Experts language model streaming fast token inference on a single desktop GPU

Image source: https://agihunt.info/p/1a1d43ffd93ddd37a00aa3049db

For enthusiasts and developers running large language models locally, video memory (VRAM) capacity and token generation throughput remain the primary bottlenecks. Following its earlier release of the 1-bit 'Boole-1.7B' model, the Recursive team developed this preview release in response to community demand for a larger, faster Mixture of Experts architecture that still fits comfortably on consumer hardware.

Combining 1.8-Bit Mixed Quantization and Expert Pruning

As its naming convention indicates, Boole-32B-A2.5B-Preview uses a Mixture of Experts (MoE) design with 32 billion total parameters, activating roughly 2.5 billion parameters per token. Instead of running every weight across an entire dense network for each word generated, an MoE model routes inputs only to the most relevant specialized sub-networks ("experts"), preserving broad knowledge capacity while keeping active compute low.

To push that efficiency further, the Recursive team stacked three optimization techniques:

  • Mixed-Precision Quantization: Rather than storing weights in standard 16-bit floating-point format (BF16), the model compresses parameters to an average of 1.8 bits while allocating varying bit widths across layers based on sensitivity.
  • Expert Pruning: Lower-contribution expert pathways inside the MoE architecture are trimmed away, cutting redundant memory overhead.
  • Custom Kernels: Specialized low-level GPU execution kernels unpack the sub-2-bit weights and run matrix operations without stalling memory bandwidth.

According to the release summary, this combination makes the model 9 times smaller than its BF16 counterpart and 2 times faster than conventional 4-bit integer (INT4) quantization.

Single RTX 5090 Throughput and Deployment Package

In reported testing on a single NVIDIA RTX 5090 GPU, Boole-32B-A2.5B-Preview reached an inference throughput of 565 tok/s. With a total footprint of just 9GB, the entire model fits easily inside the VRAM of a flagship consumer GPU with substantial room left over for context caches or concurrent workloads.

The release is packaged specifically for local inference workflows. Model weights have been uploaded to Hugging Face in GGUF—a widely used file format for local CPU/GPU model execution—alongside a companion demo repository and a custom fork of llama.cpp.

From an editorial standpoint, pairing the 9GB GGUF weights with a dedicated llama.cpp fork highlights a practical reality of sub-2-bit inference: extreme quantization only translates into real-world speed when the runtime engine natively understands the custom bit-packing layout. Shipping the modified llama.cpp fork directly alongside the weights gives local users an immediate path to reproduce the 1.8-bit execution pipeline rather than waiting for upstream runtime support.

Preview Constraints and Unverified Accuracy Trade-Offs

For local LLM users, shrinking a 32B-class MoE model to 9GB opens up experimentation on mid-to-high-tier consumer GPUs where unquantized 32B models would otherwise overflow memory. Even so, several conditions and unverified boundaries apply before adopting the build for everyday tasks.

First, Boole-32B-A2.5B-Preview is explicitly an early preview release. Running it requires setting up Recursive's custom llama.cpp fork rather than relying on an unmodified upstream llama.cpp installation.

Second, applying 1.8-bit ultra-low-precision quantization alongside expert pruning typically introduces accuracy trade-offs across reasoning, coding, and instruction-following benchmarks. The available release details establish the 9GB footprint, the 9x size reduction versus BF16, the 2x speedup over INT4, and the 565 tok/s throughput on a single RTX 5090, but formal task-by-task accuracy benchmarks and performance metrics on lower-VRAM GPUs remain to be independently verified.

Sources

Impact-Site-Verification: 9b3ebd10-6c45-44cf-a014-5b8c19b7b285