DiffusionGemma as Jev: Fast Structured Decisions via Direct Slot Probabilities

Run DiffusionGemma 26B-A4B as a Jev-compatible local decision API by reading slot probabilities directly on fixed canvases with patched vLLM instead of generati

tau · September 23, 2026

#DiffusionGemma #Jev #vLLM #StructuredOutput #NVFP4 #AIOptimization

DiffusionGemma as Jev: Fast Structured Decisions via Direct Slot Probabilities

AI and high-performance computing engineer David Hendrickson (@TeksEdge) highlighted an open-source approach from Djev-Spark that enables Google's DiffusionGemma 26B-A4B text diffusion model to function as a drop-in local alternative to TypeSafe's Jev decision engine. Instead of relying on conventional autoregressive Large Language Models (LLMs) that decode structured outputs token by token, this method leverages the unique spatial canvas of diffusion architectures to skip JSON generation entirely and read decision probabilities directly from designated slots.

High-level architectural diagram of DiffusionGemma 26B reading decision slot probabilities directly on a prefilled canvas for Jev-compatible structured outputs

Image source: David Hendrickson (@TeksEdge)

Refusing JSON Token Generation: Prefilled Canvases and Direct Slot Reads

Obtaining structured JSON output from standard language models typically requires sequential autoregressive decoding, where every structural key, bracket, and delimiter must be generated one token at a time. This process introduces significant latency and leaves room for syntax failures. Built on Google's Gemma 4 architecture, DiffusionGemma is an open-weights Mixture-of-Experts (MoE) text diffusion model that iteratively refines an entire token canvas across denoising steps, opening an alternative route for structured tasks.

As AI researcher Sakura Yuki (@sakurayukiai) observed in the technical discussion, the breakthrough here is not generating JSON faster, but refusing to generate JSON at all. Djev-Spark pre-builds the answer structure directly onto the canvas (diffusion_seed_canvas), leaving only the decision slots open for the model to resolve. Building on unmerged extensions proposed in vLLM PR #57250 (diffusion_seed_canvas and diffusion_read_only), the runtime performs a single-step forward pass to read the log probabilities and entropy of allowed label tokens directly from the model's logits, serializing the final structured result afterward.

NVFP4 Local Execution and Jev-Compatible System One Endpoints

Djev-Spark runs locally using NVFP4 quantization paired with a patched vLLM structured-read engine, currently requiring high-end NVIDIA hardware such as GB10 GPUs to support NVFP4 compute. The stack exposes a wire-compatible Jev endpoint (POST /v1/systemone and POST /v1/request), allowing existing client applications built with TypeSafe SDKs to execute typed Noul (boolean), Choice (categorical selection), and Score evaluations without any client-side code changes.

This approach contrasts sharply with prompt-based compatibility layers such as LocalJev. While LocalJev runs on Bun with an oMLX backend and requests JSON probability vectors via standard chat completions, it relies on self-reported model probabilities rather than raw output logits. In contrast, OpenJev and Djev extract genuine log probabilities and entropy straight from the diffusion canvas. On the JevBench v1.2.8 benchmark, djev reached a score of 74.3, ranking third behind Jev 1.13.0 (75.4) and SemIf (74.7). Historical benchmarks on an earlier quantized text configuration recorded a median latency of 76.87 ms (p50) and 86.40 ms (p95) over 1,000 requests, though documentation notes that this benchmark does not reflect the current BF16 release and does not apply to multimodal image queries.

Cache Reuse Dynamics and Practical Production Caveats

Production teams evaluating Djev must account for the substantial divergence in inference latency caused by prompt cache reuse. In benchmark measurements shared by Mathias Heinke (@ares_mheinke) over a 110,707-token document context, cold start inference required 105 seconds, whereas a warm run with cached prefix states took just 0.44 seconds.

Heinke emphasized that for real-world deployments processing incoming, non-repetitive documents, the cold-start figure is the primary metric of interest, advising practitioners to deliberately flush the cache during benchmarking to avoid merely measuring prefix cache reuse. Meanwhile, other community members, such as 0xBakeer, pointed to alternative implementations like Laya that claim even faster speeds. Deploying the Djev-Spark stack requires applying custom Python patches to nine core vLLM runtime files (such as those maintained in siliconflow/dgemma-jev-patch) and running on GPUs with sufficient memory bandwidth and NVFP4 support. Additionally, when decision entropy exceeds a set threshold, runtime logic can trigger extra sampling passes to verify consensus and guarantee decision reliability.

Original source