TAU-HOME.COM
LOADING

DNeo: A New Parallel Draft Model Architecture Achieving 1.3x Faster LLM Inference via MoFA

Proposed by Jameson Sandler and research colleagues, DNeo introduces Mixture-of-Features Attention (MoFA) and a working memory mechanism, achieving 1.3x faster

tau · November 22, 2026

#DNeo #LLM #InferenceSpeed #ArtificialIntelligence #arXiv

DNeo: A New Parallel Draft Model Architecture Achieving 1.3x Faster LLM Inference via MoFA

A new parallel draft model architecture named 'DNeo' has been introduced to the academic community, designed to drastically accelerate the inference speed of Large Language Models (LLMs). Authored by Jameson Sandler and seven other researchers, the study was submitted to arXiv on November 16, 2026, presenting an advanced feature encoding method and inference acceleration techniques that overcome longstanding limitations in prior models.

DNeo architectural concept overview

Image source: https://arxiv.org/abs/2611.21925

Image Credit: arXiv

This research builds upon 'speculative-diffusion decoding', a technique widely used to mitigate the bottleneck where large language models generate tokens sequentially one by one. While traditional methods rely on a small draft model generating multiple candidate tokens in a single forward pass for simultaneous parallel verification by a larger target model, previous architectures suffered from structural drawbacks in how they utilized target model features.

Limitations of Previous Parallel Draft Models and Feature Encoding

Recent parallel draft architectures like DFlash and DSpark have improved draft quality by conditioning generation on rich features computed by the target model through a mechanism known as feature encoding. However, the researchers noted that standard feature encoding mechanisms suffer from three critical limitations:

First, they are lossy, applying encoding projections that are destructive to the underlying target feature space through a discovered phenomenon known as residual mixing. Second, they are memory-less, with no record or tracking of previously processed proposals. Third, they utilize features sampled uniformly across target layers, thereby omitting crucial hidden states that significantly impact performance.

The Solution: MoFA and Working Memory Mechanisms

To address these issues, the researchers proposed Mixture-of-Features Attention (MoFA), a lightweight feature-encoding approach equipped with a dedicated working memory that encodes and retains previously rejected proposals.

By combining MoFA, rejection memory, and a suite of further enhancements—including 2-tap causal convolutions, prefix-only cross-attention, and non-uniform target layer selection—the final integrated architecture, DNeo, successfully overcomes the bottlenecks of earlier systems.

1.3x Faster Generation Speed and Practical Implications

According to the study's findings, DNeo achieves 1.3x faster generation speed compared to the previous state-of-the-art (SoTA) DFlash-2 model, without negatively impacting the drafter footprint or generation quality.

This performance boost offers direct utility for reducing latency and cutting operational server costs when deploying large language models in production environments. By accelerating generation purely through architectural efficiency rather than increasing draft model size or compromising output quality, DNeo marks a notable milestone for developers and engineers building high-performance AI infrastructure. Nevertheless, as this research currently exists as a pre-print paper submitted to arXiv, broad adaptation across various open-source model weights and production pipelines awaits further empirical validation and community testing.

Sources

Impact-Site-Verification: 9b3ebd10-6c45-44cf-a014-5b8c19b7b285