Magnitude Open-Source LLM Inference Engine Released: Auto-Tuning Kernels Deliver Over 2x Faster Decode Than Ollama

On October 15, 2026, developer Tom Greenwald (@tomgreenwald) officially announced Magnitude, a free and open-source local large language model (LLM) inference engine designed to automatically tune compute kernels to the host machine's specific hardware environment. Without requiring new hardware, Magnitude searches for the optimal execution strategy on each chip, achieving an average of 2.19x faster token decode on Mac M4 Pro and 2.17x on DGX Spark compared to Ollama.

Image source: Tom Greenwald (@tomgreenwald via X)

Distributed as a standalone desktop application for Mac, Windows, and Linux, Magnitude supports Apple Silicon, NVIDIA GPUs, AMD GPUs, and standard CPUs. The engine addresses a fundamental limitation of legacy local runtimes—which rely on static, one-size-fits-all precompiled kernels—by introducing chip-level kernel exploration and a memory architecture tailored specifically for AI coding agents.

Automated Kernel Tuning and Speculative Decoding: Over 2x Faster Decode Than Ollama

Mainstream local model runners such as Ollama, LM Studio, and llama.cpp traditionally ship with one-size-fits-all compute kernels because manually profiling and hand-tuning every model variant across hundreds of discrete chips and silicon architectures is practically impossible.

Magnitude resolves this tradeoff by automatically profiling kernels when a user downloads a model, dedicating up to one minute on the user's actual silicon to explore execution strategies:

• Automated Math Partitioning: The engine tests how GPU threads team up, evaluates chunk sizes per compute thread, and optimizes memory prefetch distances directly on local silicon.
• Output Integrity: It preserves the mathematical precision of the model, discarding any configuration that alters model outputs and keeping only the fastest valid kernel combination.
• Speculative Decoding Pairing: Each model is automatically paired with an optimal lightweight drafter model to maximize sustained token decoding throughput.

Across benchmark tests evaluating 11 popular local open models, Magnitude reported notable speedups against standard runtimes:

• Mac M4 Pro (vs. Ollama with MLX backend): 1.32x faster prompt prefill, 2.19x faster token decode
• DGX Spark (vs. Ollama with llama.cpp backend): 1.34x faster prompt prefill, 2.17x faster token decode

Addressing comparisons with data center frameworks like vLLM and SGLang, the developer noted that while those engines target massive batched throughput across clustered GPUs, Magnitude is engineered specifically for single-stream token generation on personal machines, providing a higher performance ceiling for local workflows.

Agent-Centric Architecture: Quantized KV Cache, Prefix Sharing, and Desktop Ecosystem

Beyond basic conversational chat interfaces, Magnitude is architected specifically around the resource demands of modern AI coding agents.

The engine provides an OpenAI-compatible local API server out of the box, enabling one-click integration with popular coding agents including Pi, OpenCode, Hermes, and Codex. Development environments previously configured for Ollama can transition seamlessly by pointing their existing base URL configurations to Magnitude's local server.

To mitigate workstation memory saturation when orchestrating multiple agents, Magnitude introduces dedicated memory optimizations:

• Quantized KV Cache: Quantizing Key-Value cache representations cuts per-agent memory footprint by approximately 50%.
• Dynamic Memory Allocation: Context memory expands and contracts dynamically based on active session depth, keeping the host system responsive during multitasking.
• Prefix Cache Sharing: Multiple agent instances referencing identical system instructions or repository context share common prefix caches, allowing concurrent execution without performance bottlenecks.

Hardware Compatibility, Profiling Overhead, and Future Roadmap

Magnitude is available as a completely free and open-source software (FOSS) utility. The initial release notes outline key operational considerations alongside near-term architectural milestones:

• Initial Profiling Overhead: The automatic kernel tuning routine requires up to one minute of hardware exploration per model upon initial download.
• Hardware-Dependent Variances: Published benchmarks represent an average across 11 test models; actual performance gains will vary based on user hardware configurations, memory bandwidth, and model architectures.
• Legacy GPU and High-Capacity Constraints: Older NVIDIA Pascal architectures (sm_60, such as Tesla P100 cards) currently execute via the Vulkan backend, with backward-compatible CUDA builds being evaluated based on community issue reports. In addition, tight memory margins have been noted on architectures such as Gemma 26B-A4B, with investigations ongoing via GitHub.

The development roadmap outlines three major enhancements aimed at extracting greater throughput from local hardware:

• Expert Streaming from RAM and Disk: Enabling Mixture-of-Experts (MoE) architectures exceeding local GPU VRAM to stream dynamically across system RAM and high-speed storage.
• Unified Multi-Device Model Splitting: Distributing single-model computation simultaneously across all active GPUs and CPUs to eliminate idle compute resources.
• Attention Pruning: Skipping low-impact mathematical calculations during the attention phase to accelerate prompt prefill latency.

Magnitude desktop installers are available via the official project website and GitHub repository, with developer integration documentation published on its documentation portal.

Sources

• Tom Greenwald Official Announcement on X (@tomgreenwald): Magnitude Open-Source Inference Engine Announcement
• Magnitude Official Website: magnitude.dev
• Magnitude GitHub Repository: github.com/magnitudedev/magnitude
• Magnitude API Documentation: docs.magnitude.dev/api/overview