Decision Index 0.1 Benchmark Unveiled: Over 30 Decision Models Evaluated Across 130,000 Questions

On September 22, 2026, machine learning researcher Apolinário Passos (@multimodalart) at Hugging Face officially unveiled the "Decision Index 0.1" benchmark leaderboard, providing a rigorous head-to-head evaluation of TypeSafe's commercial decision model Jev alongside more than 30 open-weights decision models. Spanning 37 granular benchmarks across five core evaluation categories, the study subjected each model to 130,000 (130K) decisions to measure their real-world capabilities in tool calling, automated workflows, and agent routing.

Image source: apolinario (@multimodalart) / Hugging Face

Unlike conventional LLM benchmarks focused primarily on free-form generative text, Decision Index 0.1 is engineered specifically to evaluate specialized decision models tasked with structured function dispatching, model routing, and lightweight autonomous operations.

37 Benchmarks and 130,000 Decisions: Standardized on a Single RTX 6000 PRO

A central methodological highlight of Decision Index 0.1 is its strict hardware normalization and intentional anti-saturation benchmark design.

To maintain an uncompromised evaluation baseline, every competing system was executed on identical hardware: a single NVIDIA RTX 6000 PRO.

• Five Core Categories Across 37 Benchmarks: Evaluating knowledge, automation and tool use, comprehension, and creativity across five categories, each model resolved 130,000 individual decision prompts.
• Anti-Saturation High Difficulty: Benchmarks were deliberately tuned with high difficulty thresholds to prevent early saturation and preserve fine-grained differentiation among top contenders.
• Open Reproducibility and Calibration: The comprehensive methodology document (methodology.html), underlying code, evaluation datasets, and probabilistic calibration metrics were published as open source, enabling full third-party reproduction and audit.

Knowledge Lead vs. Narrowing Tool Execution Gaps: Single-Pass Qwen3.8 and Decider 35B

Across the 130K decision queries, Jev sustained a noticeable margin over open-weights competitors in general knowledge, which Passos noted likely reflects a larger underlying base model. Specifically, Jev excelled on graduate-level knowledge questions, Python function selection, tool calling, and model routing.

However, in operational categories central to agent engineering—tool use, automation, retrieval, and classification—the performance margin between Jev and open models narrowed substantially ("it's close!").

• Leading Open Contenders: Trailing immediately behind Jev were inference techniques enabling single-pass execution of Qwen3.8-27B (such as Jevfire) alongside diffusion gemma approaches, achieving competitive scores without custom fine-tuning.
• Third Place Decider 35B-A3B: Decider 35B-A3B, a fine-tuned model based on Qwen3.5-35B base, secured third place on the leaderboard.

Parameter Weight Frontiers and Task-Specific Challenges

The leaderboard also delineated clear frontrunners across specific parameter brackets:

• Under 350M (<350M): Fastino's (@fastinoAI) GLiNER 2.5 led the tiny model frontier, demonstrating high efficiency for ultra-compact deployments.
• 350M–700M and 3B–10B: The Kev model family from Jared Palmer (@jaredpalmer) captured top honors across both brackets.
• Under 3B (<3B Sweet Spot): Decider 2B by @notmapika was highlighted as the sweet spot in the sub-3B segment.
• Performance vs. Latency Pareto Frontier: Openvons' Qwen3-4B stood out as a high-speed yet potent contender, while Matthew Mastracci's (@mmastrac) diffusiongemma implementation joined the Pareto frontier for fast, low-latency decision making.

At the same time, the evaluation highlighted specific task domains that presented notable difficulties:

• Task Limitations Observed for Jev: Jev was observed to struggle with strategic puzzle and reasoning games such as chess. It also faced challenges in demanding document retrieval tasks that conventionally require deep reasoning models, chord recognition from musical notes, and fine-grained sentiment classification.
• Standardized Hardware Baseline: All benchmark figures reflect evaluation on identical hardware using a single NVIDIA RTX 6000 PRO.

Sources

• Hugging Face Spaces: Decision Index 0.1 Leaderboard
• Hugging Face Methodology: Decision Index Methodology
• Apolinário Passos Official X (@multimodalart): Decision Index 0.1 Announcement Thread