GPT-6 Sol Real-World Bug Fix Benchmark: Score Plummets to 29.3 as Costs Drop by 90%

Paweł Huryn's benchmark across 105 hard bugs in two repos reveals GPT-6 Sol's score dropped to 29.3 from GPT-5.6 Sol's 43.5, while API costs fell nearly 90% to

tau · September 23, 2026

#GPT6Sol #OpenAI #ClaudeOpus55 #Benchmark #AI코딩 #Astra

GPT-6 Sol Real-World Bug Fix Benchmark: Score Plummets to 29.3 as Costs Drop by 90%

On September 23, 2026, product management and AI evaluation specialist Paweł Huryn released benchmark results evaluating frontier coding models on their ability to locate and fix 105 challenging hidden bugs across two real-world repositories. The newly released GPT-6 Sol from OpenAI, evaluated at its maximum reasoning effort setting (max), scored 29.3 points in bug resolution—a steep performance drop compared to GPT-5.6 Sol's 43.5 points. In sharp contrast, its API-equivalent execution cost came in at $9.93, achieving an 89.6% cost reduction compared to GPT-5.6 Sol's $95.25.

Comparison chart displaying bug fix scores and API costs for GPT-6 Sol and frontier coding models across 105 bugs in two repositories

Image source: @PawelHuryn

The evaluation targeted difficult real-world defects that frontier models had previously missed in early 2026, measuring practical debugging capabilities rather than synthetic code puzzle performance. The results provide empirical evidence of a sharp architectural tradeoff in GPT-6 Sol: while runtime expenses have dropped to roughly one-tenth of its predecessor, raw resolution accuracy on complex software defects has significantly degraded.

Real-World Bug Hunt Across 105 Defects: GPT-6 Sol Regresses in Frontier Model Comparison

Paweł Huryn's evaluation was conducted not on toy puzzles or synthetic benchmarks, but across 105 planted hidden bugs embedded within two production software repositories.

Each model was tested under a uniform maximum reasoning effort setting (max) with a straightforward directive: find and fix every defect possible. The resulting bug resolution scores across the compared models were:

  • GPT-6 Astra (max): 45 points
  • GPT-5.6 Sol (max): 43.5 points
  • Claude Opus 5.5 (max): 41.7 points
  • Muse Spark 1.3 (max): 32.2 points
  • GPT-6 Sol (max): 29.3 points

The most striking outcome is the cross-generational regression. While OpenAI's flagship GPT-6 Astra achieved the top score with 45 points, the new GPT-6 Sol lagged behind at 29.3 points, falling 14.2 points below the prior-generation GPT-5.6 Sol (43.5 points). It also ranked below Muse Spark 1.3 (32.2 points), underscoring a notable decline in its ability to navigate complex code context, track down elusive root causes, and author clean, comprehensive bug fixes.

90% Cost Reduction: $9.93 vs $95.25 API-Equivalent Spend

The economic picture, however, presents an inverse dynamic. Measuring the cumulative API-equivalent cost required to process all 105 bugs revealed substantial cost reductions:

  • GPT-6 Sol (max): $9.93
  • Muse Spark 1.3 (max): $18.11
  • GPT-6 Astra (max): $33.04
  • Claude Opus 5.5 (max): $58.53
  • GPT-5.6 Sol (max): $95.25

At $9.93, GPT-6 Sol reduced processing expenses by approximately 89.6% compared to GPT-5.6 Sol's $95.25, costing barely one-tenth as much. It also ran at under one-third the cost of the top-performing GPT-6 Astra ($33.04) and about 83% below Claude Opus 5.5 ($58.53).

This steep reduction aligns with OpenAI's API pricing for GPT-6 Sol ($2 per million input tokens, $10 per million output tokens, and 90% off cached inputs). While the model delivers markedly superior cost efficiency per point scored, its reduced resolution score presents a clear constraint for workflows where fixing every defect on the first pass is mission-critical.

Rigorous Blind Scoring and Developer Takeaways

The benchmark implemented an independent scoring pipeline designed to maximize measurement reliability:

  • Secret Answer Key & Cross-Family Model Judges: Solutions for the 105 planted bugs were scored against a secret answer key by a panel of judges drawn from different model families.
  • Zero Credit for Incomplete Solutions: Identified bugs left unfixed or partially implemented solutions received 0 points with no partial credit awarded.
  • Exclusion of Bugmaxing: OpenAI models frequently exhibit "bugmaxing"—reporting numerous real, theoretical, or irrelevant issues that overcomplicate the fix. These unplanted issues were strictly excluded from the scoring.

For engineering teams, these findings highlight the necessity of tiered model routing. In community discussions following the release, developers noted that mission-critical debugging and difficult root-cause investigations should remain routed to higher-tier frontier models like GPT-6 Astra or Claude Opus 5.5. Conversely, GPT-6 Sol serves as an economical engine for boilerplate creation, routine refactoring, and cost-sensitive background tasks where its $9.93 price point shines.

It is worth noting that these results represent a single benchmark across two codebases at the maximum reasoning effort setting. Paweł Huryn noted that expanded data across additional effort tiers (xhigh, high, medium, and low) will be released on bughunt.productcompass.pm.

Sources