Devin Coding Benchmark: GPT-6 Astra vs Claude Opus 5.5 Runtime and Cost Results

A real-world benchmark on Devin compares GPT-6 Astra and Claude Opus 5.5 with max thinking. Despite higher token rates, Astra cut runtime to 85 minutes and over

tau · September 23, 2026

#Devin #GPT6_Astra #ClaudeOpus5_5 #CodingAgents #AIBenchmark #CostComparison

Devin Coding Benchmark: GPT-6 Astra vs Claude Opus 5.5 Runtime and Cost Results

On September 23, 2026, real-world software engineering benchmark results surfaced pitting OpenAI's frontier model GPT-6 Astra against Anthropic's flagship Claude Opus 5.5 inside the autonomous coding agent platform Devin, using identical task prompts and maximum thinking configurations.

Performance comparison dashboard showing runtime and total cost metrics between GPT-6 Astra and Claude Opus 5.5 inside autonomous coding agent Devin Image source: Mo Elgaraihy (@EngMoElgaraihy) / X

Conducted by software engineer Mo Elgaraihy on a production development task, the evaluation tested how both models perform under extended autonomous agent loops, measuring the precise elapsed runtime and total billed API cost required to reach full completion.

Benchmark Measurements: 85 Minutes vs 130 Minutes, $61 vs $75

Evaluating autonomous execution inside Devin on a complex software engineering problem revealed distinct performance characteristics between the two frontier models.

When supplied with identical task prompts and configured with their highest reasoning settings, the models yielded the following real-world metrics:

  • GPT-6 Astra: Completed the assignment in 1 hour and 25 minutes (85 minutes), with a total billed cost of $61.
  • Claude Opus 5.5: Completed the assignment in 2 hours and 10 minutes (130 minutes), with a total billed cost of $75.

In terms of completion velocity, GPT-6 Astra completed the job 45 minutes faster, representing a 34.6% reduction in wall-clock time compared to Claude Opus 5.5. More notably, Astra also finished with a lower overall API invoice, saving $14 (an 18.7% cost reduction). This counterintuitive result demonstrates that in long-horizon autonomous agent workflows, models with higher list prices can end up delivering lower total project expenditure.

Overcoming a 2.5x Unit Rate Gap: Dense Reasoning and Fewer Loops

The core reason these benchmark figures have sparked widespread industry attention lies in the underlying token pricing structures of both models.

According to public pricing catalogs from September 2026 (including OpenRouter's published rates), the per-million (1M) token rates stand at:

  • GPT-6 Astra: $10 per 1M input tokens / $50 per 1M output tokens ($1.00 prompt cache read)
  • Claude Opus 5.5: $4 per 1M input tokens / $20 per 1M output tokens ($0.20 prompt cache read)

On a pure per-token basis, Claude Opus 5.5 is 2.5 times cheaper than GPT-6 Astra. The reason the total bill flipped in Astra's favor stems directly from the iterative operational mechanics of autonomous coding agents.

Autonomous agents operate across multi-turn feedback loops, writing code, executing shell commands, analyzing compiler diagnostics, running test suites, and debugging regressions. According to official disclosures from OpenAI, GPT-6 Astra achieves state-of-the-art results on challenging engineering benchmarks such as Agents' Last Exam (59.3%) while consuming approximately 65% fewer output tokens than Claude Opus 5.

When an agent fails to solve an issue cleanly on its first pass, every subsequent trial-and-error cycle resends massive accumulated conversation context back into the model. By reasoning more densely and requiring fewer debugging iterations, GPT-6 Astra prevented runaway context compounding, effectively neutralizing its higher unit price to deliver both lower overall cost and faster turnaround.

Practical Takeaways: Long-Horizon Agent Loops vs Interactive Chat

These real-world findings highlight that choosing an AI model for software development cannot rely solely on comparing token price sheets, but must account for the execution architecture of the task.

At the same time, practitioners must consider that these figures derive from a single real-world engineering project. Factors such as repository size, domain architectural complexity, the presence of pre-existing unit test suites, and prompt engineering structure will influence relative performance across different models.

Engineering teams should consider different models depending on their operational patterns:

  • Autonomous Multi-Turn Agent Pipelines (Devin, Codex): Because failed attempts trigger costly re-ingestion of massive context windows, a higher-tier model capable of resolving tasks with fewer cycles (such as Astra) is often more economical and faster overall.
  • Interactive Developer Chat and Code Completion: In interactive setups where human engineers direct each step and cumulative context remains lean, models with 2.5x lower token pricing such as Claude Opus 5.5, or budget models like Grok 4.7 ($2 input / $6 output), remain significantly more cost-effective.

Sources