Arize Benchmark Finds Token Price Misranks Models by an Order of Magnitude

Arize AI and Fireworks published a joint benchmark on July 23, 2026, written by Arize head of developer relations Laurie Voss, measuring what it costs to get a model to finish real work. The study, "Cost per successful task," ran ten open and closed models through 40 command-line tasks at six trials each, producing 2,400 runs on $626 of API spend, every one traced through the Arize AX observability platform.

The finding that matters most for anyone measuring AI return is that cost per token and cost per finished job can diverge by more than an order of magnitude and rank models in opposite orders. The cheapest model per completed task, an open one called gpt-oss-120b, finishes a job for 5.4 cents while passing 33 percent of the time, roughly a twelfth of GPT-5.5 per success and a twenty-third of gemini-3.5-flash.

GAIG is covering it because it puts hard numbers on the cost side of AI return measurement, the half of the equation most organizations report loosely or skip. The harness is published on GitHub, so anyone can rerun it.

Key Terms

Cost per successful task

Everything a model spent across every attempt, including runs that failed, retried, or timed out, divided by the number of times it actually completed the task.

Retry tax

The number of attempts a model needs per success, expressed as a multiple, which turns the hidden cost of failure into a single figure.

Coverage

How many distinct tasks a model solves reliably, defined in this study as passing four of six trials or better.

Silent failure

A run that finishes cleanly with every step reporting success while the answer is wrong, which no runtime alert catches.

Escalation routing

Sending work to a cheaper model first and moving it up to a more capable model when the cheaper one fails.

Key Findings

Results as published by Arize and Fireworks, ordered by cost per successful task because that ordering is the finding. Arize describes Kimi K3 as soon to be open-weights. Claude Sonnet 5 is billed at its standard rate rather than an introductory rate. GPT-5.6 Sol was not tested.

• Token price ranks models in the wrong order. The spread between the cheapest and most expensive model per completed job is roughly 23 times, and the ordering bears little resemblance to the pricing pages. Any cost figure built from token spend understates what the work cost.

• The cheapest model per success is the least capable one in the field. Counting tasks solved reliably, at four of six trials or better, gpt-oss-120b managed 8 of 40 against 25 for GPT-5.5 and 26 for Kimi K3. Part of why it looks efficient is that it only wins the work it can win.

• The frontier premium buys nothing on easy work. On easy tasks GPT-5.5 passed 69 percent while Kimi K2.6 passed 73 percent and gpt-oss-120b passed 65 percent at a fraction of the price. On the hardest tasks only the top tier competes, so the sensible unit of cost control is the task rather than the contract.

• Routing beats every single-model strategy on cost and coverage at once. A deployable escalation ladder solved 32.3 of 40 tasks at 52.5 cents per success against 25 of 40 at 63.6 cents for the best single model. Treat the routing policy as a documented design decision, since a naive ladder through all ten models cost $1.319 per success, worse than every model tested.

• The open and closed labels predict nothing useful. The two most efficient models per success were one open and one closed, and the two strongest finished in a near tie with GPT-5.5 closed and Kimi K3 open. Procurement policies written around model category rather than measured performance sort on the wrong attribute.

• Failure modes differ, and the expensive ones are invisible. Arize reports that 168 of gemini-3.5-flash's 184 failures, about 91 percent, were the model burning its whole token budget and producing nothing, while GPT-5.5 failed a trivial file conversion on three of six trials with every step reporting success. The second kind reaches production undetected, because no alert fires when the process completes and only the answer is wrong.

What the Report Covers

The tasks come from Terminal-Bench, a benchmark of real command-line jobs such as converting a file or debugging a program, each shipping its own Docker environment and test script. Arize used 40 of roughly 240 available tasks, chosen to sit where some models pass and others fail, and calls the results directional rather than a leaderboard score.

The methodology is built to keep attribution clean. Arize wrote a deliberately thin agent harness with a single tool, so a run's cost is attributable to the model rather than the scaffolding, and ran every model through one code path using OpenAI-compatible APIs across Fireworks, OpenAI, Anthropic, and Google. Grading is deterministic, using each task's own test suite with the agent never seeing the tests, and two guardrails record budget and timeout failures separately from wrong answers.

Every task-model cell holds six trials, which Arize says puts the 95 percent confidence interval on a pass rate at roughly plus or minus six points, enough to rank models and not enough to separate close neighbors. The authors decline to report Kimi K2.6 as beating GLM-5.2 for that reason, and the audience is engineering teams choosing production models, with the tool-call and trace analysis aimed there.

Two limits deserve attention. The benchmark covers a single domain where success is verifiable for free, since every Terminal-Bench task ships a test suite, and most enterprise agent work has none, so the expensive job of defining success falls on the buyer. The commercial interests also point one way, because Arize sells the observability platform the study concludes you need, Fireworks hosts the open models that win the comparison, and judge-based evaluation, which Arize also sells, is named as a cost this study avoided paying.

The ten models tested, grouped alphabetically by provider.