Model routing improves results, when it routes the task, not the prompt

Follow-up to GitHub and Manifest measured model routing wrong. Measured on the Wavemill corpus · 59 paired trials · June–August 2026.

Most model routers ask the wrong question: which model is best on average?

In 59 controlled coding trials, that approach reduced spend by 9% and improved success by nothing at all. But which model was right depended heavily on the task. On 22 of the 59 tasks, one model succeeded and the other failed.

Choosing correctly in advance would have raised success from 61.0% to 79.7% while cutting spend per attempt from $0.430 to $0.356 — 48% fewer failures at 37% lower cost per successful task.

That is a ceiling, not a router's score. What makes it worth writing down is how little data it took to find.

Fifty-nine tasks is the point

Fifty-nine tasks is not a production-scale training corpus. It comes from one repository, compares two models at a time, and is far too small to support broad claims about coding workflows in general.

That is what makes the result interesting.

Before there was anything like enough data to build a task router, model choice had already determined success on 22 of 59 tasks. Random selection left roughly half of those solvable tasks on the table. A ranking of which model performed best overall recovered none of them.

The opportunity showed up before the router was smart enough to capture it. That ordering is the whole argument: a decision layer does not have to create model complementarity, because the models already fail differently. It has to learn the conditions under which those differences matter.

The 18.7-point gap below is not a performance claim. It is a first measured addressable opportunity, and 59 tasks were enough to make it material.

Where the opportunity shows up

Wavemill is the software factory behind Hokusai — it picks up an issue, plans it, writes the code, reviews it, and opens a pull request, largely unattended. Built into it is challenge mode: two models run the same issue from the same commit at the same moment, each opening a real pull request against a real codebase.

Running two models on every task is a poor way to ship software and an excellent way to measure one. Everything that normally confounds a model comparison is held constant — same task, same repository state, same tooling, same hour of the same day.

Across 59 such pairs, the outcomes fall into three groups, and the middle one is where a router earns its keep.

The 25 on the left are where routing changes the invoice and nothing else. The 12 on the right are beyond any router's reach — no available model solved them, and a decision layer cannot manufacture a capability the pool does not contain.

The 22 in the middle are the prize. On 37% of tasks, model choice was the difference between finished work and a failed attempt. That is not a quality gradient between broadly similar models. It is complementarity: they fail on different things.

What that is worth

Because both arms actually ran on every task, the corpus can be replayed under different selection policies using real outcomes rather than projections. Every policy below picks one of the two arms that genuinely executed. Nothing is simulated.

Reported against the metric an operator actually pays — not cost per call, but cost per unit of completed work:

No routing versus the per-task ceiling, same 59 tasks

Across just 59 tasks, perfect selection would have recovered 11 additional successful outcomes without adding a single model call. The bottom rows are the business case: good routing does not merely buy cheaper calls, it buys more finished work per dollar, improving the numerator and the denominator at once.

This is available headroom, not captured value. The ceiling assumes perfect foresight about which arm will succeed, and no deployed router has that.

Ordinary routing does not collect it

Rank the models by how well they have done per dollar historically, then send each task to the winner. Fit that leave-one-out — each task routed using only the other 58, so no hindsight — and it delivers a 9% cost reduction at a success rate of 61.0%. Identical, to the decimal, to picking at random.

In this corpus, model-level averages contained too little information to improve task success. They moved cost only. And a 9% saving is thin enough that a modest amount of cache disruption erases it.

The distance between the two lower-left points and the upper-left one is the difference between knowing which model is better and knowing which model is better here.

Routing the prompt is not routing the task

Manifest ran a router in production for four months across seven thousand users and reported mixed results, conceding that "the amount saved is paid somewhere else." What it ran was a prompt-complexity classifier that switched models request by request — a design that discards the cached prefix on every switch and changes the model's assumptions mid-conversation. It did not show that routing fails. It showed that routing the prompt fails.

Three architectures get conflated under one word:

• Average routing asks which model is cheapest or best overall. It moves cost, not quality.
• Per-request routing switches models inside an active session. It breaks cache continuity and often spends more than it saves.
• Task-aware routing chooses once, at a clean task boundary, using evidence about where models actually diverge.

Only the third can reach the headroom above, because only the third decides at the granularity where the complementarity exists.

GitHub's HyDRA router, now serving auto-mode traffic in Copilot, is built on both conclusions: it selects by evaluating the task — reasoning demand, code-generation complexity, bug-diagnosis difficulty, tool-orchestration needs — and changes models only at natural cache boundaries, never mid-task. Its own explanation of why is routinely misquoted as evidence against routing: switching mid-conversation breaks the prompt cache, "which can cost more than the routing change saves." That is the reason to route at task boundaries, reached independently and confirmed at production scale.

The remaining question is how much of this task-level headroom an open, shared router can capture, and how quickly verified outcomes improve it.

What a shared router has to learn

Between picking at random and perfect foresight there is a path, and it is worth being explicit about where a decision layer actually sits on it.

Where a decision layer sits between random and perfect

The middle rows carry no numbers on purpose. Hokusai has not earned them yet, and the distance from the second row to the last is precisely what a decision layer is for.

Travelling it is a learning problem with an awkward property: the opportunity is visible locally, but the evidence needed to exploit it is not. Fifty-nine pairs from one repository can measure headroom. They cannot support a selection policy that generalises across languages, task families, or model catalogues.

Thousands of harnesses generate exactly the outcomes that would close that gap, and nearly all of them discard or silo the data. Hokusai's protocol makes those outcomes interoperable and rewards contributions that measurably improve routing on held-out tasks, so each participant can use a decision layer trained on more experience than any one of them could generate alone. More repositories expose more task types; more harnesses reveal which patterns transfer; a broader model pool creates more possible routes, and may raise the ceiling itself — a hypothesis worth testing rather than a claim.

The headroom exists locally. The intelligence needed to capture it has to be learned collectively.

Two things that will not close the gap

The obvious shortcut is an LLM judge that reads both pull requests and picks the better one. As measurement it works — given two real diffs it selects the arm that actually succeeded 86% of the time. As a router it cannot function. It is blind to cost, choosing the cheaper arm 47% of the time across the same 59 pairs, which is chance. And a verdict on two pull requests requires two pull requests: it cannot be consulted before the money is spent, which is why it sits at $0.86 in the chart above rather than $0.43. It produces training labels, not decisions.

The second is raw volume of execution logs. When a comparison in this corpus was skipped, forfeited, or exhausted its retries, the harness still wrote a record with the winner field populated by bookkeeping rather than by any judge — 35% of rows, skewed six to one toward the first arm, structurally indistinguishable from real verdicts. All are excluded from every number here. A router trained on the unfiltered logs would have learned a preference for whichever arm was listed first and had no way to detect it. Verified outcomes and large piles of traces are not the same input.

What this proves, and what it does not

The opportunity has been measured. The next milestone is showing how much of it a learned router captures on held-out tasks — which is a different claim, and one nobody should accept without seeing it.

Benchmark your own harness

Send Hokusai a representative paired sample from your harness — the same task, run on two models, with the outcome and cost of each.

The first output is not a sales promise. It is a measurement: how much success and cost headroom your current model policy leaves on the table, computed the same way as the table above. If your models turn out not to be complementary on your work, that is a real result and worth knowing.

Fifty-nine paired tasks were enough to expose the opportunity and nowhere near enough to capture it. That is the argument for a shared protocol rather than a private router: every verified outcome makes the next decision better, and improvements proven on held-out tasks can reward the contributors who made them possible.