GitHub and Manifest Measured Model Routing Wrong
Model routing doesn't work if you only measure cost-per-token. It only works if you measure cost-per-task. Uber measured this and provides some great data points.
Manifest's LLM router shut down recently — four months live, 7,000 cloud users, mixed results, a pile of GitHub issues. GitHub reached the same conclusion, at a scale no startup can match, and wrote it into the Copilot documentation: "Switching models mid-session has shown increased cost without ample improvements in quality." Between them they found three real things. Classification from the prompt doesn't work, because the prompt is only the trigger — "evaluate the tests for this repo and improve them" is a twenty-minute job on a static HTML site and a career on the Linux kernel, and nothing in the text distinguishes them. Caching matters, a lot; any switch throws away a compounding discount, and Manifest's sharpest line is that a cache-aware router ends up doing its job by not doing it. And swapping models mid-session degrades consistency, which you pay back in evals, observability, and every system prompt you own.
If you are building a decision layer and this doesn't worry you, you aren't paying attention.
But look at what they actually measured. Both measured what routing cost. Neither measured what it was worth. Manifest's mixed results, GitHub's 10% discount on model costs — every public number in this debate is denominated in dollars per token, and that is not what routing is for.
The real number is cost per task
When a cheap-tier session produces a plausible-looking PR that fails review, the token meter records a win. The loss is booked to a human reviewer, or to the second session you run at a stronger model, or to the bug that ships. Manifest says this themselves, and it's the most honest sentence in the debate: the amount saved is paid somewhere else, and the somewhere-else is harder to estimate.
There is no single right model per task. There's a Pareto frontier: cost against success rate. This is different for every task class, and it moves. It moves when a lab ships a checkpoint, when a harness changes its tool schemas, when a repo grows a new subsystem. Four static tiers are a fixed answer to a question that gets redrawn every few weeks.
Somebody built the instrument
Uber published their methodology on August 27th, and it is the missing arm of this experiment. More than 70% of their pull requests now come from agents. Weekly active users grew 7x and agent requests 9.4x between February and August, while total AI spend stayed roughly flat. Holding the model constant to isolate their own gains: cost per thousand requests down about 34% from peak, cost per session down 52%.
The interesting part isn't the savings, it's what they optimize against. Uber defines a Pareto-efficient model as the one that wins on cost per completed task, output quality, and reliability — not dollars per token. For uReview, their code-review agent, they built a benchmark out of real pull requests with known bugs, scored precision, recall and F1 against them alongside cost per review, and plotted every configuration they tested against a Pareto frontier. Their selection procedure ends with an instruction to keep moving, because "the frontier shifts every few weeks."
They also keep the record we've been saying nobody keeps: outcome-denominated cost for each managed agent — cost per merged PR, per review, per alert — paired with revert rate, F1, and MTTR, tracked through every model migration.
So cost-per-task routing isn't a thesis. It's a shipped, measured practice at a company with 5,000 engineers.
Now look at what it took. Thousands of real pull requests to build the benchmarks from. An internal SWE benchmark spanning frontier and open-weight models. A context graph of 24 million nodes and 80 million edges assembled out of more than 30 internal systems. An MCP gateway fronting over a thousand servers. Roughly 35 named engineers, led by a distinguished engineer.
Uber proved the method works and proved almost nobody else can run it.
The bet
Hokusai has a different bet about why GitHub and Manifest's routers were failures. They found that a naive decision, made once from the prompt, doesn't pay. We're not surprised.
The bet is that if you capture structured outcome data across many harnesses, three questions everyone currently guesses at become measurable:
How to classify tasks properly. Not four tiers guessed from a sentence — classes derived from what actually predicted success.
How much evidence a classification needs before it's worth acting on. Manifest proved the prompt alone is insufficient. Nobody has established what is sufficient, or how far into a session you have to get before the task has revealed itself.
Where the caching boundaries are. GitHub is right that revision at arbitrary moments is expensive. Uber moved its interactive prompt-cache TTL from five minutes to an hour because engineers idle mid-session and were forcing full-price prefix rebuilds, while leaving subagents at five minutes. That's the boundary treated as a measured design variable rather than an assumption.
Uber answered all three by building a private corpus, which is so far the only way anyone has answered them. Manifest ran four months across 7,000 users and came away with "mixed results" — the same questions, an underpowered instrument. Between those two outcomes sits every team that will never have Uber's benchmark budget, facing the same frontier, moving every few weeks, with nothing to see it by.
And this isn't a discipline problem, it's structural: the evidence that a routing decision was wrong surfaces downstream — in code review, in CI, in a different tool, often in someone's head three days later. No single gateway sees it. Every harness that does see a piece of it keeps that piece, because there's no reason to share it and no mechanism if you wanted to.
Uber's own roadmap says the quiet part: effective model routing, they write, relies heavily on comprehensive evaluation, because model capabilities vary so widely. We agree. We just don't think every team should have to build that evaluation alone, or that the ones who can afford to should be the only ones who get a router that works.
That's the layer worth building: shared, auditable outcome data owned by the people whose data improves it, rather than another vendor's internal margin. Credit where it's due — Manifest ran the experiment and published the negative result with real specifics, and Uber published the positive one in enough detail to reproduce. The number they could both name is the one worth building an instrument for.
If you maintain a coding harness and you're sitting on outcome data nobody's aggregating — that's the conversation we want. Come argue with us.