The bake-off: picking the model that does the typing

In mid-September I measured where my agent tokens went. One week came to 8.1 billion tokens. Subagents, the workers my main session sends off to do things, were 73% of it. Two lanes carried almost all of that: expensive Opus workers, and a mid-tier browser agent.

That told me what to fix. It did not tell me what to replace them with. For that I ran a bake-off.

Size the prize first

Before testing anything, I checked which lanes were worth moving. Inside the subagent week, the workers were 54.5% of tokens and the browser lane 39.2%. Together, 93.7%. The read-only scout was 2%.

So the scout stayed where it was. Putting it behind a cheaper model would have added a supervisor layer to save almost nothing. The two big lanes moved.

I had also learned that a rule written in a prompt does not hold. Earlier, the routing policy said to send routine work to a cheaper lane. In one audit window the cheap lane ran 3 times. The expensive worker ran 111. After that, the rules moved into hooks: a dispatch to an expensive worker has to state why it is escalating, or it gets blocked.

The test

The candidates had to run inside the same harness my main session uses, so skills, hooks and tools behave the same. Each one got the same implementation spec and was graded by the same verifier: 34 mechanical checks that pass or fail, with nothing left to taste. Only the model changed.

Results:

• DeepSeek V4.1 Flash: 34 of 34, in 69 seconds.
• GPT-5.6 Luna: 34 of 34, in 189 seconds.

Same correctness. Flash was 2.7 times faster.

The bigger DeepSeek model scored the same 34 of 34 and cost 2.7 times more. It also carries two documented defects in multi-turn tool calls. The lanes are pinned to Flash, and the bigger model only runs when a spec names it.

Speed was not the only reason. Flash is pay-as-you-go with no cap. My subscription lanes were hitting their limits that week, and one plan was already used up. A capped lane can stop halfway through a 400-turn run. An uncapped one cannot.

The browser lane got the same treatment. The first live check was a Hacker News read: 5 turns, 13 seconds, and the result matched the Hacker News API exactly. Long browsing runs now go in phases, each with a fresh context and a written handoff to the next. The phase loop has held across 6 phases with 5 clean handoffs.

Why judgment stays on the strong model

The cheap lane does the typing. It does not decide what gets typed.

Every dispatch starts with a spec the driver writes: the exact outcome, the files, the reference text inline, the constraints, and the commands that prove it worked. If the spec is vague, the worker improvises a different design. That failure belongs to the driver, not the worker.

The driver also reviews what comes back. A worker saying "done" is a claim. The lane shim reruns the spec's verification commands itself and reports what actually happened. When something is only proven by a typecheck or a file on disk, the status is capped at code-complete, not live-verified. Only a live check lifts it.

I also keep the author and the verifier apart. In one build, a 13-agent workflow wrote a set of quality gates, then attacked each gate with a planted defect it should have caught. Three gates missed their own defect. The builders had already passed their own work. Those three would have shipped as false negatives.

Cheap models are also worse at keeping promises written in prose. "Always check X before Y" in a prompt gets skipped under momentum. So those promises became code: gates that fail the run.

What the split looks like now

• A strong model drives. It reads the request, plans, reviews and writes the final answer.
• DeepSeek Flash is the default lane for implementation and for browsing.
• The ChatGPT and Grok lanes stay live as second vendors. I already pay for those subscriptions.
• A read-only scout handles volume retrieval.
• A reviewer with fresh context checks high-stakes work. For money copy and architecture, that reviewer runs on a different model than the driver.

If a cheap lane fails, it reports unavailable. It never falls back to the expensive model on its own. A silent fallback would undo the whole point.

What I would tell someone doing this

Measure the token split before you test a model. Build a verifier with checks that pass or fail, and change one variable at a time. Keep the model that judges separate from the model that types.

This site was built the same way. The driver wrote the specs and the copy. The cheap lane wrote most of the code. The driver reviewed every screen before it went up.