I thought I knew why my harness was slow: the model made too many decisions. Every step a coding agent takes ends in a choice, which file to open, which command to run, and every one of those choices went through a large model, on a large model’s clock. Swap the decider for something fast, I figured, and the harness picks up the pace.
I was wrong. Not a little wrong: the decision I wanted to speed up is a small slice of the time, and the first yardstick I used to measure it was wrong too.
This is the first Lab notebook entry, a series on experiments with coding agents that puts the hypothesis, the instrument and the number together, even when the hypothesis doesn’t survive. The habit of measuring before believing comes from AIvengers, the experiment where I measured what AI was doing on a real project, with a baseline and the hours counted. I’m publishing for a practical reason: I want to find people who measure the same thing and put my numbers next to theirs.
The hypothesis, and why it seemed obvious
Watch a coding agent work and you see a rhythm: it stops, thinks, calls a tool, reads the result, stops again. In the week I measured, each turn, from my request to the final answer, averaged 25.3 calls to the model. Every one of those pauses looks like a decision.
An average turn: 25.3 calls to the model
The math looked simple. If a good share of those calls only pick the next step, and a large model takes seconds to pick, the harness spends a good share of its time deciding. Taking that decision away from the large model would be the most obvious win I had within reach.
The candidate: a classifier that doesn’t write
The candidate showed up in September. TypeSafe AI launched Jev, which it calls a “System One” model: it doesn’t generate text, it returns a typed decision, yes or no, a pick among options or a score, with the probability attached. The Register covered the launch. It was exactly the shape of the decision I wanted to take away from the large model.
Vercel’s guide on when to use Jev gives the advice I followed: hand it only the bounded decision, keep whatever is already a fixed rule in code, and compare it against what you have today, on the same labeled cases, before replacing anything.
So I compared. I pulled 180 cases from my own work, all from personal projects: pieces of a plan, each labeled with the model that should run it, small, medium or large. Jev and Haiku each tried to predict the label.
| Jev | Haiku | |
|---|---|---|
| Accuracy | 0.717 | 0.722 |
| Kappa | 0.486 | 0.486 |
| p50 latency | 284 ms | 1,789 ms |
| p95 latency | 336 ms | 2,030 ms |
Jev was as accurate as Haiku and answered faster: 284 ms versus 1,789 ms at the median. Haiku’s latency includes the CLI startup, which a hook would pay too, so the comparison is fair for the use I had in mind and unfair to Haiku as a model. Neither was accurate enough to decide on its own.
But Haiku wasn’t the right baseline. The right baseline is a fixed rule, free to run. In a simulation over the same week, I put Jev and a fixed rule side by side at the three places in the harness where a rule exists: the result gatekeeper, which trims a tool’s large output before it enters the context; the circuit breaker, which stops an executor once it goes past a set number of model requests; and the /clear suggestion, which warns me when my request opens a new subject and the old context only gets in the way. In all three, the fixed rule delivered more savings per week than Jev.
Two instruments, and the shortcut that got it wrong
Before plugging Jev in anywhere, I needed to know how much of the time the decision takes up.
The first instrument was a shortcut. A model call that returns very little text, up to 150 tokens, is mostly just picking the next tool; one that returns a lot is writing. I sorted the week’s 9,316 calls by output size. There were 622 short ones, 7% of the calls, adding up to 0.4 h, 0.5% of turn time, with a median of 2.1 s. If that was the decision, the hypothesis died right there.
I didn’t trust a number that small. Output size is a shortcut: the call that picks the next read can come with a paragraph of explanation, and the shortcut counts it as writing.
The second instrument classified each call by what it produces: a command that acts, a read or search, an edit, text only, a dispatched subagent. Reading or searching, where the model decides the most without producing anything, came to 2,736 calls, 7.9 h and 10.3% of the work. The shortcut had been off by an order of magnitude.
And the conclusion still didn’t flip. If every read that took longer than 2.0 s, the median of a short read, dropped to 2.0 s, the gain would be 6.4 h, 8.3% of the work. That’s the ceiling, with an instant, perfect decider. And part of it isn’t even deciding: 259 reads came from calls with more than 2,000 output tokens, with a median of 36.4 s, and there the model is generating text, which is exactly what a classifier doesn’t do. Commands that act took 13.2% of the work and edits 9.1%: work that has to be done, not chosen.
How much of the time the decision takes up
- The shortcut0.5%622 short-output calls, 0.4 h
- Reading or searching10.3%2,736 calls, 7.9 h
- The ceiling on the gain8.3%6.4 h
Where the time actually goes
Measured by the clock, rather than by call type, the week came to 368 turns and 77.1 h.
| Where | Share of the time |
|---|---|
| Waiting for the model to respond (API) | 38.5% |
| Waiting on a background task | 22.8% |
| Compacting the context | 17.0% |
| Waiting on me | 10.0% |
| Command with a memory cap | 5.3% |
| Command without a cap | 4.9% |
Where the 77.1 h went
- 38.5% waiting for the model (API)
- 22.8% background task
- 17.0% compacting the context
- 10.0% waiting on me
- 5.3% command with a memory cap
- 4.9% command without a cap
- 1.3% hook and other
Within the 38.5% of API time, long responses, 2,000 tokens or more, account for 19.9% of the turn on their own, with a median of 35.9 s. That’s the model writing code and text, which is what I asked it to do.
After the API, what weighs the most isn’t a decision at all. It’s background tasks, and it’s compaction: 332 compactions in the week, the context filling up and the harness stopping to summarize itself, 17.0% of the time. If I want a faster harness, that’s where the time is.
What this doesn’t mean
It doesn’t mean Jev is bad. It was as accurate as Haiku, answered faster, 284 ms versus 1,789 ms, and lost to a fixed rule wherever a fixed rule exists. Where there’s no rule and the decision is closed-ended, it’s still the candidate I would test.
It doesn’t mean deciding comes free. Reading at 10.3% of the work is real time; it’s just not where most of the time goes, and the 8.3% ceiling assumes a decider that doesn’t exist.
It doesn’t apply to your harness until you measure it. This is one person, one week, personal projects only, and a harness with my quirks. Haiku’s latency includes the CLI startup. The comparison with the fixed rule is a simulation, not the harness running with each one.
Measured yours? Write to me
If you measure the time an agent harness takes, yours or your team’s, I want to see the number, especially if it contradicts mine. It doesn’t have to be polished: the hypothesis, the instrument and the result are enough, even if the hypothesis fell apart.
Write to me at eltonleao.dev@gmail.com, on LinkedIn or on WhatsApp.