The Speculation Ceiling

Alex Zhang's post proposes a clean idea. In a harness where the model writes code and the code calls tools, you don't have to wait for generation to finish before running anything. Parse the partial program as it streams, pre-launch the calls whose arguments you already know, and let the real execution collect cached results when it arrives. He calls it speculative programmatic tool calling, and he measures it at 1–1.2× on a Recursive Language Model over OOLONG.

He's upfront that the number is modest. It isn't a disappointment. It's a number you could have predicted from two measurements you already have. And once you can predict it, something more useful falls out: the post bundles two distinct techniques under one name, with very different ceilings and very different implementation costs. They are not additive, and the cheap one is worth more.

Where the wall clock actually goesA turn is two phases, and only one of them is being optimized

A turn is decode (the model writing the program, call it D), then execute, the program running its tool calls, E. Baseline wall clock is D + E. The calls inside E are usually sub-LLM or sub-agent invocations, so E is dominated by network-bound work that could have started earlier. There are three ways to arrange the same six calls.

That second bracket is the whole argument. Once the calls run concurrently, there is very little left for streaming speculation to win, because the thing it hides is already hidden. The two techniques compete for the same savings, and whichever you implement first collects most of it.

The mechanismA shadow REPL, and the valve that makes it safe

The implementation is worth understanding before deciding against half of it. Zhang forks the real namespace with a deepcopy and swaps every speculatable tool for a stub that launches the call and registers a promise. Each time a chunk of tokens arrives, the partial program is re-run in that shadow namespace.

Two design decisions deserve to be copied by anyone building in this space, independent of whether you adopt speculation at all.

The one-way valve. Shadow state is never promoted into the real namespace. A partially generated program is very often a broken program, and a speculator that mutated real state would leave the harness somewhere no retry could recover from. Treating the cell as an atomic unit costs a deepcopy and buys a system that fails cleanly.

Promises keyed by occurrence, not just arguments. If a program runs five identical llm_query calls to take a majority vote, keying the cache on arguments alone collapses them into one sample and silently changes the answer. Keying on call site plus occurrence means the speculated trajectory is identical to the unspeculated one. This is the guarantee AsyncFC gives up, and it's the difference between a latency optimization and a semantics change.

What is actually speculatablePurity is a whole-cell property in practice

Speculation is only legal when you can determine a call's arguments without running anything that touches the outside world. Zhang's implementation walks four cases.

title = llm_query("Title for: The Odyssey")
blurb = llm_query("Blurb for: The Odyssey")

a = llm_query("Triage: " + doc)
c = llm_query("Summarize: " + a)  # waits on a

side = llm_query(q, chunks[0])   # peekable
parts = [gist(c) for c in chunks]  # not

notes = open("/tmp/x").read()  # blocked
b = llm_query("Use: " + notes)  # blocked

The fourth case is the one that bites in production. Purity gets discussed as a property of individual calls, but blocking propagates through the dataflow, so it behaves like a property of the whole cell. One file read near the top of a program zeroes out the hit rate for everything downstream, and real agent programs read files near the top constantly. Expect measured hit rates well below what the case analysis suggests.

What it is worthThe ceiling is 2×, and only at one point

Now the arithmetic. Let h be the fraction of execution time you successfully speculate. You save min(hE, D): you cannot hide more work than there is decode to hide it under.

Perfect speculation, infinite engineering, zero overhead: you cannot beat 2×, and you only reach 2× when decode time and execution time are exactly equal. Plot it against E/D and you get a tent.

Two things follow that you can act on this week. First, E/D is measurable today with two timers and no new code, and it tells you your ceiling before you write a parser. Below about 0.3, your best case is under 1.3× and the shadow interpreter is not worth maintaining.

Second, and less obvious: being execution-heavy doesn't rescue you either. The tent is symmetric in log space. At E/D = 5 you get the same 1.2× as at E/D = 0.2, because now there isn't enough decode to hide the work under. Streaming speculation only pays near balance, which is a narrow place to build for.

The costs are not cappedPurity analysis protects correctness, not the bill

Zhang's safety analysis is about the inputs to a speculated call. But the calls themselves are sub-LLM invocations: they cost tokens, they occupy queue slots, and they're non-deterministic. A speculation that never gets used is not free the way a mispredicted CPU branch is free.

Zhang names the real failure mode himself: the tool serving engine clogged with speculative requests. It's nastier than it sounds. Speculative and real calls hit the same engine, so under concurrency (his own benchmark runs 4 and 8 instances against one vLLM server) aggressive speculation makes the thing you were speeding up slower, exactly when the system is busiest. Backpressure and a cap on in-flight speculation aren't a refinement here. They're load-bearing.

The half that isn't cappedBuild the scheduler, not the speculator

Zhang puts the important claim in his final paragraph: the long-run value is in "more clever JIT compilation tricks to overlap tool calls with the actual REPL execution itself." That's the lede.

Parallel dispatch, running independent calls concurrently because the model wrote them sequentially and never reached for asyncio, is bounded by the width of the dependency graph, not by decode time. Six independent calls is a 6× cut in E. Twenty is 20×. There is no tent. And it needs almost none of the machinery: no streaming API, no incremental parser, no shadow interpreter, no purity allowlist for the common case. You parse the completed cell into a dependency graph and schedule it, a problem with forty years of literature behind it.

The lineage is worth naming, because it tells you which problems to expect. This is promise pipelining, as built in the E language and Cap'n Proto: return a promise immediately, let dependent calls reference it, resolve at the boundary. The dependency analysis is dataflow analysis. The shadow REPL is speculative execution minus the rollback. What's new is the setting: the program is written by a model that doesn't know it should have parallelized, and the harness gets to fix that at runtime. It's a compiler problem living inside an agent.

If you are building thisAn order of operations

1. Measure E/D and the width of your dependency graph. Two timers and an AST walk over logged cells. An afternoon of work that tells you the ceiling of everything below.
2. Build parallel dispatch first. Highest ceiling, lowest complexity, no streaming required. If your graphs are wide, stop here. You may have collected the entire win.
3. Add streaming speculation only if E/D is still near 1 after step 2, and your remaining calls are serially dependent so parallelism can't touch them. That's the narrow band where it pays.
4. Key promises by call site and occurrence, always. Cheap, and it's the difference between a latency win and a silent change to your sampling distribution.
5. Cap in-flight speculation, shed under load, and track hit rate as a cost metric. An uncapped speculator degrades the shared engine precisely when you can least afford it.

None of this argues against the post. It's a good one, the implementation is small and readable, and the two design decisions in Fig 2 are worth stealing whatever else you conclude. The claim I'd add is that the interesting result isn't the speedup. It's that a harness executing model-written code has quietly become a runtime, and it inherits every scheduling, caching, and backpressure question runtimes have always had. Speculation is the first of those to arrive. It won't be the last.

That is the argument. Part 2 turns it into a calculator: give it your token rate and your backend's tool latency and it tells you which half, if either, is worth building for your workload.