Speculative tool calling · Part 2 of 2

Does speculation pay?

You don't need a running model to find out whether speculative tool calling is worth building. You need your token rate and your backend's tool latency: two numbers you already have. Everything below follows from them.

Best achievable

–

…

Your numbers

Decode rate tok/s Single-stream output rate for the model writing the program.

Tokens per turn Thinking plus the program it emits, per turn.

Tool calls per turn

Tool latency p50 ms One call, measured at your backend.

Program shape

Parallelism width How many of those calls are independent of each other. 1 = a strict chain.

Speculatable share % Share whose arguments resolve without touching I/O. Recoverable by parsing logged programs.

Wasted speculation % Share of speculated calls launched but never used: branch not taken, cell errored, program rewritten. Costs backend capacity, returns nothing.

Scale

Concurrent turns

Backend capacity Max tool calls your backend serves at once before latency starts rising.

Decode Ds

Tool time Es

E / D

Turn nows

Turn bests

Wasted calls/turn

AWhere your turn goes

BStreaming overlap is capped at 2×, and here is your point on it

CDoes it still pay at your scale?

The reasoning behind every formula here — why streaming overlap is capped at 2×, why parallel dispatch is not, and what the shadow REPL costs to build — is in Part 1: The Speculation Ceiling.