Benchmarking Strands Decider: multi-question latency and prefix caching

Decision models evaluate typed predicates directly without autoregressive text generation. A core architectural argument for this class of model is amortized multi-question evaluation: encoding a document once, then scoring multiple questions in parallel against the cached representation.

To evaluate how this scales in practice, I benchmarked Strands Decider 2B across 1 to 128 questions over a single document, comparing its shared-prefix cache against a naive re-encoding baseline. The benchmark code and data are available on GitHub.

Benchmark setup

• State: A 2,201-token synthetic service specification covering 48 services, each detailing route, authentication, rate limits, ownership, and latency targets.
• Questions: 128 typed yes/no questions (NoulQuestion) evaluating service properties.
• Paths tested:
  • Shared prefix (use_prefix_cache=True, default): Encodes the state once per batch (default max_batch=32), broadcasting the cached key-value states to evaluate question suffixes.
  • Naive re-encode (use_prefix_cache=False): Concatenates state and question for each query, re-encoding the full prompt every time.
• Metrics: Median wall-clock latency across repeated runs and input token counts reported by the model engine.
• Environment: Apple M3 Pro (MPS backend). The hybrid torso runs its reference implementation because causal_conv1d is not installed, so absolute latencies are conservative. The token model and relative ratios remain hardware-independent.

Latency and token scaling

Token accounting

The observed token usage follows a predictable piecewise model:

naive(n)  = n * (state + question)
          = n * 2,278

shared(n) = ceil(n / 32) * state + n * question
          = ceil(n / 32) * 2,201 + n * 77.5

The shared-prefix path eliminates redundant re-encoding of the 2,201-token state for every question within a batch. At n = 8, this reduces processed tokens from 18,296 to 2,826 (a 6.5x reduction), which directly translates to a 6.7x reduction in wall-clock latency (2.73 s vs. 18.41 s).

Marginal cost per question

While prefix caching provides substantial throughput gains, marginal questions are not free:

1. Within a batch: Adding a question costs approximately 90 ms in the shared path versus approximately 2.3 s in the naive path (a 25x difference). Each suffix must still attend across the 2,201-token cached state, so execution time grows with question count even before re-encoding.
2. Across batch boundaries: The engine enforces a batch ceiling (max_batch=32) to bound memory during KV-cache broadcast. When crossing batch boundaries (n = 33, 65, 97), the state must be re-encoded for the new batch. This produces step increases in latency at 32, 64, and 128 questions.

Conclusion

The multi-question efficiency of decision models is real and significant: 6.7x speedup at 8 questions, scaling to more than 15x asymptotically as question count increases. However, the performance profile is determined by batch boundaries and attention over the cached prefix rather than zero-marginal-cost scaling. Systems relying on high-volume evaluation batteries should align query batches with the engine's batch limit.

All benchmark scripts, raw JSON outputs, and plotting code are available at swopnilnep/strands-decider-benchmark.