Benchmarks

Headline figures, measured on AWS c7g.xlarge against the same scripted model behind every subject:

The comparison charts against other agent runtimes are in the README, and the harness that produced them is in tools/bench — one subject at a time, each driven through its own public API, so the numbers can be re-run rather than trusted.

What the benchmark measures

The engine, not a model. It drives the real HTTP and SSE paths with an instant scripted provider and an in-process echo environment, so no model latency reaches the numbers. First-token and turn figures include HTTP, request construction, writing to the session log, and dispatch.

Density and reclaim measurements need Linux /proc/*/smaps_rollup. Other platforms run the portable latency and correctness arms only. The harness refuses to substitute RSS for private memory, because that would double-count shared pages. Any probe a subject cannot honestly answer is recorded as a refusal rather than a number.

What CI enforces

Every push bounds resident memory against a live server: after 10,000 requests it must stay under 256 MiB and must not have grown by more than 16 MiB.

The same push bounds journal growth on both axes. A context grows with every decision and across every turn, so anything written per decision or per turn would cost the sum of every intermediate size rather than the final one; crates/brain/tests/journal_growth.rs holds the journal — and one page of the event stream — to a small constant multiple of the final context, and holds a whole-transcript ratio so a session's log cannot grow with the square of its turn count.

History

Figures from before the current harness are archived in BENCHMARKS.md and should not be quoted as present-day performance.