Benchmarks
Methodology & harness
This directory contains 25 self-contained candidate cases across three difficulty tiers: 7 easy, 7 medium, and 11 hard. They model real operational workflows but are independently constructed, not verified reproductions of specific upstream issues. They use Python's standard library so Latch can focus on debugging rather than downloading dependencies. Difficulty labels describe the intended reasoning and state surface; they have not yet been calibrated by repeated model runs.
The separate supervised pytest #14998 case study records a real upstream repair, repeated exploration at a 12k history budget, and a 128k CLI rerun with independent compatibility checks.
| Tier | Case | User situation | Independent acceptance |
|---|---|---|---|
| Easy | stream_records |
A log collector receives UTF-8 JSON lines in arbitrary network byte chunks. | Every byte boundary, multiple records, final line and empty reads. |
| Easy | csv_chunks |
A CSV importer receives quoted rows across network reads. | Embedded commas/newlines and all split positions. |
| Easy | path_rules |
A file picker applies ordered ignore and reinclude rules. | Last matching rule wins. |
| Easy | ansi_width |
A terminal table aligns colored Unicode cells. | ANSI controls, combining marks and wide glyphs. |
| Easy | option_values |
A deployment CLI handles explicit empty values and --. |
Value presence, positional parsing and errors. |
| Easy | header_merge |
An HTTP gateway aggregates repeated headers. | Case-insensitive names and separate cookies. |
| Easy | duration_units |
A scheduler parses millisecond and larger time units. | Units, zero and malformed inputs. |
| Medium | config_layers |
A CLI combines defaults, a project file, environment and flags. | Nested precedence, empty and false values, caller input ownership. |
| Medium | pagination_cursor |
An activity feed paginates while earlier records may be deleted. | Stable keyset cursor and invalid input. |
| Medium | http_ranges |
A download endpoint serves byte ranges. | Inclusive, suffix, open-ended and invalid ranges. |
| Medium | retry_policy |
An HTTP client retries transient responses. | Method safety, Retry-After and retry budget. |
| Medium | archive_paths |
A document importer extracts user ZIP files. | Traversal/symlink preflight and normal extraction. |
| Medium | sqlite_migration |
An address book upgrades its SQLite schema. | Preserved rows, idempotence and malformed schemas. |
| Medium | log_rotation |
An operations daemon rotates bounded log archives. | Archive order, retention and invalid limits. |
| Hard | spool_recovery |
A job runner resumes an append-only spool after failure or interruption. | Retry, incomplete line, UTF-8 byte offset and stale checkpoint. |
| Hard | lease_queue |
Workers reclaim expired jobs. | Stale acknowledgements, attempts and concurrent claims. |
| Hard | webhook_dedupe |
A receiver stores durable event claims. | Handler failure rollback and duplicate delivery. |
| Hard | dag_scheduler |
A build scheduler executes dependent tasks. | Ordering, shared dependencies, cycles and failures. |
| Hard | atomic_config |
A daemon writes JSON settings and a backup. | Invalid writes, recovery and corrupt copies. |
| Hard | incremental_sync |
A backup tool mirrors a source directory. | Content hashes, deletion, unchanged files and links. |
| Hard | stream_framing |
An RPC server decodes length-prefixed messages. | Arbitrary chunks, multiple frames and size limits. |
| Hard | cache_stampede |
Concurrent callers share slow cache loads. | Single flight, retry after failure and key independence. |
| Hard | append_index |
An event log rebuilds byte offsets after partial writes. | UTF-8 offsets, hidden tails and append refusal. |
| Hard | rate_window |
A gateway limits requests per tenant. | Rolling boundaries, isolation and denied requests. |
| Hard | transaction_outbox |
An order service persists rows and outgoing events. | Atomic rollback, duplicates and event order. |
Each workspace/ contains the buggy project and visible tests. Acceptance
checks stay outside the copied workspace in acceptance.json or check.py.
The shared checker runs before and after the agent in a networkless Bubblewrap
sandbox. That sandbox exposes only the selected case and workspace read-only,
hides the host home directory, and gives checks private scratch space. The
runner requires at least one failing baseline check and never modifies the
case template. Every case has a reference/ repair for author verification;
the runner never copies it into Latch's workspace. An isolated Git repository
and Latch state directory are created for every attempt.
Validate the complete case set without a model call:
python3 benchmark/verify_cases.py
This checks that each baseline fails at least one independent check and that the reference repair passes all independent checks and visible tests.
Run¶
Linux requires the usual Latch prerequisites (bwrap, rg, Git) and a working
DeepSeek v4.1 Flash profile. Set the credential environment variable named in
your Latch configuration. The runner copies that configuration and redirects
state to a private run directory; it does not copy secret files. If your
configuration has an explicit top-level state_dir, provide a version without
that setting. Check prerequisites with latch doctor first.
cargo build --release --locked
python3 benchmark/run.py list
python3 benchmark/run.py run --case stream_records \
--config ~/.latch/config.toml
Offline runner checks (no credentials, network, or sandbox needed):
python3 benchmark/test_runners.py covers report-directory handling, and
python3 benchmark/verify_cases.py checks the candidate cases with Bubblewrap.
For a legacy installation, pass its ~/.config/latch/config.toml instead.
Repeat --case to select several cases, or use --case all. The runner pins
the CLI invocation to provider opencode-go, model deepseek-v4.1-flash, and
WORK mode. The source config must define that provider and its credential.
Each case has a wall-clock timeout; a timeout terminates the CLI process group.
Reports go to ignored benchmark/runs/<timestamp>-<pid>/. Each case contains
result.json, CLI stdout/stderr, a Git patch including new files, an isolated state directory,
and the final workspace. summary.json aggregates case results. Correctness
requires every independent check to pass, a successful CLI exit, and the
requested provider/model. completion is recorded separately, since a passed
external acceptance suite and Latch's own Verified state answer different
questions. Event counts are read from the root session's durable SQLite log;
child-agent activity remains reflected in the CLI graph usage totals.
Token usage and cache reads come from the provider's reported graph usage.
Cost is null unless --pricing supplies a dated JSON price snapshot with
input_per_million, cached_input_per_million, and output_per_million rates
in USD. Example:
{
"input_per_million": 0.0,
"cached_input_per_million": 0.0,
"output_per_million": 0.0
}
Replace those placeholders with rates valid for the run. Reports record the
exact rates used. Timing currently measures end-to-end CLI wall time, including
model and tool waits; it does not attribute individual phases. Only the
original stream_records and config_layers cases have been smoke tested
with a real model. The full 25-case set needs repeated runs and difficulty
calibration before it can support a comparative coding-agent score.
OpenCode comparison¶
The optional local comparison runner uses the same prompts, pristine case
workspaces and independent evaluator. It requires an installed opencode,
Bubblewrap and OPENCODE_GO_API_KEY (use the same credential as Latch):
python3 benchmark/run_opencode.py --case spool_recovery \
--case stream_records --case cache_stampede --jobs 3
It starts private OpenCode servers with the build agent and the same
opencode-go/deepseek-v4.1-flash model. The entire OpenCode process runs in
Bubblewrap with the host home hidden, isolated state and a writable attempt
directory; independent acceptance checks and reference repairs are not exposed
to the agent. Provider network access remains enabled. Each attempt retains the
case's existing timeout. Reports live in ignored benchmark/runs/opencode-*.
Credentials and databases stay local and must not be included in exported
reports. Concurrency is capped at three.
The runner normalizes OpenCode's cached input into total input. OpenCode v2
separates text/tool output and reasoning; their sum is the comparable output
count. Latch already reports total input and total output. Costs remain null
without a dated price snapshot. Compare external acceptance rather than
equating OpenCode termination with Latch's kernel Verified state. Single
attempts on three cases are exploratory, not an overall agent ranking.
For a full 25-case comparison, use one scheduler for both agents:
python3 benchmark/compare.py --latch-config ~/.config/latch/config.toml --jobs 3
python3 benchmark/report_comparison.py benchmark/runs/comparison-<stamp>-<pid> \
benchmark/reports/<report-name>
The scheduler randomizes paired case order with a recorded seed and limits
total concurrency across both agents to three. It also requires GNU
/usr/bin/time and records CPU time and maximum single-process RSS (including
waited-for descendants, not aggregate process-tree memory).
OpenCode's private server is incompletely accounted by this launcher-level
measurement; raw CPU/RSS logs are retained but cannot compare agent resource
usage. A complete process-tree sampler is needed for that comparison. Source
and binary hashes, case hashes, scheduling order and progress are retained locally. A
failed attempt is recorded without silently retrying it. Model tokens for
interrupted Latch runs can be recovered from durable events, but unfinished
provider requests may remain unreported.
The exporter requires a dated deepseek-pricing.json snapshot in the run
directory with official sources and separate cache-hit, cache-miss and output
rates. It reports peak/off-peak USD/CNY estimates, averages including failed
attempts, cost per successful case including failed-attempt spending, tier
statistics and per-attempt CSV data. These estimates apply direct DeepSeek API
prices to observed OpenCode Go usage and are not actual Go invoices. Keep
original configs and SQLite databases private; the exporter checks credential
values before publishing compressed event records and logs.
Publication tables¶
benchmark/render_tables.py renders the recorded 6 October 2026 comparison
from its committed metrics.json into Times New Roman tables. It makes no
provider requests. Install Matplotlib in a separate environment, then run:
python benchmark/render_tables.py --font-dir /path/to/Times-New-Roman-fonts
The font directory must contain actual Times New Roman times.ttf and
timesbd.ttf; on WSL the default is /mnt/c/Windows/Fonts. Fonts are not
distributed in this repository. Outputs in benchmark/figures/2026-10-06/
include SVG with outlined text for GitHub display, PDF with embedded font
subsets for print, and 240-dpi PNG. The original tables remain available as downloads.
python benchmark/render_charts.py --font-dir /path/to/Times-New-Roman-fonts
uses the same optional dependencies to render the per-task time, total-token
and estimated off-peak-cost grouped bar charts from the committed
per-attempt.csv. It checks the 25 task pairs and token/time/pass aggregates
against metrics.json. All SVG, PDF and PNG charts have transparent backgrounds;
light/dark variants change text and grid colors. The root README uses a
picture element to choose SVG text colors for the viewer's theme. Hatched bars
identify failed external acceptance checks, and all attempts are included.
Working-history threshold calibration (Linux, opt-in paid)¶
run_thresholds.py reproduces the supervised pytest #14998 task from a preserved
local experiment directory containing repo/.case, repo/.venv,
PROMPT.md, config.toml, and the independent audit/test_edges.py. It reads the
original checkout and archives the pinned public baseline into fresh workspaces;
it never edits the original experiment or uses .references/. Python 3.11+,
Bubblewrap, Git, the release Latch binary and the copied development environment
are required. The config must resolve the same authorized model credential.
python3 benchmark/run_thresholds.py --source /path/to/pytest-14998-experiment \
--thresholds 32000,64000,96000,128000 --jobs 2 --live
python3 benchmark/analyze_thresholds.py benchmark/runs/thresholds-<timestamp> \
--source /path/to/pytest-14998-experiment --output /tmp/threshold-analysis.json
Only context.recent_tokens and each isolated state path change. --live is
mandatory: this calls a paid provider. The default per-attempt ceilings are 1,800
seconds, 5 million provider input tokens (including cache reads) and 150,000 output
tokens. Monitoring samples every 30 seconds, so an in-flight request can overshoot
a token ceiling. Up to three attempts can run concurrently; default concurrency
is two. --trials repeats a sweep. Token ceilings and timeouts produce censored
attempts, not completed-task timings.
The runner first confirms the baseline fault, then records the agent's edit latency, rotations, exact repeated reads/searches/shell outputs, cache usage and completion. Independent checks run the three-policy verifier, focused upstream tests, and lifecycle compatibility edges including call-phase skip retention. The analyzer also checks that auxiliary case files were not altered and reports pre-edit repeats separately. Repeated shell outputs may reflect necessary revalidation; they are not automatically classified as wasted work.
Raw configurations, state databases and transcripts stay in private ignored
benchmark/runs/. Publish aggregate reports only. Cost scenarios use the dated
DeepSeek direct-API rates recorded in the comparison report; they are hypothetical
prices applied to Go token usage, not actual OpenCode Go invoices. A single case
or one sample per threshold cannot establish a universal optimum; repeat the
promising candidates and account for independent patch correctness.
LSP model-usability pilot (Linux / WSL, opt-in and paid)¶
run_lsp.py exercises real language-server tool use with the configured OpenCode
Go account and deepseek-v4.1-flash, in disposable workspaces with private state.
It requires Python 3.11+, Git, Bubblewrap, clangd, a C++ compiler and a release
Latch binary. No language server is downloaded. Three scenarios cover overload
navigation, diagnostics plus guarded repair/validation, and a missing-server
fallback. Prompts describe semantic tasks rather than scripted RPC arguments.
# Offline runner/acceptance checks; no provider traffic:
python3 benchmark/test_lsp.py
# Paid: explicitly select your existing configuration and secrets file:
python3 benchmark/run_lsp.py --live --source-config <config.toml> --secrets <secrets.toml>
# Rerun only an interrupted scenario:
python3 benchmark/run_lsp.py --live --source-config <config.toml> --secrets <secrets.toml> --case missing_server_fallback
The default provider id is opencode-go; --provider selects another configured
instance of that family. The source credential is resolved in memory and passed
through a dedicated environment reference, never copied into generated config or
reports. File credentials require --secrets if the source has implicit state
paths; environment credentials need no secrets-file argument. Existing user
config/state is never modified. Language-server processes receive the backend's
minimal environment, not the provider credential.
Each case has a 240-second wall limit and at most 12 model turns, with retry budget
one; transient retries may issue additional requests. There is no monetary price
ceiling. Independent repair acceptance runs a hidden C++ harness in a separate,
offline, credential-free sandbox and checks additional integer/floating inputs.
Navigation acceptance also requires a user-facing answer mentioning the concrete
file, integer symbol/type and relevant lines, rather than just a completion report.
This is an answer-presence heuristic, not a semantic answer grader; review final
answers and tool traces before publishing results.
Results, configs, databases and traces stay in private ignored benchmark/runs/lsp;
publish only reviewed aggregate evidence. One run is a usability qualification,
not a success-rate or agent-ranking benchmark. See LSP design and limits.