What survives when a coding agent's conversation is cut off mid-task?
SpecSnap gives an agent a batch of new requirements, stops the session right after its first .py source edit (before that edit is committed), and starts a fresh session in the same directory with no memory of the earlier conversation.
Benchmarks like SWE-bench score whether an agent solves a task. SpecSnap scores what survives when context is compacted or reset partway through, and equally tracks where memory scaffolds fail or waste tokens.
AGENTS.md wording: Claude kept 10/36 (original Rule 4 overwrote Wave 1), Gemini 33/36 (had the detail-preservation clause), and Grok 25/36 (post-2d82680 in-place Rule 4).
git diff were caught (7-8/8), but amending a 1-cent tax bug into HEAD beat all 4 runs (0/8). A per-module SHA-256 check (Gate I) caught it 4/4.
pallets/click and pycqa/isort, unhinted memory arms spent 2.1-5.2× Vanilla's tokens on Claude (1.1-2.0× on Grok) for 0-1 extra bugs fixed. Memory keeps what you know; it doesn't find unknown bugs.
02 Results on the Synthetic Tiers (Tiers 1-4)
One run per cell across Gemini 4 Argon, Claude Opus 5.5, and Grok Build 4.7. Click any row to inspect the exact disk state at cutoff and run notes.
a Gemini Tier 3 Vanilla: 1 requirement passed raw, but the run broke original baseline behaviour and 1 module boundary, setting net retention to 0.
b Claude Tier 3 AGENTS.md Only (14/24): Not caused by AGENTS.md. Session 1 wrote a 230-line patch script (_patch1.py) covering 7 packages and was cut off as it created it; Session 2 found the script and ran it.
c Gemini Tier 4 Tuned reference (30/36): A variant of the scaffold with a 500-token limit on state.md.
d Tier 4 State.md (Claude 10/36, Gemini 33/36): Both ran with "Write ONLY the newly requested sprint requirements" in AGENTS.md, which led Claude Session 2 to overwrite Wave 1; however, Gemini's Tier 4 State.md sandbox also included the parenthetical "while preserving exact parameter names, default values, formulas, inequalities, return shapes, and ordering rules" from frontier_vibe_runner.py, which Claude's did not.
e Grok Tier 4 State.md (25/36): Ran after commit 2d82680 with the updated CLI Rule 4 (in-place wave preservation) and with git init deferred until after Session 3.
f Grok Tier 4 Tuned reference (0 / 33): First scaffold version passed all 36 raw, but 1 boundary violation zeroed the net score. Second version scored 33/36 with zero violations.
inject-poison-pill runs between Session 1 and Session 2 on Tier 2 (tier_2a_fintech). It plants two defects next to Session 1's valid uncommitted ledger/rules.py edit, and both still pass the visible tests:
- Silent regression in
billing/tax.py: a 1-cent tax error on amounts over 2,500 cents (raw_tax - 1). - Architectural violation in
rbac/rules.py: a forbiddenfrom rbac import auditimport violatingwiring.yaml.
| Variant | Model & Arm | Prompt | Raw | Net (of 8) | TES | Tokens |
|---|
When defects appear in git diff (v1 & v2), they get caught with or without marking comments: Session 2 diffs before editing and checks uncommitted hunks against state.md and TASK.md.
When defects are committed into HEAD (v3), every run without Gate I scores 0/8. Nothing in the diff, notes, or sprint pointed at billing/tax.py, so the 1-cent tax error shipped. (The rbac/rules.py import was still caught because the sprint required editing that file.)
Gate I fixes v3 under conditions (4/4 runs passed, 8/8 net): recording a SHA-256 hash of each module in app_map.md at setup and comparing after a reset flagged DRIFT on both files, including in the like-for-like control using the original neutral Session 2 prompt. Gate I was designed after seeing v3 and relies on the hash manifest itself not being tampered with.
To test whether synthetic retention gains carry over to open-ended bug-finding on real codebases, Claude Opus 5.5 (low effort) and Grok Build 4.7 (low effort) were run across two sessions on pallets/[email protected] (13 hidden bugs) and pycqa/[email protected] (11 hidden bugs taken from later upstream issues).
- No arm lost earlier work across the reset: each fix on disk is self-contained, so these tasks mainly measure search/bug-finding rather than cross-session specification memory.
- Without hints, memory arms cost 2.1-5.2× Vanilla's tokens on Claude (1.1-2.0× on Grok) for 0-1 extra bugs: an open-ended audit backlog in
state.mdkept Session 2 auditing modules without knowing what hidden tests check (and partially hinted Click v2 cost 6.2× on Claude).
| Repository | Arm | Bugs Resolved | Predicted | Turns (S1+S2) | Total Tokens | vs Vanilla |
|---|
Click v3 (3/13) is not evidence that the scaffold finds bugs on its own: its state.md backlog was written by hand and named 6 of the 13 bug areas (Option.resolve_envvar_value, argument decorator, write_usage, ZshComplete, Path.convert, echo streams).
On isort (0/11 across all arms): even though app_map.md's comment-handling and path-resolution tags pointed at modules holding 6 of the 11 subtasks, knowing the module wasn't enough when the target behaviour came from later upstream issues not specified in the 5.10.1 repo.
Pre-run predictions were too optimistic: Click predicted 1 / 1 / 3 / 5 vs actual 1 / 1 / 2 / 3; isort predicted 1 / 1 / 2 vs actual 0 / 0 / 0.
05 How a Run Works & Evaluated Configurations
Exact prompts, cutoff mechanics, and scoring definitions from EXTERNAL_AGENT_PROTOCOL.md.
Sprint Specification
The agent gets a codebase and a sprint of new requirements in the prompt. On Tiers 2-4 the baseline codebase already passes its visible tests; on Tier 1 the agent first implements and commits baseline TASK.md before starting the product-note requirements.
Cleaved After First .py Edit
The session is cleaved right after the agent's first .py source edit for the new requirements, before that edit is committed. Everything written to disk (uncommitted edits in the working tree, plus state.md if updated) stays as it is; nothing is reset.
Zero-History Continuation
A fresh session starts in the same directory with a neutral prompt: "Continue from the current workspace state, complete all remaining requirements across the packages, verify all tests pass, and commit your work." Tier 4 repeats this three times with Wave 2 and Wave 3 amendments.
Raw, Net & TES Scoring
Raw retention: hidden requirement tests passed. Net retention: raw retention, zeroed if the agent broke original hidden baseline tests, edited protected tests, or broke a wiring.yaml boundary rule. TES: net × min(1, B / tokens).
| Configuration (Arm) | Status | Files Added Before Session 1 | Key Empirical Behaviour Observed |
|---|---|---|---|
| Vanilla | Public baseline | None | Retains only the partial edit left in the working tree (0-2 on T3, 0-4 on T4); S2 reverted S1's uncommitted edit in 3/7 Claude runs and 6/8 Grok runs without notes. |
| AGENTS.md Only | Public baseline | AGENTS.md with general rules: commit as you go, and after a reset don't assume passing tests mean the work is done |
"Commit as you go" fails because the cutoff lands right after the first .py edit before a commit happens. Keeps 0-4 on T4. |
| State.md | Public baseline | A state.md scratchpad (≤ 500 tokens), plus an AGENTS.md telling the agent to write new requirements into it before editing code and check it after a reset |
Recovers most requirements (8/8 on T2, 17-23/24 on T3). Vulnerable to one-line note compression (Claude T3: 19/24) and wave overwrite on T4 under original Rule 4 (Claude T4: 10/36 vs Grok T4: 25/36 under post-2d82680 Rule 4). |
| Tuned reference | Private scaffold (Held_Out_Reference) |
More detailed AGENTS.md and state.md (≤ 500 tokens on T1-T2; ≤ 1,200 tokens on T3-T4; verbatim constants/operators + in-place wave amendments), plus AST-generated app_map.md (with SHA-256 hashes in Gate I) |
Reaches 24/24 on T3 across all 3 models and 30-36/36 on T4, plus 4/4 on Gate I v3. Not held out: its rules were written to fix the failures seen in State.md runs. |
06 Methodological Limitations & CLI Reproduction
Explicit caveats documented in README.md and commands to evaluate your own agent harness.
# 1. Create a clean, git-initialised sandbox outside the repo (arms: Vanilla, AGENTS_MD_Only, State_MD_Only) python -m sdcs_bench.cli scaffold-sandbox --tier 3 --arm State_MD_Only --dest /tmp/specsnap_t3_state # 2. Run your agent's sessions in that directory following EXTERNAL_AGENT_PROTOCOL.md # (Cleave Session 1 right after the first .py source edit, before it is committed) # 3. Optional: plant the Tier 2 regressions between Session 1 and Session 2 python -m sdcs_bench.cli inject-poison-pill --tier 2 --sandbox /tmp/specsnap_t2_state # 4. Grade the finished sandbox: raw retention, net retention, boundary violations, and TES python -m sdcs_bench.cli evaluate-sandbox --tier 3 --sandbox /tmp/specsnap_t3_state --tokens 1107107 --json-out /tmp/scorecard.json # 5. Print the recorded benchmark tables (table, markdown, or json) python -m sdcs_bench.cli leaderboard --format markdown