SWE-MARATHON / EXPLORATORY STUDY 01
LoopX keeps agents verifying after they say ‘done.’
This revision retains exploratory results for plain, native goal, and LoopX Turn on 15 tasks; the zstd case illustrates verification after visible tests pass.
00 / Core result
Among the three retained arms, LoopX Turn had the highest score and cost; repeatable gains remain unproven.
GPT-5.6 Sol · high reasoning · 15 matched tasks · 1 trial per task-arm · 30% timeout budget. The table combines treatment, outcome, cost, and the decision-relevant read.
| Arm | Continuation owner | Reward | Partial | Cost | Continues | Decision read |
|---|---|---|---|---|---|---|
plainPlain Codex | Model session | 0.267 | 0.710 | $368 | 0 | Lowest-cost baseline; stops earlier |
goalNative Goal | Codex Goal | 0.267 | 0.767 | $533 | 7 | Goal alone improves partial, not binary reward |
heartbeatLoopX Turn (external-scheduler automation) | External driver | 0.333 | 0.778 | $830 | 42 | Highest reward and partial; highest cost |
Runner noteCorrection: SSH Goal and Codex CLI group data and conclusions have been withdrawn pending validation; no capability conclusion is drawn about either mode. heartbeat is LoopX Turn driven externally through generic_cli, not a real Codex App automation host.
01 / Why this benchmark
Long enough to expose the work after ‘done.’
SWE-Marathon v1.1 contains 20 realistic, multi-hour software tasks. It combines visible and hidden tests, adversarial checks, and, for product clones, computer-use verification. This study used 15 non-GPU tasks so every arm shared one matched denominator.
02 / Statistics & mechanism insight
The zstd case: verification continues after self-reported completion.
The previous cross-arm 14/14 coverage claims and behavior charts are also withdrawn, not reinterpreted as heartbeat-only results. The mechanism below is limited to the retained zstd case and needs repeated validation.
Reject premature exit
A model-level ‘done’ becomes evidence, not the final authority.
Re-anchor the next turn
The runner rereads current worktree state instead of trusting the prior narrative.
Activate falsification
Audit, sanitizer, fuzz, regression, and edge-case work become the next action.
Convert hidden tails
Where visible success masks defects, additional verification can become score.
Completion is a control-plane decision
Treat self-reported completion as a checkpoint requiring verification and settlement, not an unconditional stop signal.
Self-verification is a leading indicator
Track verification events per step alongside score and cost. It explains whether added horizon is buying quality.
Stateful control, replaceable executors
A promising next test is durable LoopX state with ephemeral executor sessions, so each turn starts clean without losing project intent.
03 / Case analysis
zstd-decoder: the visible finish line was not the real one.
With the same model configuration and timeout budget, this plain run stopped after visible tests passed; heartbeat continued into protocol acceptance and memory-safety checks. A single comparison does not rule out sampling variation.
Declares RFC 8878 complete
6 visible fixtures pass
hidden 25/37 · partial 0.72Builds a 156-check RFC acceptance suite
audits memory safety; fixes offset-table and Huffman underflow edges
hidden 37/37 · partial 1.00This is a mechanism-revealing case, not proof that more turns always win. Many tasks were already saturated; on those tasks the extra loop tied rather than improved the baseline.
04 / Reading against official results
Official Max is higher. The experiment is not directly comparable.
The gap mixes reasoning effort, task composition, runner reliability, time budget, and trial count. It is not evidence that LoopX—or the underlying model—regressed.
| Dimension | This study | Official reporting | Interpretation |
|---|---|---|---|
| Model / reasoning | GPT-5.6 Sol · high | GPT-5.6 Sol High: 30.0% · Max: 42.5% pass@1 | Max has a larger reasoning budget; only Max exceeds this study's 33.3% best arm. |
| Task set | v1.1 · 15/20; 4 H100 + 1 A100 task excluded | v1.1 · 20/20; all GPU tasks included | The denominator and task mix differ. |
| Runtime | Research harness; three retained arms | Standardized official runner | Protocols differ; withdrawn-arm build failures are not used to explain this comparison. |
| Budget / repeats | 30% timeout; one trial per task-arm | Full task windows; eight trials per configuration | The official setup has a longer horizon and lower sampling variance. |
| Metric | Binary reward + descriptive partial | Official pass@1 | Partial is uncalibrated within-task progress; do not compare it to pass@1. |
How to read itBottom line: use this study to compare matched runner arms and inspect mechanism—not as a replacement leaderboard score. The official v1.1 leaderboard is the right external reference, but its published High/Max results follow a different protocol.
05 / Evidence boundary
Exploratory results, not a capability verdict across modes.
- SSH Goal, Codex CLI, and comparisons depending on them are withdrawn; raw records are retained and excluded from public results pending validation.
- One trial per task-arm cell; no confidence intervals or run-to-run variance estimate.
- Several mechanism variables move together across arms: prompt body, host transport, turn boundary, and recovery policy.
- The 15-task subset excludes five GPU-dependent tasks (four H100, one A100) and represents one benchmark family.
- Step and keyword-density measures are public-safe projections, not raw trajectory publication.
- Cost rose materially; the current data does not establish the best quality-per-dollar policy.
06 / Sources