Relative to Plain
+0.0731mean reward increase+17.3% relative mean gain
17 wins · 13 ties · 16 losses
≥95%: LoopX 15.2%, Plain 15.2%
>80%: LoopX 32.6%, Plain 26.1%
LHTB / LOOPX AND TWO BASELINES
What does LoopX accomplish beyond Plain and native Goal?
Across 46 tasks with the same model, compare three core metrics: mean reward, strict success rate (≥95%), and success rate above 80%. LoopX Heartbeat leads both baselines on mean reward and the 80% threshold, while matching Plain on strict success.
00 / Three core metrics
Higher mean reward than Plain and native Goal.
LoopX averages 0.4948, Plain 0.4218, and native Goal 0.4475. At reward ≥ 0.95, they solve 7, 7, and 4 tasks respectively; above 0.80, the counts are 15, 12, and 14. All three metrics use the same 46 tasks.
| Execution | Mean reward | Success ≥95% | Success >80% |
|---|---|---|---|
Plainbaseline · app-server | 0.4218 | 15.2%7/46 | 26.1%12/46 |
Native Goalbaseline · app-server | 0.4475 | 8.7%4/46 | 30.4%14/46 |
LoopX HeartbeatLoopX 1.0.3 · generic_cli | 0.4948 | 15.2%7/46 | 32.6%15/46 |
Mean reward preserves partial credit. Each success rate is the mean of binary rewards: 1 when the threshold is met, 0 otherwise. Counts below each rate show successes / 46. The ≥95% threshold retains the study’s original definition; >80% is a supplementary post-hoc view. No score is exactly 0.80 or 0.95, so > and ≥ give identical counts in this dataset.
Relative to Native Goal
+0.0473mean reward increase+10.6% relative mean gain
23 wins · 13 ties · 10 losses
≥95%: LoopX 15.2%, Native Goal 8.7%
>80%: LoopX 32.6%, Native Goal 30.4%
Deltas are computed from all 46 unrounded task rewards, then rounded for display. Relative gain = mean delta / baseline mean. Wins, ties and losses use strict comparisons of published rewards; equal values do not establish statistical equivalence.
Threshold sensitivity: at >80%, LoopX leads Plain by 3 tasks and native Goal by 1; its binary outcomes match Legacy Heartbeat on every task. One task is 2.17 percentage points. A lower threshold does not establish lower run-to-run variance: there is only one effective trial per task-arm cell.
How to read thisLoopX here means the study’s 1.0.3 Heartbeat with fresh exec, not a new evaluation of the latest release. Both baselines use app-server; LoopX uses generic_cli. Effective runs include replacement trials, some with longer limits. Historical USD is estimated; LoopX USD is recorded. This is not an equal-budget efficiency ranking.
Historical context: SSH-Goal and legacy Heartbeat
The other two arms remain available for tracing implementation history. The primary comparison throughout this page is this study’s LoopX Heartbeat against Plain and native Goal.
| Execution | Mean reward | Success ≥95% | Success >80% |
|---|---|---|---|
LoopX SSH-Goalhistorical · app-server | 0.4678 | 13.0%6/46 | 26.1%12/46 |
Legacy HeartbeatLoopX 0.5.3 · app-server | 0.4794 | 15.2%7/46 | 32.6%15/46 |
Token usage and cost · descriptive context
| Execution | Token usage | USD |
|---|---|---|
| Plain | 251.72M | $212.97 |
| Native Goal | 293.81M | $344.25 |
| LoopX Heartbeat | 505.10M in / 3.64M out | $551.73 |
| LoopX SSH-Goal | 402.86M | $425.50 |
| Legacy Heartbeat | 387.18M | $332.70 |
01 / The benchmark
LHTB grades durable artifacts after hundreds of dependent terminal actions.
Long-Horizon Terminal-Bench contains 46 containerized tasks across nine categories. Hidden verifiers grade artifacts or replayable outcomes on a continuous [0,1] scale; self-reported completion does not count. This study uses reward ≥ 0.95 as its solved threshold.
Official benchmark
Task taxonomy, leaderboard, methodology, and harness notes.
LHTB paper
Long-Horizon Terminal-Bench, arXiv:2607.08964.
What work do the nine categories cover?
Categories follow the official introduction page; the examples show the tools and deliverables involved. Official task taxonomy → · Task examples →
| Category | Representative work |
|---|---|
| Software & reverse engineering | Migrate an old LangChain application; debug a RISC-V core. |
| Scientific computing & simulation | Accelerate an N-body simulator; restore an airfoil simulation to regression tolerances. |
| Earth, climate & energy | Audit groundwater simulations and power-system optimization. |
| Multimodal & imaging analysis | Reconstruct data from scientific figures; audit medical images. |
| Research reproduction & ML | Reproduce the UNISON simulation experiment or a Foldseek paper result. |
| Systems, performance & security | Work on DuckDB optimization or construct an exploit proof of concept. |
| Interactive games | Play 2048 or Super Mario through sustained interaction. |
| APEX professional workflows | Complete investment-banking deliverables or a legal matter. |
| Logic & constraint puzzles | Search for a checkmate or solve a constrained state-space puzzle. |
For example, UNISON paper reproduction requires a runnable simulation pipeline and reports. Hidden checks examine metric fidelity, partitioning and scheduling, repeatability, and behavior under held-out seeds. Partial credit records which requirements the artifacts satisfy.
The official model comparison used Terminus-2 and a 90-minute budget. This page reports a separate LoopX five-arm study with its disclosed runtimes and replacement trials. Upstream tasks and verifiers continue to evolve; historical scores retain their original evaluation settings.
02 / Three execution settings
How does the goal persist, and who organizes the next step?
All three can use the same model and tools for long tasks. Plain advances within its session; native Goal persists the objective; LoopX also maintains work items and progress state to govern subsequent wakes. These are configuration differences, not independently established causes of the score gains.
Plain
Planning and execution within a sessionA normal app-server turn plans, uses tools, and returns. It can run for a long time, but has no native persistent Goal or LoopX control plane.
Native Goal
An objective that persists across turnsCodex maintains a “Finish the task” Goal for continued work. This arm does not use LoopX registry, Todos, or scheduling.
LoopX Heartbeat
Work state persists across executorsWorkspace, registry, and Todos persist across fresh codex exec wakes. Three qualifying advancement Todos trigger a review that can retain or revise the plan.
03 / LoopX insight
Which task types benefit, and where is the signal weak?
Research and puzzles have higher group means against both baselines. Multimodal analysis and specialist simulations show little separation; other groups need task-level inspection. Across all 46 tasks, the median paired delta is 0 against Plain and about 0.0003 against Goal: mean gains are concentrated, not universal.
These are analyst-defined, post-hoc groups based on task demands, not an official task mapping. Each of the 46 tasks appears once; groups have only 4–7 tasks. Δ is mean LoopX reward minus the baseline. Wins/ties/losses use unrounded scores and do not imply statistical significance. Select a group to inspect every member.
| Analysis group | Tasks | Δ vs Plain | Δ vs native Goal | Current signal |
|---|---|---|---|---|
| Research & modeling | 4 | +0.28024 / 0 / 0 | +0.21792 / 2 / 0 | All 4 beat Plain; ALP and Foldseek tie Goal. |
| Logic & spatial puzzles | 4 | +0.27393 / 0 / 1 | +0.13743 / 0 / 1 | 3 wins, 1 loss on each side; Sudoku trails Goal, Chess trails Plain. |
| APEX professional work | 4 | +0.10223 / 1 / 0 | +0.04073 / 0 / 1 | Positive means; law trails Goal, IB244 ties Plain. |
| Multimodal analysis | 6 | -0.00341 / 3 / 2 | +0.00583 / 3 / 0 | All 6 have absolute task deltas < 0.05 against both baselines. |
| Science & simulation | 7 | +0.01612 / 2 / 3 | +0.00993 / 2 / 2 | Similar means; median paired deltas are 0 against both. |
| Earth, climate & energy | 6 | -0.01082 / 2 / 2 | +0.03823 / 2 / 1 | No mean advantage over Plain; slightly ahead of Goal. |
| Software repair & toolchains | 6 | -0.01880 / 4 / 2 | +0.09912 / 3 / 1 | No wins over Plain: 4 ties, 2 losses; some gains are Goal-only. |
| Games & strategy | 4 | -0.01121 / 0 / 3 | +0.09713 / 0 / 1 | 2048 improves; Snake, Mario and Generals all trail Plain. |
| Systems, performance & security | 5 | +0.17321 / 1 / 3 | -0.14501 / 1 / 3 | PoC drives the gain over Plain; DuckDB drives the loss against Goal. |
Does one task drive the group mean?
Without Tabular, research still gains +0.1259 / +0.1111 over Plain / Goal. Removing any one puzzle leaves positive means against both. Systems is different: removing PoC changes its Plain delta from +0.1732 to −0.0065; removing DuckDB changes its Goal delta from −0.1450 to +0.0111. These exclusions are sensitivity checks, not an alternative leaderboard or confidence intervals.
Task structure is read from pinned upstream revision d78f5eb (accessed 2026-09-19). The study specifies only a July 2026 snapshot, without per-task source digests. Byte identity with evaluated prompts is unverified; these instructions explain task demands, not what happened in a trial.
Retained progress and repeated checks are promising conditions
Sokoban retains solved levels; Tabular requires experiments and an auditable model; Rush Hour asks for a route ledger and replay checks. These structures suit durable work state. Scores alone cannot attribute gains to Todos, fresh exec, or replanning.
Long workflows need feedback about correctness
APEX consulting beats both baselines, but law trails Goal. The law task requires late corrections and exact cross-document references; validate checks only the schema. Advancing the workflow does not ensure correct content. Persistent state still needs reliable acceptance.
Continuation has not shown a general capability lift
Multimodal tasks require segmentation, geometry/OCR and generalization; simulation tasks require correct solver use and physical constraints. Small differences do not establish that continuation resolves those bottlenecks. Perfect ties on LangChain and RISC-V also differ from the all-zero OpenRoad floor.
Test both progress and preservation
2048 retains peak score and improves, yet Snake also retains peaks and still trails Plain. DuckDB requires every query to be correct; optimization errors can zero the reward. Matched-budget repeats should isolate feedback quality, retained progress and best-version rollback.
Expand task structure and counterexamples (scores from the same data)
Gains and possible explanations
tabular-data-feature-covshift
- Plain
- 0.0000
- Native Goal
- 0.2047
- LoopX Heartbeat
- 0.7430
Sparse inverse recovery under covariate shift: recover a 200-dimensional signal from 50 observations, using experiments, validation history and an auditable model. Gains against both, but 0.743 is still below the solve threshold.
Task instructionssokoban
- Plain
- 0.2710
- Native Goal
- 0.5935
- LoopX Heartbeat
- 0.8903
155 sequential puzzles; reset the current puzzle while retaining solved levels. Ahead of both baselines, still below 0.95.
Task instructionsrush-hour-campaign
- Plain
- 0.5416
- Native Goal
- 0.5310
- LoopX Heartbeat
- 0.7716
Four long spatial-route puzzles with a ledger and replay checks. Programmatic search solvers are explicitly forbidden; the gain is not evidence of automated search-code generation.
Task instructionsunison-paper-reproduction
- Plain
- 0.3333
- Native Goal
- 0.1667
- LoopX Heartbeat
- 0.5000
Implement parallel simulation, partitioning and adaptive scheduling, with multiple artifacts and held-out seeds. Ahead of both baselines, but only 0.5.
Task instructionsCounterexamples and limits
apex-law433-matter
- Plain
- 0.4007
- Native Goal
- 0.8151
- LoopX Heartbeat
- 0.6130
70 stages with cross-document references and late corrections. Validate checks schema, not content correctness. Above Plain, below Goal: length alone does not guarantee gains.
Task instructionssnake-obstacle-campaign
- Plain
- 0.8415
- Native Goal
- 0.3639
- LoopX Heartbeat
- 0.6164
Reset and peak-score retention are available, yet LoopX trails Plain while beating Goal. Retaining progress is a candidate condition, not a sufficient one.
Task instructionsduckdb-optimizer-closure
- Plain
- 0.0000
- Native Goal
- 0.7697
- LoopX Heartbeat
- 0.0000
All 22 TPC-H queries must be correct; a wrong query can zero reward. LoopX and Plain score 0, Goal about 0.770. Terminal scores alone cannot locate the failure.
Task instructionsscientific-figure-data-reconstruction
- Plain
- 0.0400
- Native Goal
- 0.0150
- LoopX Heartbeat
- 0.0200
Reconstruct curves and tables from images, including axis calibration, line separation and numeric accuracy. Scores are close; no clear control-plane benefit is visible.
Task instructions04 / Complete task matrix
Inspect LoopX’s gains and losses against each baseline.
The default view shows three arms; blue marks the highest visible reward in each row. Filters select |LoopX − baseline| ≥ 0.05. Values display four decimals; comparisons use full precision. Historical arms can be expanded.
| Task | Plain | Native Goal | LoopX Heartbeat |
|---|---|---|---|
2048Games & strategy | 0.4109 | 0.5744 | 0.7082 |
alp-paper-reproductionResearch & modeling | 0.4000 | 0.5000 | 0.5000 |
apex-ib244-matterAPEX professional work | 0.9932 | 0.8836 | 0.9932 |
apex-investment-banking-matterAPEX professional work | 0.4366 | 0.4366 | 0.5634 |
apex-law433-matterAPEX professional work | 0.4007 | 0.8151 | 0.6130 |
apex-management-consulting-matterAPEX professional work | 0.4439 | 0.3850 | 0.5134 |
apex-openroad-ibex-signoffSoftware repair & toolchains | 0.0000 | 0.0000 | 0.0000 |
audio-visual-event-alignmentMultimodal analysis | 0.0278 | 0.0280 | 0.0280 |
chess-mateLogic & spatial puzzles | 0.2167 | 0.1167 | 0.2000 |
climate-netcdf-extreme-event-auditEarth, climate & energy | 0.0855 | 0.0855 | 0.0855 |
commit0-multilib-tddSoftware repair & toolchains | 0.9069 | 0.4311 | 0.8852 |
dicom-radiology-auditMultimodal analysis | 0.8858 | 0.8852 | 0.8852 |
document-table-layout-reconstructionMultimodal analysis | 0.0925 | 0.0725 | 0.0925 |
duckdb-optimizer-closureSystems, performance & security | 0.0000 | 0.7697 | 0.0000 |
epa-swmm-stormwater-regression-auditEarth, climate & energy | 0.0517 | 0.0517 | 0.1391 |
epidemic-inverse-control-auditScience & simulation | 0.1685 | 0.0293 | 0.1638 |
foldseek-paper-reproductionResearch & modeling | 0.7778 | 0.8889 | 0.8889 |
gdal-proj-raster-regressionEarth, climate & energy | 0.0346 | 0.0210 | 0.0384 |
generals-bot-arenaGames & strategy | 0.9750 | 0.9450 | 0.9550 |
grammar-fuzz-coverage-huntSystems, performance & security | 0.9470 | 0.9470 | 0.9451 |
great-expectations-auditSoftware repair & toolchains | 0.3636 | 0.3636 | 0.2727 |
langchain-version-migrationSoftware repair & toolchains | 1.0000 | 1.0000 | 1.0000 |
materials-phase-diagram-auditScience & simulation | 0.0902 | 0.0281 | 0.0674 |
matpower-opf-regressionEarth, climate & energy | 0.9167 | 0.9167 | 0.9167 |
microscopy-cell-count-qc-auditMultimodal analysis | 0.0200 | 0.0200 | 0.0200 |
modflow6-groundwater-regression-auditEarth, climate & energy | 0.4266 | 0.4266 | 0.3967 |
nbody-accel-iterativeScience & simulation | 1.0000 | 1.0000 | 1.0000 |
nrel-pysam-hybrid-renewables-auditEarth, climate & energy | 0.3485 | 0.0681 | 0.2221 |
opensees-seismic-structural-regression-auditScience & simulation | 0.0439 | 0.0285 | 0.0237 |
poc-exploit-craftSystems, performance & security | 0.0000 | 0.8921 | 0.8921 |
riscv-core-debugSoftware repair & toolchains | 1.0000 | 1.0000 | 1.0000 |
robotics-slam-benchmark-repairScience & simulation | 0.0313 | 0.0307 | 0.0313 |
rush-hour-campaignLogic & spatial puzzles | 0.5416 | 0.5310 | 0.7716 |
satellite-flood-change-detection-auditMultimodal analysis | 0.0400 | 0.0300 | 0.0400 |
scientific-figure-data-reconstructionMultimodal analysis | 0.0400 | 0.0150 | 0.0200 |
snake-obstacle-campaignGames & strategy | 0.8415 | 0.3639 | 0.6164 |
sokobanLogic & spatial puzzles | 0.2710 | 0.5935 | 0.8903 |
spice-ephemeris-regressionScience & simulation | 0.7879 | 0.8485 | 0.8485 |
spot-scheduler-tracesSystems, performance & security | 0.9802 | 0.9766 | 0.9633 |
su2-airfoil-regressionScience & simulation | 0.2000 | 0.4000 | 0.3000 |
sudoku-recoveryLogic & spatial puzzles | 0.2857 | 0.6199 | 0.5489 |
super-marioGames & strategy | 0.3528 | 0.2639 | 0.2558 |
tabular-data-feature-covshiftResearch & modeling | 0.0000 | 0.2047 | 0.7430 |
unison-paper-reproductionResearch & modeling | 0.3333 | 0.1667 | 0.5000 |
unknown-config-semanticsSoftware repair & toolchains | 0.2313 | 0.0000 | 0.2313 |
vector-db-iterative-buildSystems, performance & security | 1.0000 | 0.9330 | 0.9927 |
Showing 46 tasks
05 / Next experiments
Test whether the gains beyond the baselines reproduce.
This study provides outcome differences and mechanism clues. Align Plain, native Goal, and LoopX on environment and budget first, then isolate recovery, plan review, and rollback instead of changing several variables together.
Fix the comparison
Use the same model, task version, time limit, and accounting; repeat every task and report every attempt.
Separate mechanisms
Vary fresh/resumed sessions, Todo planning, and review cadence independently to identify what changes outcomes.
Measure gains and losses
Report mean reward, strict solves, per-task regressions, cost, and evidence of recovery and repeated work together.
Protect final delivery
Test independent acceptance and best-version retention so later optimization cannot erase correct results.
06 / Evidence boundary
A reason to investigate further, not proof of reliable gains at equal budgets.
- One effective run per task-arm cell, with no repeated seeds or confidence intervals. Win/loss counts are not significance tests.
- Model and reasoning effort match, but transport, session lifetime, and scheduling differ between LoopX and the baselines. No single mechanism is isolated.
- Effective aggregates include replacement trials, some with longer LoopX limits. This is not a strictly matched-budget comparison.
- Hidden evaluation grades the final result; continuation does not automatically reveal an unknown correct solution. Recovering work and solving the task remain distinct.
- Accounting differs: LoopX records $551.73, while Plain and native Goal estimate $212.97 and $344.25. A higher mean does not establish better efficiency.
07 / Sources
Public links and reproducible aggregate evidence.
01
46-task public aggregate
02Full scores, runtime settings, and cost provenance.
Original mechanism case report
03Reported S27 recovery and DuckDB regression; observational case evidence.
LoopX LHTB runner
04Current generic_cli + fresh codex exec Heartbeat implementation and fairness contract.
LHTB official site
05Benchmark definition, leaderboard, and methodology.
LHTB paper
06Long-Horizon Terminal-Bench, arXiv:2607.08964.
LoopX benchmark RFC
The broader long-horizon harness benchmark and research program.
Scores come from the public 46-task aggregate; comparisons are computed from task rewards. Mechanism cases refer to the original research brief, not new experimental findings.