EdgeBench / LHTB: notes from historical runs

1. What this round asks

For these particular tasks and configurations, the author asks whether submission order, review and closure evidence help sustain useful work. A few historical traces cannot settle that question. This article records locatable issues and explanations to test; it does not judge LoopX’s general long-horizon capability.

EdgeBench used gpt-6.1-sol / xhigh; both arms share the model, image and evaluator. LHTB used gpt-5.6-sol / max. “Native feedback” means the agent may submit and receive a public score, with no further submission during the cooldown. The original Borden and second Grid LoopX arms enabled checkpoint capture. The second Borden pair and local combined candidates disabled checkpoints but explicitly enabled TurnEnvelope. These treatments are not one default configuration; section 7 separates them.

Several runs did not finish. Experiment-controller failures included a container-count guard tripped by another cohort’s judge container and a crash while reading an exiting container; other runs were deliberately paused while writing. Pre-interruption traces can support particular fault diagnoses, but missing later windows and normal closeout cannot be filled in as completed 24-hour results.

2. What the score curves show

The following observations have different scopes within the historical runs. They suggest follow-up checks; they are not a pattern established for all LoopX runs.

First-submission timing and quality need separate checks. On Borden, native Codex first submitted at about 13.6 minutes and LoopX at about 48.2. On Grid, native first submitted at about minute 21 and LoopX at about minute 12, the latter with a 3593 placeholder. A submission timestamp is not the feedback-return time, and earlier submission alone does not establish a better candidate.

The curve flattens on a local solution. On Grid, LoopX reached 2075 at 85 minutes—better than native’s 2182 at the time—then spent 9.5 hours and nine submissions going from 2075 to 2039, under 2%. Native went from 2182 to 1791 in the same window and changed method three times: an embedded C++ accelerator, a dynamic-programming repair step, a sweep over cost parameters. On Borden, from its third submission LoopX changed little but one amplitude constant, gaining about two points per hour.

Submissions sometimes regress. In the first Borden pair, 3 of LoopX’s 11 valid submissions regressed; in the second pair it fell from 68.1 to 41.7, and the evaluator report shows why: the public error improved while the hidden-well error rose from 0.09 to 0.74—it was fitting the public data. Native regressed too (54.7 to 50.3), so this is not specific to LoopX, but every regression costs a cooldown slot.

Native’s 109 on Borden needs its own note. The evaluator report returns aggregate metrics for the hidden set, and both the native arm and the historical combined LoopX build in local short runs reached 109 by fitting those metrics. That is a protocol leak, not modelling skill; we record it as a task issue and count it as capability evidence for neither side.

Historical local supplement: timing and scores show no consistent advantage

The local report covers four tasks, 13 scored runs and 71 native submissions. They shared a four-CPU, 16-GiB host; resource ceilings were not exclusive reservations. The chart below starts each arm at its first native-session record, places each submission at submitted_at, and plots the best score through that submission. Lines only connect observations and end at the last submission; there is no fitted function, confidence band or invented zero.

* Native-goal Rust and Jagua retain valid scores, but their final Goal turns timed out and no normal final_result was produced. Codec native-goal self-completed at about minute 38; two other codec startup failures produced no score and are not zeros. ‡ The Borden 109 protocol limitation is explained above.

Using the explicit native-first-record clock here, combined LoopX first submits at about 1.7–3.0 minutes, versus official’s 2.8–9.7. Earlier submission did not ensure a higher final best: codec and both Jagua comparisons trailed, while Rust was close. Candidate/official uncached-input token ratios were about 1.06 on Borden, 1.31 on codec, 1.01 on Rust and 1.67 / 1.57 on Jagua. These runs did not show lower uncached input either; those ratios cannot be converted directly into total billed savings.

Second Borden pair: retain the early lead and the interruption

The controller crashed during container resource sampling; the solver processes were later stopped, leaving no normal final results. A stale status file still said running, which did not establish continued execution. The retained best scores are neither completed 24-hour results nor evidence that checkpoint recovery was repaired.

3. Historical planning: submission followed local preparation

The cause reads directly from the session trace and wake receipts. The planned entry has LoopX plan first, and the planner’s default profile is meant for open-ended product directions: it decomposed a bounded task with an explicit evaluator into three sequential Todos—P0 forward model, P0 inversion, P1 “once a runnable candidate exists, iterate via native evaluation”. Each execute wake finished exactly one Todo and then spent one to one-and-a-half minutes on settlement (complete the Todo, claim the next, write back vision, record quota). Submission therefore landed in wake three.

In this trace, planning took about 1.3 minutes and settlement several minutes. Todo order offers a locatable explanation for the roughly 35-minute first-submission gap in this case. But first-candidate quality, later strategy and full cost were not separately controlled. This does not establish that obtaining feedback first is always better, or attribute the entire gap to planning order.

The experimental planning constraint in #6584 put an accepted evaluation before extended local calibration. It was withdrawn in merged #6612: the current runner no longer offers --feedback-first or its planning environment switch. The submission times below describe the historical combined revisions, not a recommended activation or a validated current repair. Across the four local tasks, combined candidates first submitted at about 1.7–3.0 minutes versus official’s 2.8–9.7, using section 2’s clock. The original Borden long-run observation of about 48 minutes used different source, cooldown and operating conditions; it is not a matched “48 minutes down to 2” improvement.

The historical mechanism was a planning input on the benchmark runner’s side, not a product capability gain. It was off by default and changed work order rather than budget, submission rights or decision rules. The trace locates a delay in Todo order; the combined comparisons do not establish an isolated gain. The product question remains how shared planning balances first-deliverable quality with useful feedback; section 5 discusses the separate problem of continuing on a stalled path.

The maintainer requested withdrawal because feedback-first guidance can encourage a lower initial-quality starting point. Earlier feedback alone does not establish a better first deliverable: quality, latency and full cost need separate comparison. Submitting a competitive first candidate and a minimal one may lead later work in different directions; the historical combined runs do not isolate that effect. The original Grid trace shows prolonged iteration around a weak start, but that run did not enable the new feedback-first option. It cannot establish a side effect of that option. A matched rerun needs to test this hypothesis.

4. Review-point crashes: one status word was not accepted

All 4 review points on Grid crashed; in the first Borden pair 24 of 87 wakes failed, 8 with EffectRuntimeInternalError and 16 with ValueError. The local combined candidates did not enable checkpoint handoff and did not show this review/capture failure chain; that does not erase the two local native-goal runner timeouts or two startup failures. The direct cost of the failures was only some twenty minutes of native time, but every failure broke settlement and pushed later wakes into “recover first, then review”—review became a by-product of recovery.

The root cause is a status word that was not accepted. After the Host captures the checkpoint at a review point, the agent selects a Todo in the same Turn; that selection rereads the guard inside the CLI, and the receipt renders as upgraded—a legitimate view of the same durable receipt, meaning the selection appended its binding. The checkpoint selector accepted only committed and replayed, threw an untyped error, the Host surfaced it as an internal error, and the wake ended.

The CLI side amplified the damage. The refused handoff fell into the outer guard’s catch-all and was reported as quota_unexpected_collection_error—“collection failed”—and the saved decision file was overwritten. The native session the next wake resumed still remembered the previous Turn’s handoff socket, which had been deleted with that wake; so first quota validation failed, then a capture-root ValueError. Every failed wake on both tasks follows this one chain.

The repair is #6601, all product code: the selector admits upgraded and refuses with typed errors that keep their reason; a capture or handoff failure after the guard becomes a new typed result, quota_decision_transport_failed, returned together with the committed receipt and the saved decision path, no longer overwriting it; a stale socket reports plainly that it belonged to an ended wake and that no guard ran; failure reports keep the error class and a bounded message—wake receipts had kept only the exception class name, which is why this took two tasks to locate. Seven new tests fail before the fix and pass after; the default path (no capture, no handoff) is unchanged.

Not repaired: a resumed session can still remember an old socket, though it now gets a clear error before any guard runs; the wake after a crashed review does not re-enter review mode. Both point at the same larger design question—a new socket per wake as the transport—which section 7 returns to.

5. In-place tuning: review may retain the path, and it kept retaining

What most needs explaining on Grid is not the crashes but the 9.5 hours between them. From Todo state, review records and tool calls we read three things.

All four reviews chose “retain”. Periodic review fires every 6 settled Turns; the guidance says “a due review can retain the path; it does not force a pivot”; the novelty check requires one new hypothesis. So “push the runtime cap a little further under 28 seconds” counted as a new hypothesis, and all four reviews passed. Native’s score fell from 2182 to 1878 in the same window; every LoopX review could see that gap, and no rule asked it to do anything about it.

One catch-all Todo owned ten hours. The third Todo planning produced was “submit … and continue useful local refinement”. It never closed, its lease was renewed 12 times, and no Todo was ever added. The Todo list carried no strategy: no hypothesis families, no exit criteria, no “if three submissions do not improve, change method”.

Control-plane overhead took about a quarter of the time. 188 LoopX CLI calls plus roughly 345 reads of guard packets and decision files came to about 2.8 of 11.7 hours; 207M input tokens against native’s 146M; 20 context compactions against 7. Native had almost none of this.

Why not simply add “a stalled score forces a pivot”

The inspected core version records progress through the agent’s self-declared advanced, unchanged and blocked classes; the Grid trace declared advanced each round. Small improvements such as 2041, 2040 and 2039 do not alone decide whether to pivot. Generic core logic should not guess the meaning of a provider metric: “one fewer sorry” on Lean and “two more points” on Borden share no pivot threshold. This motivates explicit ownership of metric interpretation and replanning; it establishes neither that the model can only tune nor that LoopX lacks progress handling.

The design already reserves a slot for this: external progress review. An independent observer writes typed “drift” receipts after each work transition; in assist mode a streak of consecutive drift receipts escalates into the existing replan obligation, and answering that obligation may not restate an old hypothesis or merely rename things—it must carry new evidence. In the inspected version it is off by default, and the existing observer judges code changes, not evaluator scores. It also has a product defect: when attribution is missing (no agent id, or no typed progress record in the window), assist silently never fires while status looks healthy.

The author proposes three directions, not yet implemented or shown to improve outcomes: let a provider own metric direction, thresholds and plateau judgement while supplying drift through the existing obligation chain; expose why assist cannot fire; and give iterative Todos checkable exit criteria. These are design hypotheses for review, not promised LoopX mechanisms or an accepted roadmap. Even if implemented, their effect on decisions and results would need testing; they cannot guarantee that an agent chooses a better method.

6. Early closure: the acceptance gate changed what counts as done

LHTB’s four pilot tasks showed one pattern. In all four runs LoopX’s scheduler emitted stop_until_explicit_resume between minutes 43 and 60 of a 90-minute budget, with the identical reason: “validated closure evidence derives no-follow-up and confirms no remaining frontier”. Hidden-set rewards at that point were 0.000, 0.028, 0.086 and 0.093. Only the outer continue-until-timeout mechanism brought the agent back; the second phase never stopped on its own.

These pilots and the earlier Portfolio no_followup case both suggest an acceptance gap: completed Todos do not establish that the original goal is met. In these runs, closure evidence did not cover the full behaviour the hidden evaluation required. Two deliverables exited non-zero under the hidden wrapper manifest; another passed schema checks but scored zero on an end-to-end stress group weighted 0.89. These cases support checking verification-interface coverage, not a general judgement of LoopX’s closure capability.

We did not change the scheduler rule; we changed where closure evidence comes from. #6585, now closed without merging, tested the explicit opt-in LOOPX_ACCEPTANCE_CONTRACT=planned: during planning the model also writes a Goal acceptance document—which Todos deliver the result, which executable criteria each is bound to, criteria that must exercise the deliverable through the task’s stated verification interface—and the benchmark host, acting as the trusted local owner, configures it through the existing goal-acceptance contract. Selected Todos then cannot complete until their criteria pass, and the goal cannot close while they stay open.

Mechanism readback: planning configured 3 criteria bound one-to-one to 3 Todos; the first Todo’s todo complete ran its bound “load the submitted model and call predict on public validation rows” criterion and passed—exactly the failure the pilot hit when its deliverable could not predict in the evaluator sandbox. After the second phase replanned, it wrote a new criterion, and two completion attempts were rejected until the artifact satisfied it.

The nature of this change also needs stating. The acceptance gate is the benchmark runner writing a document on the model’s behalf and calling an existing LoopX capability; LoopX itself still declared completion at minute 38. What changed is what counts as closure evidence—verifier-shaped, owner-locked—not whether the agent may close. 0.82 is one run; whether forced continuation after that point adds reward, this experiment cannot answer. Review also found a real counterexample: re-confirmation reread the model-writable draft, so one Todo edit plus one rewritten criterion let a Todo with a missing artifact complete. The host now replays only the document it accepted and refuses when the owner contract digest moved; the counterexample is in the real-CLI tests.

7. Which numbers cannot be used

One finding, made while writing, forced us to requalify the evidence above.

LoopX has a TurnEnvelope that is off by default: it compresses the tens-of-kilobyte quota decision into an inline view of about 8 KB and writes the rest to a file read by reference. Checkpoint handoff appends --turn-envelope to the guard command whenever a capture directory is passed. The original Borden and second Grid runs therefore enabled the envelope implicitly; the local combined candidates and second Borden pair disabled checkpoints but explicitly enabled the envelope. The local receipts say turn_envelope=true and turn_envelope_source=explicit. None is a clean envelope-off control. Absence of that capture failure chain cannot show that disabling the envelope improved stability.

Grid provides a historical observation with the envelope enabled: about 345 full capture-file reads, 37–57 KB each, and roughly one-quarter control-plane overhead recorded under that configuration. Without a matched envelope-off control, this does not attribute the overhead or score plateau to the envelope. Another unmerged combined build, with relaxed re-entry rules, recorded 65 versus 25 CLI calls and 40 versus 8 quota checks. This suggests testing re-entry cadence separately; it does not identify the sole or dominant cause.

Two design questions remain to be tested: whether to decouple handoff and the envelope, and how envelope on/off changes uncached tokens, packet reads, CLI calls, time share and score on the same task under matched conditions with checkpoints off. Keeping decision text inline, reading details on demand and declaring re-entry cadence are the author’s candidate directions. These observations establish neither their optimality nor a validated product conclusion.

Three more kinds of numbers must be set aside. Experiment-controller incidents—a container-count guard tripped by a judge container, a controller crashing on an exiting container—shortened runs but are not LoopX failures. Borden’s 109 comes from an evaluation protocol leak, as section 2 explains. And every “after the fix” local number carries several simultaneous changes: feedback-first, the relaxed re-entry rule, an explicit envelope; none can be attributed alone.

8. How to judge this batch of repairs

The table separates historical observations, local contract validation and hypotheses still to test. When product code, harness inputs and configuration change together, score differences cannot be attributed to one factor. A merged patch, passing local checks or a changed runner curve does not validate the current LoopX version’s general capability or efficiency.

The engineering value is in retaining checkable failure chains, exit criteria and acceptance counterexamples. Corresponding entrypoint reproductions and regression checks can qualify local repairs; a versioned, configuration-explicit rerun retaining failures is still needed to assess long-horizon outcomes. These fragments establish neither that the main cause lies in the product rather than the model nor that LoopX is generally better or worse than native execution.

9. The author’s proposed follow-up validation

At the historical run cutoff, the remote runs were paused and their evidence retained. Since then #6584 and #6601 reached main, while #6612 withdrew feedback-first. The sequence below is the research proposal, not execution authority: new runs need their own authorization, and #6585 was closed without merging. Its retained branch evidence does not establish a supported mainline strategy.

This article leaves engineering leads from historical configurations, not formal LoopX capability conclusions. The author proposes checking local repairs and then task quality and efficiency under declared source, model, task, feedback, budget and resource conditions. Earlier feedback does not establish better delivery; a locally verified repair does not establish long-horizon gains. A withdrawn strategy is not current activation advice.