Independent App first-use and repeat-use evaluation¶
Status: prepared protocol; no participant trials or passing results. This turns the roadmap's alpha/beta targets into a bounded study using the golden-query pack. It adds no runner, runtime obligation, telemetry or recruitment automation. Source tests and scripted browser fixtures do not qualify an installed product.
The question is whether a new person can obtain and find a useful result through the App, then choose to return, without a maintainer operating their machine. Start with one supported host/profile and one public task. Small-team, Lark, cross-host and other-model claims retain their separate acceptance.
Freeze before the first attempt¶
The evaluator copies this sheet into a private study note and fills every applicable blank before a participant starts. The note remains private until its owner approves a redacted summary. A document with blanks is preparation, not a runnable or qualified cohort.
| Frozen item | Required value |
|---|---|
| Study / package | Study revision, evaluation date/time zone, exact release artifact URL and digest, App/runtime revisions, previous known-good rollback package |
| Prerequisite qualification | Evidence that the selected installed package passed the supported entry/result/recovery path; an open implementation PR is insufficient |
| Supported configuration | OS/version, host, exact execution profile/model/effort, account type, documented prerequisites and supported install steps |
| Participants | Five consenting first-time users, anonymous slots U1–U5, prior relevant experience and declared independence from implementation; recruiting/contact details stay private |
| Task and sources | Fixed GQ01 request below, latest three completed fiscal years at the evaluation date, official report URLs/revisions or hashes, accessible public input copies |
| Independent oracle | Evaluator's checked fiscal periods, units, operating cash flow, cash purchases of property/equipment, definition of free cash flow and calculation; keep expected values away from the executing Agent |
| Envelope | Numeric model/tool call, token and time ceilings; monetary ceiling for metered usage, measurement source and unavailable-cost policy; no silent model substitution |
| Observation | Start/end definitions below, consented observation method, evaluator and scoring criteria, recovery/stop variant assigned before testing |
| Follow-up | Later calendar date/window for the same five users, repeat-task variant, consent and withdrawal handling; no reminder or invitation sent by this protocol |
Do not start this cohort until the package, permitted usage and willing users are available. A participant uses a fresh disposable project/Goal, not an active personal or organizational Goal. Credentials stay in the normal machine-owned login/configuration flow; the evaluator never copies them into a fixture. Preconfiguration must match the shipped instructions. List every setup action; do not hide maintainer-created bindings, repaired profiles or manual state edits.
Freeze proposed targets of 10 minutes of configuration after prerequisites and login, 15 minutes from readiness to the first useful result, at least 4/5 successful first-use attempts, and at least 3/5 successful voluntary repeat users on another day. These are targets, not current measurements. Do not adjust the task, thresholds or oracle after viewing candidate results. A fix or package/profile change starts a separately identified cohort; preserve the original five rows.
For a subscription profile without task-level billing, freeze enforceable usage and time ceilings and retain task cost as unknown. The artifact and usability checks may pass; monetary-budget qualification remains unproven. Do not convert an unknown subscription allocation into zero cost or a monetary-efficiency claim.
Participant handout / 给参与者的话¶
Use the selected release's ordinary install/login instructions. Open the LoopX App and say:
建个目标：研究微软近三年的现金流，给我一页报告。
Start a goal to study Microsoft's cash flow over the last three years. Give me a one-page report.
The evaluator supplies the same pinned public reports and task budget through the supported setup, not a hidden prompt that routes around the App. Equivalent natural phrasing is welcome; the participant does not need Agent IDs, CLI commands, internal receipt names or a prescribed answer. Confirm the actual creation preview when it correctly represents the request. Ask an ordinary clarification or change the request if needed; record it rather than coaching the participant toward a successful score.
结果留在这里，先别发布。卡住了就按界面提示处理；不想继续可以停止。
Keep the result here; don't publish it. Use the interface if something gets stuck, and stop whenever you want.
Once the answer arrives, find its evidence, open the full result and reload the conversation. The same result should still be readable. Say what you understood, what was confusing and whether you would use LoopX again. Screen observation is optional and requires consent; redact account details, paths and unrelated work.
On the separately agreed later day, the participant may choose to return to the same work and ask:
接着上次的，看看资本开支增长对现金流的影响。
Pick up the previous report. Look at how growing capital expenditure affects cash flow.
The repeat request should reuse the retained work and its constraints. It needs checked calculations and limits, not a guaranteed investment forecast. Declining or not returning is recorded with a volunteered reason, not replaced by another user. This handout does not authorize public posting, trading, broader account access or additional background work.
What counts as a useful result¶
An evaluator who did not implement the candidate checks the stored result and owning state after the participant finishes. Appearance or an Agent's claim of completion does not decide the score.
| Check | Evidence needed |
|---|---|
| Request survives | Original request, one confirmed Goal/execution path, no lost message, duplicate creation or competing driver; return stays in the initiating conversation |
| Work is understandable | Participant can identify actual owner and waiting/running/failed state, locate the result and source evidence, and explain any decision requested of them |
| Report is correct | All three frozen fiscal periods, consistent units, cash-flow/cash-capex table, explicit free-cash-flow definition, checked arithmetic, source links and uncertainty; leases/commitments are not silently counted as cash spending |
| Report is usable | Leading answer, a bounded comparison and readable Markdown or report; the participant can open it, find evidence and recover it after reload without a shell or manually relaying results |
| Envelope is respected | Observed usage fits the frozen enforceable ceilings and profile; report monetary coverage separately, and never use unknown cost to certify a monetary limit |
| Repeat actually works | A new later-day request uses the retained source/report version and no-publication constraint, produces a checked incremental result, and returns to the same conversation |
Record pass / fail / blocked / not run per required check and surface. Only pass counts as success. Record an unavailable prerequisite/profile as blocked, a lost request or incorrect result as fail, and an unattempted task as not run. Preserve abandonment and timeout at their observed stage. A result rescued by maintainer shell work is not an independent first-use pass, even if the repaired report is subsequently correct.
Run GQ08 stop and GQ09 recovery companion variants on the same pinned package in a disposable operator rehearsal before recruitment. Observe one interruption and reconnection in the preassigned participant variant when consented. Stop must affect the targeted work and report actual readback; do not infer team stop from stopping the chat. Missing native evidence stays unqualified. These checks do not replace G1's two real small-team cycles or authorize fault injection into a participant's active work.
Observation sheet and honest denominators¶
Use a monotonic elapsed clock during an attempt; record its date/time zone for later-day repeat qualification. Keep these intervals separate:
- Complete onboarding: first install/setup action to readiness, including prerequisites and authentication. A pre-readiness failure still has a row.
- Configuration: prerequisites and login complete to readiness. Readiness requires the actual supported profile/entry to be usable, not a green label.
- First result: submission after readiness to availability of the exact result version subsequently accepted by the independent checker in the original App conversation. Exclude retrospective scoring delay; a rejected earlier version does not stop the clock. Report waiting and active attention too.
- Recovery / repeat: intervention to verified recovery, and later-day request to its accepted incremental result. Unknown times remain unknown.
For each intervention record stage, cause and actor. Separate required login, explicit authorization, goal changes and voluntary deep discussion from avoidable owner finding, context repetition, reminders and result transport. Count one input once, retaining secondary tags. Normal documented UI actions are not maintainer rescue; undocumented CLI/state repair or a maintainer taking over the task is. An observer can ask for retrospective feedback after the attempt, but hints during execution count as assistance.
Copy this unrun ledger; do not treat its blank numbers as zero:
| Slot | Onboarding / configuration / result time | Outcome / blocked stage | Assistance and attention | Measured cost or unknown | Evidence / later-day repeat |
|---|---|---|---|---|---|
| U1 | unknown | not run | unknown | unknown | none / not run |
| U2 | unknown | not run | unknown | unknown | none / not run |
| U3 | unknown | not run | unknown | unknown | none / not run |
| U4 | unknown | not run | unknown | unknown | none / not run |
| U5 | unknown | not run | unknown | unknown | none / not run |
Publish only an approved, redacted summary: exact candidate/configuration, attempted count out of the frozen five, accepted count out of five, each failure/blocked/not-run row, later-day accepted repeat count out of those same five, times, interventions and measured/unknown costs. Report timing coverage; do not omit slow/failed samples or impute zero cost. For the 4/5 and 3/5 targets, blocked, abandoned and not-run slots stay in the denominator. A useful-artifact rate alone does not prove all safety, budget or advertised-platform gates passed. Five users provide diagnostic feedback, not a statistical reliability claim.
Existing feedback route and next repair¶
Keep detailed observations in the private study note. Participants may optionally use the existing First run or Usage story forms for a short public-safe outcome or failure description. Public issue submission and anonymous citation are separate opt-ins; neither is required for participation or a passing result. Do not paste Goal contents, raw transcripts, account data, local paths or private evidence into the forms.
Map each observed failure to the existing golden-query case and its owning product task: creation/connection (GQ01/02), responsibility/return (GQ03/04), continuity/controls (GQ07–09), readable result or evidence. Carry the reproduction, expected result, package/profile and independent readback into that task rather than creating a parallel adoption queue. Re-test on a new pinned candidate and keep previous attempts visible. Preparing this protocol, filing feedback or installing a package does not close the roadmap's product or cohort acceptance.