Spike findings — T4 autonomous loop (HLD §11 de-risk)

Field	Value
Issue	#1422 (T4)
HLD	docs/hld/HLD-autonomous-loop-langgraph.md (on main, fdb5f0c3)
Author	Principal Technical Architect Agent
Date	2026-06-18
Scope	Time-boxed, throwaway prototype. No prototype code merged (lives in /tmp/t4-spike, deleted after).
Verdict	GO-WITH-CHANGES — all three High risks retire with concrete, cheap mitigations the LLD must bake in.

This spike answers the three High risks the HLD flagged (R1 re-plan thrash, R3 compaction timing, R5 interrupt()-resume double-execution). It used the real code where cheap — checkpoint_schema.py (verbatim), the real maybeCompact() decision tree from internal/context/assembly/pipeline.go (#1426), the real checkpoint OQ-5 header (internal/runtime/checkpointstore/store.go), and the observed execute_tools / ingest_result / checkpoint / restore semantics from the worker — and stubbed only the LLM and MCP egress side effects.

Cross-validation that the prototype is grounded in live contracts:

• go test ./internal/context/assembly/ -run Compact → ok
• go test ./internal/runtime/checkpointstore/ → ok
• Python encode/decode_checkpoint round-trip: header 00000001, schema_version 1 == Go SchemaVersion — match.

---

R1 — Re-plan thrash → RETIRED

Question: does an orchestrator-driven plan/replan loop oscillate or converge, and does a stall/stop condition fire before budget exhaustion?

Prototype: implemented the HLD §5.1–5.4 orchestrator loop (Plan{goal, todos, cursor, progress[], replans}, StepProgress, three terminate axes) and drove it with three synthetic worker behaviours.

Evidence:

Worker behaviour	Terminated by	Loops (budget=25)
healthy (completes a todo each step)	complete (goal)	2
looping model (identical output, no artifacts)	stalled:no_progress	3 (= stall window)
thrash model (re-plans every step, novel output each time)	stalled:replan_cap	5

• The loop converges in every case; none oscillate.
• The §5.3 no-progress detector fires exactly at the stall window (3 steps) — and is distinct from budget (stops at 3, not 25).
• Critical nuance the spike surfaced: a thrash model that emits novel text each step defeats the response_delta_hash novelty check (the HLD's primary stall signal). The replans cap sub-axis (HLD R1 mitigation, §5.3) is what actually catches it. So the replan cap is load-bearing, not belt-and-braces — the LLD must implement it, not treat it as optional.

Mitigation to bake into LLD:

1. Implement BOTH stall sub-axes — no_progress (window over artifacts/completion/novelty) AND replans > cap. They catch different failure shapes; neither alone is sufficient.
2. The no-progress window AND replan cap must be execution_mode-tunable defaults (HLD OQ deferred item 3). Suggested seed: window=3, replan_cap=4.
3. goal_complete requires len(todos) > 0 — guard against the empty-plan-reads-as-complete artifact the harness exposed.

---

R3 — Compaction timing → RETIRED

Question: now that compaction is live (#1426 assemble-time trigger + worker + Summaries slot), does it fire correctly across a multi-step run without starving context or thrashing? Specifically the interaction between the per-turn assemble trigger and a long loop.

Prototype: mirrored the real maybeCompact() decision tree (headroom, budgetBlown, thresholdByCount; async-enqueue-for-next-call + sync-fallback-only-when-blown) over a 12-step run.

Evidence (normal — async worker runs between steps):

• Count threshold (6 events) fires the async producer one step before the budget is ever blown. The worker compacts between steps; packed tokens never exceed budget; 0 sync fallbacks, 0 truncations; earliest turn preserved as a summary.
• Steady state is a benign sawtooth (live events oscillate 4↔6 as summaries accrue) — no thrash: each turn is summarized at most once (the summarized flag + sliding-window live-tail prevent re-compacting the same turns).

Evidence (starvation probe — async worker never runs):

• The sync fallback is the safety net: it fires exactly when budgetBlown (steps 6 and 11), compacts inline so the current call serves a summarized prompt instead of a truncated tail. 0 hard truncations, packed always ≤ budget even with a fully-starved async worker.

The dual-path design (async-ahead-of-budget producer + sync-when-already-blown fallback) is robust to async lag — the exact long-loop interaction the HLD worried about. The per-turn assemble trigger does not thrash and does not starve.

Mitigation to bake into LLD:

1. Keep compaction_threshold_events strictly below the budget-blown point so the async path leads and the sync path stays a rare fallback. The default (manual-only, threshold_events=0) is wrong for autonomous mode — autonomous sessions MUST ship a non-zero count threshold or they lean entirely on the sync-blown path (higher per-call latency on the blown step). LLD: set a non-zero default compaction_threshold_events for execution_mode=autonomous.
2. Cap progress[] and store completed-todo summaries not full transcripts (HLD R4) so the checkpoint blob itself doesn't grow unbounded across a long plan — the same compaction discipline applies to the plan state, not just context.
3. Sync compaction runs on the assemble hot path; for autonomous loops add a metric/alert if sync-fallback rate is non-trivial (signals the count threshold is mis-tuned).

---

R5 — interrupt()-resume double-execution → RETIRED (with mandatory LLD constraints)

Question (riskiest): when an approval-required tool mid-loop suspends and later resumes from checkpoint, does the step re-execute (double-fire) or resume exactly-once?

First finding — the HLD's in-step path does not exist yet. Today (graph/nodes.py _governance_block_tool, governance/client.py) a requires_approval verdict fails closed as a hard deny — it returns an error ToolResult and the tool is blocked. There is no interrupt() anywhere in the worker (grep confirmed). So HLD §7.4-item-2 (in-step interrupt() → approval.Service → resume) is net-new build, not a wiring of something present. The HLD should say so explicitly.

Prototype: modelled the in-step approval pause against the real checkpoint round-trip and the real state-strip semantics, with a side-effecting "MCP tool" (a ledger) to observe double-fires. Tested two candidate resume designs:

Design	read_file (already-run sibling) fires	gated delete_db fires	Verdict
NAIVE (checkpoint only the original input; full re-run on resume)	2×	1×	DOUBLE-EXECUTION BUG
HLD (persist tool_results into checkpoint + skip-done + idempotency key on the gated call)	1×	1×	exactly-once

• The naive design reproduces the double-execution bug: the already-executed read_file sibling re-fires on resume because its result was not persisted.
• The HLD design achieves exactly-once for the gated call and the executed siblings.
• Second finding the spike surfaced: under at-least-once resume delivery (pod churn / accidental double-resume of the same step), a non-gated side-effecting tool without an idempotency key (send_email in the harness) fired twice. So idempotency keys must cover every side-effecting tool call in the step, not only the gated one. Persist-results-into-checkpoint protects the clean single resume; idempotency keys protect against duplicate resume delivery. Both are required — they defend different failure modes.
• Checkpoint hygiene holds: provider_keys and events are stripped from the blob (confirmed — SECRET/noise absent from serialized bytes), consistent with executor.checkpoint() and the #993 key-strip contract.

Mitigation to bake into LLD (R5 — these are mandatory, not advisory):

1. The interrupt checkpoint must persist the already-executed tool_results of the in-flight step (not just the original current_tool_calls input). Resume skips tool calls whose tool_call_id is already in tool_results.
2. Idempotency key on every side-effecting MCP call in the step, keyed by (session_id, loop_iteration, tool_call_id). This is the un-bypassable backstop at the MCP-proxy egress — it makes resume safe even under at-least-once / duplicate delivery, independent of the worker getting (1) right.
3. The in-step LangGraph checkpointer must be scoped to the single step only (HLD §7.4) — never the whole plan; the macro plan state stays in Go + agent_checkpoints.
4. Reuse the existing OQ-3 idempotency: agent_checkpoints unique (session_id, loop_iteration) + deriveCheckpointKey already give per-iteration uniqueness; the pause/resume checkpoint at the interrupt point must use a distinct loop_iteration (or sub-key) so it does not collide with the next step's checkpoint and trip a false HashMatch=false P1.
5. QA gate (mandatory): a test that pauses on a gated tool mid-batch, recycles the worker, resumes, and issues a duplicate resume — asserting each side-effecting tool fired exactly once.

---

Overall recommendation: GO-WITH-CHANGES

The orchestrator-side plan-loop placement (HLD §4) is sound — all three High risks retire with cheap, concrete mitigations, and the per-step overhead is dominated by the checkpoint write + rehydrate the substrate already does today (no new persistence store, plan rides the existing blob). Proceed to the T4 LLD, conditioned on the changes below.

HLD adjustments to make before/with the LLD

1. §7.4 — state plainly that the in-step interrupt() path is net-new. Today requires_approval hard-fails-closed; there is no interrupt() in the worker. The LLD designs this from scratch; it is not a re-wire. (Does not change the placement decision; corrects an implication.)
2. R5 — upgrade the mitigation from "idempotency keys on MCP calls" to "idempotency keys on every side-effecting tool call in the step, keyed by (session_id, loop_iteration, tool_call_id), PLUS persist executed tool_results into the interrupt checkpoint." The spike showed each defends a different failure mode (clean resume vs duplicate resume); the single-line HLD mitigation is necessary but not sufficient.
3. R1 — mark the replans cap as load-bearing, not optional. A novel-output thrash model defeats the response_delta_hash novelty signal; only the replan cap catches it.
4. R3 — autonomous mode must ship a non-zero compaction_threshold_events default. The platform default (manual-only, 0) forces reliance on the sync-blown path; set a non-zero count threshold for execution_mode=autonomous so the async producer leads.
5. Add to §12 OQ: plan-state checkpoint-blob growth (apply the same compaction discipline to progress[] / completed-todo summaries, HLD R4) and the interrupt-checkpoint loop_iteration sub-keying (avoid OQ-3 HashMatch false positives).

These are LLD-shaping refinements, not blockers — hence GO-WITH-CHANGES, not NO-GO. Founder approval of the HLD direction (§13) remains the gate before LLD per ADR-0019; this spike de-risks the engineering, not the founder call.