Memory extraction — dead-letter replay

Issue #2308 · milestone #27 (Agent Memory Loop, PRD #2037) · tool: cmd/tools/memory-extraction-replay · ledger: memory_extraction_runs

When you need this

An extraction run that fails 3 times lands in memory_extraction_runs.state = 'dead_letter'. Before #2308 that was permanent: the consumer acks the job so Redis never redelivers it, and claimRun drops any later delivery for a terminal run. The session's memories were simply gone.

On beta-dev 2026-07-28, four runs dead-lettered inside one ~2-minute window from pure local-model contention — runs either side of them completed in 13.1s and 10.6s against a 30s cap. Day tally: 11 succeeded, 4 dead_letter. Nothing was structurally wrong; a brief load spike cost four sessions their memories.

The source transcript is still in session_events. Only the attempt was lost. This runbook is the path back.

Related: #2298 / PR #2306 makes the extraction timeout tunable, which reduces how often this triggers. It does not address what to do when it does.

Detect

-- dead letters by day and failure mode
SELECT date_trunc('day', updated_at) AS day,
       count(*) FILTER (WHERE state = 'dead_letter') AS dead,
       count(*) FILTER (WHERE state = 'succeeded')   AS ok
FROM memory_extraction_runs
GROUP BY 1 ORDER BY 1 DESC LIMIT 14;

-- what actually killed them
SELECT session_id, org_id, attempts, replays, last_replay_at, left(last_error, 160)
FROM memory_extraction_runs
WHERE state = 'dead_letter'
ORDER BY updated_at DESC LIMIT 50;

Metrics (Context Engine, OTel → Prometheus):

Metric	Meaning
memory_extraction_dlq_total{org_id}	runs dead-lettered after the retry cap
memory_extraction_replay_total{org_id,outcome}	deliberate replays — recovered / failed / skipped / no_transcript

increase(memory_extraction_dlq_total[1h]) is the alert signal. The permanent loss is dlq_total − replay_total{outcome="recovered"}.

Replay

Always dry-run first. It costs nothing, calls no model, and writes nothing.

export DATABASE_URL=postgres://...      # a role that reads across orgs (owner/BYPASSRLS)
export REDIS_ADDR=...                   # for the embed-on-write enqueue
export MEMORY_EXTRACTION_API_KEY=...    # or COMPACTION_API_KEY

# 1. What would be replayed?
go run ./cmd/tools/memory-extraction-replay -dry-run

# 2. Replay transient failures from the last 24h, at most 20 of them
go run ./cmd/tools/memory-extraction-replay -max-age 24h -limit 20

# 3. One specific session, whatever the failure reason
go run ./cmd/tools/memory-extraction-replay -session <uuid> -include-nonretryable

Flag	Default	Notes
-dry-run	off	list only; no model call, no writes
-max-age	168h	horizon on enqueued_at; 0 disables. Replaying a months-old transcript against today's prompt produces memories nobody reviewed in context
-limit	0	caps the blast radius — each replay is one model call
-max-replays	3	poison cap; a run replayed 3× is not a blip
-org / -session	all	narrow the sweep
-include-nonretryable	off	also replay failures not classified transient (parse errors, malformed envelopes)

Outcomes in the summary line:

Outcome	Meaning
recovered	re-extracted and the run reached succeeded; memories are in agent_memory
failed	failed again; row is back in dead_letter with the new error, replays incremented
skipped	not dead-lettered any more, or at the replay cap. Nothing changed
no_transcript	session_events is empty — nothing to recover. Row left dead-lettered on purpose, so the loss stays visible

Why it is safe to run

• It cannot duplicate a good run. The only exit from dead_letter is a CAS guarded on state = 'dead_letter', so a succeeded run is structurally unreachable. The online consumer's claim path is unchanged.
• It cannot double-run. The CAS is one UPDATE ... WHERE; concurrent invocations produce exactly one winner.
• It cannot loop forever. replays < -max-replays.
• It is resumable. Work is found by DB predicate and keyset-paged, so interrupting and re-running resumes; recovered runs drop out of the predicate.
• It refuses to run blind. Without a real extractor configured it exits non-zero rather than stamping every run succeeded with zero candidates.

Gotchas

• Run it as an admin DB role. Like cmd/tools/memory-backfill it reads across orgs. Every write re-derives app.org_id from the row, so writes stay scope-correct under FORCE RLS regardless of the read role.
• A replay is a fresh model call, not a stored intermediate — the extractor output of a failed run is never persisted. Candidates may differ from what the original run would have produced; the persist path dedups on redacted content_hash, so overlap converges to a no-op.
• Fix the cause first. If dead letters are still arriving (a wedged upstream, a too-tight MEMORY_EXTRACTION_TIMEOUT), replaying just re-burns quota. Replay once the spike is over.