Runbook — `memory-status-reevaluate` (backlog re-evaluation of quarantined memories)

Issue: #2399 · Arc: PRD #2037 · HLD #2064 · LLD #2069 · tracker #2063 · milestone #27 · Rulings: #2322 (classifier) · #2394 (implementation) · #2095 (quarantine-on-redaction-hit) · #2323/#2329 (stranded approvals) · #2325 (backfill discipline)

Founder approval: 2026-08-05 on #2399 — "approved — sweep." The approval is conditional on the dry-run diff being reviewed before any live write. Do not skip step 4.

What this does

agent_memory.status is stamped at write time. #2394 replaced the model-assigned high_stakes vote with classify.StatusFor, a deterministic server-side rule — but only for new writes. This tool applies that same rule, late, to the backlog, so the reviewer's queue holds the rows the rule actually says belong there and nothing else.

It is a replay, not a re-decision. There is no model anywhere in the status path. That is the entire basis on which the retroactive sweep was approved: if a model ever enters this decision, the approval no longer covers it.

For a row that now derives active, in one transaction:

Table	Write
`agent_memory`	`status` pending → active (compare-and-set)
`governance_approvals`	the paired still-pending row → `expired`, `reason = 'reclassified_by_2322'`
`approval_events`	one `expired` event carrying the same reason

then, post-commit: the embed job the pending insert skipped, and one agent_audit_log row with action memory_reclassified (never memory_approved — nobody approved these rows).

For a row that still derives pending it writes nothing at all.

Preconditions

Migration 197 must be applied to the target database. The tool fail-closes on this (preflight FAILED: … does not admit "memory_reclassified"). Without it every audit row fails its enum cast and is silently dropped — a bulk activation with no trail, which is worse than not running.
A DB role that can read agent_memory across the orgs you intend to sweep. Exactly two invocations are accepted, and since #3498 the tool refuses anything else before it reads a row:
- a role that BYPASSES RLS (the migration role today) — the full cross-org sweep; or
- both -org and -agent, which set app.org_id AND app.agent_id on the read transaction.
Both, not either. The policy is org_id = app.org_id AND (agent_id = app.agent_id OR app_has_share_grant(id)), so an org-only GUC matches nothing. Measured on app_rw before the guard landed, with three quarantined rows waiting, the tool printed pending before : 0 / examined : 0 and exited 0 — i.e. it said the backlog this runbook exists to drain was already gone. Not the schema owner either: FORCE ROW LEVEL SECURITY removes the owner's exemption.

After the #3259 cutover the app DSN is app_rw; give this invocation the migrate DSN explicitly rather than inheriting the service environment. See docs/runbooks/migration-role-rls-posture.md.
REDIS_ADDR (optional). Without it the transition still lands; the activated row simply is not semantically recallable until a memory-backfill run.

Procedure

export DATABASE_URL='postgres://…'

# 1. Snapshot the corpus BEFORE. Record these numbers.
psql "$DATABASE_URL" -c "SELECT status, count(*) FROM agent_memory GROUP BY 1 ORDER BY 1;"
psql "$DATABASE_URL" -c "SELECT status, count(*) FROM governance_approvals WHERE action_type='memory_review' GROUP BY 1;"

# 2. Dry run. Writes nothing. Save the output.
go run ./cmd/tools/memory-status-reevaluate -dry-run | tee /tmp/reeval-dry-1.txt

# 3. Review the per-row diff. For every keep_pending row, the ARM and TOKEN
#    columns say WHY it stays in the queue. For every activate row, satisfy
#    yourself the classifier's verdict is one you would have made.

# 4. Re-run the dry run IMMEDIATELY before the live pass and diff the DECISIONS.
go run ./cmd/tools/memory-status-reevaluate -dry-run | tee /tmp/reeval-dry-2.txt
diff /tmp/reeval-dry-1.txt /tmp/reeval-dry-2.txt

# 5. Live.
go run ./cmd/tools/memory-status-reevaluate

# 6. Verify.
psql "$DATABASE_URL" -c "SELECT status, count(*) FROM agent_memory GROUP BY 1 ORDER BY 1;"

Step 4 is not optional, and not a formality

This is the #2325 discipline. Content is mutable via correct, so an identical set of candidate ids does NOT imply an identical set of decisions. Diff the derived verdicts, not just the id column.

The live pass re-derives once more inside the write transaction on the locked row, so a correction landing mid-sweep is classified rather than ignored — but that is a backstop against a race, not a substitute for a human reading the diff.

Verifying afterwards

-- Acceptance: pending equals the classifier's own verdict on the corpus.
SELECT status, count(*) FROM agent_memory GROUP BY 1;

-- Acceptance: no orphaned approval. MUST return 0.
SELECT count(*) FROM agent_memory m
  JOIN governance_approvals g
    ON g.action_type='memory_review'
   AND (g.target = m.id::text OR g.metadata->>'memory_id' = m.id::text)
 WHERE m.status='active' AND g.status='pending';

-- Acceptance: the genuinely quarantined rows still await a human.
SELECT m.id, m.memory_type FROM agent_memory m
 WHERE m.status='pending';

ACCEPTANCE GATE — the audit-row count must equal `activated`

This is a gate, not a query to glance at. It is the only detector for a dropped audit row (see Known limitation 3: emission is post-commit and best-effort, so a PG failure after the retries logs and drops while the sweep still reports success). Run it immediately after the live pass:

SELECT count(*) AS audit_rows
  FROM agent_audit_log WHERE action_type = 'memory_reclassified';

SELECT count(*) AS resolved_approvals
  FROM governance_approvals
 WHERE action_type = 'memory_review' AND reason = 'reclassified_by_2322';

Expected
`audit_rows`	== the run's reported `activated`
`resolved_approvals`	== the run's reported `approvals resolved` (= `activated` − `approvals missing`)

If audit_rows < activated, STOP and escalate. Memories have been moved out of quarantine with no durable record of it. Recover the missing rows from the structured logs — every decision is emitted at INFO as msg="memory review audit" before the durable write, so the full detail survives even when the INSERT does not. Grep for "action":"memory_reclassified" in the run's stderr.

Reading the counts

approvals missing counts activated rows that had no pending approval to resolve — either the #2323 stranded class, or a memory whose approval was already denied/approved/expired. Neither is a failure and neither is repaired here.

One distortion to know about (#2506 review, N1): ix_gov_approvals_dedup is UNIQUE on (org_id, session_id, tool_name, args_sha256) WHERE status='pending' and does not key on target, so nothing at the schema level prevents two pending memory_review approvals pointing at one memory. If that ever happens the join fans out and the memory is examined twice, so would activate / still quarantined over-count. The safety properties are unaffected — resolvePairedApproval has no LIMIT and resolves every returned id, so no orphan appears, and the second pass no-ops on the CAS. Live instances at the time of writing: 0 (all 45 pending rows have exactly one approval). Verify with:

SELECT m.id::text, count(g.id) AS approvals
  FROM agent_memory m
  JOIN governance_approvals g
    ON g.action_type='memory_review' AND g.status='pending'
   AND (g.target = m.id::text OR g.metadata->>'memory_id' = m.id::text)
 WHERE m.status='pending'
 GROUP BY 1 HAVING count(g.id) > 1;

Known limitations — read these before running

1. The audit rows carry `session_id = NULL` (#2493)

A review-gate transition is tenant- and agent-scoped but not session-scoped (see internal/context/memory/review/audit.go), so every row this tool writes to agent_audit_log lands with a NULL session_id — exactly like every approve/reject/correct/forget row that precedes it. Under #2493 the hash chain is keyed on session_id, so these rows are unchained and invisible to the chain verifier.

This is a pre-existing shape, not a new defect, but the sweep adds a batch of such rows for a governance action, which is the worst place to have them.

What survives, stated precisely — this buys ANSWERABILITY, not tamper-evidence, and the distinction matters. The record of why no human answered lives on governance_approvals.reason and in the approval_events ledger, both keyed to the approval rather than to a session, so both are unaffected by #2493. If you need to answer "why was this resolved without a human?", ask the approval, not the audit chain.

But do not read that as closing #2493. approval_events has no append-only enforcement anywhere in the migration set — no trigger, no rule, no REVOKE — and it carries ON DELETE CASCADE from governance_approvals, so deleting the approval silently deletes its whole event history. This substitutes redundancy across three mutable surfaces for integrity on one. Redundancy raises the cost of a consistent forgery; it does not make forgery detectable.

The property that actually holds here is recomputability, and it is stronger than chaining. memory_reclassified records the output of a pure, deterministic function of (memory_type, agent_memory.content), and the sweep modifies neither input. Anyone can re-run classify.StatusFor over the corpus and reproduce the decision set exactly:

go run ./cmd/tools/memory-status-reevaluate -dry-run   # reproduces the verdicts

A hash chain proves a row was not edited; recomputation proves the row is true. That is emphatically not available for a human memory_approved row, whose content exists nowhere but the audit row itself. Deletion also leaves a detectable residue — an active memory whose approval vanished is #2323's stranded shape, which memory-approval-repair already finds.

2. A rationale-only redaction escalation is not recoverable

The extraction gate escalates to pending on a redaction hit in the candidate's rationale as well as its content, and the rationale is not persisted on agent_memory at all. A row escalated only that way, whose content is clean under both the classifier and the redaction-marker read, is indistinguishable from one the old model vote quarantined, and this sweep will activate it.

Bounding facts: the rationale is never recalled and never rendered, so the content being activated is clean by every test that reads the row; and on the beta-dev corpus measured for #2399, 0 of 45 pending rows carried a redaction marker at all, so the redaction escalation is not what put any of them in the queue.

3. The audit write is post-commit and best-effort

Consistent with every other review-gate transition: the status flip is authoritative and a failed audit insert does not roll it back (retried 3×, then logged at ERROR with the full entry). The preflight in Preconditions 1 removes the one systematic failure mode. The structured slog line (msg="memory review audit") carries the complete decision regardless, so a durable drop is still recoverable from logs.

The only detector for this path is the acceptance gate above (audit_rows == activated). Nothing else fails, and the run still exits 0 — so if you skip that gate, a silent drop is indistinguishable from a clean sweep.

Reverting

There is no bulk undo, by design. pending → active is the same transition a human approval produces, so nothing in the schema distinguishes a machine-reclassified row from an approved one — only the append-only audit trail does, and a rollback has no business consuming it. Migration 197.down deliberately reverses nothing for this reason.

To revert one memory, use the audited governed path:

go run ./cmd/tools/memory-review -op forget -org <o> -agent <a> -memory <m>

To find the candidates:

SELECT detail->>'memory_id' AS memory_id, org_id, agent_id, created_at
  FROM agent_audit_log
 WHERE action_type = 'memory_reclassified'
 ORDER BY created_at;

cmd/tools/memory-review hangs on exit (#2490 — a never-cancelled context handed to audit.Writer.Start). The work commits; budget a timeout and do not read the hang as a failure. memory-status-reevaluate does not have this bug: it owns the cancel.

Tool	Use when
`memory-status-reevaluate`	the backlog is quarantined under a superseded rule (this tool)
`memory-approval-repair`	a pending memory has no approval row at all (#2323)
`memory-summary-backfill`	an approval card renders as a bare UUID (#2325)
`memory-review`	you are deciding one memory as an operator

What this does​

Preconditions​

Procedure​

Step 4 is not optional, and not a formality​

Verifying afterwards​

ACCEPTANCE GATE — the audit-row count must equal activated​

Reading the counts​

Known limitations — read these before running​

1. The audit rows carry session_id = NULL (#2493)​

2. A rationale-only redaction escalation is not recoverable​

3. The audit write is post-commit and best-effort​

Reverting​

Related tools​