Runbook: the vault passphrase

Audience: whoever enters a real provider key, a GitHub App private key or any other tenant secret into a stack, and whoever changes that stack's vault passphrase. Origin: #3476 (2026-09-19).

1. What this protects

The platform vault is two columns, both pgp_sym_encrypt'ed by pgcrypto under the stack's VAULT_PASSPHRASE:

Table	Column	Written by	Read by
provider_keys	encrypted_key	context-engine (SetProviderKey, SetLLMEndpointCredential, MCP and Slack credentials), agent-orchestrator, vault-seed, siem-export-worker	context-engine, agent-orchestrator, model-gateway (endpoint credentials)
scm_credentials	encrypted_payload	agent-orchestrator / context-engine (GitHub App and SCM credentials)	same

pgcrypto has no key escrow. If you lose the passphrase you lose every row. Keep it in the founder's password manager.

2. The rule

One passphrase per stack. It comes from outside the repo, and every vault reader and writer on that stack gets the identical value.

Stack	Where the value comes from	What enforces identity
beta-dev (compose project upsquad-dev on the devbox)	VAULT_PASSPHRASE= in the box-local, untracked /opt/upsquad/upsquad-core/.env. That is the file the reconciler already reads.	One x-vault-passphrase anchor in docker-compose.dev.yml (pinned by test/lint/vault_passphrase_posture_test.go), plus the reconciler's vault preflight: scripts/dev-reconcile.sh refuses with exit 4, before any pull or recreate, when the rendered config gives the consumers the dev literal, an empty value, disagreeing values or no value at all.
throwaway local stack	nothing: the anchor falls back to the repo-public dev-vault-passphrase-not-for-production	same anchor
on-prem tenant	.env.tenant (${VAULT_PASSPHRASE:?}). scripts/tenant/tenant-preflight.sh refuses empty and CHANGE_ME values.	one variable, three services
GKE (mnet overlay)	ExternalSecret remote key mnet-vault-passphrase, one key for all three services	one remote key
ICS sandbox	its own local literal (ENVIRONMENT=sandbox), throwaway data only	one anchor in docker-compose.ics.yml

The startup gate

vault.MustCheckPassphraseEnv() (internal/vault/passphrase.go) runs at the top of run() in context-engine, agent-orchestrator, model-gateway, siem-export-worker and vault-seed. When ENVIRONMENT is anything other than exactly development, and an absent ENVIRONMENT counts as not development, the process exits 1 if:

• VAULT_PASSPHRASE is the repo-public dev literal, or
• VAULT_PASSPHRASE is present but empty. That is the signature of an interpolation that resolved to nothing.

An absent VAULT_PASSPHRASE is still admitted, because it means the vault is disabled. Every consumer then fails closed at the credential step. Refusing it would crash-loop the GKE base manifests, which deliberately carry none for context-engine and model-gateway.

beta-dev runs ENVIRONMENT=development, so the gate does not protect it. On beta-dev the control is the reconciler instead.

The reconciler preflight (beta-dev)

vault_passphrase_preflight() in scripts/dev-reconcile.sh runs first in every dev-deploy reconcile. It renders docker compose config (which includes the box .env) and exits 4 without pulling or recreating anything if:

• no service carries VAULT_PASSPHRASE;
• any consumer's value is empty;
• the consumers disagree, which was the live #3476 defect; or
• every consumer would get the repo-public dev literal, meaning the .env line is missing.

It logs sha256 8-char prefixes only. The deploy job fails and the Discord notifier fires on every sweep until this is fixed. A missing .env line therefore produces a red deploy, never a stack running on the public literal.

There is one override, and it covers only the dev-literal case: DEV_RECONCILE_ALLOW_DEV_VAULT_PASSPHRASE=1, read from the job env or else the box .env. With it set, the run proceeds and logs a WARNING. It exists for the §5 rollback. Empty, disagreeing or missing values are refused regardless. It is deliberately an opt-out: an opt-in "this is beta-dev" marker would live in the same .env as the passphrase and vanish with it.

3. Measuring (read-only, safe at any time)

cd /opt/upsquad/upsquad-core
python3 scripts/vault-rekey.py --check

It prints the passphrase class (empty / dev-literal / target / other) and the sha256 8-char prefix of every running vault consumer, whether they all agree with the value compose resolves, and a per-table census: rows total, rows decryptable under the target, under the source (by default the dev literal), and under neither. It never prints a passphrase or a plaintext. Every other part of this procedure follows the same rule: compare passphrases by sha256 prefix, never by value, in terminals, logs, issue comments and PRs:

printf '%s' "$VALUE" | sha256sum | cut -c1-8

The tool refuses when the connected role cannot bypass RLS. provider_keys is FORCE ROW LEVEL SECURITY, so such a role sees only part of the table and would under-report.

Measured on beta-dev, 2026-09-19, before this procedure existed: context-engine and agent-orchestrator held the dev literal (d625c33d), and model-gateway held an empty value (e3b0c442). There were 5 rows (provider_keys 4, scm_credentials 1), and all 5 decrypted under the dev literal. The gateway booted with credentials_enabled=false.

4. Cutover: giving beta-dev a real passphrase

Founder-run. The founder supplies the secret. Do the steps in this order. Merging #3476 deploys automatically, so step 1 must come before the merge. If the merge lands first anyway, nothing unsafe happens: every dev-deploy sweep exits 4 at the vault preflight, and all beta-dev deploys stop, not just this one, until step 1 is done. The next sweep after step 1 then deploys normally.

Step 1: create the passphrase outside the repo (before merging #3476)

umask 077
P="$(openssl rand -hex 32)"      # hex: nothing compose's .env parser can misread
# Store "$P" in the password manager NOW, before anything else.
printf '\nVAULT_PASSPHRASE=%s\n' "$P" >> /opt/upsquad/upsquad-core/.env
printf '%s' "$P" | sha256sum | cut -c1-8     # record THIS prefix, not the value
unset P

The 5-minute dev-deploy sweep then recreates model-gateway only. Before #3476 the two writers hardcode the literal and ignore .env. That is harmless, because the gateway could not decrypt anything before this and still cannot.

Step 2: merge #3476 and let it deploy

dev-deploy syncs /opt/upsquad/upsquad-core to main and reconciles. context-engine and agent-orchestrator are recreated on the new value. Confirm that the running consumers, not just the files, agree:

python3 scripts/vault-rekey.py --check
#   every consumer: "target sha8=<your prefix>", "parity with target: OK"
#   census: under_target 0, under_source 5  (rows not re-keyed yet)

From this point until step 3 finishes, existing vault rows cannot be read. Provider-key reads and the GitHub App credential fail closed. Go straight to step 3.

Step 3: re-key the existing rows

python3 scripts/vault-rekey.py --apply

This runs as one transaction. It re-encrypts every row that decrypts under the source (the dev literal) and not yet under the target, then COMMITs only if every row now decrypts under the target. Otherwise it ROLLs BACK. It refuses before touching anything if a running consumer does not hold the target, if the target is empty or the dev literal, or if any row decrypts under neither passphrase. It is idempotent: a second run re-keys 0 rows.

Step 4: verify

python3 scripts/vault-rekey.py --check
#   under_target 5, under_source 0, neither 0, parity OK
docker logs upsquad-model-gateway 2>&1 | grep -m1 'controls wired' | grep -o 'credentials_enabled=[a-z]*'
#   credentials_enabled=true

Step 5: write-then-read round trip (the proof the issue asks for)

1. Write through the product path. Set the FastRouter key on its endpoint with AIGatewayEndpointCredentialService.SetLLMEndpointCredential, either through the portal's credential slot or with the call in docs/runbooks/model-gateway-external-agent-onboarding.md §1 step 3.
2. Read in the writer. ListLLMEndpointCredentials shows the masked hint (last 4 characters) for that endpoint. That hint is decrypted inside Postgres by context-engine's passphrase.
3. Read at rest. python3 scripts/vault-rekey.py --check shows provider_keys total up by one, and every row under_target.
4. Read in the reader. Make one governed call through the model gateway, per the onboarding runbook §2–§3 (agent token, then POST to :8091). A relayed response proves the gateway decrypted what context-engine wrote. credential_missing means the reader and writer disagree: go back to step 2's parity check.

Step 6: rotate what sat under the public passphrase

Re-keying protects a secret from now on. It does not undo the time it spent under a published string. Rotate each real credential that was in the vault before the cutover at its issuer. The rows were: an anthropic provider key (2026-07-02), a github_app SCM credential (2026-07-20), an MCP bearer token (2026-07-21) and two llm_endpoint_* keys (2026-09-10/11). Then re-enter each one through the product. Postgres on the devbox binds 127.0.0.1 only, so the exposure was box users and anyone in the docker group, together with anyone who can read the repo. It was not the internet.

5. Rollback (under 5 minutes)

Once #3476 is merged, going back to the dev literal needs the reconciler override. Without it, every sweep refuses with exit 4. The refusal is safe, because nothing gets recreated, but it stays red. Swap the passphrase line for the override in the box .env:

cd /opt/upsquad/upsquad-core
sed -i '/^[[:space:]]*VAULT_PASSPHRASE=/d' .env
printf 'DEV_RECONCILE_ALLOW_DEV_VAULT_PASSPHRASE=1\n' >> .env

The fast path takes under a minute. It recreates the three consumers from /opt so that .env applies. It is config-only: no pull, no GHCR login, no dependencies touched. The next sweep then sees no drift and, because of the override, logs a WARNING instead of refusing:

cd /opt/upsquad/upsquad-core
docker compose -f docker-compose.dev.yml up -d --no-deps --pull never context-engine agent-orchestrator model-gateway

The slow path is to dispatch dev-deploy (workflow_dispatch). It can queue behind an in-flight sweep, so it sits at the 5-minute edge.

Where you are	Do this
after step 1, before the merge	Delete the VAULT_PASSPHRASE line from .env. The next sweep puts the gateway back to empty. No override is needed, because the pre-#3476 reconciler has no preflight.
after step 2, before step 3	Swap in the override (above), then run the fast path. Every consumer falls back to the dev literal, and the rows never left it.
after step 3	read -rs VAULT_REKEY_OLD; export VAULT_REKEY_OLD (the stack passphrase, from the password manager). Swap in the override, run the fast path, then python3 scripts/vault-rekey.py --apply --to-dev-literal. That is one transaction with the same post-check, and it rolls back on failure.

Remove the override once you go forward again. It is a WARNING on every sweep for a reason.

Do not revert only the compose hunk of #3476. That restores the reader/writer split itself. Revert the whole PR or none of it.

6. Traps

• The reconciler preflight guards only dev-deploy. A hand-run docker compose up bypasses it. Run hand-runs from /opt/upsquad/upsquad-core, as the fast path does.
• docker compose up from a worktree without --env-file /opt/upsquad/upsquad-core/.env starts the consumers on the dev fallback. It resets the Clerk and auth settings the same way (see devbox.md). Rows written in that window are sealed under the public literal. The reconciler restores the right value within 5 minutes, but those rows stay where they are: --check shows them as under_source > 0. Re-run --apply.
• .env survives the deploy's git reset --hard. It does not survive git clean -x. The password manager is the backup.
• Never pass a passphrase in argv (psql -v, -e VAR=value, a URL) and never echo it. vault-rekey.py hands values to psql through docker exec -e NAME (names only), \getenv and \bind, so they stay out of ps, out of statement text and out of the server log.
• docker inspect shows container env to anyone in the docker group. On this box that is the trust boundary for every secret in .env, not just this one.