Temporal — operations runbook (dev + beta)

Issue #1796 (WF-10), Wave W0.5 of the Workflows-on-Temporal MVP (PRD #1783 / ADR-0023, tracker #1786). Temporal is the workflow execution substrate, adopted now and kept behind the Dispatcher seam (internal/bus/dispatcher.go) so the engine choice is reversible.

Scope of this runbook: operating the dev Temporal (the compose substrate shipped by WF-06, #1792) and the procedures for standing up beta. This runbook does NOT enable anything on beta and changes no flags anywhere — beta enablement and the TEMPORAL_ENABLED flip are separate, deliberate go-live actions gated on the Ashik kickoff ratification and the founder.

---

0. Decisions of record

Decision	Value	Source
Server image (dev)	temporalio/auto-setup:1.29.7 (digest-pinned)	docker-compose.dev.yml (WF-06 #1792)
UI image (dev)	temporalio/ui:2.52.1 (digest-pinned)	docker-compose.dev.yml
Persistence (dev)	Shared postgres:5432, DIRECT (not pgbouncer), separate temporal + temporal_visibility DBs	WF-06
Default namespace	upsquad-dev (auto-created on first boot)	WF-06
Retention — dev	72h	WF-06 / this runbook
Retention — beta	30d (720h) — set via namespace update, never in dev compose	this runbook (§3)
Dev frontend addr	127.0.0.1:7233 (loopback-only)	WF-06
Dev Web UI	http://127.0.0.1:8233 (loopback-only)	WF-06
App kill-switch	TEMPORAL_ENABLED (default false)	PRD #1783 (§5)
Beta shape (self-host GKE vs Temporal Cloud)	Ashik ratifies at kickoff — PRD #1783 question (j)	§7

Why Postgres-direct (not pgbouncer): Temporal's persistence layer relies on session-scoped state that breaks under PgBouncer transaction pooling, so the temporal container connects straight to postgres:5432. Keep this in mind for any manual DB work — point psql at 5432, never 6432.

Why a separate database, not new tables in upsquad: Temporal owns its schema end-to-end via the temporal-sql-tool baked into the auto-setup image (NOT golang-migrate). The upsquad DB and its migrations (internal/context/store/migrations) are untouched.

---

1. Start / stop

The Temporal server + UI are ordinary services in docker-compose.dev.yml. No app service depends on either, so the rest of the stack boots with Temporal stopped (the UI just shows a connection banner).

Make targets (preferred)

cd /opt/upsquad/upsquad-core

make dev-up               # brings up the whole dev stack incl. temporal + temporal-ui
make dev-temporal-status  # cluster health + `namespace describe upsquad-dev`
make dev-down             # stops + removes the whole dev stack

make dev-up prints the Temporal endpoints on completion:

  Temporal frontend → 127.0.0.1:7233   Temporal UI → http://127.0.0.1:8233
  Namespace: upsquad-dev  ('make dev-temporal-status' to verify)

Compose targets (just Temporal, without touching the rest of the stack)

cd /opt/upsquad/upsquad-core
COMPOSE="docker compose -f docker-compose.dev.yml"

# Start ONLY temporal + its UI (postgres must already be healthy)
$COMPOSE up -d temporal temporal-ui

# Stop ONLY temporal + its UI (rest of stack keeps running)
$COMPOSE stop temporal temporal-ui

# Restart temporal (e.g. after a config change)
$COMPOSE up -d temporal

# Follow logs
$COMPOSE logs -f temporal

The server has a Docker healthcheck (operator cluster health); it typically reports healthy ~40s after first boot (schema setup runs once). The UI has a soft dependency (service_started, not service_healthy) so a stopped server never blocks the UI container from starting.

Prod co-residence: the dev box also hosts the prod compose stack. The temporal container is capped at mem_limit: 512m (settles ~150–250 MiB after setup) and the UI at 128m. Before/after any manual bring-up, snapshot docker stats --no-stream so you can confirm you left headroom.

---

2. UI access

• URL (on the box): http://127.0.0.1:8233

• The port is bound to loopback only (127.0.0.1:8233:8080); it is not reachable from the network. Reach it from your laptop via the devbox SSH tunnel / WARP path (see docs/runbooks/devbox-ssh-warp.md), e.g.:

  ssh -N -L 8233:127.0.0.1:8233 devbox   # then browse http://localhost:8233

• The UI is scoped to the upsquad-dev namespace by default; use the namespace switcher (top bar) to view temporal-system.

---

3. Namespaces — create / describe / retention

All commands run the temporal CLI from inside the server container (no CLI install needed on the host). -T disables the pseudo-TTY so they work in scripts and CI.

cd /opt/upsquad/upsquad-core
T() { docker compose -f docker-compose.dev.yml exec -T temporal temporal "$@" --address 127.0.0.1:7233; }

Describe (read-only) — the everyday check

T operator namespace describe -n upsquad-dev
T operator namespace list          # all namespaces (upsquad-dev + temporal-system)
T operator cluster health          # -> SERVING

describe prints Config.WorkflowExecutionRetentionTtl — for dev this is 72h0m0s.

Create a namespace (this is the beta bootstrap command)

Dev's upsquad-dev is auto-created by the auto-setup image, so you rarely create by hand in dev. On beta the operator creates the namespace explicitly with the 30-day retention:

# BETA: create the app namespace with 30-day (720h) retention.
# Run against the beta Temporal frontend address (see §7 for what that is
# under each variant).
temporal operator namespace create \
  --retention 720h \
  --description "UpsQuad beta workflow namespace" \
  -n upsquad-beta \
  --address <beta-frontend:7233>

Retention is expressed as a Go duration: 72h (dev), 720h = 30d (beta). Temporal enforces a minimum of 24h.

Change retention on an existing namespace

# Example: pin a namespace to 30 days.
T operator namespace update --retention 720h -n <namespace>
# then confirm:
T operator namespace describe -n <namespace> | grep RetentionTtl

Delete a namespace (rare; requires typed confirmation)

T operator namespace delete -n <namespace> --yes

Validated 2026-07-15 against the live dev container (create @720h → describe → update to 72h → delete) — all four commands succeed as written.

Configurable dev defaults (WF-10)

docker-compose.dev.yml parameterises the default namespace + dev retention so an operator can override them from the repo-root .env without editing YAML:

# .env (box-local, gitignored) — OPTIONAL overrides. Defaults reproduce WF-06.
TEMPORAL_NAMESPACE=upsquad-dev     # default
TEMPORAL_DEV_RETENTION=72h         # default

The defaults reproduce the WF-06 values exactly, so docker compose config renders byte-identically and the reconciler sees no drift until you set them. Do not encode the beta 30d value here — beta retention is set on the beta cluster via namespace update (above), never through the dev compose.

---

4. Kill-switch semantics — TEMPORAL_ENABLED

The Temporal server container running is NOT the same as workflows being live. Whether the application dispatches workflows to Temporal is gated by the app-level flag TEMPORAL_ENABLED (default false), per PRD #1783:

Flag	Behaviour
TEMPORAL_ENABLED=false (default)	New triggers are rejected (the trigger path returns a "workflows disabled" error). Any in-flight runs park durably — Temporal persists their history in the temporal/temporal_visibility DBs; nothing is lost, execution simply does not advance. This is the instant-disable / rollback path (PRD NFR-5).
TEMPORAL_ENABLED=true	The app dispatches new triggers to Temporal via the Dispatcher seam; parked runs resume as workers poll.

This flag is app-level (read by the coordinator / Dispatcher — WF-11+), not a compose gate: the substrate container runs regardless so it can be exercised in dev before go-live. It composes with the existing staged-enablement flags BUS_ENABLED / COORDINATOR_ENABLED / HARNESS_ENABLED (all default off, never run in prod during MVP).

Where the flag lives: the box-local, gitignored repo-root .env. There is no repo-tracked default that turns it on — flipping it is a deliberate operator action, and beta before prod (PRD NFR-5). This runbook and its accompanying PR change no flags.

dev-deploy reachability gate (inert until the flag flips)

.github/workflows/dev-deploy.yml ("Deploy Dev Stack") has a Verify Temporal namespace step that:

• soft-skips and passes when TEMPORAL_ENABLED is unset/false (the default) — so it is inert on the live dev box and cannot break a deploy today;
• asserts temporal operator namespace describe -n upsquad-dev succeeds when TEMPORAL_ENABLED=true, failing the deploy (and firing the Discord notifier) if the substrate is enabled but the namespace is unreachable.

To exercise the assertion at go-live, set TEMPORAL_ENABLED=true in the box .env and let the next reconcile (or a manual workflow_dispatch) run.

---

5. Worker-liveness monitoring

MVP runs a single-replica worker harness (PRD NFR-6); the durable harness ledger (#1259) is out of scope, so a worker crash means in-flight steps stall until it restarts — the run history is safe in Temporal, but liveness must be watched.

What to watch:

1. Cluster health — make dev-temporal-status (or T operator cluster health → SERVING). This is the substrate, not the worker.

2. Task-queue pollers — once the worker registers (WF-11+), a live worker shows up as a poller with a recent LastAccessTime:

   T task-queue describe --task-queue <role-or-team-queue> --namespace upsquad-dev
   # look for a non-empty Pollers list with a fresh LastAccessTime

   No pollers on a queue that should have a worker == worker down / not deployed. This is the primary liveness signal.

3. Stuck runs — runs sitting in Running with no recent event advance:

   T workflow list --query 'ExecutionStatus="Running"' --namespace upsquad-dev

Namespace gotcha: the CLI's built-in default namespace is default, which we do not create. operator namespace ... commands already name the namespace with -n; for workflow / task-queue commands pass --namespace upsquad-dev (or -n upsquad-dev) explicitly, or you'll hit a "namespace default not found" error.

4. Grafana/Prometheus — the dev stack already runs Prometheus + Grafana. Temporal's SDK/server metrics wiring (worker poll success rate, task latency, schedule_to_start) is a follow-on; until it lands, use (1)–(3).

Determinism / worker-versioning (PRD NFR-4, Ashik's ownership — question j): workflow bodies must be deterministic (no time.Now()/rand/direct I/O — those go through activities), and worker deploys must follow worker-versioning discipline with continue-as-new for long histories. A determinism-lint CI gate is coordinated with the Temporal PRD (#623). Operationally: never hot-swap worker code for a build-id that has in-flight workflows without a version set.

---

6. Schema upgrades (auto-setup image bump)

Temporal's DB schema is versioned by the temporal-sql-tool baked into the auto-setup image, not by golang-migrate. On boot, auto-setup runs setup-schema / update-schema idempotently against the temporal and temporal_visibility databases — so upgrading the schema == bumping the auto-setup image tag and recreating the container.

Procedure

1. Pick the target version. Read the Temporal server release notes for the target and every intermediate version between the current pin (1.29.7) and the target. Note any entry that says "schema change" — Temporal supports rolling schema upgrades but you should not leap across a schema-incompatible boundary blindly. Prefer stepping one minor line at a time when a release flags a schema bump.

2. Resolve the new digest (we digest-pin every image):

   docker manifest inspect temporalio/auto-setup:<NEW_VERSION> \
     | grep -m1 -i '"digest"' || \
   docker pull temporalio/auto-setup:<NEW_VERSION> \
     && docker inspect --format '{{index .RepoDigests 0}}' temporalio/auto-setup:<NEW_VERSION>

3. Update the pin in docker-compose.dev.yml (both the tag AND the @sha256: digest must move together):

   temporal:
     image: temporalio/auto-setup:<NEW_VERSION>@sha256:<NEW_DIGEST>

   Do the same for temporalio/ui when bumping the UI, using its release notes for CLI/UI compatibility with the server version.

4. Open a PR (branch → PR → review by devops-engineer + principal-architect per the routing table; infra change → architect must review). Never push to main directly.

5. On merge, the dev-deploy reconciler pulls the new digest and recreates the temporal container; auto-setup applies update-schema on boot. Watch:

   docker compose -f docker-compose.dev.yml logs -f temporal   # look for schema update lines
   make dev-temporal-status                                    # -> SERVING + namespace intact

6. Rollback (schema upgrades are forward-compatible within a supported window, so app/image rollback is the fast lever): revert the pin to the prior @sha256: digest and reconcile. The reconciler already snapshots the prior digest and auto-rolls-back on an unhealthy recreate (scripts/dev-reconcile.sh). Do not attempt a schema downgrade — pin the image back and, if a schema change is genuinely incompatible, restore the temporal/temporal_visibility DBs from the postgres backup taken before the upgrade.

Before any beta schema upgrade, snapshot the Temporal databases first (Cloud SQL automated backup / on-demand backup, or pg_dump of temporal + temporal_visibility). Under Temporal Cloud (§7 variant b) schema is managed by Temporal — there is no auto-setup image to bump; you only track SDK/server-feature compatibility.

---

7. Phase-1 — beta stand-up (BOTH variants; seam unchanged)

Ashik ratifies at kickoff — PRD #1783 question (j). Temporal timing is decided (adopt now, founder 2026-07-11). What Ashik ratifies is the shape: self-host on GKE vs Temporal Cloud (he carries ops; Cloud keeps the calendar tighter and is ADR-0013-compliant behind the seam), plus worker-versioning/deploy-discipline ownership.

The application seam is identical either way. The app talks to a Temporal frontend address + namespace + (optional) mTLS creds, all injected via env behind internal/bus/dispatcher.go. Nothing above the Dispatcher knows or cares which variant is live — so this decision does not fork application code, only the infra + connection env. Both variants below stop at the same seam.

Connection seam (common to both variants)

The Dispatcher is configured purely from env — this is the ONLY surface that differs between variants:

TEMPORAL_ENABLED=false              # stays false until go-live (§4)
TEMPORAL_ADDRESS=<frontend>:7233    # dev: temporal:7233
TEMPORAL_NAMESPACE=upsquad-beta     # dev: upsquad-dev
# mTLS (Temporal Cloud, and optionally self-host with TLS):
TEMPORAL_TLS_CERT=/secrets/temporal/client.pem
TEMPORAL_TLS_KEY=/secrets/temporal/client.key

Client certs/keys are secrets — mounted from the secret store (GCP Secret Manager / mounted key file), never committed, never in .env in the repo, never in compose. Same discipline as the Vertex SA key (docs/runbooks/beta-vertex-embeddings.md).

Variant (a) — Self-host on GKE (temporalio/helm-charts + separate Cloud SQL)

• Chart: temporalio/helm-charts (the official chart). Deploy the 4 Temporal roles (frontend, history, matching, worker) — NOT the chart's bundled Cassandra/Elasticsearch/Prometheus/Grafana; disable those subcharts.
• Persistence: a separate Cloud SQL Postgres instance (its own, not the app DB), private-IP, in the beta VPC. temporal + temporal_visibility databases. Postgres visibility (no Elasticsearch) is the MVP posture.
• Schema: run temporal-sql-tool setup-schema / update-schema (the chart's schema job) against that Cloud SQL — the Helm equivalent of the auto-setup boot step in §6. Direct connection (no PgBouncer), same reasoning as dev.
• Namespace: temporal operator namespace create --retention 720h -n upsquad-beta against the chart's frontend Service (§3).
• TLS: terminate mTLS at the frontend (chart supports it) or run inside the mesh; inject client certs via the seam above.
• Ops burden (why this is Ashik's call): cluster upgrades, schema jobs, scaling the 4 roles, backup/restore of the Cloud SQL, and worker-versioning are all self-operated.
• Seam: TEMPORAL_ADDRESS = the frontend Service DNS; namespace = upsquad-beta. Unchanged app code.

Variant (b) — Temporal Cloud (managed)

• Provision: a Temporal Cloud account + namespace (upsquad-beta.<acct>), region-matched to beta. ~$100–500/mo, ~$0.0003–0.0005 per agent step (PRD).
• Auth: mTLS client certificate issued for the namespace; store the cert/key in the secret store and inject via TEMPORAL_TLS_CERT / TEMPORAL_TLS_KEY (seam above). No cluster to run.
• Schema: managed by Temporal — no auto-setup image, no schema job, no Cloud SQL to back up. §6's image-bump procedure does not apply; you only track SDK/server-feature compatibility.
• Retention: set on the namespace via the Cloud UI/API (or temporal operator namespace update --retention 720h against the Cloud address with the mTLS creds).
• Ops burden: minimal infra ops; Temporal Cloud runs the cluster. Worker fleet + worker-versioning are still ours.
• Seam: TEMPORAL_ADDRESS = <namespace>.<account>.tmprl.cloud:7233; namespace = the Cloud namespace; mTLS creds required. Unchanged app code.

Decision matrix (for the Ashik kickoff)

Axis	(a) Self-host GKE	(b) Temporal Cloud
Infra to run	4 roles on GKE + separate Cloud SQL	none (managed)
Schema ops	self (§6-style jobs)	managed
Cost	GKE + Cloud SQL compute	~$100–500/mo usage-based
Calendar	slower (stand-up + burn-in)	tighter
Backups	self (Cloud SQL)	managed
mTLS	optional / mesh	required
ADR-0013 compliance	behind seam	behind seam (PRD notes Cloud is compliant)
App code change	none	none

No beta enablement is performed here. Standing up beta under either variant is a separate action gated on this ratification and the founder; the TEMPORAL_ENABLED flip is separate again (§4).

---

8. Quick reference

cd /opt/upsquad/upsquad-core
COMPOSE="docker compose -f docker-compose.dev.yml"
T() { $COMPOSE exec -T temporal temporal "$@" --address 127.0.0.1:7233; }

make dev-up / make dev-down / make dev-temporal-status
$COMPOSE up -d temporal temporal-ui      # start just temporal
$COMPOSE stop temporal temporal-ui       # stop just temporal
T operator cluster health                # SERVING
T operator namespace describe -n upsquad-dev
T operator namespace update --retention 720h -n <ns>   # 30d
T task-queue describe --task-queue <q> --namespace upsquad-dev   # worker liveness
T workflow list --query 'ExecutionStatus="Running"' --namespace upsquad-dev
# UI: http://127.0.0.1:8233  (loopback; tunnel from laptop)

9. CI — Temporal test jobs (WF-09)

Three CI surfaces exercise Temporal; all are path-filtered so unrelated PRs report success in seconds.

9.1 temporal_integration tests — .github/workflows/temporal-tests.yml › integration

Boots an in-memory, headless Temporal dev-server (the temporal CLI's server start-dev, NOT the compose auto-setup image) and runs the temporal_integration-tagged Go tests against it. The CLI is installed from a pinned release tarball with a sha256 check (temporalio/cli v1.8.0) — no third-party action, no curl | sh. Health must land inside a 30s budget (the dev-server is typically SERVING in ~1s).

Canonical env vars — every tagged test reads TEMPORAL_SERVER_ADDRESS (default 127.0.0.1:7233) and TEMPORAL_NAMESPACE (default upsquad-dev); the CI job exports both. Standardise on these when adding new tests.

Integration package set (mirrored in Makefile TEMPORAL_INTEGRATION_ROOTS) — the dirs that ship tagged tests, filtered to those that exist:

Dir	Tagged coverage
internal/bus	WF-08 TemporalDispatcher start→signal→complete e2e (+ non-workflow rides Redis via in-process miniredis — no external Redis needed)
cmd/agent-orchestrator	WF-07 startTemporal connects/registers a worker + fail-loud on unreachable
internal/temporalclient	WF-07 client wrapper unit tests
internal/temporalwf	WF-09 SDK-free reachability smoke

Write a Temporal test that needs a live frontend behind the build tag:

//go:build temporal_integration

package bus
// dial TEMPORAL_SERVER_ADDRESS (default 127.0.0.1:7233), TEMPORAL_NAMESPACE …

Run locally (needs a reachable frontend — the dev compose temporal works):

make test-temporal-integration                       # dials 127.0.0.1:7233
TEMPORAL_SERVER_ADDRESS=127.0.0.1:7233 make test-temporal-integration
# or boot a throwaway dev-server (needs the `temporal` CLI on PATH):
make temporal-dev-server        # shell A — foreground, in-memory, :7233
make test-temporal-integration  # shell B

go test ./internal/temporalclient/... is a hard error — not a no-op — while a dir is absent, so the set is filtered to existing dirs at run time. Adding a new top-level substrate dir means extending the loop + TEMPORAL_INTEGRATION_ROOTS; new sibling packages under an existing root are picked up automatically.

9.2 Replay / determinism gate — temporal-tests.yml › replay

worker.WorkflowReplayer replays committed history fixtures under internal/temporalwf/testdata/histories/*.json:

• No fixtures → SKIP (green). State until WF-11 lands the interpreter.
• Fixture present + replay diverges → FAIL. An interpreter change that breaks replay of an old history must ship a workflow.GetVersion patch and a new fixture (never edit an old one). See that dir's README.md for how to export a history and the naming convention.

Run locally: make test-replay.

The same job runs Temporal's workflowcheck static analyzer over internal/temporalwf/interpreter/ (pinned go.temporal.io/sdk/contrib/tools/workflowcheck@v0.5.0). Skip-if-absent until WF-11 creates the interpreter package. Run locally: make workflowcheck.

WF-11 follow-up: once real fixtures exist, add go.mod/go.sum to the temporal-tests.yml trigger filter so an SDK bump re-runs the replay gate.

9.3 Golden-flow E2E compose — .github/workflows/golden-flow-e2e.yml

The E2E job adds the compose temporal service (pinned auto-setup image) to its postgres redis migrate set only when a temporal-substrate path changes (internal/temporalwf/**, internal/temporalclient/**). The image is pre-pulled before the boot/health window and health-waited (120s budget — auto-setup runs schema on first boot). Until the golden flow exercises temporal-backed runs (W2 SessionExecutor bridge) this only proves the infra comes up alongside the suite.

Related

• docs/runbooks/dev-migrate-dirty-recovery.md — dev-box /opt sync + migrate.
• docs/runbooks/beta-vertex-embeddings.md — secret-handling pattern for beta.
• docs/runbooks/devbox-ssh-warp.md — reaching loopback ports from your laptop.
• PRD #1783 / ADR-0023, tracker #1786, WF-06 (#1792) compose substrate.