Workflow determinism — operations + deploy-drill runbook

Issue #1945 (WF-32), Wave W5 of the Workflows-on-Temporal MVP (PRD #1783 / ADR-0023, tracker #1786, LLD #1798 §6 + §9). This runbook covers the replay/determinism gate for the DSL interpreter (internal/temporalwf/interpreter): how the golden history fixtures work, how to version the interpreter safely with workflow.GetVersion, and how to debug a replay failure.

Companion: docs/runbooks/temporal-ops.md (running the dev/beta Temporal substrate). This runbook is about code correctness under replay, not about operating the server.

0. Why this gate exists (read once)

Temporal re-executes workflow code from the recorded history on every worker restart, continue-as-new, and failover. The interpreter (RunWorkflow) MUST produce the exact same sequence of commands (schedule-activity, start-timer, signal-wait, …) it produced the first time, or Temporal raises [TMPRL1100] nondeterministic workflow and the run is stuck. A change that alters that command sequence for an already-running workflow is a determinism break.

Two gates catch this before merge, both wired into the Temporal replay + determinism CI job (.github/workflows/temporal-tests.yml) and both runnable locally:

Gate	Command	What it catches
Replay	`make test-replay`	An interpreter change that no longer reproduces a recorded history — replays every golden fixture against the current code, fails on divergence.
workflowcheck	`make workflowcheck`	A statically non-deterministic call reachable from a registered workflow (`time.Now`, `rand`, `map` range, direct I/O, goroutines, …) — Temporal's `workflowcheck` analyzer.

The golden line: LLD §6.5 / §9.5 — never edit an existing fixture. A new interpreter shape adds a new _vN.json fixture behind a workflow.GetVersion patch. Editing a fixture to make a red gate go green re-writes history that real running workflows still replay against — it defeats the entire gate.

1. Decisions of record

Decision	Value	Source
Fixture location	`internal/temporalwf/testdata/histories/*.json`	`replay_test.go` (`historyFixtureGlob`)
Fixture format	Temporal JSON history (`temporal workflow show --output json`)	README in that dir
Naming	`<scenario>_v<N>.json`, `N` = interpreter shape the fixture pins	README + §2
Replayer	`worker.WorkflowReplayer` over `interpreter.RunWorkflow`, registered as `api.RunWorkflowType`	`replay_test.go` `replayOne`
Coverage guard	`replay_coverage_test.go` — every W1 node type / control path MUST have a fixture	WF-18 #1821
Static analyzer	`go.temporal.io/sdk/contrib/tools/workflowcheck` v0.5.0 (latest published tag)	`Makefile` `WORKFLOWCHECK_VERSION`
Analyzer build toolchain	repo go.mod version (`GOTOOLCHAIN=go<go.mod>`), NOT `local`/`auto`	`Makefile` `workflowcheck` (see §5)
CI check-run names	`Temporal replay + determinism`, `Temporal integration (dev-server)`	`temporal-tests.yml` (stable → branch-protection-ready)
Version-marker convention	`wf<wave>-<kebab-summary>` (e.g. `wf16-cancel-voids-pending-approval`)	LLD §6.4, interpreter constants

There is no workflowcheck "v1.13.0"

The analyzer submodule …/contrib/tools/workflowcheck tops out at v0.5.0. The 1.x version line belongs to the top-level go.temporal.io/sdk module (this repo pins v1.46.0 in go.mod) — a different module that versions independently. Do not try to pin the analyzer to a v1.x tag; it does not exist and go install …@v1.13.0 will fail.

2. The committed fixtures

One per shipped node-type / control-path scenario, all at interpreter shape v1 except the two version-gated ones:

Fixture	Exercises	Version marker it pins
`linear_dag_v1.json`	two `agent_action` → `terminal`	— (DefaultVersion)
`gate_approve_v1.json`	`approval_gate` → decision signal → approve → execute	—
`gate_reject_v1.json`	`approval_gate` → deny → `on_reject` branch	—
`govern_deny_v1.json`	`agent_action` blocked by governance deny (fail-closed)	—
`cancel_v1.json`	drain-cancel during a parked gate (pre-WF-16 shape, no VoidApproval)	`wf16-…` DefaultVersion branch
`cancel_void_v2.json`	WF-16 cancel that VOIDS the pending approval	`wf16-cancel-voids-pending-approval` = 1
`conditional_true_v1.json`	WF-26 CEL `EvalBranch` → true → `on_approve`	`wf26-conditional-cel-routing` = 1
`conditional_false_v1.json`	WF-26 CEL `EvalBranch` → false → `on_reject`	`wf26-conditional-cel-routing` = 1

The pre-patch fixtures (cancel_v1, and any pre-WF-26 conditional history) are the whole reason the version markers exist: they replay through the DefaultVersion arm of the GetVersion branch. The _vN fixtures replay through the new arm. Keeping both is what proves the patch is backward-compatible.

3. Exporting / versioning a new fixture

You add a fixture whenever you ship an interpreter change that produces a new command shape (see §4 — it always rides a GetVersion patch). Fixtures are exported from a dev-server run with FAKE activities (canned outputs, no DB), so they are deterministic and reproducible.

3.1 Boot an ephemeral dev-server

# Loopback-only in-memory dev-server, no Web UI. (Same substrate the CI job and
# the temporal_integration tests use — see docs/runbooks/temporal-ops.md §1.)
temporal server start-dev \
  --headless --namespace upsquad-dev --port 7233 --log-level error &
temporal operator cluster health --address 127.0.0.1:7233   # wait for healthy

For a fully throwaway run in a test, prefer testsuite.StartDevServer (ephemeral port, torn down with the test) — never point a generator at live/beta infra.

3.2 Run one scenario with fake activities

Register interpreter.RunWorkflow on wf-runs and a fake activity fleet on wf-activities (BootRun returns the scenario Definition; Govern returns allow/deny; ExecuteAgentStep returns a canned output; EvalBranch returns the CEL verdict you want; the terminals + Project + VoidApproval are no-ops). Start one run per scenario and signal it (approval_decision / cancel / pause) exactly as the scenario requires. This mirrors how the WF-13/WF-16/WF-26 fixtures were made.

There is deliberately no committed generator binary — the fake fleet is a handful of closures written inline for the scenario. The w1_exit_integration_test.go and e2e_integration_test.go suites are the reference for wiring a fake fleet.

3.3 Export the history to a versioned file

temporal workflow show \
  --workflow-id <run:org:run_id> \
  --namespace upsquad-dev \
  --address 127.0.0.1:7233 \
  --output json \
  > internal/temporalwf/testdata/histories/<scenario>_v<N>.json

N is the interpreter shape this fixture pins — bump it (_v1 → _v2) for a new shape; never overwrite an existing _vN (LLD §9.5).
Update the table in internal/temporalwf/testdata/histories/README.md and §2 above, and — if it's a new W1 node type / control path — the requiredW1Fixtures map in replay_coverage_test.go.

3.4 Verify

make test-replay        # the new fixture must replay clean against current code
make workflowcheck      # interpreter must stay statically deterministic

4. The `workflow.GetVersion` patch procedure (interpreter changes)

Any interpreter change that alters the command sequence for an already-running workflow MUST be gated by workflow.GetVersion, or it breaks replay of in-flight runs (and the golden fixtures). This is the single most important rule in this file.

4.1 The pattern

// 1. Declare a version-marker constant (convention: wf<wave>-<kebab>, LLD §6.4).
const versionMyChange = "wf27-my-new-hop"

// 2. Branch on GetVersion. DefaultVersion = the OLD behaviour (what recorded
//    histories replay against); >= 1 = the NEW behaviour (new runs only).
if workflow.GetVersion(ctx, versionMyChange, workflow.DefaultVersion, 1) >= 1 {
    // NEW shape: e.g. an extra activity hop, a new timer, a different route.
    _ = exec(ctx, api.ActMyNewHop, in, nil)
} else {
    // OLD shape: byte-for-byte what pre-patch histories recorded.
}

Two live examples in interpreter.go: versionCancelVoidsApproval (WF-16 — adds the VoidApproval hop on cancel) and versionConditionalCEL (WF-26 — adds the CEL EvalBranch route for conditional).

4.2 Why it works

GetVersion records a marker event in the history the first time it runs for a new workflow. On replay:

an old history has no marker → GetVersion returns DefaultVersion → the else arm runs → the recorded commands match. ✅
a new history has the marker → GetVersion returns 1 → the new arm runs → matches the new recorded commands. ✅

So the same code replays both the old and the new fixture. That is exactly why you keep the pre-patch fixture and add a new _vN one (§2).

4.3 Checklist for a versioned interpreter change

New behaviour is behind workflow.GetVersion(ctx, "<marker>", DefaultVersion, 1).
The else/DefaultVersion arm is identical to the pre-patch commands.
Existing fixtures still make test-replay clean (they hit DefaultVersion).
A new _vN.json fixture pins the new arm (exported per §3).
make workflowcheck is clean.
README + §2 tables + replay_coverage_test.go updated if it's a new node type.

Never delete or renumber a version marker once shipped — a run created under it still replays against it forever.

5. Running the gates locally

cd /opt/upsquad/upsquad-core

make test-replay     # replay every fixture against the current interpreter
make workflowcheck   # Temporal static determinism analyzer over the interpreter

The workflowcheck build-toolchain trap (do not "fix" it back)

make workflowcheck installs the analyzer with GOTOOLCHAIN=go<go.mod version> (e.g. go1.25.4). This is load-bearing. workflowcheck's own go.mod declares an older go directive, so a bare go install …workflowcheck@v0.5.0 (under GOTOOLCHAIN=auto/local) builds the analyzer binary with that older toolchain. The binary embeds go/packages + shells out to go list, which then rejects this repo:

workflowcheck: err: exit status 1: stderr: go: go.mod requires go >= 1.25.4 (running go 1.24.2; GOTOOLCHAIN=local)
workflowcheck: failed prerequisites: … (every interpreter dep)

— i.e. it loads nothing and the gate silently passes on an empty analysis. Building the analyzer with the repo's own go.mod toolchain lets its internal go list accept the repo. The Makefile derives that version from go mod edit -json so it never drifts. (WF-32 replaced the old, wrong GOTOOLCHAIN=local here — see the workflowcheck target comment.)

6. Debug workflow — a replay test just failed

make test-replay (or the CI Temporal replay + determinism job) went red with replay diverged for <fixture>: [TMPRL1100] nondeterministic workflow: ….

6.1 Read the divergence line

The error names both sides of the mismatch. Example (the WF-32 proof case, an accidental extra CompleteRun):

history event is  ActivityTaskScheduled: (ActivityId:95, ActivityType:(Name:Project), …)
replay command is ScheduleActivityTask:  (ActivityId:95, ActivityType:(Name:CompleteRun), …)

Read it as: "at command #95 the recorded history did Project, but the current code wants to do CompleteRun." The pair (recorded vs. wanted) points straight at the interpreter statement that changed the command order.

6.2 Localise it

# Replay just the failing fixture, verbose, to see the exact subtest + panic.
go test -count=1 -v -run 'TestWorkflowReplayDeterminism/<fixture>.json' \
  ./internal/temporalwf/

# Inspect the recorded history around the diverging ActivityId to see what the
# committed run actually did (jq over the JSON fixture).
jq '.events[] | select(.eventType | test("ActivityTaskScheduled"))
      | {id:.eventId, activity:.activityTaskScheduledEventAttributes.activityType.name}' \
  internal/temporalwf/testdata/histories/<fixture>.json

Then diff the interpreter against origin/main to find the command-order change:

git diff origin/main -- internal/temporalwf/interpreter/

6.3 Decide: bug or intended change?

It's a bug (you did not mean to change the command sequence): fix the interpreter so it reproduces the recorded commands. Green = fixed. Do not touch the fixture.
It's an intended change (new hop / new route / new timer): it MUST ride a workflow.GetVersion patch (§4). Put the new behaviour behind the marker, keep the DefaultVersion arm byte-identical to the old commands, and add a new _vN fixture for the new shape. The old fixture must still pass (DefaultVersion).
NEVER edit or delete an existing fixture to make the gate green — that silently breaks every in-flight production run that replays against it (LLD §9.5).

6.4 workflowcheck failed instead

…/interpreter.RunWorkflow is non-deterministic, reason: calls non-deterministic function time.Now
  time.Now is non-deterministic, reason: declared non-deterministic

A statically non-deterministic call is now reachable from the workflow. Replace it with the workflow-safe equivalent:

Don't (in workflow code)	Do
`time.Now()`	`workflow.Now(ctx)`
`time.Sleep`	`workflow.Sleep(ctx, d)`
`rand`, `uuid.New()`	`workflow.SideEffect` / pass in via input
`for k := range m` (map)	iterate a keyed slice / sorted keys
goroutine, channel, mutex	`workflow.Go`, `workflow.Channel`, cooperative state
direct DB/HTTP/file I/O	do it in an activity, call via `exec(ctx, …)`

7. Making the gate a required branch-protection check

The two check-runs are already stable and always-reporting (both jobs run on every PR; the changes filter gates only the inner steps), so they are safe to add to main branch protection as required contexts. As of WF-32 they are not yet required (branch protection lists only proto/DB/container/compose). Adding a required status check is a branch-protection change → founder decision (see CLAUDE.md escalation list); this runbook records the exact context strings to add:

Temporal replay + determinism
Temporal integration (dev-server)

Until then the gate still runs and blocks nothing automatically — but a red Temporal replay + determinism on a PR is a hard stop for the reviewer.

8. Quick reference

# Run both determinism gates locally
make test-replay
make workflowcheck

# List / inspect fixtures
ls internal/temporalwf/testdata/histories/*.json
jq '.events | length' internal/temporalwf/testdata/histories/<fixture>.json

# Export a NEW fixture from a throwaway dev-server (fake activities)
temporal server start-dev --headless --namespace upsquad-dev --port 7233 --log-level error &
#   … run one scenario with a fake activity fleet, then:
temporal workflow show --workflow-id <id> --namespace upsquad-dev --output json \
  > internal/temporalwf/testdata/histories/<scenario>_v<N>.json

# Replay a single failing fixture, verbose
go test -count=1 -v -run 'TestWorkflowReplayDeterminism/<fixture>.json' ./internal/temporalwf/

Golden rule: an interpreter change that alters commands rides a workflow.GetVersion patch + a new _vN fixture. Never edit an existing fixture.

0. Why this gate exists (read once)​

1. Decisions of record​

There is no workflowcheck "v1.13.0"​

2. The committed fixtures​

3. Exporting / versioning a new fixture​

3.1 Boot an ephemeral dev-server​

3.2 Run one scenario with fake activities​

3.3 Export the history to a versioned file​

3.4 Verify​

4. The workflow.GetVersion patch procedure (interpreter changes)​

4.1 The pattern​

4.2 Why it works​

4.3 Checklist for a versioned interpreter change​

5. Running the gates locally​

The workflowcheck build-toolchain trap (do not "fix" it back)​

6. Debug workflow — a replay test just failed​

6.1 Read the divergence line​

6.2 Localise it​

6.3 Decide: bug or intended change?​

6.4 workflowcheck failed instead​

7. Making the gate a required branch-protection check​

8. Quick reference​