KT — Workflow Orchestration
Audience: an engineer picking up anything that touches workflows — the WorkflowService API, the execution engine, the live-DAG UI, or the reconciler.
Goal: after this you can trace a run end-to-end, know exactly what is live vs gated-off vs parked, and find the code fast.
Status legend: 🟢 LIVE (merged, in the prod code path) · 🟡 STUBBED (placeholder, works but not final) · 🔵 DEV-ONLY (only wired for dev/local/CI) · 🟠 PENDING/PARKED (designed and/or coded but gated off / blocked).
The one caveat you must hold: the entire execution engine is code-complete, tested, and ADR-accepted, but gated OFF by default (
TEMPORAL_ENABLED=false) and present in no prod/beta Kubernetes manifest. Throughout this doc, "🟢 LIVE, flag-gated" means code-complete with production callers that activate only underTEMPORAL_ENABLED. See §7.
This doc was last fully re-verified on 2026-07-27, before Workflow Runtime v2 (PRD #2081, milestone #28) landed. Two consequences:
- The DSL section below is out of date. There are now 11 node kinds, not 5 —
loop,parallel,wait,question,user_inputandswitchall shipped. In particular the claim thatparallelis "rejected at create time (parked)" is no longer true: it is live,2..16branches with four join policies. - The default-off caveat above still holds in code, but the beta-dev deployment overrides it — the running orchestrator at revision
70853560hasTEMPORAL_ENABLED=trueandTEMPORAL_WORKER_ENABLED=true, so v2 workflows do execute there. The "no prod/beta Kubernetes manifest" half remains accurate.
For the v2 constructs, the identity-key discipline, the observation-admission fence and the current honest gap list, see Workflow Runtime v2. Sections of this doc covering the API plane, the coordinator substrate, run projection and the reconciler remain current.
0. TL;DR — the mental model
- One engine, not two. The real executor is the Temporal DSL interpreter (
internal/temporalwf/interpreter/interpreter.go→RunWorkflow). The oldinternal/workflow/engine.gois demoted — dead for running workflows; it only still serves the manualApprove/Reject/Escalateaction RPCs for legacy (non-Temporal) runs, and even those are fenced off for Temporal-owned runs. internal/coordinator/is not a rival engine — it is the shared choke-point substrate. Its ports (Governor · Auditor · CostAccruer · ApprovalGate · RunProjector) are reused as adapters by the Temporal activities. Governance, audit, and projection are centralized there (ADR-0015).- What runs in production today: the
WorkflowServicedefinition/CRUD plane + the live-DAG UI (StreamRunStatus/StreamRunLogsover PostgresLISTEN/NOTIFY). WithTEMPORAL_ENABLED=false(the default, and the state of every committed manifest),TriggerWorkflowandResumeRunreturnFAILED_PRECONDITION— no run executes. - Go-live is blocked on one decision, not more code. ADR-0023 is Accepted; the only open sub-item is the Phase-1 production deployment shape (self-host Temporal vs Temporal Cloud), owned by Ashik. Flip the flag + ship a manifest and the built engine runs.
- When enabled, the guarantees are real: deterministic replay (CI-gated), durable approval pause/resume, per-hop governance (deny/approve/allow), hash-chained audit (
coordinator_audit_log), and per-workflow cost recomputed fromllm_usage_events. On the legacy path the cost meter is a wired no-op (always0). - DSL: a workflow is a JSON DAG of typed steps —
agent_action,approval_gate,conditional(CEL),tool_action,terminal.parallelis rejected at create time (parked).
1. Architecture & control flow (numbered, colored)
Colour key: blue = API/definition plane (context-engine) · purple = Temporal (server, worker, interpreter) · amber = shared choke-point ports (internal/coordinator) · green = data / projection tables · red = real execution egress (agent-worker LLM turn) · grey = live-DAG UI stream.
Walkthrough (numbers match the diagram)
TriggerWorkflowvalidates the request, resolves the requester, and RLS-scopesGetWorkflow(cross-tenant guard) —internal/workflow/service.go:291,324,332,347.- The gate.
if s.trigger == nil→FAILED_PRECONDITION "workflow execution disabled"—service.go:301-303.s.triggeris wired only viaWithTemporalTrigger, built only underTEMPORAL_ENABLED—cmd/context-engine/temporal.go:115-118,144;main.go:672. Off = stop here (prod default). TemporalTrigger.Triggercommits theworkflow_runsrow (status=running), creates the canonical Task +StampWorkflowRun/LinkWorkflowRun, thenExecuteWorkflow—internal/workflow/temporal_trigger.go:172,185-213. Exactly-once idrun:{org}:{run_id},REJECT_DUPLICATE.- Temporal dispatches to the wf-runs worker (a two-queue split so a 30-min LLM activity never starves a workflow task) —
cmd/agent-orchestrator/temporal.go:96-140. RunWorkflowruns;BootRunsnapshots the stored definition into Temporal history (in-flight-edit immune, replay-deterministic) —interpreter/interpreter.go:100,114;activities/boot.go:27-70.- The interpreter walks the DAG deterministically (keyed-map reads, hop budget
maxHops=200) —interpreter.go:162-215. - Per hop: the
Governactivity callsgovernance.Engine.Check—interpreter/steps.go:45-63;activities/steps.go:17-29; wired ata2acoord.go:131-175. - Verdict:
Deny→ fail-closed;RequiresApproval→ forced approval gate;Allow→ dispatch.agent_actionruns a real LLM turn via the step bridge → agent-workerExecuteStep—cmd/agent-orchestrator/stepbridge.go:63-270. - An
approval_gate(or a governance-forced approval) opens a durable approval and parks the run (awaiting_approval) on a signal + server timer —interpreter/steps.go:475-568. - A human
Approve/Rejectrecorded onApprovalServicepublishes a decision signal →SignalWorkflow(resume)(routed byMetadata["source"]=="temporal-workflow") —temporaldecisionpublisher.go:116-134;approval/service.go:848. - Each agent hop accrues cost + writes a hash-chained audit hop (
coordinator_audit_log) —activities/steps.go:188-255;audit_pg.go:6-8,144. - Every transition emits a typed
projection.Event; theProjectactivity applies it —interpreter.go:229;runprojector.go:90-94. PgRunProjectorupsertsworkflow_actions/workflow_runs/workflow_run_logs(idempotent, RLS, write-once terminal guard) and firespg_notify—cmd/agent-orchestrator/runprojector.go:133-283.StreamRunStatus/StreamRunLogsfan the NOTIFY out to the UI (subscribe → snapshot → live) —connect_adapter_live_dag.go:133,192.
2. Component catalog & status
| Component | Path | Status | Note |
|---|---|---|---|
WorkflowService (22 RPCs) | internal/workflow/service*.go, served cmd/context-engine/main.go:2150 | 🟢 LIVE | Definition/CRUD + run-control + streaming. 3 RPCs Temporal-gated (below). |
TemporalTrigger | internal/workflow/temporal_trigger.go | 🟢 LIVE, flag-gated | Mints run row + StartWorkflow. nil when TEMPORAL_ENABLED=false. |
Temporal interpreter RunWorkflow | internal/temporalwf/interpreter/interpreter.go:100 | 🟢 LIVE, flag-gated | The only real DAG executor. Deterministic walk, maxHops=200. |
| Activity inventory (13) | internal/temporalwf/api/activities.go:20-34 | 🟢 LIVE, flag-gated | Boot/Govern/Audit/OpenApprovalGate/ExecuteAgentStep/CompleteStepAndAccrue/EvalBranch/VerifyOutcome/InvokeTool/VoidApproval/Complete/Fail/CancelRun. |
| Real agent-step bridge | cmd/agent-orchestrator/stepbridge.go:63 | 🟢 LIVE when vault present | Real LLM turn (BYOK) → usage record. |
FakeTurnExecutor | internal/temporalwf/stepbridge/fake.go:19 | 🔵 DEV/CI-ONLY | Canned deterministic turn; only when WF_FAKE_EXECUTOR=true. |
StubExecutor / StubToolInvoker | internal/temporalwf/activities/executor.go:19,38 | 🟡 STUB (fail-loud) | Non-retryable error if real deps unwired — never silent. |
RunProjector port + PgRunProjector | internal/coordinator/projector.go:52; cmd/agent-orchestrator/runprojector.go:66 | 🟢 LIVE, flag-gated | Derived UI tables; never source of truth (re-derivable). |
Governance choke-point (Govern) | governance.Engine via a2acoord.go:145-175 | 🟢 LIVE | 4-layer cascade, per hop. |
Audit (coordinator_audit_log, hash-chain) | internal/coordinator/audit_pg.go (migration 093) | 🟢 LIVE | ADR-0015 SHA-256 chain, hop_key dedup. |
| Cost (Temporal) | internal/temporalwf/activities/cost.go:36 | 🟢 LIVE, flag-gated | Σ of llm_usage_events (authoritative). |
Cost (legacy) NoopCostMeter | internal/workflow/cost_meter.go:23; wired main.go:640 | 🟡 STUBBED | Always 0; seam exercised, value inert. |
Schedules (WF-28) + ScheduleReconciler | internal/workflow/service_schedule.go; temporal.go:146 | 🟢 LIVE, flag-gated | Temporal Schedules, 5-min floor. |
| Workflow reconciler (WF-31) | internal/workflow/wfreconcile/; cmd/reconciler/wfreconcile.go | 🟢 LIVE, flag-gated | 3-way drift heal; no-op when flag off. |
SchemaValidator (DSL) | internal/workflow/validator.go | 🟢 LIVE | JSON-Schema + semantic checks (refs, cycles, contracts). |
| DSL leaf types | internal/temporalwf/dsl/dsl.go | 🟢 LIVE | Pure, stdlib-only (determinism-safe). |
Legacy Engine | internal/workflow/engine.go:32 | 🟢 LIVE (fenced) | Demoted; serves manual approve/reject/escalate for legacy runs only. |
3. The execution substrate — coordinator ports vs Temporal interpreter
Two flags, one master switch. All defaults are in code and OFF:
| Flag | Default | Where | Gates |
|---|---|---|---|
TEMPORAL_ENABLED | false | cmd/context-engine/config.go:350; cmd/agent-orchestrator/config.go:91 | Client dial + trigger + signaler. Off ⇒ TriggerWorkflow/ResumeRun error; no Temporal connection. |
TEMPORAL_WORKER_ENABLED | false | cmd/agent-orchestrator/config.go:298 | The activity worker + RunWorkflow registration. |
WF_FAKE_EXECUTOR | false | cmd/agent-orchestrator/config.go:299 | Swaps the real bridge → FakeTurnExecutor (dev/CI). |
There is no runtime router that picks an engine per request. Selection happens once at boot: if TEMPORAL_ENABLED is off, dialWorkflowTemporal returns nils, s.trigger stays nil, and the kill-switch fires (service.go:301). No legacy fallback executes a run.
The internal/coordinator relationship. The coordinator is the deterministic task substrate whose ports — Governor, Auditor, CostAccruer, ApprovalGate, RunProjector — are built once (buildTemporalActivityDeps, cmd/agent-orchestrator/a2acoord.go:131) and handed to the Temporal activities. So the Temporal path reuses the same governance/audit/projection code the coordinator defined; it does not reimplement it. The coordinator's own async executor (COORDINATOR_ENABLED, default off) is a separate, also-dormant path.
No silent stubs in the hot path. If the worker is enabled with no vault and no explicit WF_FAKE_EXECUTOR, the process refuses to start rather than completing workflows with fake output — cmd/agent-orchestrator/main.go:1150-1156.
Determinism is CI-enforced (WF-32, #1945 closed). make test-replay replays every internal/temporalwf/testdata/histories/*.json fixture against RunWorkflow and fails on command divergence; make workflowcheck runs Temporal's static analyzer. Both are required checks — .github/workflows/temporal-tests.yml:273-314. New node behaviors are introduced behind GetVersion patches (interpreter.go:263-310); CEL/acceptance are evaluated inside activities so a cel-go bump can't poison replay.
4. Run lifecycle & governance
Run state machine (Temporal path — RunProjector is the single status writer)
pending/running ⇄ paused (pause/resume signals) → awaiting_approval (gate) → resume → … → terminal ∈ completed · failed · cancelled.
Constants interpreter/events.go:16-21; terminal set service_temporal.go:207-214.
Action/step status
- Temporal projection:
in_progress(dispatched) →awaiting_approval(blocked) →completed/approved·failed/rejected·skipped(un-taken conditional arm) —internal/workflow/projection.go:33-43. - Legacy engine: initial
pending(agent/conditional) orawaiting_approval(gate) →approved/rejected/escalated/completed/failed—engine.go:566-580.
Approval pause / resume (real, Temporal-only)
The legacy engine has no resumable pause (ResumeRun → FAILED_PRECONDITION, service_temporal.go:136-139). On the Temporal path: a gate opens a durable approval and parks the run on a signal + server timer (interpreter/steps.go:475-568); a human decision recorded on ApprovalService publishes a decision signal → SignalWorkflow(resume) (temporaldecisionpublisher.go:116-134). Timer expiry voids the approval and fails the run; a cancel of a parked gate voids the pending approval. There is no in-step interrupt/suspend — mid-step tool approval / delegation is deliberately fail-closed unsupported (activities/steps.go:134-137).
Governance (per hop)
Every agent/tool hop calls the Govern activity → governance.Engine.Check (4-layer org→team cascade) before any side effect — interpreter/steps.go:45-63,364-379. Deny fails closed; RequiresApproval routes into the same gate machinery as a self-gate.
Cost & audit
- Cost: real only on the Temporal path, recomputed as
SUM(llm_usage_events)per step and rolled to the run (activities/cost.go:36-70,steps.go:188-255). The legacy meter is a wired no-op (NoopCostMeter), soworkflow_runs.cost_usd/workflow_actions.cost_usdare0off the Temporal path. - Audit: every hop appends to the hash-chained
coordinator_audit_log(row_hash = SHA256(prev_hash ‖ canonical(row)),hop_keydedup) —audit_pg.go:6-8,144. The live-DAG projection is derived and re-derivable, never the audit source of truth.
5. Operate — define, trigger, monitor (RPC checklist)
| # | Step | Surface (RPC) | Effect |
|---|---|---|---|
| 1 | Author a workflow | CreateWorkflow | Validates the DAG (SchemaValidator) → workflows row (status=draft). |
| 2 | Edit the DAG | UpdateWorkflowDefinition | Re-validates via the same path; run-immune (running runs use their boot snapshot). |
| 3 | Activate | UpdateWorkflowStatus | draft → active. |
| 4 | (optional) Schedule | SetWorkflowEnabled | Writes trigger_type/trigger_config/enabled; reconciles a Temporal Schedule (fires only when TEMPORAL_ENABLED). |
| 5 | Run it | TriggerWorkflow | Temporal on: mints a running run + StartWorkflow. Off: FAILED_PRECONDITION. |
| 6 | Watch it | StreamRunStatus / StreamRunLogs | Live node/log stream via PG NOTIFY (works regardless of engine, driven by projection). |
| 7 | Inspect a node | GetNodeIO / GetNodeTranscript | Redacted input/output JSON; agent turn transcript (clearance-gated). |
| 8 | Approve a gate | ApproveAction / RejectAction / EscalateAction (legacy runs) or ApprovalService.RecordDecision (Temporal runs) | Legacy RPCs are fenced for Temporal-owned runs; Temporal approvals flow through the governance gate + resume signal. |
| 9 | Pause / cancel / resume | PauseRun / CancelRun / ResumeRun | Pause/Cancel are dual-path (legacy DB write or Temporal signal); Resume is Temporal-only. |
| 10 | History | ListRuns | Cursor-paginated run history with a status filter. |
6. Data model & API reference
Tables (all FORCE ROW LEVEL SECURITY, org-isolated on app.org_id):
| Table | Key columns / enums | Migration |
|---|---|---|
workflows | definition JSONB · status ∈ {draft, active, archived} · trigger_type ∈ {manual, schedule} · enabled (default false) | 014_workflows.up.sql; triggers 151_workflow_triggers.up.sql; updated_at 166 |
workflow_runs | status ∈ {pending, running, awaiting_approval, paused, completed, failed, cancelled} · cost_usd NUMERIC(18,6) default 0 · temporal_workflow_id (non-NULL ⇒ Temporal-owned; fence key) | 014 base; 086_workflow_live_dag.up.sql (+paused, cost); 146 (+awaiting_approval); 148 (temporal id) |
workflow_actions | status ∈ {pending, in_progress, awaiting_approval, approved, rejected, escalated, completed, failed, skipped} · cost_usd · elapsed_ms · approval_id · resolved_agent_id/resolved_team_id · UNIQUE(run_id, step_name) | 014; 086; 095; 146; 147 (+skipped); 167 |
workflow_run_logs | level ∈ {INFO, WARN, ERROR} · append-only | 086 |
workflow_triggers | type ∈ {manual, schedule} · config JSONB | 151 |
DSL node types (internal/workflow/validator.go, internal/temporalwf/dsl/dsl.go):
| Node | Meaning | Status |
|---|---|---|
agent_action | One bounded governed agent turn (optional acceptance CEL, model override, I/O contracts) | 🟢 supported |
approval_gate | Park run on durable approval until a decision/timeout; requires required_clearance | 🟢 supported |
conditional | CEL boolean route (true→on_approve, false→on_reject); compiled at create | 🟢 supported (WF-26) |
tool_action | Governed tool call, no LLM turn; whitelisted tool/action (only github today) | 🟢 supported (WF-30) |
terminal | Explicit success sink | 🟢 supported |
parallel | Concurrent execution | 🟠 rejected at create — parked, never executed |
Semantic validation (beyond JSON-Schema): entry-point + edge-reference resolution, per-type fences, I/O contract compatibility, and circular-dependency detection (DFS over all routing edges) — validator.go:309-500. UpdateWorkflowDefinition runs the identical path.
Full RPC list: proto/upsquad/workflow/v1/workflow.proto. All 22 handlers are implemented and served by context-engine; the three that require TEMPORAL_ENABLED are TriggerWorkflow, ResumeRun, and the schedule-firing half of SetWorkflowEnabled (the DB write always works).
7. Gaps, pending, dev-only, parked — the honest state
| Item | Ref | State |
|---|---|---|
Temporal engine not in production — flags default off; zero Temporal refs in deployments/, deploy/, infra/; prod configmaps don't even carry the TEMPORAL_* vars | grep deployments/ = 0; cmd/context-engine/config.go:350 | 🟠 PARKED — code-complete, gated off |
| The blocking decision: ADR-0023 Phase-1 deployment shape (self-host Temporal on GKE ~$400-900/mo vs Temporal Cloud ~$100-500/mo) — the only open sub-item, owned by Ashik | docs/adrs/ADR-0023-workflow-execution-substrate.md:93-102,216-225 | 🟠 PENDING decision |
Workflow reconciler (WF-31) folded into the live reconciler daemon but no-op unless TEMPORAL_ENABLED | cmd/reconciler/wfreconcile.go:15-19 | 🟠 flag-gated |
parallel node unimplemented — recognized name, rejected at create | validator.go:459-463; dsl.go:49-51 | 🟠 PARKED |
Legacy-engine timeout_minutes parsed but enforced by nothing (Temporal-side timeouts are enforced) | engine.go:98 | 🟡 STUBBED (legacy only) |
Per-workflow cost = 0 on the legacy/context-engine path (NoopCostMeter); real only on the Temporal path | cost_meter.go:23; wired main.go:640 | 🟡 STUBBED |
FakeTurnExecutor / StubExecutor / StubToolInvoker — dev/CI and fail-loud placeholders | stepbridge/fake.go, activities/executor.go | 🔵 DEV-ONLY / 🟡 STUB |
Stale ADR text: ADR-0023's Context ("TriggerWorkflow sequences but never executes", "timers do not exist", "no schedule system") describes the pre-build state. Current code executes via the interpreter, and schedules (WF-28) + Temporal-side timeouts shipped. Only parallel + legacy timeout remain genuinely unimplemented. | ADR-0023 Context vs code | ⚠️ doc drift — trust the code |
WF_FAKE_EXECUTOR doc/impl mismatch: struct comment says "implicitly true when no vault"; code instead fails startup in that case | config.go:108-113 vs main.go:1150-1156 | ⚠️ stale comment |
No CI gate uses a real LLM — the golden-flow E2E (WF-33) and daily-wave2 suites drive the real interpreter/activities with a scripted/usage-seeding executor (keyless), so they prove the orchestration spine and a meaningful non-zero Σ-cost, but not a live model turn.
Closed MVP tickets (built): WF-30 #1983 (tool node), WF-31 #1944 (reconcile), WF-32 #1945 (determinism gate), WF-33 #1943 (golden-flow E2E). Still open: PRD #1783, tracker #1786 (gated on the deployment decision above).
8. Key files, flags & environment specifics
Feature flags / env (defaults in code, all OFF):
TEMPORAL_ENABLED— master switch (client dial + trigger + signaler). Defaultfalse. Off ⇒TriggerWorkflow/ResumeRun→FAILED_PRECONDITION.TEMPORAL_WORKER_ENABLED— activity worker +RunWorkflowregistration. Defaultfalse.WF_FAKE_EXECUTOR— dev/CI deterministic turn. Defaultfalse.WF_RECONCILE_DISABLED— kill-switch for the workflow reconciler daemon.TEMPORAL_SERVER_ADDRESS/_NAMESPACE/_TASK_QUEUE_PREFIX— dial target / namespacing (dev:temporal:7233/upsquad-dev).- Note: these are env/config-struct booleans, not DB-flippable —
internal/config/feature_flags.goholds only Org-v2.3/v2.4 + Memory flags.
Source map
- API / service plane:
internal/workflow/service*.go,connect_adapter*.go,validator.go,tool_catalog.go; servedcmd/context-engine/main.go:2150. - Trigger / schedule:
internal/workflow/temporal_trigger.go,service_trigger.go,service_schedule.go,service_temporal.go. - Interpreter / activities:
internal/temporalwf/interpreter/,internal/temporalwf/activities/,internal/temporalwf/dsl/,internal/temporalwf/projection/,internal/temporalwf/stepbridge/. - Worker wiring:
cmd/agent-orchestrator/{temporal.go,a2acoord.go,stepbridge.go,runprojector.go,main.go}. - Choke-point substrate:
internal/coordinator/{projector.go,audit_pg.go,ports.go},internal/governance/. - Reconciler:
internal/workflow/wfreconcile/,cmd/reconciler/wfreconcile.go. - Migrations:
internal/**/store/migrations/{014,086,146,147,148,151,166,167}_*workflow*.sql. - Proto:
proto/upsquad/workflow/v1/workflow.proto. - CI:
.github/workflows/{temporal-tests.yml,golden-flow-e2e.yml,daily-wave2-e2e.yml};test/integration/goldenflow/temporal_e2e_test.go.
Related ADRs / docs: ADR-0023 — Workflow Execution Substrate · [ADR-0015 — coordinator choke-point/audit] · Temporal Architecture & Licensing · Governed MCP & the Team Gateway.
Doc reflects upsquad-core@51f0490f (origin/main, 2026-07-25). Invalidated by: TEMPORAL_ENABLED becoming the production default or landing in a k8s manifest (flip §7 status); a resolution of the ADR-0023 Phase-1 deployment decision; implementation of the parallel node; or any change to the DSL node set or the run/action status enums.