KT — Workflow Orchestration
Audience: an engineer picking up anything that touches workflows — the WorkflowService API, the execution engine, the live-DAG UI, or the reconciler.
Goal: after this you can trace a run end-to-end, know exactly what is live vs gated-off vs parked, and find the code fast.
Status legend: 🟢 LIVE (merged, in the prod code path) · 🟡 STUBBED (placeholder, works but not final) · 🔵 DEV-ONLY (only wired for dev/local/CI) · 🟠 PENDING/PARKED (designed and/or coded but gated off / blocked).
The one caveat you must hold: the entire execution engine is code-complete, tested, and ADR-accepted, but gated OFF by default (
TEMPORAL_ENABLED=false) and present in no prod/beta Kubernetes manifest. Throughout this doc, "🟢 LIVE, flag-gated" means code-complete with production callers that activate only underTEMPORAL_ENABLED. See §7.
0. TL;DR — the mental model
- One engine, not two. The real executor is the Temporal DSL interpreter (
internal/temporalwf/interpreter/interpreter.go→RunWorkflow). The oldinternal/workflow/engine.gois demoted — dead for running workflows; it only still serves the manualApprove/Reject/Escalateaction RPCs for legacy (non-Temporal) runs, and even those are fenced off for Temporal-owned runs. internal/coordinator/is not a rival engine — it is the shared choke-point substrate. Its ports (Governor · Auditor · CostAccruer · ApprovalGate · RunProjector) are reused as adapters by the Temporal activities. Governance, audit, and projection are centralized there (ADR-0015).- What runs in production today: the
WorkflowServicedefinition/CRUD plane + the live-DAG UI (StreamRunStatus/StreamRunLogsover PostgresLISTEN/NOTIFY). WithTEMPORAL_ENABLED=false(the default, and the state of every committed manifest),TriggerWorkflowandResumeRunreturnFAILED_PRECONDITION— no run executes. - Go-live is blocked on one decision, not more code. ADR-0023 is Accepted; the only open sub-item is the Phase-1 production deployment shape (self-host Temporal vs Temporal Cloud), owned by Ashik. Flip the flag + ship a manifest and the built engine runs.
- When enabled, the guarantees are real: deterministic replay (CI-gated), durable approval pause/resume, per-hop governance (deny/approve/allow), hash-chained audit (
coordinator_audit_log), and per-workflow cost recomputed fromllm_usage_events. On the legacy path the cost meter is a wired no-op (always0). - DSL: a workflow is a JSON DAG of typed steps —
agent_action,approval_gate,conditional(CEL),tool_action,terminal.parallelis rejected at create time (parked).
1. Architecture & control flow (numbered, colored)
Colour key: blue = API/definition plane (context-engine) · purple = Temporal (server, worker, interpreter) · amber = shared choke-point ports (internal/coordinator) · green = data / projection tables · red = real execution egress (agent-worker LLM turn) · grey = live-DAG UI stream.
Walkthrough (numbers match the diagram)
TriggerWorkflowvalidates the request, resolves the requester, and RLS-scopesGetWorkflow(cross-tenant guard) —internal/workflow/service.go:291,324,332,347.- The gate.
if s.trigger == nil→FAILED_PRECONDITION "workflow execution disabled"—service.go:301-303.s.triggeris wired only viaWithTemporalTrigger, built only underTEMPORAL_ENABLED—cmd/context-engine/temporal.go:115-118,144;main.go:672. Off = stop here (prod default). TemporalTrigger.Triggercommits theworkflow_runsrow (status=running), creates the canonical Task +StampWorkflowRun/LinkWorkflowRun, thenExecuteWorkflow—internal/workflow/temporal_trigger.go:172,185-213. Exactly-once idrun:{org}:{run_id},REJECT_DUPLICATE.- Temporal dispatches to the wf-runs worker (a two-queue split so a 30-min LLM activity never starves a workflow task) —
cmd/agent-orchestrator/temporal.go:96-140. RunWorkflowruns;BootRunsnapshots the stored definition into Temporal history (in-flight-edit immune, replay-deterministic) —interpreter/interpreter.go:100,114;activities/boot.go:27-70.- The interpreter walks the DAG deterministically (keyed-map reads, hop budget
maxHops=200) —interpreter.go:162-215. - Per hop: the
Governactivity callsgovernance.Engine.Check—interpreter/steps.go:45-63;activities/steps.go:17-29; wired ata2acoord.go:131-175. - Verdict:
Deny→ fail-closed;RequiresApproval→ forced approval gate;Allow→ dispatch.agent_actionruns a real LLM turn via the step bridge → agent-workerExecuteStep—cmd/agent-orchestrator/stepbridge.go:63-270. - An
approval_gate(or a governance-forced approval) opens a durable approval and parks the run (awaiting_approval) on a signal + server timer —interpreter/steps.go:475-568. - A human
Approve/Rejectrecorded onApprovalServicepublishes a decision signal →SignalWorkflow(resume)(routed byMetadata["source"]=="temporal-workflow") —temporaldecisionpublisher.go:116-134;approval/service.go:848. - Each agent hop accrues cost + writes a hash-chained audit hop (
coordinator_audit_log) —activities/steps.go:188-255;audit_pg.go:6-8,144. - Every transition emits a typed
projection.Event; theProjectactivity applies it —interpreter.go:229;runprojector.go:90-94. PgRunProjectorupsertsworkflow_actions/workflow_runs/workflow_run_logs(idempotent, RLS, write-once terminal guard) and firespg_notify—cmd/agent-orchestrator/runprojector.go:133-283.StreamRunStatus/StreamRunLogsfan the NOTIFY out to the UI (subscribe → snapshot → live) —connect_adapter_live_dag.go:133,192.
2. Component catalog & status
| Component | Path | Status | Note |
|---|---|---|---|
WorkflowService (22 RPCs) | internal/workflow/service*.go, served cmd/context-engine/main.go:2150 | 🟢 LIVE | Definition/CRUD + run-control + streaming. 3 RPCs Temporal-gated (below). |
TemporalTrigger | internal/workflow/temporal_trigger.go | 🟢 LIVE, flag-gated | Mints run row + StartWorkflow. nil when TEMPORAL_ENABLED=false. |
Temporal interpreter RunWorkflow | internal/temporalwf/interpreter/interpreter.go:100 | 🟢 LIVE, flag-gated | The only real DAG executor. Deterministic walk, maxHops=200. |
| Activity inventory (13) | internal/temporalwf/api/activities.go:20-34 | 🟢 LIVE, flag-gated | Boot/Govern/Audit/OpenApprovalGate/ExecuteAgentStep/CompleteStepAndAccrue/EvalBranch/VerifyOutcome/InvokeTool/VoidApproval/Complete/Fail/CancelRun. |
| Real agent-step bridge | cmd/agent-orchestrator/stepbridge.go:63 | 🟢 LIVE when vault present | Real LLM turn (BYOK) → usage record. |
FakeTurnExecutor | internal/temporalwf/stepbridge/fake.go:19 | 🔵 DEV/CI-ONLY | Canned deterministic turn; only when WF_FAKE_EXECUTOR=true. |
StubExecutor / StubToolInvoker | internal/temporalwf/activities/executor.go:19,38 | 🟡 STUB (fail-loud) | Non-retryable error if real deps unwired — never silent. |
RunProjector port + PgRunProjector | internal/coordinator/projector.go:52; cmd/agent-orchestrator/runprojector.go:66 | 🟢 LIVE, flag-gated | Derived UI tables; never source of truth (re-derivable). |
Governance choke-point (Govern) | governance.Engine via a2acoord.go:145-175 | 🟢 LIVE | 4-layer cascade, per hop. |
Audit (coordinator_audit_log, hash-chain) | internal/coordinator/audit_pg.go (migration 093) | 🟢 LIVE | ADR-0015 SHA-256 chain, hop_key dedup. |
| Cost (Temporal) | internal/temporalwf/activities/cost.go:36 | 🟢 LIVE, flag-gated | Σ of llm_usage_events (authoritative). |
Cost (legacy) NoopCostMeter | internal/workflow/cost_meter.go:23; wired main.go:640 | 🟡 STUBBED | Always 0; seam exercised, value inert. |
Schedules (WF-28) + ScheduleReconciler | internal/workflow/service_schedule.go; temporal.go:146 | 🟢 LIVE, flag-gated | Temporal Schedules, 5-min floor. |
| Workflow reconciler (WF-31) | internal/workflow/wfreconcile/; cmd/reconciler/wfreconcile.go | 🟢 LIVE, flag-gated | 3-way drift heal; no-op when flag off. |
SchemaValidator (DSL) | internal/workflow/validator.go | 🟢 LIVE | JSON-Schema + semantic checks (refs, cycles, contracts). |
| DSL leaf types | internal/temporalwf/dsl/dsl.go | 🟢 LIVE | Pure, stdlib-only (determinism-safe). |
Legacy Engine | internal/workflow/engine.go:32 | 🟢 LIVE (fenced) | Demoted; serves manual approve/reject/escalate for legacy runs only. |
3. The execution substrate — coordinator ports vs Temporal interpreter
Two flags, one master switch. All defaults are in code and OFF:
| Flag | Default | Where | Gates |
|---|---|---|---|
TEMPORAL_ENABLED | false | cmd/context-engine/config.go:350; cmd/agent-orchestrator/config.go:91 | Client dial + trigger + signaler. Off ⇒ TriggerWorkflow/ResumeRun error; no Temporal connection. |
TEMPORAL_WORKER_ENABLED | false | cmd/agent-orchestrator/config.go:298 | The activity worker + RunWorkflow registration. |
WF_FAKE_EXECUTOR | false | cmd/agent-orchestrator/config.go:299 | Swaps the real bridge → FakeTurnExecutor (dev/CI). |
There is no runtime router that picks an engine per request. Selection happens once at boot: if TEMPORAL_ENABLED is off, dialWorkflowTemporal returns nils, s.trigger stays nil, and the kill-switch fires (service.go:301). No legacy fallback executes a run.
The internal/coordinator relationship. The coordinator is the deterministic task substrate whose ports — Governor, Auditor, CostAccruer, ApprovalGate, RunProjector — are built once (buildTemporalActivityDeps, cmd/agent-orchestrator/a2acoord.go:131) and handed to the Temporal activities. So the Temporal path reuses the same governance/audit/projection code the coordinator defined; it does not reimplement it. The coordinator's own async executor (COORDINATOR_ENABLED, default off) is a separate, also-dormant path.
No silent stubs in the hot path. If the worker is enabled with no vault and no explicit WF_FAKE_EXECUTOR, the process refuses to start rather than completing workflows with fake output — cmd/agent-orchestrator/main.go:1150-1156.
Determinism is CI-enforced (WF-32, #1945 closed). make test-replay replays every internal/temporalwf/testdata/histories/*.json fixture against RunWorkflow and fails on command divergence; make workflowcheck runs Temporal's static analyzer. Both are required checks — .github/workflows/temporal-tests.yml:273-314. New node behaviors are introduced behind GetVersion patches (interpreter.go:263-310); CEL/acceptance are evaluated inside activities so a cel-go bump can't poison replay.
4. Run lifecycle & governance
Run state machine (Temporal path — RunProjector is the single status writer)
pending/running ⇄ paused (pause/resume signals) → awaiting_approval (gate) → resume → … → terminal ∈ completed · failed · cancelled.
Constants interpreter/events.go:16-21; terminal set service_temporal.go:207-214.
Action/step status
- Temporal projection:
in_progress(dispatched) →awaiting_approval(blocked) →completed/approved·failed/rejected·skipped(un-taken conditional arm) —internal/workflow/projection.go:33-43. - Legacy engine: initial
pending(agent/conditional) orawaiting_approval(gate) →approved/rejected/escalated/completed/failed—engine.go:566-580.
Approval pause / resume (real, Temporal-only)
The legacy engine has no resumable pause (ResumeRun → FAILED_PRECONDITION, service_temporal.go:136-139). On the Temporal path: a gate opens a durable approval and parks the run on a signal + server timer (interpreter/steps.go:475-568); a human decision recorded on ApprovalService publishes a decision signal → SignalWorkflow(resume) (temporaldecisionpublisher.go:116-134). Timer expiry voids the approval and fails the run; a cancel of a parked gate voids the pending approval. There is no in-step interrupt/suspend — mid-step tool approval / delegation is deliberately fail-closed unsupported (activities/steps.go:134-137).
Governance (per hop)
Every agent/tool hop calls the Govern activity → governance.Engine.Check (4-layer org→team cascade) before any side effect — interpreter/steps.go:45-63,364-379. Deny fails closed; RequiresApproval routes into the same gate machinery as a self-gate.
Cost & audit
- Cost: real only on the Temporal path, recomputed as
SUM(llm_usage_events)per step and rolled to the run (activities/cost.go:36-70,steps.go:188-255). The legacy meter is a wired no-op (NoopCostMeter), soworkflow_runs.cost_usd/workflow_actions.cost_usdare0off the Temporal path. - Audit: every hop appends to the hash-chained
coordinator_audit_log(row_hash = SHA256(prev_hash ‖ canonical(row)),hop_keydedup) —audit_pg.go:6-8,144. The live-DAG projection is derived and re-derivable, never the audit source of truth.
5. Operate — define, trigger, monitor (RPC checklist)
| # | Step | Surface (RPC) | Effect |
|---|---|---|---|
| 1 | Author a workflow | CreateWorkflow | Validates the DAG (SchemaValidator) → workflows row (status=draft). |
| 2 | Edit the DAG | UpdateWorkflowDefinition | Re-validates via the same path; run-immune (running runs use their boot snapshot). |
| 3 | Activate | UpdateWorkflowStatus | draft → active. |
| 4 | (optional) Schedule | SetWorkflowEnabled | Writes trigger_type/trigger_config/enabled; reconciles a Temporal Schedule (fires only when TEMPORAL_ENABLED). |
| 5 | Run it | TriggerWorkflow | Temporal on: mints a running run + StartWorkflow. Off: FAILED_PRECONDITION. |
| 6 | Watch it | StreamRunStatus / StreamRunLogs | Live node/log stream via PG NOTIFY (works regardless of engine, driven by projection). |
| 7 | Inspect a node | GetNodeIO / GetNodeTranscript | Redacted input/output JSON; agent turn transcript (clearance-gated). |
| 8 | Approve a gate | ApproveAction / RejectAction / EscalateAction (legacy runs) or ApprovalService.RecordDecision (Temporal runs) | Legacy RPCs are fenced for Temporal-owned runs; Temporal approvals flow through the governance gate + resume signal. |
| 9 | Pause / cancel / resume | PauseRun / CancelRun / ResumeRun | Pause/Cancel are dual-path (legacy DB write or Temporal signal); Resume is Temporal-only. |
| 10 | History | ListRuns | Cursor-paginated run history with a status filter. |
6. Data model & API reference
Tables (all FORCE ROW LEVEL SECURITY, org-isolated on app.org_id):
| Table | Key columns / enums | Migration |
|---|---|---|
workflows | definition JSONB · status ∈ {draft, active, archived} · trigger_type ∈ {manual, schedule} · enabled (default false) | 014_workflows.up.sql; triggers 151_workflow_triggers.up.sql; updated_at 166 |
workflow_runs | status ∈ {pending, running, awaiting_approval, paused, completed, failed, cancelled} · cost_usd NUMERIC(18,6) default 0 · temporal_workflow_id (non-NULL ⇒ Temporal-owned; fence key) | 014 base; 086_workflow_live_dag.up.sql (+paused, cost); 146 (+awaiting_approval); 148 (temporal id) |
workflow_actions | status ∈ {pending, in_progress, awaiting_approval, approved, rejected, escalated, completed, failed, skipped} · cost_usd · elapsed_ms · approval_id · resolved_agent_id/resolved_team_id · UNIQUE(run_id, step_name) | 014; 086; 095; 146; 147 (+skipped); 167 |
workflow_run_logs | level ∈ {INFO, WARN, ERROR} · append-only | 086 |
workflow_triggers | type ∈ {manual, schedule} · config JSONB | 151 |
DSL node types (internal/workflow/validator.go, internal/temporalwf/dsl/dsl.go):
| Node | Meaning | Status |
|---|---|---|
agent_action | One bounded governed agent turn (optional acceptance CEL, model override, I/O contracts) | 🟢 supported |
approval_gate | Park run on durable approval until a decision/timeout; requires required_clearance | 🟢 supported |
conditional | CEL boolean route (true→on_approve, false→on_reject); compiled at create | 🟢 supported (WF-26) |
tool_action | Governed tool call, no LLM turn; whitelisted tool/action (only github today) | 🟢 supported (WF-30) |
terminal | Explicit success sink | 🟢 supported |
parallel | Concurrent execution | 🟠 rejected at create — parked, never executed |
Semantic validation (beyond JSON-Schema): entry-point + edge-reference resolution, per-type fences, I/O contract compatibility, and circular-dependency detection (DFS over all routing edges) — validator.go:309-500. UpdateWorkflowDefinition runs the identical path.
Full RPC list: proto/upsquad/workflow/v1/workflow.proto. All 22 handlers are implemented and served by context-engine; the three that require TEMPORAL_ENABLED are TriggerWorkflow, ResumeRun, and the schedule-firing half of SetWorkflowEnabled (the DB write always works).
7. Gaps, pending, dev-only, parked — the honest state
| Item | Ref | State |
|---|---|---|
Temporal engine not in production — flags default off; zero Temporal refs in deployments/, deploy/, infra/; prod configmaps don't even carry the TEMPORAL_* vars | grep deployments/ = 0; cmd/context-engine/config.go:350 | 🟠 PARKED — code-complete, gated off |
| The blocking decision: ADR-0023 Phase-1 deployment shape (self-host Temporal on GKE ~$400-900/mo vs Temporal Cloud ~$100-500/mo) — the only open sub-item, owned by Ashik | docs/adrs/ADR-0023-workflow-execution-substrate.md:93-102,216-225 | 🟠 PENDING decision |
Workflow reconciler (WF-31) folded into the live reconciler daemon but no-op unless TEMPORAL_ENABLED | cmd/reconciler/wfreconcile.go:15-19 | 🟠 flag-gated |
parallel node unimplemented — recognized name, rejected at create | validator.go:459-463; dsl.go:49-51 | 🟠 PARKED |
Legacy-engine timeout_minutes parsed but enforced by nothing (Temporal-side timeouts are enforced) | engine.go:98 | 🟡 STUBBED (legacy only) |
Per-workflow cost = 0 on the legacy/context-engine path (NoopCostMeter); real only on the Temporal path | cost_meter.go:23; wired main.go:640 | 🟡 STUBBED |
FakeTurnExecutor / StubExecutor / StubToolInvoker — dev/CI and fail-loud placeholders | stepbridge/fake.go, activities/executor.go | 🔵 DEV-ONLY / 🟡 STUB |
Stale ADR text: ADR-0023's Context ("TriggerWorkflow sequences but never executes", "timers do not exist", "no schedule system") describes the pre-build state. Current code executes via the interpreter, and schedules (WF-28) + Temporal-side timeouts shipped. Only parallel + legacy timeout remain genuinely unimplemented. | ADR-0023 Context vs code | ⚠️ doc drift — trust the code |
WF_FAKE_EXECUTOR doc/impl mismatch: struct comment says "implicitly true when no vault"; code instead fails startup in that case | config.go:108-113 vs main.go:1150-1156 | ⚠️ stale comment |
No CI gate uses a real LLM — the golden-flow E2E (WF-33) and daily-wave2 suites drive the real interpreter/activities with a scripted/usage-seeding executor (keyless), so they prove the orchestration spine and a meaningful non-zero Σ-cost, but not a live model turn.
Closed MVP tickets (built): WF-30 #1983 (tool node), WF-31 #1944 (reconcile), WF-32 #1945 (determinism gate), WF-33 #1943 (golden-flow E2E). Still open: PRD #1783, tracker #1786 (gated on the deployment decision above).
8. Key files, flags & environment specifics
Feature flags / env (defaults in code, all OFF):
TEMPORAL_ENABLED— master switch (client dial + trigger + signaler). Defaultfalse. Off ⇒TriggerWorkflow/ResumeRun→FAILED_PRECONDITION.TEMPORAL_WORKER_ENABLED— activity worker +RunWorkflowregistration. Defaultfalse.WF_FAKE_EXECUTOR— dev/CI deterministic turn. Defaultfalse.WF_RECONCILE_DISABLED— kill-switch for the workflow reconciler daemon.TEMPORAL_SERVER_ADDRESS/_NAMESPACE/_TASK_QUEUE_PREFIX— dial target / namespacing (dev:temporal:7233/upsquad-dev).- Note: these are env/config-struct booleans, not DB-flippable —
internal/config/feature_flags.goholds only Org-v2.3/v2.4 + Memory flags.
Source map
- API / service plane:
internal/workflow/service*.go,connect_adapter*.go,validator.go,tool_catalog.go; servedcmd/context-engine/main.go:2150. - Trigger / schedule:
internal/workflow/temporal_trigger.go,service_trigger.go,service_schedule.go,service_temporal.go. - Interpreter / activities:
internal/temporalwf/interpreter/,internal/temporalwf/activities/,internal/temporalwf/dsl/,internal/temporalwf/projection/,internal/temporalwf/stepbridge/. - Worker wiring:
cmd/agent-orchestrator/{temporal.go,a2acoord.go,stepbridge.go,runprojector.go,main.go}. - Choke-point substrate:
internal/coordinator/{projector.go,audit_pg.go,ports.go},internal/governance/. - Reconciler:
internal/workflow/wfreconcile/,cmd/reconciler/wfreconcile.go. - Migrations:
internal/**/store/migrations/{014,086,146,147,148,151,166,167}_*workflow*.sql. - Proto:
proto/upsquad/workflow/v1/workflow.proto. - CI:
.github/workflows/{temporal-tests.yml,golden-flow-e2e.yml,daily-wave2-e2e.yml};test/integration/goldenflow/temporal_e2e_test.go.
Related ADRs / docs: ADR-0023 — Workflow Execution Substrate · [ADR-0015 — coordinator choke-point/audit] · Temporal Architecture & Licensing · Governed MCP & the Team Gateway.
Doc reflects upsquad-core@51f0490f (origin/main, 2026-07-25). Invalidated by: TEMPORAL_ENABLED becoming the production default or landing in a k8s manifest (flip §7 status); a resolution of the ADR-0023 Phase-1 deployment decision; implementation of the parallel node; or any change to the DSL node set or the run/action status enums.