Skip to main content

ADR-0023 — Workflow Execution Substrate (Temporal, adopted now, behind the Dispatcher seam)

  • Status: Accepted (founder 2026-07-15; Ashik ratifies Phase-1 production deployment shape only)
  • Date: 2026-07-11 (drafted); accepted 2026-07-15
  • Deciders: Vaisakh (founder — flipped the default 2026-07-11, ratified adopt-now 2026-07-15), Ashik (runtime ops — owns the Phase-1 production deployment-shape call), Principal Architect (technical validation)
  • Numbering: ADR-0023, not 0022 — see the footnote at the end of this document.1
  • Related:
    • ADR-0015 — A2A communication model; establishes the Temporal-shaped dispatcher interface (internal/bus/dispatcher.go) this decision sits behind. This ADR exercises the swap ADR-0015 pre-declared.
    • ADR-0013 — abstraction principle; the Temporal Cloud option is only admissible because it sits behind the Dispatcher/coordinator seam (open primitive we call, not a cloud service woven into the hot path).
    • ADR-0012 — agnostic middleware positioning; the moat (governance choke-point + hash-chained audit) must remain ours regardless of substrate.
    • PRD #623 (Temporal Adoption — Durable Agent Workflows) — this ADR resolves its open timing question: adopt now, not at a deferred tripwire.
    • PRD #1783 ("Workflows — governed DAG execution on one engine") — the Workflow PRD this ADR is the written buy-vs-build ruling for.
    • Build breakdown: WF issues #1791–#1800 — the adopt-now implementation, incl. #1791 (WF-05, go.temporal.io/sdk dependency — cited by Appendix A) and #1792 (WF-06, dev-compose temporal-server + temporal-ui, the $0 Phase-0 local Temporal).
    • #1127 (workflow per-node cost rollup) — the cost seam the substrate feeds.

Context​

Workflows is the next workstream. The front-end is founder-validated and already ported 1:1 into the live app (16-RPC WorkflowService proto vendored, adapter-wired); the backend spine is built-but-dormant. The scope anchor is the product contract at https://demo.upsquad.ai/workflows: defs/instances tabs, a live DAG (trigger / agent / tool / branch / governance / terminal nodes), approval gates that pause runs, per-node cost/timing, a 3-step create wizard, and the SDLC 13-node showcase. This is a wiring and projection problem, not a construction problem.

Ground truth verified in code:

  • internal/workflow/ sequences but never executes — TriggerWorkflow creates rows; nothing runs them ("synchronous MVP" by its own doc).
  • internal/coordinator/ is a complete async executor (govern → hash-chained audit → cost → bus dispatch; approval pause/resume already wired to ApprovalService) — but dormant behind flags (BUS_ENABLED, COORDINATOR_ENABLED, HARNESS_ENABLED all default off; never run in prod).
  • Timers do not exist. timeout_minutes is validated and enforced by nothing. There is no trigger/schedule system. parallel is accepted by the validator but unimplemented.
  • The live-DAG UI streams from workflow_actions via PG LISTEN/NOTIFY — anything that writes those rows lights the founder-validated UI for free.

The open question this ADR settles: what runs the workflow? Three families were on the table — finish the native engine on Redis Streams; adopt an external OSS workflow tool (n8n came up internally, hence this ADR exists); or adopt a durable-execution substrate (Temporal) behind the seam ADR-0015 already declared. ADR-0015 pre-committed to Temporal task-queues "as a backend swap, not a redesign" but left the timing open — interim Redis Streams now, Temporal "later (#623)." The founder's 2026-07-11 challenge flipped that default. This ADR records the flip honestly and closes the n8n discussion with a written ruling.

Decision​

Native definitions / governance / UX, with Temporal as the execution substrate, adopted NOW — behind the existing internal/bus/dispatcher.go seam per ADR-0015's original intent.

The split is deliberate and load-bearing:

  • Ours (the moat, untouched): the workflow definition plane (CRUD, validation, listing, streaming, pause/cancel intents in internal/workflow); the governance choke-point and hash-chained audit in the Coordinator; clearance enforcement; cost accrual (llm_usage_events authoritative); and the validated client UX (frozen WorkflowService contract, NOTIFY-driven live DAG). None of this moves to Temporal.
  • Temporal's (undifferentiated plumbing only): durable step-advance, retries, timers, schedules, replay-recovery. This is exactly the "hard durability" ADR-0015 said it wanted to defer to Temporal "rather than hand-built on raw pub/sub."

Why flip the default from tripwire-deferral to adopt-now (recorded honestly):

  1. The Redis Streams dispatch path was never-run-in-prod code. Its burn-in learning would be discarded at the eventual swap. We would be soaking a transport we had already decided to replace. The coordinator's governance/audit/cost choke-point logic survives the swap and is worth exercising early; the Redis dispatch soak is disposable.
  2. The MVP's own schedule trigger (W4) is a timer tripwire ~3–4 weeks in. The standing rule — nobody builds a timer-wheel on Redis — trips inside this very MVP, not in some hypothetical future. Deferral does not avoid the timer; it just guarantees we hit it mid-build with the wrong substrate underneath.
  3. Adopting once is cheaper than adopting twice. Adopt-now totals ≈ 13.5–18.5 eng-weeks once; the tripwire-deferral path totals ≈ 16–21 eng-weeks lifetime (native Redis glue + later Temporal migration + ~3 ew of discarded scaffolding). The price of adopting now is calendar, not total effort: MVP lands at ~7–9 calendar weeks instead of ~5. The founder accepts that trade.
  4. The swap touches only undifferentiated plumbing. Temporal replaces step-advance, retries, timers, and schedules. The governance choke-point, hash-chained audit, clearance, cost, and the validated UX are untouched. End-state under either path is identical — Temporal-under-our-moat. The founder's ruling just gets there without paying twice.

Adoption shape (staged, all behind existing seams): Temporal infra (W0.5) → TemporalDispatcher behind internal/bus/dispatcher.go → a deterministic DSL-interpreter workflow hosted by the Coordinator that walks the same JSON definitions (agent_action → activity on a role/team task queue; approval_gate → durable signal from ResumeFromApproval with timeout_minutes as a real timer; governance / audit / cost as wrapping activities) → Temporal Schedules API replacing the hand-built W4 scheduler. Carried over untouched: the SessionExecutor→ExecuteStep bridge (activity body verbatim), RunProjector, the definition plane, the frozen client contract, the approval wiring, and the tenant model. This is consistent with ADR-0015's keystone ruling: the Coordinator remains the sole executor and sole governance choke-point — Temporal is the durable engine inside it, not a second brain beside it.

Deployment shape — the one open sub-item (Ashik's Phase-1 production call)​

Temporal is decided; how we run it in production is not. Phase-0 dev is already settled — a $0 local Temporal via docker-compose (WF-06 / #1792). For Phase-1 production, two admissible options, both ADR-0013-compliant because both sit behind the Dispatcher/coordinator seam. This choice is assigned to Ashik — he owns runtime ops and carries the burden of whichever is chosen.

OptionShapeCost (indicative)Notes
A. Self-hostedHelm on GKE, 4 Temporal services, Postgres persistence on a separate Cloud SQL instance — explicitly no Cassandra, no Elasticsearch at MVP scale~$400–900/mo infra + our ops timeFull control; a new stateful subsystem we operate; tighter data-residency story for enterprise/on-prem tenants (ADR-0002/ADR-0012 dual model).
B. Temporal CloudManaged namespaces; we run only workers~$100–500/mo entry, ~$0.0003–0.0005 per agent-step (actions-based)Keeps the calendar tighter (no infra stand-up on the critical path); a new external dependency; still ADR-0013-compliant since it lives behind the seam.

Multi-tenancy stance (applies to both): single Temporal namespace + per-tenant task queues, per Temporal's own SaaS multi-tenancy guidance. Namespace-per-tenant is reserved for enterprise isolation tiers (dedicated-namespace / on-prem customers), not the default. This mirrors the per-tenant-queue keying already in ADR-0015's bus model, so tenant attribution and backpressure are unchanged by the substrate choice.

n8n ruling — the reason this ADR exists​

n8n came up in the internal discussion as a candidate embed/substrate. Two independent disqualifiers; either alone is fatal.

REJECTED as embed / execution substrate:

  1. Licensing. n8n ships under the Sustainable Use License, which prohibits offering n8n's functionality to third parties as part of a paid/hosted service without the commercial Embed program. Embedding it in UpsQuad's paid SaaS is exactly the prohibited use.
    • Source: https://docs.n8n.io/sustainable-use-license/ and the repo LICENSE.md.
    • The Embed program is reportedly ~$50k/yr entry per third-party sources — mark as unofficial; not a published price. Even at $0 this would still be rejected on (3).
  2. No first-class multi-tenancy. n8n's operational model is instance-per-customer; there is no supported single-instance multi-tenant isolation. That contradicts our tenant model (ADR-0015 per-tenant queues, single control plane) and multiplies ops cost linearly with tenants.
  3. Governance bypass — fatal regardless of licensing. Execution inside n8n workers runs outside the Coordinator, which means it bypasses the governance choke-point + hash-chained audit — the core moat (ADR-0012, ADR-0015). A workflow engine that can act without passing through Governance.Check and the audit chain is disqualifying on architecture alone. This is the decisive reason — the licensing point merely makes it moot to argue.
  4. Stack fit. A TypeScript runtime fleet in a Go shop; a second worker runtime to operate and secure for zero differentiated benefit.

ADOPTED as a customer-edge pattern (zero cost, distribution upside): customers running their own n8n is fully SUL-permitted. We publish documented integration recipes — inbound webhooks (customer n8n → CreateTask / trigger a run) and outbound run events (run lifecycle → customer n8n webhook). This turns n8n into a distribution channel with zero licensing exposure and no governance bypass (the customer's n8n calls our governed API from outside; execution still happens inside our Coordinator).

Alternatives considered and killed​

Each killed with its single decisive disqualifier:

  • Windmill — AGPL core, and the features we'd actually need (RBAC depth, multi-tenant controls) live in the EE tier. AGPL tarpit for a paid SaaS + paywalled essentials.
  • Conductor / Orkes — JVM runtime; overlaps our engine without Temporal's durable-timer / replay depth. Overlap without the one capability we're adopting for.
  • Argo Workflows — pod-per-step execution shape; wrong for cheap agent hops and for month-long paused approval gates (a pod cannot idle for a month waiting on a human).
  • Prefect — Python-shaped orchestration; impedance mismatch with the Go Coordinator and its deterministic state machine.
  • Dify — its license prohibits multi-tenant use; non-starter for our platform model.
  • Flowise — no durable execution at all, and it duplicates the validated UX we already own. Nothing gained, a second UX to reconcile.

Status-quo+ (finish the native engine on Redis Streams) — documented as the retained fallback (Option B in PRD #1783) in case the ops burden is rejected. Its failure ladder, recorded so we don't relearn it: timers (no timer-wheel — timeout_minutes needs real durable timers) → backoff/retry (hand-built at-least-once + DLQ semantics we'd own forever) → fan-out (parallel join, sagas, compensation — the hardest to hand-roll correctly). The standing rule holds: nobody builds a timer-wheel on Redis. If deferral is chosen, the schedule trigger (W4) is the tripwire that forces this conversation again ~3–4 weeks in.

Consequences​

Positive

  • The hard durability is bought, not built — durable timers, replay recovery, retry policies, schedules, and (later) parallel/sagas come from Temporal instead of hand-rolled on raw pub/sub (ADR-0015's stated intent, now realised).
  • The moat is untouched — governance choke-point, hash-chained audit, clearance, cost, and the validated UX all remain ours; the swap is invisible to the client (frozen WorkflowService contract; NOTIFY-driven live DAG unchanged).
  • We adopt once — no discarded Redis-dispatch scaffolding, no second migration; end-state is reached directly.
  • timeout_minutes becomes real — approval gates pause on a durable timer-backed signal rather than an unenforced validation field.

Negative / trade-offs (accepted)

  • Determinism discipline — workflow-definition-interpreter code must be deterministic (no wall clock, no un-seeded randomness, no non-deterministic map iteration in the workflow body). This is a new class of bug for the team; guarded by replay tests in CI.
  • Worker-versioning discipline — activity/worker signatures must be versioned; a deploy runbook is required (W5 deliverable). Breaking an in-flight run's contract is a production hazard we now own.
  • Continue-as-new for long histories — month-long paused runs will hit Temporal history limits; the interpreter must continue-as-new to bound history. Designed in from W1.
  • A new stateful subsystem OR a new external dependency — self-host adds a Temporal cluster + Cloud SQL we operate (HA, backups, upgrades); Temporal Cloud adds a vendor on the critical path. Either way, ops surface grows. This is the burden Ashik weighs in the Phase-1 production call.
  • Calendar cost — MVP lands at ~7–9 weeks instead of ~5. Total effort is lower; wall-clock is higher.

What gets deleted / not-cloned later

  • The internal/bus Redis Streams implementation demotes to a dev/local fallback behind the Dispatcher interface — kept for laptop dev and CI, not a production path.
  • The approval-scheduler lease-loop pattern is NOT cloned for workflows — the schedule trigger uses Temporal's Schedules API (no scheduler process, no lease loop). We do not hand-build a second scheduler.
  • Any temptation to add a parallel definition→execution translator is foreclosed here — that is the split-brain pattern that produced the coordinator_tasks / tasks divergence. One engine, one executor (ADR-0015 keystone).

Status and ratification​

Accepted. The substrate decision — native definitions/governance/UX + Temporal as the execution substrate, adopted now, behind the Dispatcher seam — is final. The founder flipped the default on 2026-07-11 and ratified adopt-now on 2026-07-15; the build has started, with the WF breakdown filed as #1791–#1800 (incl. #1791 WF-05 go.temporal.io/sdk, which Appendix A of this document cites). A decision record whose downstream implementation issues are already filed and referenced inside it is not "Proposed" — it is Accepted.

One scoped open sub-item remains, owned by Ashik:

  • Phase-1 production deployment shape — self-hosted (Helm/GKE + separate Cloud SQL) vs Temporal Cloud — plus ownership of worker-versioning / deploy-discipline thereafter. This is a deliberately bounded operational choice within the accepted decision, not a re-litigation of it; an explicitly-scoped open sub-question with a named owner is fully compatible with Accepted.

Not open: the Phase-0 development substrate is a $0 local Temporal (docker-compose, temporal-server + temporal-ui, WF-06 / #1792) — already decided and landing. Dev does not wait on Ashik's Phase-1 call.

This ADR resolves the timing question left open by PRD #623 (adopt now, not at a deferred tripwire) and is the written buy-vs-build ruling — including the n8n answer — that PRD #1783 cross-references.

Appendix A — Verified license text (fetched from source repositories 2026-07-12; re-confirmed at adoption 2026-07-15)​

All Temporal components in scope are under The MIT License, verified directly from each repository's LICENSE file: temporalio/temporal (server), temporalio/sdk-go, temporalio/ui, temporalio/helm-charts. Operative text, verbatim (bold spans below are emphasis added, not present in the source):

The MIT License

Copyright (c) 2025 Temporal Technologies Inc. All rights reserved. Copyright (c) 2020 Uber Technologies, Inc.

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software (emphasis added), and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Commercial/enterprise-use analysis (engineering due diligence):

  1. Selling a product/service built on the Software is expressly granted ("sublicense, and/or sell", "without restriction"). Operating Temporal inside UpsQuad's paid multi-tenant SaaS is within the license.
  2. SaaS operation distributes no copies to customers → the sole MIT condition (notice preservation) barely engages.
  3. On-prem/enterprise bundles that DO ship Temporal must include the notice above in the distribution — the entire obligation. No copyleft, no network/SaaS clause, no field-of-use restriction (contrast: n8n Sustainable Use License, Dify multi-tenant prohibition, AGPL network copyleft).
  4. License changes are not retroactive; adopted versions remain MIT permanently. Version + license text pinned in the dependency PR (WF-05 / core#1791) with a transitive-dependency license scan.
  5. Trademark ("Temporal" name/marks) is not granted by MIT — do not brand UpsQuad features with Temporal marks; describing the capability is fine.
  6. Temporal Cloud (if selected at Phase 1) is a commercial managed service under its own terms — a vendor contract matter, independent of this code license.
  7. For enterprise customer paper requiring license schedules, this appendix + the WF-05 scan are the source; specific contract language remains a counsel review item.

Footnotes​

  1. This document is ADR-0023, deliberately skipping 0022. "ADR-0022" is reserved for the MCP registry/gateway/credential track — it is referenced 17× across docs/api/{mcpserver,toolpolicy,agent}.md (and in issues such as #1753/#1754, "W3/B4/B8") with no ADR-0022 file landed yet. Taking 0022 here would collide with that in-flight track, so the workflow-substrate ADR takes the next unambiguously-free number. ↩

Edit this page
Last updated on by upsquad-backend-sme[bot]