Skip to main content

ADR-0018: Activate Class A Cloud Infra for Beta + Prod (VPN-only, hard-isolated)

  • Status: Accepted (founder approval 2026-06-12 — issue #1274)
  • Date: 2026-06-12
  • Deciders: Vaisakh (founder), Principal Architect (review), DevOps Engineer (proposal)
  • Supersedes: ADR-0001 §3.7 / §8 deferral of Class A managed infra (for beta + prod only; dev/beta-dev stays on ADR-0001)
  • Related: #1274 (this ADR), ADR-0017 (staged promotion pipeline — the app delivery path this foundation carries), milestone #16 (Cloud Foundation: Beta + Prod), #1275–#1288 (implementation issues), #250 (Redis TLS+AUTH), #59 (CloudSQL deferral, closed not planned)

Renumbered 2026-06-12. This ADR was briefly merged as a second "ADR-0017". ADR-0017 — Staged Promotion Pipeline (merged first) keeps the 0017 number; this document is renumbered to ADR-0018. The two are complementary: ADR-0018 (this) governs the platform foundation (hierarchy, network, firewall, VPN-only access plane, registry isolation, data layer); ADR-0017 governs the app delivery path (stages, gates, build-once promotion). See the Reconciliation (2026-06-12) section at the end.

1. Context​

ADR-0001 stood up a hybrid local dev stack (docker-compose, "Flavor A") and explicitly deferred all other Class A managed infrastructure — managed CloudSQL, a permanent GKE cluster, ArgoCD bootstrap, observability deployment — until a tripwire fired. Section 8 of ADR-0001 listed those tripwires: a first external design partner, an exposed ingest API alpha, portal integration hitting the Context Engine, or any SLO commitment. Until then, the compose stack was the sole authoritative dev environment and infra/pulumi/upsquad-infra/index.ts remained an empty scaffold.

A tripwire has now fired. The founder has directed standing up the real GCP environment for pre-prod and prod, with three hard requirements:

  1. Hard isolation between pre-prod and prod — network, container registry, Kubernetes, secrets — with zero code-contamination paths between them.
  2. A highly secure, default-deny network — no public surface anywhere.
  3. VPN-only access — engineers reach everything through a private access plane, never the public internet.

ADR-0001's deferral remains correct for dev (the devbox). This ADR records the activation of Class A infra for beta + prod only, superseding ADR-0001's deferral for those two environments while leaving dev/beta-dev governed by ADR-0001.

1.1 Environment naming (founder-confirmed)​

  • beta — the pre-prod GCP environment (NOT "staging"). Projects, VPCs, clusters, AR, tunnels, and Pulumi stacks all use beta.
  • prod — the production GCP environment.
  • beta-dev — the existing devbox docker-compose environment. It stays out of GCP and keeps its self-hosted Postgres. ADR-0001 still governs it.

This supersedes the design draft's "staging" naming throughout (staging → beta, upsquad-stg → upsquad-beta).

2. Decision​

Activate Class A managed infrastructure on GCP (region asia-south1, Mumbai) for beta and prod only, provisioned via Pulumi (TypeScript), accessed exclusively via Cloudflare Zero Trust, with Artifact Registry per env and digest-only promotion. The four anti-contamination tenets are normative:

  1. No network path beta↔prod. Separate VPCs, no peering, no Shared VPC, no shared subnets, distinct projects, distinct service accounts, distinct secrets.
  2. Prod runs only prod-built images. Binary Authorization on the prod cluster requires a prod-AR attestation; the prod node SA can read the prod AR only (no beta AR, no GHCR).
  3. No public surface. No external IPs, private GKE control plane, private CloudSQL, internal-only LBs. The only ingress is Cloudflare tunnels initiated outbound from inside each VPC.
  4. CI never touches clusters. Promotion is a digest copy + GitOps pull; humans approve via a GitHub production Environment + ArgoCD.

2.1 Resource hierarchy​

Org: upsquad.ai (existing)
└── folder: upsquad-platform        ← org policies + hierarchical firewall bind here
    ├── project: upsquad-core        ← SHARED: Pulumi state, KMS, GitHub WIF pool, audit/billing sinks (no env workloads)
    ├── folder: env-beta
    │   └── project: upsquad-beta     ← apps, GKE, CloudSQL, Redis, AR
    └── folder: env-prod
        └── project: upsquad-prod     ← apps, Cloud Run (GKE on trigger), CloudSQL, Redis, AR

The shared upsquad-core GCP project (directly under upsquad-platform) holds all cross-cutting, non-env resources: the Pulumi state bucket, the KMS key for state secrets, the GitHub Workload Identity Federation pool, the SPA-releases bucket, and the BigQuery audit/billing sinks. There is deliberately NO dedicated upsquad-iac project — splitting state out of upsquad-core is a phase-2 item (#1288). Env projects (upsquad-beta, upsquad-prod) stay purely beta/prod workloads. Container images live in a per-env Artifact Registry inside each env project (not a hub AR — see Reconciliation), so registry IAM never crosses the env boundary.

Org policies bind at upsquad-platform (both envs inherit; tighten on env-prod): deny external IPs (compute.vmExternalIpAccess), block VPC peering (compute.restrictVpcPeering), no default network, disable SA key creation (iam.disableServiceAccountKeyCreation — Workload Identity only), domain-restricted sharing to the upsquad.ai customer ID, resource locations locked to asia-south1, CloudSQL public-IP ban (sql.restrictPublicIp), sql.restrictAuthorizedNetworks, storage public-access prevention, OS Login, deny vmCanIpForward. Per-project budgets + a BigQuery billing export (in upsquad-core) give per-env cost visibility without cross-project IAM.

Two independent custom-mode VPCs, no peering, no Shared VPC. Non-overlapping CIDRs so WARP can route both simultaneously without collision and neither collides with devbox docker (172.17/16), home LANs (192.168/16), or WARP CGNAT (100.64/10). The table below is the source of truth and ships as docs/network/ip-allocation.md plus the Docusaurus page (#1277, #1287):

Purposebeta (upsquad-beta)prod (upsquad-prod)
VPCvpc-betavpc-prod
Subnet (nodes)10.10.0.0/2010.20.0.0/20
Pods (secondary)10.11.0.0/1610.21.0.0/16
Services (secondary)10.12.0.0/2010.22.0.0/20
GKE control plane172.16.0.0/28172.16.0.16/28
CloudSQL PSA10.13.0.0/2410.23.0.0/24
Memorystore10.13.1.0/2410.23.1.0/24
Internal LB / PSC10.14.0.0/2410.24.0.0/24

Private Google Access on every subnet; Cloud NAT per VPC for controlled egress only (no internet inbound); Private Service Connect to the Google restricted VIP (199.36.153.4/30) so API traffic never traverses the internet; PSA ranges for CloudSQL + Memorystore.

2.3 Firewall posture​

Hierarchical firewall policy at upsquad-platform (evaluated before VPC rules): a logged deny-all-ingress catch-all, allow-gke-health-checks (35.191.0.0/16, 130.211.0.0/22), and allow-google-restricted-vip egress. Per-VPC default-deny ingress and egress with explicit allows only (control plane → nodes, health checks, intra-VPC, internal LB → backends; egress to Cloudflare edge ranges on 7844/443, Google restricted VIP, CloudSQL/Memorystore PSA ranges, control-plane CIDR). Firewall Rules Logging on every allow + the deny; VPC Flow Logs on all subnets.

2.4 Compute — Cloud Run now, GKE on trigger + Binary Authorization​

Per the ADR-0017 reconciliation (2026-06-12), the initial compute plane is Cloud Run, not GKE. Each env runs the core API on Cloud Run (private ingress, internal LB), pulling images from that env's Artifact Registry. Binary Authorization is enforced on Cloud Run from day one: prod requires a prod-AR attestation (deploy is denied without it), and beta also requires attestation so the mechanism is exercised before prod relies on it. The Cloud Run deploy SA per env can read that env's AR only.

GKE Autopilot + ArgoCD activate on ADR-0017's documented revisit triggers (>N long-lived services, complex rollout topologies, or the agent worker fleet outgrowing Cloud Run's model). When that trigger fires, the GKE clusters gke-beta / gke-prod are stood up as regional private Autopilot in asia-south1: private nodes, private control-plane endpoint with the public endpoint disabled, master authorized networks limited to internal ranges + the cloudflared subnet, Workload Identity, shielded nodes, security posture + workload vuln scanning, and Binary Authorization carried over to GKE admission (REQUIRE_ATTESTATION, default-admission DENY, attestor keyed to a prod KMS key only the promotion pipeline can use). The node SA per cluster gets artifactregistry.reader on that env's AR repo only. The network/VPC/firewall/VPN/org-policy foundation in §2.1–§2.3, §2.5–§2.7, §2.9 builds now and is compute-agnostic — it carries Cloud Run today and GKE later with no redesign.

2.5 VPN-only access — Cloudflare Zero Trust​

cloudflared runs as a managed connector per env (a minimal Managed Instance Group or equivalent non-GKE home while compute is Cloud Run; it moves to a 2+ replica Kubernetes Deployment in a cloudflare-system namespace per cluster once GKE activates — no canIpForward exception, creds via Secret Manager / ESO). One named tunnel per env (tunnel-beta, tunnel-prod) advertises that VPC's private CIDRs as WARP private-network routes. Engineers enrolled in the upsquad.ai Zero Trust org reach the private control plane (kubectl), CloudSQL (psql), and internal Grafana over WARP — nothing is public. Access policies: beta allows email_domain == upsquad.ai + WARP device-posture; prod allows only a dedicated prod-operators group + MFA + a shorter session. This reuses the established eng.upsquad.ai pattern. (Cloudflare ZT design tracked in #1282; while GKE is deferred the connector runs off a non-GKE home as above.)

2.6 Registry & promotion (the only beta→prod path)​

Artifact Registry per env, inside each env project (asia-south1-docker.pkg.dev/upsquad-beta/upsquad and asia-south1-docker.pkg.dev/upsquad-prod/upsquad), immutable tags. This replaces the hub-AR-in-upsquad-core layout in the original draft (and in ADR-0017's first cut) so registry IAM is isolated per env — see Reconciliation. CI (WIF) has artifactregistry.writer on beta AR only; the prod compute deploy SA (and the prod node SA, once GKE activates) has artifactregistry.reader on prod AR only; a promotion SA (WIF, behind a GitHub production Environment with required reviewers) has reader-on-beta-AR + writer-on-prod-AR and no cluster/Cloud-Run-admin access beyond the gated promotion. Promotion is gcloud artifacts docker images copy by digest beta→prod, then attest the prod digest with the prod KMS key, then re-point prod compute at the attested digest (Cloud Run revision now; GitOps pin once GKE activates). This is the only sanctioned beta→prod movement — image bytes by digest, not a network path. Prod compute cannot pull from the beta AR or GHCR (IAM + BinAuthz both deny).

2.7 Data layer — CloudSQL for prod + beta only​

CloudSQL Postgres 16 + pgvector, private IP only (ipv4Enabled = false), is provisioned for prod and beta only: prod REGIONAL HA + read replica + PITR (30 retained backups); beta ZONAL, smaller tier. beta-dev (the devbox) keeps its self-hosted Postgres (docker-compose) — no GCP DB, per ADR-0001. Memorystore Redis per env, private, with prod on rediss:// TLS + AUTH (#250) and beta on TLS too; AUTH strings in Secret Manager. Secret Manager per project; ESO ClusterSecretStore gcp-secret-store authenticates via Workload Identity to the same-project Secret Manager only — no cross-env secret references are possible (different projects, different WI pools).

2.8 Deploy flow​

Initial (Cloud Run): beta deploys are a Cloud Run revision re-point at the freshly built beta-AR digest on merge to main; prod deploys are the gated promotion job re-pointing the prod Cloud Run service at the attested prod-AR digest. Rollback = re-point at the previous digest (< 5 min, satisfies tenet 5). Image paths are the per-env AR paths (no gcr.io/upsquad-<env>).

On GKE activation (ADR-0017 revisit triggers): ArgoCD runs per env, in-cluster (one in gke-beta, one in gke-prod) — no central ArgoCD reaches across env boundaries. Beta auto-syncs main (selfHeal on), filling the placeholder deployments/<svc>/overlays/staging renamed to overlays/beta. Prod's ArgoCD Application syncs only pinned digests written by the approved promotion job. Rollback = revert the pin (< 5 min).

2.9 Observability & audit​

Cloud Audit Logs with Data Access logs enabled on both env projects (esp. Secret Manager, CloudSQL, AR), plus VPC Flow Logs and Firewall Rules Logging. Log sinks → per-env locked-retention buckets + an aggregated org-level security sink → BigQuery in upsquad-core (feeds the existing siem-export-worker). Per-env Grafana/Prometheus (OpenTelemetry) are internal-only, exposed via cloudflared/Access; prod dashboards sit behind the prod-operators policy.

2.10 Pulumi structure​

One program, two stacks (beta, prod) in infra/pulumi/upsquad-infra/, behind component resources (NetworkStack, GkeStack, RegistryStack, DataStack, IamStack, CloudflareStack). Env differences (tiers, HA, BinAuthz mode, Access groups) come from stack config only → structural parity. State lives in gs://upsquad-pulumi-state (versioned, public-access-prevention, uniform bucket-level access) in the shared upsquad-core project; the Pulumi secrets provider is a GCP KMS key (not the passphrase provider). Splitting state into a dedicated IaC project is phase 2 (#1288).

3. Scope​

In scope (this ADR / milestone #16): beta + prod GCP environments per §2, the IP allocation register, and the Docusaurus architecture page.

Out of scope:

  • dev / beta-dev — stays on the devbox docker-compose stack, governed by ADR-0001. No upsquad-dev GCP project is provisioned.
  • CloudSQL for beta-dev — beta-dev keeps self-hosted Postgres.
  • VPC Service Controls, Secure Web Proxy FQDN egress allowlist, and splitting Pulumi state into a dedicated IaC project — all phase 2 (#1288).

4. Consequences​

4.1 What we gain​

  • Real beta + prod environments with structurally guaranteed isolation (separate projects/VPCs/SAs/secrets; no peering).
  • Defense-in-depth against prod running non-prod images (IAM scoping and Binary Authorization).
  • No public attack surface; all human access is identity-gated through Cloudflare Zero Trust.
  • A single Pulumi program producing both envs from config → parity by construction.
  • Sub-5-minute rollback via GitOps pin reverts (satisfies the platform's rollback tenet).

4.2 What we accept​

  • Managed-infra spend begins (budgeted per project; beta capped lower).
  • Operational footprint grows: two clusters, two CloudSQL instances, tunnels, ArgoCD, KMS, attestors to operate and patch.
  • Egress in v1 is Cloud NAT + default-deny firewall + tightened K8s NetworkPolicies; FQDN-level egress control (Secure Web Proxy) and VPC-SC perimeters are deferred to phase 2 — a residual exfil-surface risk accepted for v1.
  • Pulumi state co-locating in upsquad-core (shared with KMS/WIF/sinks) is a slight blast-radius compromise accepted until the phase-2 split.

4.3 What this ADR does NOT do​

  • It does not change dev/beta-dev (ADR-0001 still governs).
  • It does not weaken RLS, audit logging, or the governance model.
  • It does not approve VPC-SC, Secure Web Proxy, or a dedicated IaC project — those are explicitly phase 2.

5. Phase 2 (tracked, not approved here) — #1288​

  • VPC Service Controls perimeters per env project (AR, Secret Manager, CloudSQL admin, Storage).
  • Secure Web Proxy FQDN egress allowlist replacing the v1 NAT+firewall+NetworkPolicy egress posture.
  • Split Pulumi state out of upsquad-core into a dedicated IaC project.

6. Alternatives rejected​

  • "staging" naming — rejected by the founder in favour of beta / beta-dev.
  • Dedicated upsquad-iac project for state in v1 — deferred to phase 2; upsquad-core hosts state for now to reduce v1 moving parts.
  • CI connecting to clusters for deploys — rejected; pull-based GitOps + digest promotion keeps CI off the clusters entirely.
  • ALWAYS_ALLOW Binary Authorization on beta for CI velocity — rejected; beta enforces attestation so the prod-critical mechanism is proven before prod depends on it.
  • VPC Service Controls in v1 — deferred to phase 2 due to operational friction; the default-deny network + private-everything posture is the v1 control.

Reconciliation (2026-06-12)​

This ADR was briefly merged as a second "ADR-0017", colliding with ADR-0017 — Staged Promotion Pipeline (merged first, which keeps the 0017 number). The founder reconciled the two ADRs in-session on 2026-06-12; this document is renumbered to ADR-0018 and the four decisions below are now normative across both ADRs and the HLD (docs/hld/cloud-foundation-beta-prod.md):

  1. Canonical prod project ID is upsquad-prod (not upsquad-prd). All project, VPC (vpc-prod), cluster (gke-prod), tunnel (tunnel-prod), and AR-path tokens use the -prod form.
  2. Compute is Cloud Run now, GKE on trigger. Per ADR-0017's documented revisit criteria, the core API runs on Cloud Run initially; GKE Autopilot + ArgoCD activate when those triggers fire. The network / VPC / firewall / VPN / org-policy foundation is compute-agnostic and builds now. Binary Authorization is enforced on Cloud Run deploys from day one and carries over to GKE admission later.
  3. Registry is per-env Artifact Registry — beta AR in upsquad-beta, prod AR in upsquad-prod, with gated digest-copy promotion between them. This replaces the hub-AR-in-upsquad-core layout (registry-IAM isolation requirement). The upsquad-core hub keeps the WIF pool, the pulumi-state bucket, the spa-releases bucket, and the audit/billing sinks.
  4. Division of governance. ADR-0018 (this) governs the platform foundation — hierarchy, network, firewall, VPN-only access plane, registry isolation, and the data layer. ADR-0017 governs the app delivery path — stages, human-approval gates, and build-once / by-digest promotion. The two cross-reference each other.
Edit this page
Last updated on by upsquad-backend-sme[bot]