Skip to main content

Cloud Foundation — Beta + Prod (VPN-only, hard-isolated)

Status

Architecture approved by founder 2026-06-12. ADR-0018 is Accepted (#1274, PR #1303). Implementation is tracked in milestone 16 — "Cloud Foundation: Beta + Prod (VPN-only, isolated)". Sections marked pending describe the target state; some controls land issue-by-issue as the milestone progresses. Compute is Cloud Run initially; GKE Autopilot + ArgoCD activate on ADR-0017's revisit triggers.

FieldValue
TypeInfrastructure architecture (high-level design)
StatusApproved 2026-06-12 · ADR-0018 Accepted (#1274)
ADRADR-0018 — activate Class A cloud infra for beta+prod (supersedes ADR-0001's deferral); paired with ADR-0017 (staged promotion pipeline)
Milestone16 — Cloud Foundation: Beta + Prod (VPN-only, isolated)
Regionasia-south1 (Mumbai), everywhere
IaCPulumi (TypeScript) — one program, two stacks (beta, prod)
Source of truth (CIDRs)docs/network/ip-allocation.md (lands with #1277)

Overview​

UpsQuad runs two hard-isolated GCP environments:

  • beta — pre-production, GCP project upsquad-beta.
  • prod — production, GCP project upsquad-prod.

A third "environment", beta-dev, is the existing devbox running the docker-compose stack with self-hosted Postgres. beta-dev stays out of GCP and is governed by ADR-0001; no upsquad-dev GCP project is provisioned.

Both cloud environments are VPN-only with no public surface: no external IPs, private GKE control planes, private CloudSQL, internal-only load balancers. The only way in is the Cloudflare Zero Trust access plane — engineers reach kubectl, psql, and Grafana over the WARP client, never over the public internet.

The defining property is zero code-contamination paths between beta and prod: separate VPCs (no peering, no Shared VPC), separate Artifact Registries, separate clusters, separate secrets. The only sanctioned beta→prod movement is a gated, by-digest image promotion (image bytes, not a network path).

Isolation tenets​

  1. No network path beta↔prod. Independent VPCs, no peering, no Shared VPC, no shared subnets, distinct projects, distinct service accounts.
  2. Prod runs only prod-built images. Binary Authorization on prod requires a prod-AR attestation; the prod node SA can read the prod AR only (no beta AR, no GHCR).
  3. No public surface. No external IPs, private GKE control plane, private CloudSQL, internal-only LBs; the only ingress is outbound-initiated Cloudflare tunnels from inside each VPC.
  4. CI never touches clusters. Promotion is digest-copy + pull-based GitOps; humans approve via a GitHub Environment.

Network diagram​

CIDR / IP allocation​

The table below is documented here for the architecture view. The in-repo source of truth is docs/network/ip-allocation.md (the versioned IP allocation register, landing with #1277). When that file lands, this page links to it and both must stay in sync; if they ever disagree, docs/network/ip-allocation.md wins.

Purposebeta (upsquad-beta)prod (upsquad-prod)
VPCvpc-betavpc-prod
Subnet (nodes)10.10.0.0/2010.20.0.0/20
Pods (secondary)10.11.0.0/1610.21.0.0/16
Services (secondary)10.12.0.0/2010.22.0.0/20
GKE control plane172.16.0.0/28172.16.0.16/28
CloudSQL PSA10.13.0.0/2410.23.0.0/24
Memorystore10.13.1.0/2410.23.1.0/24
Internal LB / PSC10.14.0.0/2410.24.0.0/24

Ranges are non-overlapping so WARP can route both environments simultaneously without collision, and were chosen to avoid the devbox docker network (172.17.0.0/16), typical home LANs (192.168.0.0/16), and the WARP CGNAT range (100.64.0.0/10). Every subnet has Private Google Access enabled; Google API traffic uses the Private Service Connect restricted VIP (199.36.153.4/30) so it never traverses the internet. Cloud NAT per VPC provides controlled egress only — there is no inbound from the internet.

Resource hierarchy + org policies​

Org: upsquad.ai
└── folder: upsquad-platform
    ├── project: upsquad-core      (shared: Pulumi state, KMS, GitHub WIF pool,
    │                               audit/billing BigQuery sinks — no env workloads)
    ├── folder: env-beta
    │   └── project: upsquad-beta  (apps, Cloud Run [GKE on trigger], CloudSQL, Redis, beta AR)
    └── folder: env-prod
        └── project: upsquad-prod  (apps, Cloud Run [GKE on trigger], CloudSQL, Redis, prod AR)

The shared upsquad-core project holds the cross-cutting, non-environment resources: the Pulumi state bucket (gs://upsquad-pulumi-state), the KMS key for state secrets, the GitHub Workload Identity Federation pool, and the audit/billing BigQuery sinks. (Splitting these into a dedicated upsquad-iac project is a deferred phase-2 item — there is no upsquad-iac today.) The env-beta / env-prod projects stay purely environment workloads.

Org policies bind at folder: upsquad-platform so both environments inherit them:

  • compute.vmExternalIpAccess → deny all (no external IPs anywhere)
  • compute.restrictVpcPeering → empty allowlist (no peering)
  • compute.skipDefaultNetworkCreation → enforced (no default VPC)
  • iam.disableServiceAccountKeyCreation → enforced (Workload Identity only; no exportable SA keys)
  • iam.allowedPolicyMemberDomains → restricted to the upsquad.ai customer ID
  • gcp.resourceLocations → asia-south1 only
  • sql.restrictPublicIp → enforced (CloudSQL private IP only)
  • storage.publicAccessPrevention → enforced

Each project carries its own budget + alerts (beta capped lower) with a BigQuery billing export so per-env cost is visible without cross-project IAM.

Access model — Cloudflare Zero Trust​

Connectors. cloudflared runs as a managed connector per env. While compute is Cloud Run, the connector rides a minimal Managed Instance Group (or equivalent non-GKE home, designed in #1282); once GKE activates it moves to a 2+ replica Kubernetes Deployment in a cloudflare-system namespace per cluster (rides Autopilot autoscaling/HA, no VMs to patch). Credentials come from Secret Manager (via External Secrets Operator on GKE). One named tunnel per env: tunnel-beta, tunnel-prod.

Private routing (WARP). Each tunnel advertises its VPC's private CIDRs into Cloudflare as private network routes scoped to that tunnel. Engineers run the WARP client enrolled in the upsquad.ai Zero Trust org with split-tunnel including only the 10.10–10.24/x ranges. Result: kubectl to the private control-plane endpoint, psql to CloudSQL private IP, and internal Grafana all work over WARP, and nothing is public.

Access policies (identity layer).

  • beta — allow email_domain == upsquad.ai (whole team) plus WARP device posture.
  • prod — allow only a dedicated prod-operators group (founder + on-call), require WARP + MFA, with a shorter session lifetime than beta.

This reuses the established eng.upsquad.ai / demo.upsquad.ai provisioning pattern (Access applications + policies created via the Cloudflare API).

CI never reaches clusters. GitHub Actions never connects to either control plane. CI authenticates to GCP via Workload Identity Federation (GitHub OIDC → GCP, no SA keys) and its only jobs are: build + push to the beta AR, and — on approval — run the digest promotion to the prod AR, write the attestation, and re-point prod compute at the attested digest. Deployment is a Cloud Run revision re-point initially (per ADR-0017; not an ArgoCD pin); once GKE activates it becomes pull-based GitOps with ArgoCD running inside each cluster pulling manifests, and nothing pushes into the clusters from outside.

Image promotion flow​

GitHub Actions (WIF/OIDC)
   │ build + push by digest
   ▼
beta Artifact Registry  (asia-south1-docker.pkg.dev/upsquad-beta/upsquad)
   │  gcloud artifacts docker images copy  <SVC>@sha256:DIGEST   (by digest only)
   │  ── gated by GitHub `production` Environment required reviewers ──
   ▼
prod Artifact Registry  (asia-south1-docker.pkg.dev/upsquad-prod/upsquad)
   │  attest the prod digest with the prod KMS key (promotion pipeline only)
   │  re-point prod compute at the attested digest
   ▼
prod Cloud Run revision (now) / ArgoCD pin in prod cluster (once GKE activates)
   │  Binary Authorization: REQUIRE_ATTESTATION → admits only attested prod digests
   ▼
prod workloads

Promotion is by digest, never by tag-following, and is the only sanctioned beta→prod movement. It runs behind a GitHub production Environment with required reviewers, so human approval is the gate. Binary Authorization is set to REQUIRE_ATTESTATION on prod (and on beta too, to exercise the mechanism before prod relies on it); the attestor is keyed to a prod KMS key, and only the promotion pipeline can mint attestations. The prod compute deploy SA (and the prod node SA, once GKE activates) has artifactregistry.reader on the prod repo only — so a prod pull of a beta-AR image fails on both IAM and Binary Authorization. Rollback is re-pointing prod compute at the previous digest (< 5 min).

Compute — Cloud Run now, GKE on trigger​

Initial compute is Cloud Run (private ingress, internal LB) in each env, pulling images from that env's Artifact Registry. Binary Authorization is enforced on Cloud Run deploys from day one (prod requires a prod-AR attestation; beta requires attestation too, to exercise the mechanism early). The Cloud Run deploy SA per env can read that env's AR only.

GKE Autopilot + ArgoCD activate on ADR-0017's documented revisit triggers (>N long-lived services, complex rollout topologies, or the agent worker fleet outgrowing Cloud Run's model). The network / VPC / firewall / VPN / org-policy foundation is compute-agnostic and builds now — it carries Cloud Run today and GKE later with no redesign. When GKE activates, both clusters are private regional Autopilot clusters (gke-beta, gke-prod) in asia-south1:

  • Private nodes and a private control-plane endpoint (public endpoint disabled) — reachable only from inside the VPC, i.e. over WARP.
  • Master authorized networks limited to internal ranges only (no public CIDRs).
  • Workload Identity for KSA→GSA bindings; the node SA is a dedicated minimal GSA per cluster with artifactregistry.reader scoped to that env's AR only.
  • Binary Authorization carried over to GKE admission; GKE security posture + workload vulnerability scanning enabled.

Data layer​

  • CloudSQL Postgres 16 + pgvector, private IP only, for prod and beta only. Prod is REGIONAL HA with a read replica and PITR; beta is ZONAL and smaller. beta-dev keeps its self-hosted Postgres on the devbox — no GCP DB.
  • Memorystore Redis per env, private. Prod uses rediss:// TLS + AUTH (per #250); beta uses TLS for parity. AUTH strings live in Secret Manager.
  • Secret Manager per project. The External Secrets Operator gcp-secret-store authenticates via Workload Identity to the same-project Secret Manager only — no cross-env secret references are possible.

Deploy​

Initially (Cloud Run): beta deploys re-point the beta Cloud Run service at the freshly built beta-AR digest on merge to main; prod deploys are the gated promotion job re-pointing the prod Cloud Run service at the attested prod-AR digest. Rollback is a revision re-point at the previous digest (< 5 min).

On GKE activation (ADR-0017 revisit triggers): ArgoCD runs per env, in-cluster (no central ArgoCD reaching across the env boundary). Beta auto-syncs main (filling the placeholder deployments/<svc>/overlays/staging, renamed to overlays/beta). The prod ArgoCD Application syncs only pinned digests written by the approved promotion job.

Observability & audit​

Cloud Audit Logs (including Data Access logs) on both projects, an org-level security sink to BigQuery feeding the siem-export-worker, VPC Flow Logs + Firewall Rules Logging everywhere, and per-env Grafana/Prometheus exposed internal-only behind Cloudflare Access.

Pulumi structure​

One program, two stacks (beta, prod) in infra/pulumi/upsquad-infra/, built from component resources (NetworkStack, ComputeStack — Cloud Run now, the GKE path activated on ADR-0017's revisit triggers — RegistryStack, DataStack, IamStack, CloudflareStack). Environment differences (tiers, HA, Binary Authorization mode, Access groups) come from stack config only, not code forks — guaranteeing structural parity. State lives in gs://upsquad-pulumi-state (project upsquad-core) with the secrets provider backed by a GCP KMS key.

Phase 2 (deferred)​

  • VPC Service Controls perimeters around AR, Secret Manager, CloudSQL admin, and Storage.
  • Secure Web Proxy FQDN egress allowlist (v1 uses Cloud NAT + default-deny firewall + tightened K8s NetworkPolicies).
  • Splitting Pulumi state / shared resources out of upsquad-core into a dedicated IaC project.

References​

  • ADR-0018 (#1274) — activate Class A cloud infra for beta+prod (Accepted 2026-06-12; this HLD's governing ADR)
  • ADR-0017 — staged promotion pipeline (the app delivery path; reconciled with ADR-0018 on 2026-06-12)
  • ADR-0001 — hybrid local dev stack (governs beta-dev / devbox)
  • ADR-0002 — dual business model (cloud + on-prem)
  • docs/network/ip-allocation.md — versioned IP allocation register (source of truth; lands with #1277)
  • Milestone 16 — Cloud Foundation: Beta + Prod (VPN-only, isolated)
Edit this page
Last updated on by upsquad-backend-sme[bot]