Runbook: Governed code-execution sandbox substrate (WR2-3 Slice 2)

Owner: DevOps Engineer · Refs: LLD #2184 §2/§3, HLD #2113 §2/§3 (founder security review PASSED), milestone #28 · Task: #2187

This runbook covers the SUBSTRATE the SandboxRunner seam (#2188) runs on: the gVisor RuntimeClass + pod template, the forced-proxy egress topology, the docker-compose dev equivalent, and the fail-closed proof.

---

1. What this is (and is not)

• Is: the isolation + egress substrate — RuntimeClass, ephemeral pod template, the forced egress-proxy, NetworkPolicies, and the dev-box equivalent.
• Is not: the Go SandboxRunner / code_exec provider (#2188), the workspace store (#2185), or the security-assertion suite (#2191). This slice delivers what those run on.

2. The load-bearing invariant (HLD Finding A)

There is no code path in which a sandbox reaches the network without an allow decision, because the network is unreachable by construction otherwise.

Enforcement is at the network boundary, not the Go http.RoundTripper (GetEgressTransport is invisible to raw sockets inside agent code). A sandbox pod runs with no default route; a NetworkPolicy permits egress ONLY to DNS + the forced proxy on 3128. Every failure mode (proxy down, misconfigured, allowlist error) collapses to no egress — fail-closed by topology, the anti-WF-30 design.

3. Components

Component	Prod (GKE)	Dev (docker-compose)
Isolation	runtimeClassName: gvisor (GKE Sandbox)	runtime: runsc if present, else runc fallback
No-route netns	per-pod netns + sandbox-egress-forced-proxy-only NetworkPolicy	sbx-internal network (internal: true)
Forced proxy	egress-proxy Deployment (squid, default-deny)	sbx-egress-proxy container (same squid.conf shape)
Allowlist DATA	tenant_egress_allowlist patterns -> /etc/squid/allowlist.txt (per-run grant)	dev/sandbox/allowlist.txt
Deny CIDRs	deny.go two tiers (floor + RFC1918)	floor only (see §5)
Resource bounds	pod resources.limits (cpu/mem/ephemeral-storage/pids) + activeDeadlineSeconds	cpus/mem_limit/pids_limit/tmpfs

Files:

• deployments/sandbox/base/ — RuntimeClass, namespace, proxy (ConfigMap/Deployment/Service), NetworkPolicies.
• deployments/sandbox/overlays/{dev,prod}/ — dev = floor-only + single proxy; prod = HA proxy (2 + PDB), RFC1918 private-deny injected, RuntimeClass removed (Autopilot-managed).
• deployments/sandbox/templates/sandbox-pod.yaml — the per-run-step Pod the runner (#2188) stamps out. NOT a standing object (excluded from kustomize).
• dev/sandbox/ + docker-compose.sandbox.dev.yml — dev topology.
• scripts/sandbox-egress-smoke.sh — the fail-closed proof.

4. runsc-on-devbox feasibility

runsc is NOT installed on the current devbox (only runc; no /dev/kvm, so Firecracker is also out — consistent with HLD §2). The dev fallback (LLD §2.3) therefore runs: the governed path is identical — netns-no-route + forced proxy + cgroups + cap-drop + non-root + read-only-root + RuntimeDefault seccomp. Only the kernel-isolation strength downgrades (userspace-kernel gVisor -> host-kernel runc). The smoke prints a loud WARN and treats it as sandbox_degraded. To run the prod-faithful path on a runsc-capable box:

# install runsc + register the docker runtime, then:
SANDBOX_RUNTIME=runsc bash scripts/sandbox-egress-smoke.sh

Config assertion (owned by #2188): the gvisorRunner MUST refuse to start in CI/prod when the resolved runtime is not runsc (assert on SANDBOX_RUNTIME/RuntimeClass), so the degraded fallback can never run outside a developer box. This substrate surfaces the signal; the runner enforces it.

5. Prod-vs-dev delta (and why dev is still prod-faithful)

The only substantive delta is the RFC1918 private-CIDR deny:

• SSRF FLOOR (neverOverridableCIDRs in internal/mcp/egress/deny.go: metadata 169.254.0.0/16, loopback, link-local, 0.0.0.0/8) — enforced identically in dev + prod (squid ssrf_floor ACL, denied first).
• RFC1918 (overridableCIDRs: 10/8, 172.16/12, 192.168/16, fc00::/7) — prod only (overlays/prod injects 10-private-deny.conf into squid conf.d). Dev omits it because dev upstreams legitimately live on the docker RFC1918 bridge. This mirrors deny.go's own ADR-0025 allow_private in-cluster-upstream override — dev is exactly the "reach an in-cluster upstream on a private IP" case that tier is designed to permit.

Everything else — no-route topology, forced-proxy default-deny, allowlist DATA plumbing, cap-drop/non-root/read-only/seccomp/pids/cpu/mem bounds — is byte-identical in shape. Only the pod-create vs docker-run launch call differs, and that is isolated in the runner's platform adapter (#2188).

6. Fail-closed proof (run it)

bash scripts/sandbox-egress-smoke.sh

Asserts (exit 0 iff ALL hold):

#	Test	Expect
P0	direct dial to a public IP (no proxy)	blocked — no route
P1	direct dial to metadata 169.254.169.254	blocked — no route
P2	proxied + allowlisted host	HTTP 200
P3	proxied + non-allowlisted host	HTTP 403 (default-deny)
P4	proxied + metadata IP	HTTP 403 (SSRF floor)

Serve-mode preview (LBE slice 1, #2239 — the architect Flaw-1 correction)

The same script ALSO runs the P0–P4 proof from a preview-sandbox-labeled pod (the serve-mode sibling) plus a router-reachability check, so a preview pod's egress is proven byte-for-byte the batch posture:

#	Test	Expect
PP0	preview pod direct dial to a public IP	blocked — no route
PP1	preview pod direct dial to metadata	blocked — no route / lateral+metadata
PP2	preview pod proxied + allowlisted host	HTTP 200 — it reaches the forced proxy
PP3	preview pod proxied + non-allowlisted	HTTP 403 (default-deny)
PP4	preview pod proxied + metadata IP	HTTP 403 (SSRF floor)
PP5	served whoami dialed from a router-net container on $PORT	HTTP 200 — reachable in-network

Why PP2 is the Flaw-1 assertion (GKE-only): on GKE, a preview pod is labeled upsquad.io/component: preview-sandbox (so the batch NetworkPolicy's ingress: [] stays untouched). But the forced-proxy's own ingress policy (egress-proxy-netpol, deployments/sandbox/base/networkpolicy.yaml) previously admitted only code-exec-sandbox on 3128 — a preview-sandbox pod could not reach the proxy AT ALL (fail-closed, but its egress was fully blocked, not "identical to batch"). This slice amends egress-proxy-netpol to admit preview-sandbox too (shared egress infra — the one deliberate touch), and adds preview-sandbox-netpol.yaml (ingress from the preview-router on $PORT; egress = DNS + 3128 only). PP2 is the assertion that would have caught the gap. The dev-box caveat: docker egress is network-membership-based (both sandbox types join sbx-internal), not label-selector-based, so the compose stack was never affected — the gap was GKE/prod-only.

Last local run (runc fallback, sandbox_degraded=1) — batch P0–P4 + preview PP0–PP5:

  PASS: P0 direct public IP blocked (no route): [000 rc=7]
  PASS: P1 direct metadata IP blocked (no route): [000 rc=7]
  PASS: P2 proxied allowlisted host allowed: HTTP 200
  PASS: P3 proxied non-allowlisted host denied: HTTP 403
  PASS: P4 proxied metadata IP denied (SSRF floor): HTTP 403
  PASS: PP0 preview direct public IP blocked (no route): [000 rc=7]
  PASS: PP1 preview direct metadata IP blocked (no route): [000 rc=7]
  PASS: PP2 preview reaches forced proxy for allowlisted host: HTTP 200
  PASS: PP3 preview proxied non-allowlisted denied: HTTP 403
  PASS: PP4 preview proxied metadata IP denied (SSRF floor): HTTP 403
  PASS: PP5 served whoami reachable from router-net container on $PORT: HTTP 200
  ==> RESULT: 11 passed, 0 failed
  ==> FAIL-CLOSED PROOF: OK

Serve-mode CLI smoke (slice-1 acceptance, no proxy needed)

cmd/preview-serve-smoke is the standalone Go caller of the ServiceRunner seam — NewGVisorServiceRunner → StartService (launch + readiness) → ProbeService (liveness) → StopService (idempotent teardown) — over the dev docker ServiceLauncher, using buildDockerArgs verbatim (the served whoami runs the IDENTICAL hardening as a batch run; only the launch shape is detached):

go run ./cmd/preview-serve-smoke \
  -network upsquad-sandbox-dev_sbx-internal \
  -image traefik/whoami:v1.10.2 -args "--port,8080"

Last local run (runc fallback, sandbox_degraded=1):

  PASS: became ready at sbx-prev-...:8080 (runtime=runc degraded=true)
  PASS: liveness probe reports the service live
  PASS: stopped cleanly
  PASS: re-stop is idempotent
  ==> SERVE-MODE SMOKE: OK

CI: .github/workflows/sandbox-egress-smoke.yml runs P0–P4 (+ preview PP0–PP5) on every PR touching the substrate (non-authoritative — no runsc on GH runners; #2191 adds the authoritative gVisor gate on a runsc-capable runner).

The HLD's P1 netns-escape / P2 proxy-down / P3 DNS-rebind pen-tests are hardened further by QA in #2191; this script is their seed.

7. Validate the manifests

kustomize build deployments/sandbox/overlays/prod | kubeconform -strict -summary
kustomize build deployments/sandbox/overlays/dev  | kubeconform -strict -summary -skip RuntimeClass
kubeconform -strict deployments/sandbox/templates/sandbox-pod.yaml
docker compose -f docker-compose.sandbox.dev.yml config

8. Deploy / rollback (prod)

Deploy is GitOps via ArgoCD (never kubectl apply by hand). Point an ArgoCD Application at deployments/sandbox/overlays/prod. Rollback = revert the commit; ArgoCD syncs the prior revision (< 5 min). Because egress is fail-closed by topology, a partial/failed sync degrades to no egress, never to open egress.

9. Secrets

No secrets enter the sandbox (HLD §3.3). The only in-sandbox credential is the per-run, prefix-scoped object-store signer (workspace #2185), injected by the runner (#2188) — never baked into any manifest here. The forced proxy holds no tenant credentials; it consults allowlist DATA only.