Skip to main content

ADR-0025 — MCP egress: allow approved private/in-cluster upstreams + close the initializer SSRF gap

Status: Accepted (founder-approved 2026-07-22, issue #2019) Author: Principal Architect Refs: #2013, #1988, #2019 · ADR-0021 (T4 group fan-out, T7 termination), ADR-0022 (team gateway), ADR-0024 (MCP-tool governance)

Context​

Completing the governed refund E2E on beta-dev (#2013) surfaced two coupled egress defects in the MCP team-gateway path.

(1) Legit private/in-cluster upstreams are unreachable by governed tools/call. internal/mcp/egress/deny.go always-denies RFC1918/loopback/link-local/ULA (alwaysDenyCIDRs), enforced twice in roundtripper.go — at resolve and at DialContext (ReasonPrivateIP). This is the correct SSRF defense for arbitrary public egress, but an MCP upstream is legitimately private: in GKE a ClusterIP/headless Service, on the dev docker stack the fixture resolves to 172.18.x (inside 172.16.0.0/12). The allow-list model (tenant_egress_allowlist, migration 035) is hostname-pattern-only and rejects IP literals, and always-deny is code, not data — so there was no vetted way to reach ANY private upstream through the guarded client.

(2) Initializer SSRF hole + inconsistent guarding. The initialize fan-out was constructed as mcpproxy.NewHTTPUpstreamInitializer(nil) (cmd/context-engine/main.go) — a plain &http.Client{Timeout:15s}, no egress guard. The tools/call Forward path uses newEgressHTTPClient (guarded). This asymmetry is why #2013 observed init reaching a private upstream while call did not — init was silently bypassing SSRF defenses and could hit any private IP including cloud metadata 169.254.169.254. Both hops must traverse the same guard.

Decision​

D1 — Unify the guard (close the SSRF hole). Route the initialize fan-out through the SAME egress-guarded client as tools/call.

  • cmd/context-engine/main.go: NewHTTPUpstreamInitializer(newEgressHTTPClient(pool, 15*time.Second)).
  • HTTPUpstreamInitializer.Initialize stamps the tenant onto the outbound ctx via egress.WithTenant(ctx, member.OrgID). Member gained an OrgID field (resolver-derived — GroupUpstream.OrgID, the same value as ForwardParams.OrgID). Without it the guard fails-closed ReasonNoTenant.
  • Intended consequence: init now ALSO fails-closed on private upstreams — making the D2 override the single unblock point for both hops. Redirects stay refused (factory default ErrUseLastResponse).

D2 — Vetted private-upstream override (option (a): FQDN + allow_private flag, approval-scoped).

  • Migration 168 extends tenant_egress_allowlist: allow_private BOOLEAN NOT NULL DEFAULT false, pinned_cidr CIDR NULL, a new reason mcp_private_upstream_approval in the CHECK, and a CHECK (allow_private = false OR pinned_cidr IS NOT NULL). Rows stay keyed on domain_pattern (an FQDN — dotted, ParsePattern-valid; single-label hosts and raw IP literals still rejected). Default false ⇒ zero behavior change for every existing row. Migration 168 also adds mcp_server_units.private_upstream_approved BOOLEAN NOT NULL DEFAULT false + private_pinned_cidr CIDR (the binding capability + captured CIDR).
  • deny.go: alwaysDenyCIDRs is split into overridableCIDRs = {10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, fc00::/7} and neverOverridableCIDRs = {0.0.0.0/8, 127.0.0.0/8, 169.254.0.0/16, fe80::/10, ::1/128}. alwaysDenyCIDRs is now the union of the two (never drifts). New IsNeverOverridable(ip). IsAlwaysDenied unchanged for the non-override path.
  • Allowlist model (patterns.go, allowlist.go): Pattern carries AllowPrivate bool + PinnedCIDR netip.Prefix; PostgresAllowlist SELECTs the new columns; the Redis cache marshal is extended (with legacy []string tolerance).
  • guard.go: DefaultGuard.PrivateOverride(ctx, tenant, host) returns the matching row's pinned CIDR.
  • roundtripper.go: at resolve + at DialContext, if the request host matches an allow-list row with AllowPrivate=true, an IsAlwaysDenied IP is permitted ONLY when it is NOT IsNeverOverridable AND falls inside that row's PinnedCIDR (privateAllowed). Everything else stays ReasonPrivateIP.
  • Pin at approval, re-validate at connect (DNS-rebinding defense): the pin is captured on the binding at approval time. Each request re-resolves the FQDN; at connect the pinned IP is re-validated ∈ pinned_cidr in DialContext. An attacker repointing the FQDN to metadata is denied (metadata ∈ neverOverridable, ∉ pinned_cidr) at BOTH the resolve and the dial hop.
  • Two gates, per-host: only a Manager-approved binding may seed the row (SeedFromMCPServerApproval), AND only when the binding's private_upstream_approved capability is set — writing THAT capability requires L5 clearance (UpdateMCPServerBinding, privateUpstreamMinClearance = 5). Never "allow all private."

What STAYS denied (unchanged, even WITH the flag): cloud metadata 169.254.0.0/16, loopback 127.0.0.0/8/::1, IPv6 link-local fe80::/10, 0.0.0.0/8; any private IP outside an approved row's pinned_cidr; raw IP literals in mcp_servers.url; 3xx redirects; cross-tenant (RLS on tenant_egress_allowlist + tenant-scoped guard). Composes with ADR-0024 governance: the edge clearance floor / allowlist / tool-policy still gate the call BEFORE egress; this ADR only widens the network reachability for an explicitly-approved host. A seed whose pinned_cidr overlaps the SSRF floor is refused (ValidatePinnedCIDR), so a floor-overlapping pin never reaches the DB.

Consequences​

Security posture — SSRF vectors CLOSED: the initializer no longer bypasses the guard (both hops guarded); metadata/loopback/link-local unreachable even with the flag; private reach is per-approved-FQDN + pinned-CIDR + Manager-approval + L5 capability; DNS-rebinding defeated by pin + connect-time re-validate; redirects refused; passthrough ban and RLS unchanged.

GKE / prod path (build-for-prod): a real in-cluster upstream is a ClusterIP/headless Service. Register mcp_servers.url = https://svc.ns.svc.cluster.local:port/mcp; set private_upstream_approved=true + private_pinned_cidr = <Service CIDR or ClusterIP /32> on the binding (L5); Manager approves; the approval seeds an allow_private=true egress row. Identical code path to dev — no dev-only branch.

Dev-unblock (one-step config, same mechanism): on the beta-dev refund binding set private_upstream_approved=true + private_pinned_cidr='172.18.0.0/16' (L5); the Manager-approval seeder writes tenant_egress_allowlist(domain_pattern='betadev-refund-mcp.mcp.internal', allow_private=true, pinned_cidr='172.18.0.0/16', reason='mcp_private_upstream_approval'). This unblocks BOTH the (now-guarded) initialize and tools/call through the single override, closing the #2013 dev-config chain.

Rollout​

  1. Land D1 guard-unification + migration 168. Default allow_private=false ⇒ no behavior change; inert until a row sets allow_private.
  2. Enable private_upstream_approved + private_pinned_cidr on the beta-dev refund binding (L5) → seed override row → verify init+call reach the 172.18.x fixture end-to-end (re-run #2013 E2E).
  3. Document GKE in-cluster upstream registration in the MCP runbook.
Edit this page
Last updated on by upsquad-backend-sme[bot]