KT — Governed MCP & the Team Gateway

Audience: a new engineer joining the MCP / gateway effort. Goal: understand the whole system end‑to‑end, know what is real vs stubbed vs parked, be able to onboard an MCP server, and be able to hand external‑agent operators the exact steps to connect and authenticate.

Status legend used throughout: 🟢 LIVE (merged, in the prod code path) · 🟡 STUBBED (placeholder, works but not final) · 🔵 DEV‑ONLY (only wired for beta‑dev/docker) · 🟠 PENDING/PARKED (designed and/or coded but gated off / blocked).

---

0. TL;DR — the mental model

• An agent never talks to a customer's MCP server directly. It talks to our team gateway at /mcp/team/{unit}, which is the single Policy Enforcement Point (PEP) for every MCP tool call.
• There are two authentication hops, and keeping them straight is the whole game:
  • Hop‑A — agent → our gateway ("prove who you are to us"). A short‑lived RS256 JWT, either platform‑minted (agents running inside our platform) or obtained via POST /oauth/token (agents running outside our platform, pre‑registered as external agents).
  • Hop‑B — our gateway → the upstream MCP server ("we authenticate to the tool on your behalf"). One of none / bearer / oauth2, using credentials we hold in a vault. The agent's inbound token is never forwarded upstream (passthrough ban).
• Between the PEP and the upstream there is a network egress guard (SSRF defence): private IPs are denied by default and only reachable through an explicit, approval‑scoped allowlist row.
• Governance (clearance floor, per‑agent allowlist, HITL approval, tiered guardrails, manager subtree‑withhold) is enforced at the edge, in order, before anything leaves our process. This is ADR‑0024.

---

1. Architecture & control flow (numbered, colored)

Colour key: 🔵 blue = agent runtime · 🟡 amber = auth/trust · 🟢 green = gateway PEP · 🔴 red = egress/SSRF · 🟣 purple = upstream MCP · ⚪ grey = data stores.

Walkthrough (numbers match the diagram)

0. (External agents only) The operator registers the agent once, then exchanges client_credentials at POST /oauth/token to get a short‑lived JWT identical to what an internal agent would carry. Internal agents skip this.
1. Session freeze. When a session starts, the Session Initializer (internal/runtime/session/initializer.go) resolves the agent's effective config through the Org‑v2.4 multi‑parent resolver.
   • 1a. D6 clearance‑floor drop: any MCP server whose min_clearance exceeds the agent's clearance is removed from the snapshot (dropMCPBelowClearanceFloor → filterMCPByClearanceFloor). Surviving tools are frozen as mcp:<serverID> and exposed to the model as {server}__{tool}.
2. Hop‑A mint. On each ExecuteStep, the orchestrator mints a ~10‑minute RS256 JWT (RS256AgentTokenMinter.Mint, internal/auth/oauth/agent_token.go) — but only if the snapshot actually contains MCP servers. Claims: aud=<base>/mcp/team/{unit}, azp=agent-runtime, org_id, agent_id, unit_id.
   • 2b. The agent calls tools/call on the gateway carrying that JWT as Authorization: Bearer.
3. Edge PEP — enforced in this exact order (internal/gateway/mcpproxy/dispatch.go, gates in edge_policy.go). All gates fail closed.
   • 3.1 Reverse‑route {server}__{tool}; unknown → -32602 (never misroute). Validate the JWT signature (against JWKS) and aud.
   • 3.2 OAuth terminate + confused‑deputy consent; resolve/mint the upstream credential for Hop‑B. require_per_agent bindings fail closed here if no agent identity.
   • 3.3 EdgeClearanceFloor (D6) — coarsest gate; deny → -32602 + mcp_clearance_floor_denied audit.
   • 3.4 EdgeAgentAllowlist (#1744) — server not in the agent's creation‑time allowlist → deny.
   • 3.5 EdgeToolPolicy (D3, tiered CEL guardrails) — tri‑state: allow / deny (-32602) / requires‑approval (-32001 + a governance_approvals row — a HITL pause, never a silent deny).
   • (Manager subtree‑withhold, D4b, is applied earlier at resolution time — a withheld server never even enters the member/upstream set — see pg_group_resolver.go / resolver.go.)
   • 3.7/3.8 Rewrite to the raw upstream tool name and forward through the shared, egress‑guarded upstream proxy.
4. Egress guard (Hop‑B network layer) (internal/mcp/egress). The upstream HTTP client always‑denies RFC1918/ULA private IPs and the never‑overridable SSRF sinks (metadata 169.254/16, loopback, link‑local, 0.0.0.0/8). Private upstreams are reachable only via an approval‑scoped tenant_egress_allowlist row (allow_private=true + pinned_cidr). The connect‑time IP is re‑validated at DialContext to defeat DNS‑rebind. Host patterns must be dotted FQDNs (a bare docker service name is rejected by design).
5. Upstream call. The proxy does initialize → captures Mcp-Session-Id → tools/call, injecting the Hop‑B credential (or omitting Authorization entirely for type=none). Result flows back to the agent.

---

2. Component catalog & status

Component	Path	Status	Notes
Session Initializer + D6 snapshot drop	internal/runtime/session/initializer.go	🟢 LIVE	mcp:<serverID> freeze; #1744 narrow; #2027/#2028 cache‑poisoning fixed
Org‑v2.4 multi‑parent resolver + cache	internal/orgunit/resolver.go	🟢 LIVE	clone‑on‑get/put (#2028), invalidation (#2024), 45s TTL (#2026)
Hop‑A RS256 minter	internal/auth/oauth/agent_token.go	🟢 LIVE (code)	~10m TTL; prod blocked by #1864 (no prod key)
JWKS / issuer routes	internal/auth/oauth/issuer_routes.go (on context‑engine)	🟢 LIVE (code)	nil key → mounts nothing (fail‑safe)
/oauth/token (external agents)	internal/auth/oauth/token_endpoint.go	🟢 LIVE (code)	client_credentials; parked for prod (#1864)
RegisterExternalAgent + trust table	internal/agent/external_service.go, external_agent_clients (mig 155)	🟢 LIVE (code)	DB trust‑table, not per‑runtime JWKS
Team gateway edge dispatch (PEP)	internal/gateway/mcpproxy/dispatch.go	🟢 LIVE	ordered gates, all fail‑closed
EdgeClearanceFloor / AgentAllowlist / ToolPolicy CEL	internal/gateway/mcpproxy/edge_policy.go	🟢 LIVE	wired in cmd/context-engine/main.go
Manager subtree‑withhold	pg_group_resolver.go, resolver.go	🟢 LIVE	enforced at every resolution surface (#1992)
Egress guard + SSRF floor + rebind re‑validate	internal/mcp/egress/{roundtripper,deny,patterns}.go	🟢 LIVE	ADR‑0025 (#2019/#2020)
Upstream proxy + passthrough ban	internal/gateway/mcpproxy/upstream_proxy.go	🟢 LIVE	injects Hop‑B cred only; never forwards client token
Idempotency dedup	upstream_proxy.go + pg_idempotency.go	🟢 LIVE	folded in from ContextForge; fails OPEN (correctness backstop, not a gate)
GovernanceService #288 on the MCP path	internal/governance	🟢 EXITED (D2)	the residue default‑deny was removed; the store survives only as the CEL policy store
ContextForge (services/mcp-proxy)	—	🟠 REMOVED	deleted in #1909/#1912; idempotency folded into the Go edge
Per‑tool full inputSchema passthrough	dispatch.go (permissive {"type":"object"})	🟡 STUBBED	arg validation deferred to CEL edge + upstream; follow‑up

---

3. The two hops in detail (auth mechanisms)

Hop‑A — agent → our gateway (authenticate to us)

One signing key, one validator, two ways to obtain the token.

Mechanism	Who uses it	How it works	Status
Platform‑minted	agents running inside our platform	orchestrator mints the RS256 JWT itself each ExecuteStep; gateway verifies against JWKS and trusts only the signed agent_id claim	🟢 code LIVE · 🟠 prod parked (#1864)
External‑agent	agents running outside our platform (BYO)	operator pre‑registers (RegisterExternalAgent → external_agent_clients row: client_id → org_id/agent_id/unit_id, plus a public JWK or a bcrypt secret), then calls POST /oauth/token grant client_credentials; the endpoint resolves the binding and mints the same JWT stamped runtime=external	🟢 code LIVE · 🟠 prod parked (#1864)

• Client‑auth methods for the external path: private_key_jwt (preferred, RFC 7523) or client_secret_post.
• Signing key (beta‑dev): host /opt/upsquad-secrets/platform-issuer/platform-signing-key.pem → container /var/secrets/platform-issuer/platform-signing-key.pem (PLATFORM_SIGNING_KEY_FILE), kid dev-platform-YYYYMMDD, RSA‑2048, generated by scripts/dev-secrets/gen-platform-signing-key.sh. Issuer default http://context-engine:8080.
• Validation: JWKSValidator checks the signature against the cached JWKS, then the terminator checks aud + confused‑deputy consent (consent is per‑team azp=agent-runtime, not per individual agent). A worker cannot forge a peer's agent_id.
• #1864 (OPEN, blocker/arch‑concern/approved): production has no real platform signing key / JWKS and MCP_GATEWAY_ENABLED defaults false. So both Hop‑A paths are exercisable on dev but parked for prod until the platform issuer is provisioned. The "per‑runtime JWKS / BYO‑trust" third option is recorded on #1864 only — not implemented; external trust today is the DB trust‑table.

Hop‑B — our gateway → the upstream MCP server (we authenticate to the tool)

Upstream type	Meaning	What we send	Status
none	anonymous MCP server	no Authorization header	🟢 LIVE
bearer	static shared token	Authorization: Bearer <vault token>	🟢 LIVE
oauth2	gateway identity via token exchange	RFC 8693 exchange → minted token → Bearer	🟢 LIVE

• Passthrough ban: the agent's inbound token is never forwarded upstream; an empty upstream credential fails closed to no header, never to the client's cred.
• Credential granularity (answers "are creds per agent?"): creds live in the vault keyed by team_id at three grains (credential/scope.go): org _default, team <unit_id>, agent _agent_<agent_id>. The binding carries credential_mode (team_shared default | per_agent) plus require_per_agent. Resolver precedence (credential/resolve.go):
  • require_per_agent → agent‑slot only, fail closed if the agent has no cred;
  • per_agent → agent → team → org fallback;
  • team_shared → team → org.
  • So: per‑server/binding by default, optionally per‑team, optionally per‑agent — and per‑agent can be made mandatory (fail‑closed).

Responsibilities per mechanism

Mechanism	Platform / product side	Customer side	External‑agent operator
Hop‑A platform‑mint	provision real signing key + JWKS (#1864); flip MCP_GATEWAY_ENABLED	—	n/a (internal)
Hop‑A external‑agent	same key/gate; expose RegisterExternalAgent + /oauth/token	register the agent, hand out client_id + secret or register the agent's public JWK	hold client_id + private key/secret; exchange client_credentials for a fresh JWT (~every 10m)
Hop‑B bearer / oauth2	vault + resolver (live)	register the MCP server, set type, run SetMCPServerCredential, choose credential_mode / require_per_agent, upload per‑agent creds if desired	n/a
Hop‑B none	—	register server with type=none	n/a

---

4. Onboarding an MCP server (operator checklist)

Do these in order. Each row is one RPC/UI action and its DB effect.

#	Step	Surface	Effect
1	Register the server on a unit (team/pillar) — name, URL, min_clearance (0–5), visibility, credential_mode, require_per_agent, oauth type	RegisterMCPServer · client SPA mcp-registry.jsx register form 🟢	inserts mcp_servers (status=pending) + mcp_server_units (approval_status=pending, binding_kind=grant); a registration probe captures tool_schema
2	Lifecycle approval (T1) — a unit Manager flips the server live	SetMCPServerStatus (Manager‑only, fail‑closed)	mcp_servers.status pending→approved; only approved projects to the upstream view; evicts resolver cache
3	Per‑team binding approval (T2)	UpdateMCPServerBinding / decideBinding · team-gateway/pending-bindings-panel.jsx	mcp_server_units.approval_status pending→approved; inert until approved
4	(optional) Subtree‑withhold a server for a subtree	SetMCPServerWithhold (Manager‑only)	binding_kind='withhold' — subtracts the server from that unit + descendants (D4b)
5	Guardrails (3‑tier) — registry/server → team → agent tool policies	GuardrailService / ToolPolicyService · team-gateway/entitlement-matrix.jsx	mcp_tool_policies (CEL argument_rule / response_filter); unit_id NULL = org‑wide
6	Per‑agent tool selection (narrow‑only) at agent creation	AgentService.SetAgentTools (L4 admin or L3 Manager, IDOR‑guarded)	agent_configurations.tools — a subset of the team's tools
7	Credential (fail‑closed) — bearer token or oauth client secret	SetMCPServerCredential · team-gateway/credential-dialog.jsx	writes vault (mcp_bearer_token_* / mcp_oauth_client_secret_*)
8	Egress allowlist — only for a PRIVATE upstream (L5 + Manager gated)	auto‑seeded on binding approval when private_upstream_approved=true (seed_mcpapproval.go)	tenant_egress_allowlist row allow_private=true + pinned_cidr, reason mcp_private_upstream_approval. Dev: docker bridge 172.18.0.0/16, dotted alias betadev-refund-mcp.mcp.internal. Prod: the GKE ClusterIP CIDR. SSRF sinks are never overridable.

Verification after onboarding: start a fresh session for an agent whose clearance ≥ min_clearance, confirm the tool appears (GetTeamEntitlementMatrix verdict allowed), then run a real tools/call.

---

5. Connecting an EXTERNAL agent (hand these steps to the operator)

Use this when the agent runs on infrastructure we don't control (customer's LangGraph/Claude‑SDK/etc.). Prereq (platform side): the platform issuer must be live — currently dev‑only, see #1864.

One‑time registration (done by the customer's admin in our platform / by us on their behalf):

1. Create the agent in the target team (unit) and note its agent_id, unit_id, org_id.
2. Register the external client: RegisterExternalAgent → returns a client_id. Choose the client‑auth method:
   • private_key_jwt (recommended): the operator generates a keypair and registers the public JWK with us; the private key never leaves their side.
   • client_secret_post: we issue a client secret (store it in their secret manager).

Per‑run (the agent does this itself, every ~10 minutes): 3. Obtain a gateway token:

POST /oauth/token
Content-Type: application/x-www-form-urlencoded

grant_type=client_credentials
&client_id=<client_id>
# private_key_jwt:
&client_assertion_type=urn:ietf:params:oauth:client-assertion-type:jwt-bearer
&client_assertion=<signed JWT>
# …or client_secret_post:
&client_secret=<secret>

→ returns a short‑lived RS256 JWT (aud=<base>/mcp/team/{unit}, runtime=external). 4. Call our gateway with it:

POST /mcp/team/{unit}
Authorization: Bearer <token from step 3>
Content-Type: application/json

{ "jsonrpc":"2.0", "id":1, "method":"tools/call",
  "params": { "name":"{server}__{tool}", "arguments": { … } } }

5. Refresh the token before it expires (~10m) and retry on 401.

What the operator must understand about errors (they are JSON‑RPC, not just HTTP):

• -32602 → denied by a governance gate (clearance floor, not in allowlist, or a tool‑policy deny). Not retryable without a config change.
• -32001 → requires human approval (HITL); a governance_approvals row was created. The call succeeds once approved — poll/retry.
• 401 → token missing/expired/invalid aud → re‑mint at /oauth/token.

About Hop‑B credentials for external agents: the external agent authenticates to us (Hop‑A). It does not hold the upstream MCP server's credentials — we inject those (Hop‑B) from our vault. If the binding is require_per_agent, an agent‑scoped upstream credential must have been uploaded for that agent_id (step 7 above with agent_id set), otherwise the call fails closed at gate 3.2.

---

6. Data model reference

Table	Key columns / enums	Migrations
mcp_servers	status (pending|approved|disabled), min_clearance 0–5, url, transport, tool_schema JSONB, oauth_config_json (vault‑ref — no raw tokens), visibility, credential_mode, require_per_agent	128, 162
mcp_server_units (binding)	approval_status (pending|approved|denied), binding_kind (grant|withhold), visibility (inherit_down|private|shared_siblings), credential_mode (team_shared|per_agent), require_per_agent, private_upstream_approved, private_pinned_cidr, requested_by/decided_by/decided_at	126, 130, 143, 164, 168
tenant_egress_allowlist	domain_pattern, reason (incl. mcp_private_upstream_approval), allow_private, pinned_cidr (CHECK: allow_private ⇒ pinned_cidr NOT NULL); enforcement_phase (onboarding_log_only|enforce)	035, 168
mcp_tool_policies	policy_kind (argument_rule|response_filter), cel_expr, filter_spec JSONB, action (deny|escalate), scope unit_id/server_id/agent_id	132
external_agent_clients	client_id → org_id/agent_id/unit_id, public JWK or bcrypt secret	155
governance_approvals	HITL pause rows created on -32001	—
audit (audit_action_type)	mcp_binding_*, mcp_server_approved/_disabled, tool_policy_denied/_escalated, mcp_subtree_withheld (165), mcp_clearance_floor_denied (163)	130, 143, 150, 163, 165

---

7. Governance model quick reference (ADR‑0024, all 9 decisions LIVE)

#	Decision	Where enforced
D2	#288 GovernanceService exits the MCP path; the gateway edge is the sole PEP	dispatch.go (no governance import)
D3	HITL / requires‑approval at the edge → -32001 + governance_approvals (never silent deny)	edge_policy.go
D4a	Org kill‑switch = registry L5 mcp_servers.status='disabled'	SetMCPServerStatus
D4b	Manager subtree‑withhold = binding_kind='withhold', subtract‑after‑union at every resolution surface	resolver.go, pg_group_resolver.go
D5	Guardrails = 3‑tier ToolPolicyService (registry→team→agent), edge‑enforced	edge_policy.go
D6	Registry clearance floor = mcp_servers.min_clearance, enforced at snapshot and edge	initializer.go, edge_policy.go
D7	Per‑agent MCP selection at creation (narrow‑only)	SetAgentTools
D9	Entitlement‑matrix denial reasons (most‑global‑first)	GetTeamEntitlementMatrix

Egress/SSRF is ADR‑0025 (#2019/#2020): always‑deny private IPs, approval‑scoped allow_private+pinned_cidr override, never‑overridable SSRF floor, DNS‑rebind re‑validation.

---

8. Gaps, pending, dev‑only, parked — the honest state

Item	Ref	State
MCP gateway shipped feature‑OFF — MCP_GATEWAY_ENABLED default false, per‑tenant opt‑in AgentGatewayMCPTenants	grpc.go	🟠 dark / ship‑then‑flip
External‑agent + governed‑MCP prod enablement PARKED — needs a real platform signing key + JWKS (the "Decision B reuse" premise was false)	#1864 (OPEN, blocker)	🟠 parked; revisit with GCP migration
FlagMultiParentResolver — the sole prod resolver path; compile‑default TRUE (GA #1981). Kill‑switch: platform_feature_flags["orgv24.multiparent_resolver"]=false → falls back to v2.3 → empty MCP set (governed MCP invisible)	feature_flags.go	🟢 on; ⚠️ do not flip off without knowing this
Dev‑only private‑CIDR path — 172.18.0.0/16 docker bridge + *.mcp.internal dotted alias; prod uses the GKE ClusterIP pinned CIDR	deny.go, seed.go	🔵 dev‑only
Egress enforcement default = onboarding_log_only (not enforce) per tenant	mig 035	🔵 dev‑default
Per‑tool full inputSchema passthrough	dispatch.go	🟡 stubbed (permissive)
ContextForge services/mcp-proxy removed; idempotency folded into Go edge	#1909/#1912	🟠 removed (done)
HTTP‑proxy passthrough mode	—	🟠 deferred

---

9. Key files, flags & beta‑dev specifics

Feature flags / env

• orgv24.multiparent_resolver (FlagMultiParentResolver) — resolver path; default ON, kill‑switch to v2.3.
• MCP_GATEWAY_ENABLED — gateway on/off, default false; AgentGatewayMCPTenants — per‑tenant allowlist.
• PLATFORM_SIGNING_KEY_FILE — Hop‑A signing key path (absent → minting + JWKS off, fail‑safe).

beta‑dev topology (docker‑compose + host Envoy + Cloudflare tunnel; beta-dev.app.upsquad.ai)

• Gateway host port 127.0.0.1:8083; the SPA reaches it via Envoy route /rpc/ → app_rpc_dev (8083).
• Deploy = docker compose -f docker-compose.dev.yml up -d --no-deps --force-recreate context-engine agent-orchestrator after the GHCR image publishes; verify the container image org.opencontainers.image.revision label matches the merge SHA.
• Postgres: docker exec upsquad-postgres psql -U upsquad -d upsquad.
• Signing key: /opt/upsquad-secrets/platform-issuer/platform-signing-key.pem, kid dev-platform-YYYYMMDD.

Source map

• Runtime/session: internal/runtime/session/initializer.go, internal/orgunit/resolver.go
• Hop‑A auth: internal/auth/oauth/{agent_token,issuer_routes,token_endpoint,jwks_validator,signing_key}.go, internal/agent/external_service.go
• Gateway PEP: internal/gateway/mcpproxy/{dispatch,edge_policy,upstream_proxy,pg_group_resolver}.go
• Egress: internal/mcp/egress/{roundtripper,deny,patterns,seed,seed_mcpapproval}.go
• Registry/credential: internal/mcpserver/service.go, internal/mcpserver/credential/{service,scope,resolve}.go
• Client SPA: app/src/pages/mcp-registry.jsx, app/src/components/team-gateway/*.jsx, app/src/data/mcp-clearance-floor.ts

Related ADRs: ADR‑0021/0022 (MCP gateway + trust root), ADR‑0024 (MCP governance realignment), ADR‑0025 (egress/SSRF). Trust‑root blocker: #1864.

---

Doc reflects main as of the #2028/#2029 merges (2026‑07). If you flip MCP_GATEWAY_ENABLED or resolve #1864, update §3 and §8.