Web URL connector — egress hardening (SSRF defence-in-depth)

The Web URL RAG connector (internal/knowledge, issue #1375) fetches tenant-supplied URLs server-side from the context-engine pod. A tenant-controlled fetch target is a classic SSRF lever, so the connector enforces application-layer egress controls and the deploy MUST add a Kubernetes egress NetworkPolicy as a second, independent layer.

Layer 1 — application (in webfetch.go, always on)

The SafeFetcher enforces, on the initial request and on every redirect hop:

• scheme allowlist — https only (ErrSchemeNotAllowed);
• DNS-pinning — the host is resolved in-process, every candidate IP is screened, and the connection is dialled to the exact vetted IP so net/http never re-resolves (TOCTOU / DNS-rebinding safe);
• IP block-list (ErrBlockedIP) — loopback, link-local (incl. the 169.254.169.254 cloud-metadata address), RFC-1918 private ranges, CGNAT (100.64.0.0/10), IPv6 ULA (fc00::/7), unspecified, and multicast — in IPv4, IPv6, and IPv4-mapped-IPv6 forms;
• redirect cap (ErrTooManyRedirects), whole-request + dial timeouts, and a max-body cap (ErrBodyTooLarge).

Proven by webfetch_test.go (metadata-IP, private-IP, DNS-rebind, redirect→ metadata, and non-https are all refused).

Layer 2 — Kubernetes egress NetworkPolicy (deploy MUST add)

Application controls can be bypassed by a future code path or a dependency, so the pod itself must be denied network access to the cluster-internal and metadata ranges. Apply an egress NetworkPolicy to the context-engine workload that:

1. denies egress to the GKE/EC2 metadata address 169.254.169.254/32;
2. denies egress to RFC-1918 ranges (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) and link-local 169.254.0.0/16, except the specific in-cluster service CIDRs the pod legitimately needs (Postgres, Redis, the OTel collector, kube-dns);
3. allows egress to 0.0.0.0/0 on tcp/443 for the public-internet fetch, with the private ranges above carved out via except: blocks.

Example (illustrative; reconcile CIDRs with the live cluster before applying):

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: context-engine-egress
spec:
  podSelector:
    matchLabels: { app: context-engine }
  policyTypes: [Egress]
  egress:
    # DNS to kube-dns only.
    - to: [{ namespaceSelector: {}, podSelector: { matchLabels: { k8s-app: kube-dns } } }]
      ports: [{ protocol: UDP, port: 53 }, { protocol: TCP, port: 53 }]
    # Postgres / Redis / OTel — pin to their service CIDRs (fill in).
    - to: [{ ipBlock: { cidr: <DATA_TIER_CIDR> } }]
    # Public internet for the web connector, with private + metadata carved out.
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
            except:
              - 169.254.169.254/32
              - 169.254.0.0/16
              - 10.0.0.0/8
              - 172.16.0.0/12
              - 192.168.0.0/16
      ports: [{ protocol: TCP, port: 443 }]

Until this policy is in place the application layer is the only guard — it is strong (and tested) but should not be the sole control for an internet-fetching workload.