Beta — Real embeddings via Vertex AI (GCP-native)

Issue #1403. Founder decision (2026-06-15): run real embeddings on beta using GCP-native Vertex AI (not OpenAI). The OpenAI path (#1405) was superseded/closed — GCP can't mint an OpenAI key, whereas Vertex is provisionable end-to-end through GCP (service account + ADC / Workload Identity), with no third-party key to paste/rotate and no egress to api.openai.com.

Split: backend ships the VertexEmbedder (EMBEDDING_PROVIDER=vertex case in cmd/context-engine); devops (this runbook) provisions GCP, wires beta compose, re-embeds, and validates.

Decisions of record

Decision	Choice	Why
Provider	Vertex AI (`EMBEDDING_PROVIDER=vertex`)	GCP-native, no third-party key
Project	`upsquad-geap-eval`	The eval/beta GCP project (billing enabled). Workload Identity moves to the cloud spoke at #1274.
Location	`us-central1`	Matches `gcloud config region`; Vertex embeddings GA there
Model	`gemini-embedding-001`	Native 3072-d but supports `outputDimensionality=1536` → matches the `vector(1536)` schema with no migration. `text-embedding-005` is native 768-d and cannot reach 1536, so it is not usable here.
Output dim	`1536`	Matches `context_embeddings.embedding vector(1536)`
`model_version`	`vertex-gemini-embedding-001-1536` (backend-owned string)	Distinct from `dev-hash-embedding-v1` and any prior vectors so spaces never mix
Auth (beta/local)	ADC via a mounted SA key, `GOOGLE_APPLICATION_CREDENTIALS`	Beta is the LOCAL containerized devbox stack — not on GCP yet
Auth (cloud)	Workload Identity (#1274) — no key file	Replaces the mounted key once on the GCP spoke

GCP resources provisioned (already done, 2026-06-15)

All self-serve via gcloud on the devbox (operator account vaisakh@upsquad.ai).

API enabled: aiplatform.googleapis.com on upsquad-geap-eval (was already on).
Service account: context-engine-vertex@upsquad-geap-eval.iam.gserviceaccount.com with project role roles/aiplatform.user (least privilege for the embeddings predict endpoint).
SA key (ADC): JSON key generated and stored outside the repo at /opt/upsquad-secrets/vertex/context-engine-vertex-sa.json on the beta devbox — mode 0600, owned by uid:gid 65532:65532 (the context-engine container's distroless nonroot user, so the read-only bind mount is readable in-container). The key is never committed, never in .env, never in compose, and never echoed to logs.

Re-create the SA + key from scratch if ever needed:

PROJECT=upsquad-geap-eval
gcloud services enable aiplatform.googleapis.com --project=$PROJECT
gcloud iam service-accounts create context-engine-vertex --project=$PROJECT \
  --display-name="Context Engine Vertex AI embeddings (beta, #1403)"
gcloud projects add-iam-policy-binding $PROJECT \
  --member="serviceAccount:context-engine-vertex@$PROJECT.iam.gserviceaccount.com" \
  --role="roles/aiplatform.user" --condition=None
sudo install -d -m 0750 -o vb -g vb /opt/upsquad-secrets/vertex
KEYFILE=/opt/upsquad-secrets/vertex/context-engine-vertex-sa.json
( umask 0177; gcloud iam service-accounts keys create "$KEYFILE" \
    --iam-account=context-engine-vertex@$PROJECT.iam.gserviceaccount.com --project=$PROJECT )
sudo chown 65532:65532 "$KEYFILE" && sudo chmod 0600 "$KEYFILE"   # never cat this file

Verify the SA + model work (no secret printed)

export CLOUDSDK_CONFIG=$(mktemp -d)
sudo cat /opt/upsquad-secrets/vertex/context-engine-vertex-sa.json > "$CLOUDSDK_CONFIG/key.json"
chmod 0600 "$CLOUDSDK_CONFIG/key.json"
gcloud auth activate-service-account --key-file="$CLOUDSDK_CONFIG/key.json" --project=upsquad-geap-eval
TOKEN=$(gcloud auth print-access-token)
curl -s -X POST \
  "https://us-central1-aiplatform.googleapis.com/v1/projects/upsquad-geap-eval/locations/us-central1/publishers/google/models/gemini-embedding-001:predict" \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"instances":[{"task_type":"RETRIEVAL_DOCUMENT","content":"hello"}],"parameters":{"outputDimensionality":1536}}' \
  | python3 -c "import sys,json;print('dims:',len(json.load(sys.stdin)['predictions'][0]['embeddings']['values']))"
rm -rf "$CLOUDSDK_CONFIG"   # expect: dims: 1536

Confirmed 2026-06-15: HTTP 200, dims: 1536.

Beta compose wiring (`docker-compose.dev.yml`, context-engine service)

EMBEDDING_PROVIDER=vertex (was dev); removed the fake OPENAI_API_KEY.
VERTEX_PROJECT=upsquad-geap-eval, VERTEX_LOCATION=us-central1, VERTEX_EMBEDDING_MODEL=gemini-embedding-001 (all ${VAR:-default} so they can be overridden via repo-root .env).
GOOGLE_APPLICATION_CREDENTIALS=/var/secrets/vertex/sa.json with a read-only bind mount of the host key at /opt/upsquad-secrets/vertex/....
ENVIRONMENT=development is unchanged (auth-bypass + the app/ SPA depend on it; the vertex provider is allowed in any env — only dev is env-gated).

Egress: deployments/context-engine/base/networkpolicy.yaml now documents Vertex AI (aiplatform.googleapis.com / *.googleapis.com) as the embeddings egress target and drops the OpenAI reference. (Local compose has no egress firewall; the NetworkPolicy governs the future GCP spoke.)

Deploy — GATED + BATCHED (do not deploy ad hoc)

This must NOT deploy standalone. It is gated and batched per #1403:

#1401 green — the Org/Pillar/Team/RAG E2E gate (golden-flow) passing.
Batched with #1402 (DB-role hardening) — one coordinated beta redeploy.
Backend VertexEmbedder merged AND ghcr.io/upsquad-ai/upsquad-context-engine:latest republished with the vertex case. The old image rejects EMBEDDING_PROVIDER=vertex at config-validate and crash-loops — so this compose change must reach main only when the vertex-aware image is live and the SA key is on the box. (The hands-off dev-deploy.yml reconcile does git reset --hard origin/main → docker compose up -d; a mis-ordered merge would crash-loop context-engine on the next reconcile.)

The compose-service-def change (new env + volume) alters the service hash, so the reconcile recreates context-engine exactly once and picks up the mount + config.

Re-embed (purge dev-hash, re-ingest with Vertex)

Chunks embedded by the offline dev hasher (model_version=dev-hash-embedding-v1) live in a different vector space and are incompatible with Vertex queries. Beta has little data, so re-ingest rather than a backfill worker.

# 1) Purge stale dev-hash embeddings (chunks/sources are kept; only vectors drop)
docker exec -i upsquad-postgres psql -U upsquad -d upsquad -c \
  "DELETE FROM context_embeddings WHERE model_version = 'dev-hash-embedding-v1';"

# 2) Re-ingest each beta knowledge source via KnowledgeService.IngestDocument
#    (re-chunk + re-embed with Vertex). Auth-bypass headers as below.

If a future beta accumulates real corpora, add a one-shot re-embed job that re-runs the embedding pipeline over rag_chunks lacking a current-model row (NOT EXISTS (SELECT 1 FROM context_embeddings WHERE chunk_id=... AND model_version='vertex-...')). Not needed yet.

Validate (run at the batched deploy, after the key + image are in place)

Connect HTTP on http://localhost:8083 (host → container 8080). Auth-bypass is on (ENVIRONMENT=development + DISABLE_AUTH/ICS_MODE), so stamp the scope headers the SPA transport uses.

ORG=acme-corp-test            # seeded dev tenant
H='-H Content-Type:application/json -H X-Org-Id:'"$ORG"' -H X-Clearance:5'

# A) confirm the embedder selected — logs should show the Vertex model, not dev hash
docker logs upsquad-context-engine 2>&1 | grep -i "embedd" | tail
#   expect: vertex model gemini-embedding-001 (model_version vertex-gemini-embedding-001-1536)
#   NOT:    "dev deterministic offline hash embedder"

# B) ingest sample content (needs an ingestable source_id from KnowledgeService)
curl -s $H http://localhost:8083/upsquad.knowledge.v1.KnowledgeService/IngestDocument \
  -d '{"source_id":"<SOURCE_UUID>","filename":"smoke.md","content_type":"md",
       "content":"'"$(printf 'Our refund policy lets customers return items within 30 days for a full reimbursement.' | base64 -w0)"'"}'

# C) exact-term query returns ranked results
curl -s $H http://localhost:8083/upsquad.context.v1.ContextEngine/Search \
  -d '{"query":"refund policy","top_k":5,"min_score":0.3}' | python3 -m json.tool

# D) SEMANTIC query — synonyms/paraphrase, no shared keywords with the source —
#    must still rank the refund chunk highly. This is the real-embeddings proof
#    (the dev lexical hasher would miss it).
curl -s $H http://localhost:8083/upsquad.context.v1.ContextEngine/Search \
  -d '{"query":"how do I get my money back after buying something","top_k":5,"min_score":0.3}' \
  | python3 -m json.tool

Capture B/C/D output as the deploy evidence on #1403.

Rollback (< 5 min)

The provider is a single env var, so rollback is a config flip + one reconcile:

# Fastest: pin the provider back to the offline dev hasher on the beta box
echo 'EMBEDDING_PROVIDER=dev' >> /opt/upsquad/upsquad-core/.env   # ENVIRONMENT=development permits it
docker compose -f docker-compose.dev.yml up -d context-engine     # recreates once
docker exec -i upsquad-postgres psql -U upsquad -d upsquad -c \
  "DELETE FROM context_embeddings WHERE model_version LIKE 'vertex-%';"   # optional: clear vertex vectors

Full rollback is reverting the compose/egress PR (image pin returns to the prior :latest). No DB migration is involved (dim stays 1536), so there is nothing to downgrade. To revoke access entirely: disable/delete the SA key (gcloud iam service-accounts keys delete) — the embedder then fails ADC and the context-engine degrades to recency-only.

Decisions of record​

GCP resources provisioned (already done, 2026-06-15)​

Verify the SA + model work (no secret printed)​

Beta compose wiring (docker-compose.dev.yml, context-engine service)​

Deploy — GATED + BATCHED (do not deploy ad hoc)​

Re-embed (purge dev-hash, re-ingest with Vertex)​

Validate (run at the batched deploy, after the key + image are in place)​

Rollback (< 5 min)​