KT — CI/CD Supply-Chain Security
Audience: an engineer about to touch a privileged workflow, add a secret, or wonder why a job is shaped oddly. Goal: understand which controls actually hold, which only appear to, and why the answer is counterintuitive.
Revision: verified against upsquad-core 31069fb1, upsquad-client 4549a69e, upsquad-admin cfb32eff — all 2026-08-27.
The configuration survey ran at 3ad06271; scripts/check-privileged-job-isolation.py landed 8.4 hours later in 31069fb1 (#2613). §7 is verified at 31069fb1 — the script does not exist at 3ad06271, so anything claiming otherwise was checked at the wrong revision.
Status legend
🟢 LIVE · 🟡 PARTIAL · 🔵 DEV-ONLY · 🟠 PENDING, PARKED or NOT ENFORCED
0. TL;DR — the one thing to take away
A control that lives in a file can be edited away by anyone who can push a branch — and git history makes that permanent. A control in repo settings cannot be, by that actor.
The qualifier is load-bearing. Settings are not immune — they are immune to workflows: write. An actor holding administration:write, environments:write or secrets:write can edit the settings-level controls directly, and devops-engineer holds all three (measured 2026-08-27; §8 tabulates the first and third). The asymmetry this document rests on is between permissions, not between mechanisms: the pattern in §4 defeats a branch-pusher, not a repo-admin.
This is not obvious, and it cost four attempts to learn. The remediation of the founding incident walked four layers, and each of the first three was defeated by the layer below it:
| # | Control | Why it was insufficient |
|---|---|---|
| 1 | Revoke workflows: write | The payload was an ordinary repo file, not a workflow file. Reach needed only contents: write — held by all seven Apps |
| 2 | Pin the checkout (ref:) | Not retroactive. create runs the definition from the created ref; 326 of 342 branches still carried the unpinned file |
| 3 | Move to workflow_run | Edits main's definition only. Cannot reach trees that already exist |
| 4 | Scope the secret in repo settings | This closed it. Settings are not in any tree, so history cannot route around them |
The founding incident: board-sync.yml on main checked out the PR merge ref and ran scripts/board-sync.py after minting the project-manager App private key. Nothing in .github/workflows/** needed modifying — the payload was an ordinary repository file.
1. Architecture & control flow
Walkthrough. 1 The trigger decides which definition runs — from the branch, or from base. 2 The deployment-branch policy is evaluated on github.ref and hard-fails if it does not match; 2a a branch-side github.ref is refused outright. 3 An admitted job proceeds. 4 The checkout pin decides which tree executes — a separate question from which definition started. 5 The environment releases the secret.
The two decisions at 2 and 4 are independent, and conflating them is the single most common error here. See §5.
2. What workflows: write actually enables 🟢
It is a push permission, not a merge permission. Branch protection on main is untouched by it: 1 approving review, dismiss_stale_reviews, enforce_admins, strict — all true in all three repos.
(Precisely: strict is true in core and client. upsquad-admin has no required_status_checks key at all — zero required contexts, so nothing gates a merge there. See §10.)
That reads reassuring, and it is incomplete in a way that reverses the conclusion:
Branch protection gates merge. It does not gate execution.
A workflow on a pushed branch runs from its own definition, on pull_request, before any review, with same-repo secrets in scope. Nothing in the required-review machinery runs before a workflow does.
Demonstrated, not argued. PR #2585 added .github/workflows/migration-ceiling.yml — a workflow absent from main — and it executed on that PR seven times before review or merge. The guard PR that started this thread is itself the proof.
3. Why file-level fixes cannot be retroactive 🟢
Git history is immutable. Any commit that has ever existed can be pushed as a new ref, and create runs the definition from that ref.
Proven live, 10.7 days after the pin merged: run 32942869294 — event=create, conclusion=success, head_sha=0d51e9ba (main immediately pre-pin), branch evidence-2603-stale-tree, 2026-08-26T07:27Z. The log shows persist-credentials: true, no ref: line, app-id: 3281205 minted, and board-sync.py run from that tree.
Full census at the time: 326 of 342 branches (95.3%) carried create: + secrets.PJM + an unpinned checkout, from only five distinct blobs.
The trees are in history. A fix that edits main protects refs whose tree contains the fix, and nothing else — including refs that do not exist yet but could be created from any historical commit.
4. The pattern that worked: environment secret + deployment-branch policy 🟢
Two properties, and both are required:
- An environment secret is invisible to a job that does not declare
environment:. All 326 stale copies declare none, sosecrets.PJMresolves to the empty string there;actions/create-github-app-tokenfails and no token is minted. - The deployment-branch policy lives in repo settings, not in a workflow file. Without this half, a branch simply adds
environment: board-syncto its own copy and the secret comes back.
Property 2 is the one that makes this the first control in this repo that survives workflows: write.
Current configuration 🟢
| repo | environment | protection rules | branch policy | secret |
|---|---|---|---|---|
| core | board-sync | [branch_policy] | exactly main | PJM |
| client | board-sync | [branch_policy] | main | PJM |
| admin | board-sync | [branch_policy] | main | PJM |
Declared by four privileged jobs: core board-sync.yml:131 and ready-to-pick-guard.yml:71, client board-sync.yml:103, admin board-sync.yml:115.
No approval prompt is ever raised — all three environments carry one branch_policy rule and no required_reviewers, and the declaring jobs trigger only on issues / workflow_run / pull_request_target, all of which report github.ref = refs/heads/main.
5. The invariants this rests on 🟠
None of these is enforced mechanically yet (tracked as #2626). All are currently true and were measured.
5.1 allow_forking = false
Verified: all three repos private: true, forks: 0, allow_forking: false.
pull_request_target grants secrets to fork PRs; pull_request does not. That is the one property traded away in §6 — and it costs nothing only while forking is off. Enabling forking is a founder decision with this section as its blast radius.
5.2 The branch policy is NOT a backstop for the checkout pin
This is easy to get backwards. Measured: under pull_request_target, github.ref reports refs/heads/main regardless of what is checked out. So the policy will happily release the secret to a job that someone later modifies to check out a PR head.
github.refconstrains which definition may start. It never constrains what that definition then does.
The ref: pin is the sole control on the executing tree. Removing it is worse under pull_request_target than under pull_request — which is precisely what makes that trigger dangerous in the wild.
5.3 Re-scoping an org secret is silent
The org-level PJM was scoped to zero repositories, not deleted (2026-08-27T16:27Z, founder decision). Functionally equivalent — an org secret with Selected repositories and none selected is readable by nothing.
Two differences that matter:
- Adding a repository back re-opens the hole with no signal. Deletion would fail loudly.
- The value left in it is the OLD key. It was never updated to the rotated one, so a future re-scope reintroduces the possibly-exposed credential rather than the current one.
6. pull_request_target — safer here, footgun in general 🟢
The two portals use it. That was forced, not preferred: the deployment-branch policy is evaluated on github.ref, which under pull_request is refs/pull/N/merge and is hard-refused — the job fails in 0 steps, ~2s.
Why it is safer than what it replaced. Under pull_request the definition comes from the merge ref, so a PR can delete the ref: pin and persist-credentials: false from its own copy and run that. The pin was self-defeating. Under pull_request_target the definition comes from base, so removing the pin requires a merged, reviewed commit.
The residual, and it is real. An attacker controls the event payload, and it now arrives alongside secrets. So what the job does with github.event.* becomes security-relevant.
Payload trace for board-sync.py (traced end-to-end): EVENT_PAYLOAD: ${{ toJSON(github.event) }} passed via env:, never spliced into a run:. Imports json, os, re, sys, urllib.request only — no subprocess, no eval, no shell=True. handle_pr_event reads exactly four fields — pr.merged, pr.draft, pr.number, pr.base.repo.full_name — all GitHub-set. title, body, head.*, user.login and labels are never read. GraphQL uses bound variables.
One pre-existing primitive: closingIssuesReferences derives from the PR body, so any author can drive an arbitrary issue's board Status to Review. Done requires merged: true, which is unforgeable.
pull_request_targetPin the checkout, set persist-credentials: false, narrow permissions:, and trace every github.event.* field the job reads. The combination that causes real-world compromises is pull_request_target plus a checkout of the PR head — which the pin exists to prevent.
7. The isolation guard, and the guard that recommended its own bypass 🟡
scripts/check-privileged-job-isolation.py (#2613, merged) walks every workflow and flags privileged jobs that are branch-reachable or execute an unpinned tree. Unconditional lane, verdict wired to the exit code, 49-case suite.
It shipped with two holes, both found by executing fixtures rather than reading:
Hole 1 — it recommended its own bypass. job.get("environment") was truthy, so any name discharged R1 — and the violation message told authors to "gate the job with environment:". An author who tripped the guard was steered directly into a no-op fix: add the line, go green, protect nothing, because an environment that does not exist has no branch policy.
Closed by testing membership of SETTINGS_BACKED_ENVIRONMENTS = {"board-sync"}, rewriting the message to lead with "FIX IT BY REMOVING THE REACHABILITY, not by relabelling", and adding --verify-environments to re-derive the set live.
A pre-existing "clean" test case had encoded the hole — it asserted that
environment: prodplus a pin was fine. The suite was defending the blind spot.
Hole 2 — spelling-dependence. CHECKOUT_RE matched only uses: actions/checkout@, so run: git fetch … && git checkout FETCH_HEAD && npm ci was invisible — the exact shape of the sibling instance in upsquad-client. Closed with a second arm over parsed run: scalars.
The auto-created-environment class
Six environments across the three repos have protection_rules: [] — upsquad-eng-docs in core, and five Cloudflare Pages environments in client. All are integration-created. None holds a secret and none is declared by any workflow, so nothing is exposed today — but each is a name that would discharge a truthiness check.
Correction on the record: upsquad-eng-docs was described during review as a live bypass. It was latent — verified: no workflow declares it, zero secrets. The fix was right; the severity was inflated in the retelling.
8. Agent App permission map 🟢
Measured 2026-08-27 via scripts/show-app-permissions.py.
| App | workflows:w | actions:w | secrets:w | administration:w |
|---|---|---|---|---|
| product-manager | — | — | — | — |
| project-manager | — | — | — | — |
| principal-architect | — | — | — | — |
| backend-sme | ✓ | ✓ | — | — |
| frontend-sme | — | ✓ | — | — |
| qa-engineer | — | ✓ | — | — |
| devops-engineer | ✓ | ✓ | ✓ | ✓ |
All seven hold contents:write — which is the permission the founding incident actually required.
workflows: write was revoked from frontend-sme and project-manager on 2026-08-15 (both had zero recorded uses; backend-sme had 15, devops-engineer 8).
🟠 administration:write on devops-engineer can edit branch protection — the control every other guarantee rests on. An identity that can edit the gate is not gated by it. Unreviewed inherited default; tracked as #2612.
9. Operator checklist
Adding a secret to a workflow
| Step | Why |
|---|---|
| 1. Does the job need to be branch-reachable at all? | If not, use issues / workflow_run / schedule and stop here |
| 2. Put the secret in an environment, not org or repo scope | Invisible to jobs without environment:, so stale trees cannot reach it |
| 3. Set the environment's deployment branch policy to the default branch | This is the half that survives workflows: write |
4. Pin the checkout: ref: ${{ github.event.repository.default_branch }} | The policy does not constrain the tree — §5.2 |
5. persist-credentials: false | Stops the git credential outliving the checkout step |
6. Narrow permissions: to the minimum | pull_request_target defaults to write |
7. Trace every github.event.* field the job reads | Under pull_request_target the payload is attacker-controlled |
Touching an existing privileged workflow
Run the guard before you push: python3 scripts/check-privileged-job-isolation.py. It is unconditional in core CI but not a required check — a red does not block your merge.
10. Honest state — deliberately not done 🟠
| Item | State |
|---|---|
Old project-manager App private key | Retained as rollback until 2026-09-09. Accepted low-probability risk: if it was exfiltrated during the window it remains usable |
Org PJM secret | Scoped to zero repos, not deleted. Silent re-scope risk; stored value is the old key |
Privileged Job Isolation lane | Not a required check (#2628) |
| Guard coverage | Core only. Client has 4 real violations never checked (client#809); admin has none installed |
| Guard header scope | Core-scoped and core-correct, but names one auto-created no-op environment where six exist org-wide. §7 states the class properly; the header has not caught up (#2629) |
| Nightly environment re-verification | Not wired — needs environments:read |
| App permission scope review | Not started (#2612) |
upsquad-admin required checks | Zero. Everything reports, nothing gates |
Forensics behind the rotation decision
Exposure window 2026-04-17 → 2026-08-15 ≈ 120 days; Actions retention floor 2026-07-15 ⇒ ~89 days unauditable. Within the retained window: 4000 Board Sync runs, 940 on branch-controlled triggers, 528 distinct head SHAs, and 6 off-main script executions — all the team's own deliberate #2603 markers. No evidence of abuse. The rotation was approved on the unauditable window, not on observed compromise.
Already-minted installation tokens remain valid for ~60 minutes. The retirement is not atomic.
11. Source map
| Concern | Path |
|---|---|
| The isolation guard | scripts/check-privileged-job-isolation.py |
| Its lane | .github/workflows/privileged-job-isolation.yml |
| Privileged jobs (core) | .github/workflows/board-sync.yml, ready-to-pick-guard.yml |
| The inert relay | .github/workflows/board-sync-trigger.yml (permissions: {}) |
| Payload consumer | scripts/board-sync.py |
| App permission measurement | scripts/show-app-permissions.py |
| Migration-ceiling guard | scripts/check-migration-ceiling.py |
12. Related
- Workflow Orchestration · Workflow Runtime v2
- Issues: #2603 (the P1) · #2624 (the pattern) · #2626 (invariants) · #2612 (permission scope) · #2592 (
workflows: write) - Sibling repos: client#809 · client#807 · admin#12
Reflects upsquad-core 31069fb1 (§7) / 3ad06271 (configuration survey), upsquad-client 4549a69e, upsquad-admin cfb32eff, all 2026-08-27. Invalidated by: enabling forking on any repo; adding a repository to the org PJM scope; adding an environment to SETTINGS_BACKED_ENVIRONMENTS; changing branch protection. Every status claim here was measured, not inferred — re-measure before relying on it.