Ekho-Labs / infrastructure-tf-kubernetes · open

PR #87 - Bind the reviewed manifest to the executed remote plan

Plans run remotely on HCP Terraform, so the artifact the apply gate hashed was never a plan. It was an 86-byte pointer to a run. The digest could match byte for byte while the run behind it planned something else. This change re-derives the reviewed policy from that pointer at apply time and fails closed unless it matches.

Base
main
Head
ci/bind-reviewed-plan-to-remote-run @ c9961009
Diff
5 files, +512 / -10
CI
run 31924407290 attempt 1 - success
86 B
bytes in the saved tfplan, carrying three keys and no change set.
3
new fail-closed gate steps between artifact download and apply.
8 / 8
executable apply-gate fixtures land on their own step, twice.
10
distinct contract failures the new tests raise against origin/main.
0
existing controls weakened, and no raw plan JSON written to disk.

01What was reviewed and what actually ran

Five files, +512 / -10. One workflow gains three fail-closed steps between artifact download and apply; the checked-in contract, its failure fixtures and a new executable harness lock every one. The finding came from a pre-apply deploy review, not from a production failure.

Bound by this change new

  • Reviewed policy to executed change set. The redacted manifest is re-derived from the saved pointer at apply time through the checked-in plan-review-manifest.jq projection and must equal the dispatched plan_manifest_sha256.
  • Pointer shape and host. Exactly three keys, remote_plan_format == 1, the pinned HCP host, a run reference matching ^run-[A-Za-z0-9]{16}$.
  • Remote run lifecycle. Must resolve to workspace kubernetes-infra, be reported confirmable by HCP, and hold a status in the allowlist planned, cost_estimated, policy_checked.
  • CLI to workspace. TF_VERSION reconciled 1.9.0 → 1.10.2, the version the workspace reports; all four setup-terraform steps resolve that single pin.
  • Reviewability. Both plan-producing jobs publish the remote run_id beside their digests.

Explicitly not asserted said so in the job summary

  • Any approval recorded outside this workflow. The gate reads HCP for run state; it does not treat an out-of-band sign-off as authority to apply.
  • Any workspace change made after this read. The run is confirmable when queried; a later mutation is outside the window this gate can close.
  • The pin regression on origin/main. Pre-existing, from another team's merged PR, left unfixed deliberately - see section 08.
  • Nothing is retained on disk. The projection streams into jq and sha256sum; the contract rejects any raw plan JSON intermediate.

02The artifact the gate was hashing

tf/terraform.tf declares a cloud {} block bound to organization ekho-labs and workspace kubernetes-infra. Plans therefore execute remotely, and terraform plan -out=tfplan does not write a self-contained plan. It writes this, and nothing else.

// tf/tfplan — 86 bytes, run reference redacted {"remote_plan_format":1,"run_id":"run-XXXXXXXXXXXXXXXX","hostname":"app.terraform.io"}

remote_plan_format
Number 1. The only structural marker that this file is a pointer rather than a plan.
run_id
A 20-character HCP run reference. The entire change set lives behind it, on HCP, and never enters this file.
hostname
app.terraform.io. Which HCP instance resolves the reference at apply time.

Shape taken from the pointer the main-push plan run actually produced, not from documentation. The live PR evidence comment now carries a real run_id of the same pattern.

What tfplan_sha256 certified the gap

That these 86 bytes were unaltered. A genuine control, and it is kept. But the digest pins a run reference, so it can match byte for byte while the remote run behind it planned something else entirely.

The reviewed plan-review-manifest.json was a sibling produced at plan time and carried in the same zip. It was compared against its own recorded digest and never re-derived from the remote run at apply time. Nothing tied reviewed policy to the change set that would execute.

86 bytes3 keys0 resource changes

Corroborating signal from the same pass: the workflow pinned TF_VERSION: "1.9.0" while the workspace ran 1.10.2 - a silent CLI/workspace split that only a remote-execution reading explains.

03The apply gate, before and after

Same dispatch, same artifact, same digests. The difference is what has to be true between downloading the plan and running it. Hover or focus a step to trace its path; click to pin, Escape to release.

  • Permitted — asserted before advancing
  • Unbound — advances on a run reference alone
  • Fail closed — no apply

Before — origin/main digest-bound, policy-unbound

Four steps, and not one of them asks what the referenced run actually plans. Every assertion here is about the bytes, the artifact and the branch - never about the change set.

After — PR #87 three fail-closed steps, then apply

Both lanes end at the identical terraform apply of the identical saved pointer. Nothing was added to the apply itself; what changed is which runs can reach it.

04The hardened apply sequence, step by step

Steps 01-03 and 07 are unchanged and stated inline; expand a new step, 04 to 06, for exactly what it asserts and which checked-in fixture fails if someone removes it. The contract also requires this order, requires every new step to begin set -euo pipefail, forbids continue-on-error and || true inside them, and requires apply to be the last step in the job.

  1. 01Bound manual dispatchretained · workflow_dispatch on refs/heads/main, four required inputs, environment: production, concurrency group terraform-production-apply. Unchanged by #87.
  2. 02Source-run provenanceretained · repository, workflow path, event push, ref refs/heads/main, head SHA equal to commit_sha, conclusion success, exactly one unexpired artifact of the exact name. Detail in section 06.
  3. 03Exact artifact set and both digestsretained · exactly tfplan and plan-review-manifest.json as regular files, both matching the dispatched digests. The pointer digest is a real integrity control; it is just not a policy control.

  7. 07Recheck main, then apply the saved planretained · remote refs/heads/main must still equal the dispatched commit, immediately followed by the apply and nothing else. The contract compares this step's active lines to an exact three-line list, locked by mutable-step-after-main-recheck and unbound-second-apply.

05The test that separates what a digest cannot

The apply gate's real shell steps run against synthetic pointers with terraform and curl replaced by fixture shims, so each control is proven by behaviour rather than by a string match on the workflow file. Two fixtures hold the pointer bytes constant and move only the remote change set.

valid accepted

Pointer digest 9be342a86d5e1abfe5aea899409959876e4f9693f3123e3830cce74b364f64cd

The re-derived manifest equals the dispatched plan_manifest_sha256. The run clears pointer, run_state and rebind, and reaches apply.

rc=0failed_step=none

manifest-drift rejected

Pointer digest 9be342a86d5e1abfe5aea899409959876e4f9693f3123e3830cce74b364f64cd

Byte-identical to the accepted case. Behind it the first resource change has become a delete. tfplan_sha256 cannot tell them apart; the re-derived manifest does, and the job exits at rebind.

rc=1failed_step=rebind

Those digests are the harness's synthetic pointer, not a production run. That is the point: the two cases are constructed to be indistinguishable to the old control.

All eight executable apply-gate fixtures

Each is rejected at its own step for its own reason, so one over-broad check cannot be mistaken for five specific ones. Identical on both independent green passes.

FixtureOutcomeStops atRejected because
validappliednoneEvery assertion holds.
extra-keyrejectedpointerKey set does not carry exactly the three reviewed keys.
missing-keyrejectedpointerKey set does not carry exactly the three reviewed keys.
renamed-keyrejectedpointerKey set does not carry exactly the three reviewed keys.
wrong-hostnamerejectedpointerHost is not the expected HCP host.
run-discardedrejectedrun_stateHCP does not report the run confirmable.
run-wrong-workspacerejectedrun_stateRun belongs to a workspace other than the reviewed one.
manifest-driftrejectedrebindRe-derived manifest does not match the reviewed plan_manifest_sha256.

06Controls that were already there and stayed there

No existing gate was weakened to make room for the new ones. Each is still asserted by the same checked-in contract, and each still has a failure fixture that turns red if it is removed.

Speculative plans stay ineligible retained

A PR plan uploads under terraform-speculative-plan-<head sha> on the pull_request event. Apply accepts only a push source run, so a PR artifact can never be applied.

Apply material comes from main retained

The deployable artifact is produced only by a successful push to refs/heads/main, named terraform-plan-<github.sha>, holding exactly the pointer and the redacted manifest.

Four bound dispatch inputs retained

source_run_id, commit_sha, tfplan_sha256, plan_manifest_sha256, all required. The new rebind step consumes the fourth rather than adding a fifth.

Source-run provenance retained

Repository, workflow path, event, ref, head SHA, conclusion and exactly one unexpired artifact are verified before anything is downloaded.

Digest verification retained

Exact regular-file set equality, then both SHA-256 digests. Pointer integrity is still checked - now the first of four assertions instead of the only one.

Main recheck and concurrency retained

Remote main is re-read immediately before apply, and applies serialize in group terraform-production-apply with cancel-in-progress: false. Non-dispatch runs stay cancellable.

Also unchanged: no plan contents, values, outputs or variables are ever published. The redacted manifest carries only format and Terraform versions and, per resource, address, module, mode, type, name, actions, replace paths and sensitivity booleans.

07Evidence

Every control is locked by a test that failed before the change. Red proof was measured against origin/main's workflow at sha256 2cacbef5…; the green suite ran twice, independently.

10
distinct static-contract failures the extended checker reports against origin/main - the missing version pin, the missing host pin, four unrecorded-run-id assertions, and the three absent gate steps.
8 / 8
executable apply-gate fixtures fail against origin/main with apply job is missing bound gate steps: pointer, run_state, rebind.
14 / 15
new negative fixtures have no marker to mutate on origin/main, because the code they would break does not exist there yet.
48
failure fixtures in the Task 3 suite after this change, all parsing and all rejected as intended.
2
independent green passes, identical: contract checker, legacy characterization still correctly failing, 3 runtime source-run fixtures, 8 apply-gate fixtures, the Task 3, 5, 6, 7 and 9 suites, amtool, both temporary-root suites.
0 / 0
terraform fmt and validate findings; actionlint reports 0 on main and 0 on the branch; basedpyright reports 0 errors on both sides.
31924407290
CI run, attempt 1, event pull_request, conclusion success. Security Check, Static Checks and Terraform Plan pass; Main Plan and Apply correctly skipped.
run-…
the live PR evidence comment now carries a real remote_run_id matching ^run-[A-Za-z0-9]{16}$ - independent confirmation that the shape the gate asserts is the shape the workspace emits.

Merge state at review: no conflicts, merged tree 145ec015. All eight apply-gate controls added here hold on that merged tree; its single failure is the pre-existing pin regression described in section 08.

08Residual risk

Severity is the reviewer's calibration of what is still exposed with this change open and unapplied. Two of these are not defects in PR #87 at all; they are conditions it does not remove.

Showing all 6 risks.