fix: bake openssl and jq into the runner image, and assert them

The release tag gate parses the Control Plane quorum verdict and verifies its
Ed25519 DSSE signature by shelling out to jq and openssl. Neither is in this
image, and both call sites read cmd ... || { echo <security message>; exit 1; },
so command-not-found took the failure branch: the gate reported a valid, signed
approval first as "not a signed DSSE envelope" and then as "signature FAILED
verification". That blocked v0.0.134 and v0.0.135 on 2026-08-31 and pointed the
investigation at the Tag Request rather than at this image — the same shape as
the git-daemon gap already documented above the package list.

Also adds both to the toolchain assertion. That check already existed and still
missed them, because it only covers what someone thought to list; the comment
now says so, and says to extend it whenever a gate grows a shell dependency.

Verified by building this image and running the gate script inside it against
the live Control Plane: openssl 3.5.8, jq 1.8.1, and the v0.0.135 verdict
verifies and binds to its commit. The unpatched image fails the same script.

.github/workflows/build-image.yml

@@ -0,0 +1,147 @@
+name: build image
+
+# Publishes the runner toolchain image to ghcr.io.
+#
+# Why GitHub rather than our own Gitea Actions: the runners are host mode
+# (`ubuntu-latest:host`, no job containers) and their container drops ALL
+# capabilities with allowPrivilegeEscalation:false. Rootless buildah needs
+# newuidmap/user namespaces for its RUN steps, so building images there would
+# mean weakening the posture that hardening established. GitHub runners have
+# docker + buildx + QEMU already.
+#
+# Why ghcr.io rather than the Gitea registry: gitea.langenoja.se sits behind
+# Cloudflare, whose request-body cap returns 413 on a registry layer. The
+# cluster already pulls zegit's own images from ghcr.io and the runner
+# deployments already carry a github-registry-credentials pull secret, so this
+# reuses an established, trusted path instead of inventing one.
+
+on:
+  workflow_dispatch:
+    inputs:
+      tag:
+        description: 'Image tag, e.g. 12.11.1-ci4 (upstream runner version + ci revision)'
+        required: true
+        type: string
+  push:
+    branches: [master]
+    paths:
+      - 'Dockerfile'
+      - '.github/workflows/build-image.yml'
+
+permissions:
+  contents: read
+  packages: write
+
+env:
+  IMAGE: ghcr.io/${{ github.repository }}
+
+jobs:
+  build:
+    # GitHub-only: the Gitea mirror has no ghcr.io and no GITHUB_TOKEN.
+    if: ${{ github.server_url == 'https://github.com' }}
+    runs-on: ubuntu-latest
+    timeout-minutes: 45
+    steps:
+      - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
+
+      # arm64 is not optional: forgejo-runner runs on arm64 nodes and
+      # zegit-runner on amd64, neither deployment sets a nodeSelector, and both
+      # use this one tag.
+      - uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3
+      - uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
+
+      - uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
+        with:
+          registry: ghcr.io
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+
+      # A master push VALIDATES the Dockerfile still builds on both arches but
+      # publishes nothing — only an explicit dispatch names a tag and pushes.
+      # Guessing a tag from a branch push would silently mint versions.
+      - id: meta
+        run: |
+          if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
+            echo "push=true" >> "$GITHUB_OUTPUT"
+            echo "tags=${IMAGE}:${{ inputs.tag }}" >> "$GITHUB_OUTPUT"
+          else
+            echo "push=false" >> "$GITHUB_OUTPUT"
+            echo "tags=${IMAGE}:ci-validation" >> "$GITHUB_OUTPUT"
+          fi
+
+      - uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8 # v6
+        with:
+          context: .
+          platforms: linux/amd64,linux/arm64
+          push: ${{ steps.meta.outputs.push }}
+          tags: ${{ steps.meta.outputs.tags }}
+          provenance: false
+          sbom: false
+
+      # Read the published index back rather than trusting the push's exit
+      # code, and fail if either architecture is absent — a single-arch image
+      # under this tag would CrashLoop on half the nodes.
+      - name: Verify both architectures were published
+        if: ${{ steps.meta.outputs.push == 'true' }}
+        run: |
+          set -euo pipefail
+          out=$(docker buildx imagetools inspect "${IMAGE}:${{ inputs.tag }}")
+          echo "$out"
+          for want in amd64 arm64; do
+            echo "$out" | grep -q "linux/$want" || { echo "::error::published image is missing linux/$want"; exit 1; }
+          done
+          echo "✓ ${IMAGE}:${{ inputs.tag }} carries linux/amd64 and linux/arm64"
+
+      # The image is a TOOLCHAIN, so "it built" is not the property that matters
+      # — "it still contains what the gates shell out to" is. This exists because
+      # a hand-built image had drifted from this Dockerfile: the deployed ci3
+      # carried gcc, musl-dev, python3 and py3-yaml, the Dockerfile did not, and
+      # rebuilding from the Dockerfile silently dropped them. CI runs then failed
+      # at "credentials-coverage: need python3" instead of running the tests.
+      #
+      # Asserted on BOTH architectures, because a package that resolves on amd64
+      # and not on arm64 would otherwise reach half the nodes unnoticed.
+      #
+      # openssl and jq were added after this check already existed and still
+      # missed them: the release tag gate shells out to both to verify the
+      # Control Plane's signed quorum verdict, and their absence surfaced as
+      # "verdict not signed" / "signature FAILED verification" rather than as a
+      # missing tool, blocking v0.0.134 and v0.0.135. The list only covers what
+      # someone remembered to add, so add to it whenever a gate grows a new
+      # shell dependency.
+      - name: Verify the image still contains the toolchain the gates need
+        if: ${{ steps.meta.outputs.push == 'true' }}
+        run: |
+          set -euo pipefail
+          REF="${IMAGE}:${{ inputs.tag }}"
+          fail=0
+          for plat in linux/amd64 linux/arm64; do
+            echo "── $plat"
+            out=$(docker run --rm --platform "$plat" --entrypoint sh "$REF" -c '
+              set -u
+              py=$(python3 -c "import yaml,sys;print(sys.version.split()[0])" 2>/dev/null) \
+                && echo "python3+yaml=$py" || echo "python3+yaml=MISSING"
+              command -v gcc      >/dev/null && echo "gcc=ok"      || echo "gcc=MISSING"
+              command -v node     >/dev/null && echo "node=ok"     || echo "node=MISSING"
+              command -v bash     >/dev/null && echo "bash=ok"     || echo "bash=MISSING"
+              command -v make     >/dev/null && echo "make=ok"     || echo "make=MISSING"
+              command -v curl     >/dev/null && echo "curl=ok"     || echo "curl=MISSING"
+              command -v tar      >/dev/null && echo "tar=ok"      || echo "tar=MISSING"
+              command -v openssl  >/dev/null && echo "openssl=ok"  || echo "openssl=MISSING"
+              command -v jq       >/dev/null && echo "jq=ok"       || echo "jq=MISSING"
+              [ -x "$(git --exec-path)/git-http-backend" ] \
+                && echo "git-http-backend=ok" || echo "git-http-backend=MISSING"
+            ' 2>&1) || { echo "::error::could not run $REF on $plat"; fail=1; continue; }
+            echo "$out"
+            if echo "$out" | grep -q MISSING; then
+              echo "$out" | grep MISSING | while read -r l; do
+                echo "::error::$plat is missing a required tool: $l"
+              done
+              fail=1
+            fi
+          done
+          [ "$fail" = 0 ] || {
+            echo "::error::The published image lost a tool the repo's gates depend on. Restore it in the Dockerfile — do not weaken this check."
+            exit 1
+          }
+          echo "✓ toolchain intact on both architectures"

Dockerfile

@@ -6,15 +6,20 @@
 # The runner pod is non-root with all caps dropped, so it cannot apt/apk-add at
 # runtime -- the tools must be baked in here.
 #
-# Build multi-arch (cluster is mixed amd64/arm64) and push to the Gitea registry:
+# PUBLISHING IS AUTOMATED. Run the "build image" workflow on GitHub with a tag
+# (e.g. 12.11.1-ci4); it builds both architectures and pushes to
+# ghcr.io/zegit-zoo/zegit-forgejo-runner. ./push-image.sh remains for building
+# by hand. README.md explains why the registry moved off Gitea (Cloudflare's
+# request-body cap returns 413 on a registry layer) and why the build cannot run
+# on our own runners (host mode, all capabilities dropped).
 #
-#   docker buildx create --use --name zegit-ci 2>/dev/null || true
-#   docker login gitea.langenoja.se        # username + a Gitea personal access token
-#   docker buildx build --platform linux/amd64,linux/arm64 \
-#     -t gitea.langenoja.se/zegit-zoo/zegit-forgejo-runner:12.11.1-ci --push .
+# Then in flux-config set the image on BOTH deployment-zegit-runner and
+# deployment-forgejo-runner, digest-pinned.
 #
-# Then in flux-config set both containers' image: to the tag above and commit.
-# Package visibility is PUBLIC, so the runner namespace pulls without a secret.
+# The image is NOT public. Pulls need a registry credential in
+# forgejo-runner-system -- github-registry-credentials for ghcr.io. An earlier
+# version of this comment claimed the package was public and needed no secret;
+# that was wrong for the Gitea registry too, which is private.
 FROM code.forgejo.org/forgejo/runner:12.11.1
 
 USER root
@@ -33,6 +38,25 @@ USER root
 # curl,tar -> setup-go / gitleaks download + extract their own arch-specific binaries
 # make     -> zegit Makefile targets
 # ca-certificates -> trusted roots for HTTPS to gitea/github/goproxy
+# openssl  -> the release tag gate verifies the Control Plane's SIGNED quorum
+#             verdict (Ed25519 DSSE) with `openssl pkeyutl -verify -rawin`.
+#             Absent, that command failed "command not found" and the gate's
+#             `|| { echo ...; exit 1; }` reported it as "Quorum verdict
+#             signature FAILED verification" -- a TOOLING gap narrated as a
+#             SECURITY verdict, pointing the investigation at a perfectly
+#             valid signed approval instead of at this image. It blocked
+#             v0.0.134 and v0.0.135 on 2026-08-31.
+# jq       -> the same story one step earlier: the gate parsed that envelope
+#             with `jq -e`, and a missing jq was reported as "not a signed
+#             DSSE envelope". jq is the reflex tool for JSON in shell steps,
+#             so its absence keeps resurfacing as a misleading failure in
+#             whichever step reaches for it next.
+# gcc,musl-dev -> C toolchain for cgo; `go test -race` (the Makefile test
+#             target) requires cgo, which needs a C compiler + libc headers
+#             (musl on Alpine)
+# python3,py3-yaml -> the repo's own gates are Python: credentials-coverage.sh,
+#             validate-admission.sh and feature-lint.sh all shell out to
+#             python3, and credentials-coverage parses manifests with yaml
 #
 # Go is intentionally NOT baked in: actions/setup-go fetches the pinned version
 # per-run (node + tar are present), so the image needs no rebuild on Go bumps.
@@ -45,7 +69,13 @@ RUN apk add --no-cache \
       curl \
       tar \
       make \
-      ca-certificates
+      ca-certificates \
+      gcc \
+      musl-dev \
+      python3 \
+      py3-yaml \
+      openssl \
+      jq
 
 # Match the runner pod's runAsUser (1000); keep the image non-root by default.
 USER 1000

README.md

@@ -73,3 +73,24 @@ Two traps the script exists to prevent: pushing to the bare MagicDNS name
 namespace**, since a registry host is only recognised when it contains a dot or
 colon; and `docker push $_` pushes `.`, because `$_` is the last ARGUMENT of the
 build command, not its tag.
+
+### Automated builds
+
+`.github/workflows/build-image.yml` builds and publishes to
+**ghcr.io/zegit-zoo/zegit-forgejo-runner** on GitHub.
+
+Run it from the Actions tab (**build image** → Run workflow) with the tag, e.g.
+`12.11.1-ci4`. A push to `master` that touches the Dockerfile only VALIDATES
+that both architectures still build — it publishes nothing, because guessing a
+tag from a branch push would silently mint versions.
+
+It builds on GitHub rather than our own Gitea Actions because the runners are
+host mode (`ubuntu-latest:host`, no job containers) and drop ALL capabilities
+with `allowPrivilegeEscalation: false`; rootless buildah needs user namespaces
+for its RUN steps, so building there would mean weakening that posture. It
+publishes to ghcr.io rather than the Gitea registry because
+`gitea.langenoja.se` sits behind Cloudflare, whose body cap returns **413** on a
+registry layer — and the cluster already pulls zegit's own images from ghcr.io
+with an existing `github-registry-credentials` pull secret.
+
+`push-image.sh` remains for building by hand when you need it.