zegit's arc-runner-pin.sh pins a runner image only when cosign verifies
its digest against the runner-image public key (zegit#143), but nothing
signed the images, so the pin refused every day. Sign the digest with
the RUNNER_IMAGE_COSIGN_KEY repository secret after the publish and
verify the signature in the same run. sign_only re-signs an image
published before signing existed.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_0147GvtJgsTjDGcrfvJsx5jS
The publish path failed at registry login on every dispatch (runs 18,
20) because it read `secrets.ZEGIT_PACKAGE_TOKEN`, a name that exists
nowhere. The package-write token every zegit-zoo image workflow uses is
ZEGIT_CI_PACKAGE_TOKEN, held above repository scope, so this repository
can read it under that name — no second token, no second copy to rotate.
buildctl v0.32 has no "cacheonly" exporter, so every push-triggered
validation build since this workflow was added failed with 'exporter
"cacheonly" could not be found' (run 16, 2026-09-08) — unnoticed, since
nothing depended on it. A build with no --output still builds every
stage on both architectures. The post-publish toolchain assertion now
also requires chromium-browser and psql (12.11.1-ci7).
zegit's `browser tests` (chromedp + CDP virtual authenticator) and
`lighthouse` jobs were github.com-only, so when `ci` was routed to this
runner on 2026-08-31 they stopped running anywhere. Chromium 149 (Alpine
3.23) with font-noto lets both run here; psql lets the lighthouse job
create and drop its own database on ci-postgres instead of migrating the
shared maintenance database from under the integration suite's template.
Verified locally (docker build, amd64): chromium-browser 149.0.7827.53,
psql 18.6, 45 font files, node 24.18.1, still uid 1000.
The toolchain check's own comment said the quiet part: "the list only covers what
someone remembered to add". So the list was derived instead of remembered — every
command invoked from a `run:` block across zegit's Gitea-executed workflows and
composite actions was extracted and checked against this image.
Result: no missing apk package. Everything resolves to one of three sources —
baked in here, provided by busybox/coreutils, or downloaded per-run and
immediately version-checked (go via setup-go, buildctl, flux, cosign, syft,
goreleaser). The download path needs nothing from this image beyond curl and tar,
both already asserted.
What the audit did find is that the busybox/coreutils layer was entirely
unasserted. Dropping `coreutils` from the apk line would pass this check while
base64, mktemp, seq and sort silently vanished — and on this repo's own history,
their absence would be narrated as whatever the gate reaching for them happened
to be doing, not as a missing tool. That is how openssl became "signature FAILED
verification" and jq became "not a signed DSSE envelope".
Verified by executing the assertion, not by reading it: extracted the inner
script and ran it against a PATH with base64 deliberately absent. It printed
base64=MISSING and the outer `grep -q MISSING` failed the build. Against a full
PATH it reports zero MISSING.
docker manifest create refuses a source that is itself a manifest list:
ERROR: ...:<tag>-amd64 is a manifest list
With Docker containerd image store, docker build --platform linux/amd64
produces an OCI index rather than a plain manifest, so BOTH per-arch tags are
lists and the combine fails — after the slow multi-hundred-MB pushes have
already succeeded. The failure mode is nasty: the per-arch tags are published
and the PLAIN tag, the one both Deployments reference, is absent. Hit on
12.11.1-ci6, 2026-08-31.
buildx imagetools create accepts indexes as sources and combines registry-side,
so it works with either image store. Verification switches to imagetools
inspect and now asserts both linux/amd64 and linux/arm64 specifically, rather
than counting two distinct architecture strings.
build-image.yml is GitHub-only, so the image the Gitea runners are built FROM
could only be produced by GitHub. When GitHub Actions went unavailable on
2026-08-29 that circularity bit: the fix for openssl/jq missing from this image
— the absence that blocked v0.0.134 and v0.0.135 — could not be published on the
forge the project is moving to.
This publishes from the self-hosted runners through the in-cluster rootless
buildkitd, the path zegit images already use. buildah needing user namespaces
does not apply: buildkitd is a separate hardened workload and the job only
speaks to it over the network, so nothing is built inside the runner container.
Same contract as build-image.yml: a master push validates both arches and
publishes nothing, only an explicit dispatch names a tag and pushes, and a
dispatch without a package-write token fails loudly instead of reporting success
having pushed nothing.
The toolchain assertion is kept but done with buildkit rather than docker run,
which these runners lack: a stage FROM the published image whose RUN fails on a
missing tool, built natively for both arches. Verified it passes against an
image built from current master and fails with MISSING: openssl against the
deployed ci5.
The release tag gate parses the Control Plane quorum verdict and verifies its
Ed25519 DSSE signature by shelling out to jq and openssl. Neither is in this
image, and both call sites read cmd ... || { echo <security message>; exit 1; },
so command-not-found took the failure branch: the gate reported a valid, signed
approval first as "not a signed DSSE envelope" and then as "signature FAILED
verification". That blocked v0.0.134 and v0.0.135 on 2026-08-31 and pointed the
investigation at the Tag Request rather than at this image — the same shape as
the git-daemon gap already documented above the package list.
Also adds both to the toolchain assertion. That check already existed and still
missed them, because it only covers what someone thought to list; the comment
now says so, and says to extend it whenever a gate grows a shell dependency.
Verified by building this image and running the gate script inside it against
the live Control Plane: openssl 3.5.8, jq 1.8.1, and the v0.0.135 verdict
verifies and binds to its commit. The unpatched image fails the same script.
This image is a TOOLCHAIN, so "it built" is not the property worth checking —
"it still contains what the repo's gates shell out to" is.
It exists because the hand-built image had drifted from this Dockerfile: the
deployed ci3 carried gcc, musl-dev, python3 and py3-yaml, the Dockerfile did
not, and rebuilding from the Dockerfile silently dropped them. Gitea CI then
failed at "credentials-coverage: need python3" without reaching the tests at
all. Nothing in the pipeline objected, because every check asked whether the
image built, not what was in it.
Asserts python3 with a working yaml import, gcc, node, bash, make, curl, tar
and git-http-backend, on BOTH architectures — a package that resolves on amd64
but not arm64 would otherwise reach half the nodes unnoticed.
Verified the check can actually FAIL, which is the only property that makes it
worth having: against the bare forgejo/runner:12.11.1 it reports python3+yaml,
gcc, node and git-http-backend all MISSING; with the packages installed it
reports 3.12.14 and ok for the rest.
REGRESSION, mine. The committed Dockerfile had drifted from the image actually
in service: the running ci3 image contained these four packages, the Dockerfile
did not, and I rebuilt ci4 from the Dockerfile. The rollout therefore removed
tools zegit's CI depends on.
Proof the deployed ci3 had them: its Gitea runs got PAST `make credentials-check`
— which shells out to python3 — and failed later at cover-check. The ci4 run
fails earlier, at `credentials-coverage: need python3`.
What each is for, so the next rebuild does not drop them again:
python3, py3-yaml the repo's own gates are Python — credentials-coverage.sh,
validate-admission.sh and feature-lint.sh all shell out to
it, and credentials-coverage parses manifests with yaml
gcc, musl-dev cgo, which `go test -race` in the Makefile test target
requires: a C compiler plus libc headers (musl on Alpine)
Verified on the real base image rather than assumed — on
code.forgejo.org/forgejo/runner:12.11.1, `apk add git git-daemon gcc musl-dev
python3 py3-yaml` yields Python 3.12.14, a working `import yaml`, gcc 15.2.0,
and git-http-backend on the git exec path.
The deeper lesson is that a hand-built image can drift from its Dockerfile
silently, and nothing notices until someone rebuilds. Publishing via CI (#2)
is what stops that recurring, since every future image comes from the
committed file.
Two things were wrong, and the second actively misled.
The publish instructions described pushing to the Gitea registry by hand.
Publishing now runs on GitHub Actions to ghcr.io, because Cloudflare's
request-body cap returns 413 on a registry layer and our own runners cannot
build images (host mode, all capabilities dropped). The example tag had also
been mangled to "12.11.1-ci".
"Package visibility is PUBLIC, so the runner namespace pulls without a secret"
was false. The Gitea package is private — flux-config's kustomization says so
in as many words — and the runner Deployments have always carried a pull
secret. Reading that line led me to conclude the ghcr package would need no
credential; it does, and forgejo-runner-system now holds
github-registry-credentials for exactly that.
Verified with `docker build --check`: no warnings.
Publishing this image by hand kept hitting the same four walls — Cloudflare's
413, Docker parsing a bare MagicDNS name as a Docker Hub namespace, a PAT
missing write:package, and multi-arch. push-image.sh made that survivable;
this removes the manual step entirely.
It builds on GitHub rather than our own Gitea Actions because the runners
cannot build images: their labels are `ubuntu-latest:host` (host mode, no job
containers) and the container drops ALL capabilities with
allowPrivilegeEscalation:false, while rootless buildah needs newuidmap/user
namespaces for RUN steps. Building there would mean weakening the posture the
hardening established.
It publishes to ghcr.io rather than the Gitea registry because Cloudflare's
body cap is what caused the 413 in the first place, and because the cluster
already pulls zegit's own images from ghcr.io through an existing
github-registry-credentials pull secret — an established path rather than a
new one.
A master push validates that both arches still build but publishes nothing;
only an explicit dispatch names a tag and pushes, so a branch push cannot
silently mint a version. The final step reads the published index back and
fails if either architecture is absent, since a single-arch image under this
tag would CrashLoop on half the nodes.
Action SHAs match the pins zegit already uses, rather than introducing a
second set of versions to track.
Publishing this image by hand hit four obstacles in a row, each of which
recurs on every rebuild:
413 Payload Too Large gitea.langenoja.se is behind Cloudflare, which
caps request bodies below a registry layer
Docker Hub auth refusal the bare MagicDNS name was parsed as a Docker Hub
NAMESPACE — a registry host is only recognised
when it contains a dot or a colon
insufficient_scope the PAT needs write:package, not read:package
wrong-arch pulls forgejo-runner runs on arm64 and zegit-runner on
amd64, neither pins a nodeSelector, and both use
this one tag
The image reference must keep the hostname gitea.langenoja.se, because that is
what the cluster pulls. So the script keeps the NAME and changes only the
ROUTE: it pins that hostname to the Istio gateway on the LAN for the duration
of the push. The gateway serves a *.langenoja.se wildcard cert, so TLS
validates normally — no insecure-registries needed.
The pin carries a marker comment, which makes three cases distinguishable: a
leftover from a run that died before its trap fired (reuse, then remove), a
pin the operator added deliberately (never touch), and one pointing somewhere
other than the gateway (refuse to run). Without the marker a leftover pin was
indistinguishable from a deliberate one, so it survived every later run and
had to be reverted by hand. `--unpin` clears a leftover without building, and
the trap covers INT and TERM as well as EXIT.
It also verifies the published manifest really lists two architectures rather
than trusting the push's exit code.
Alpine splits git-http-backend OUT of the `git` package: git alone ships
git-remote-http/https but not the CGI backend. The runner therefore had git
2.52.0 and no git-http-backend, so zegit's internal/forgeprobe smart-HTTP tests
failed at request time with "fork/exec git-http-backend: no such file or
directory" and reported three FAILURES for a missing tool.
That turned Gitea CI red on every zegit master commit while GitHub stayed
green, for hours, with nobody looking — GitHub is the release oracle, which is
exactly why a red mirror is invisible.
Verified against the real base image, not just alpine:3 — on
code.forgejo.org/forgejo/runner:12.11.1 (Alpine 3.23), `apk add git git-daemon`
puts git-http-backend at /usr/libexec/git-core/git-http-backend, and
`git --exec-path` resolves to that directory, which is the lookup the test
harness performs.
Adds the Dockerfile (FROM code.forgejo.org/forgejo/runner:12.11.1, plus
node/git/bash/coreutils/curl/tar/make/ca-certificates) and a README with
the multi-arch build/push workflow. Built and pushed as
gitea.langenoja.se/zegit-zoo/zegit-forgejo-runner:12.11.1-ci; consumed by
flux-config applications/forgejo-runner.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude
markus