With openssh-keygen in the image (#17, 12.11.1-ci11), zegit's SSHSIG
oracle test still failed on this runner: `ssh-keygen -Y verify` answered
"No user exists for uid 1000" (exit 255). The pod runs as uid 1000, which
has no /etc/passwd entry here, and ssh-keygen looks the current user up
(zegit#575, Gitea run 25467 job 56050).
The image adds a `runner` user and group for uid/gid 1000 (when the base
has none) with a home directory, and a build step running as uid 1000
signs and verifies an SSHSIG, so a missing entry fails the build.
Checked on the operator workstation: zegit's test fails in ci11 run as
--user 1000 ("No user exists for uid 1000") and passes in this build.
Refs: zegit-zoo/zegit#575
Co-Authored-By: Claude Opus 5.5 <[email protected]>
zegit's TestSSHSIG_ChecksumsVerifyWithSSHKeygen_441 verifies release
checksum signatures with `ssh-keygen -Y verify` and fails under CI when
ssh-keygen is missing. zegit's Gitea jobs run in host mode on this
runner image, which has no ssh-keygen, so Gitea `lint + test` stayed red
on zegit master 639871e0 and the v0.1.56 cut is refused (zegit#575).
GitHub's ARC pool was fixed separately (zegit-arc-runner#21, zegit#578).
openssh-keygen joins the apk list, and a build step signs and verifies
an SSHSIG in namespace "file", so a build without a working ssh-keygen
fails here.
Checked on the operator workstation: the image builds, and zegit's test
passes inside it with CI=1 (`ok cmd/z-backend`, ssh-keygen at
/usr/bin/ssh-keygen).
Refs: zegit-zoo/zegit#575
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Gitea 28 adds job-index and job-total under strategy in the workflow it
sends for a matrix job, and forgejo-runner's schema refuses them, so
every matrix job dies before its first step (zegit image-gitea.yml run
23992, the v0.1.52 tag build). No forgejo-runner release accepts them
(checked through 13.2.1). Build the runner from v12.11.1 at its commit
with a one-file schema patch and two tests, one of them the real payload
of that failed job; the build runs them. Bases are pinned by digest.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_018wWyCiV27KmNSauYTdLyp7
The forgejo/runner:12.11.1 base ships brotli-libs 1.2.0-r0, and `apk add`
never upgrades a package the base already has. Upgrade it explicitly
(-> 1.2.1-r0) and fail the build if it is still older, so a base bump
cannot bring the vulnerable version back. Markus approved 2026-11-07.
Tested locally: builds for linux/amd64 and linux/arm64,
"Upgrading brotli-libs (1.2.0-r0 -> 1.2.1-r0)", gate passes.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
zegit's `browser tests` (chromedp + CDP virtual authenticator) and
`lighthouse` jobs were github.com-only, so when `ci` was routed to this
runner on 2026-08-31 they stopped running anywhere. Chromium 149 (Alpine
3.23) with font-noto lets both run here; psql lets the lighthouse job
create and drop its own database on ci-postgres instead of migrating the
shared maintenance database from under the integration suite's template.
Verified locally (docker build, amd64): chromium-browser 149.0.7827.53,
psql 18.6, 45 font files, node 24.18.1, still uid 1000.
The release tag gate parses the Control Plane quorum verdict and verifies its
Ed25519 DSSE signature by shelling out to jq and openssl. Neither is in this
image, and both call sites read cmd ... || { echo <security message>; exit 1; },
so command-not-found took the failure branch: the gate reported a valid, signed
approval first as "not a signed DSSE envelope" and then as "signature FAILED
verification". That blocked v0.0.134 and v0.0.135 on 2026-08-31 and pointed the
investigation at the Tag Request rather than at this image — the same shape as
the git-daemon gap already documented above the package list.
Also adds both to the toolchain assertion. That check already existed and still
missed them, because it only covers what someone thought to list; the comment
now says so, and says to extend it whenever a gate grows a shell dependency.
Verified by building this image and running the gate script inside it against
the live Control Plane: openssl 3.5.8, jq 1.8.1, and the v0.0.135 verdict
verifies and binds to its commit. The unpatched image fails the same script.
REGRESSION, mine. The committed Dockerfile had drifted from the image actually
in service: the running ci3 image contained these four packages, the Dockerfile
did not, and I rebuilt ci4 from the Dockerfile. The rollout therefore removed
tools zegit's CI depends on.
Proof the deployed ci3 had them: its Gitea runs got PAST `make credentials-check`
— which shells out to python3 — and failed later at cover-check. The ci4 run
fails earlier, at `credentials-coverage: need python3`.
What each is for, so the next rebuild does not drop them again:
python3, py3-yaml the repo's own gates are Python — credentials-coverage.sh,
validate-admission.sh and feature-lint.sh all shell out to
it, and credentials-coverage parses manifests with yaml
gcc, musl-dev cgo, which `go test -race` in the Makefile test target
requires: a C compiler plus libc headers (musl on Alpine)
Verified on the real base image rather than assumed — on
code.forgejo.org/forgejo/runner:12.11.1, `apk add git git-daemon gcc musl-dev
python3 py3-yaml` yields Python 3.12.14, a working `import yaml`, gcc 15.2.0,
and git-http-backend on the git exec path.
The deeper lesson is that a hand-built image can drift from its Dockerfile
silently, and nothing notices until someone rebuilds. Publishing via CI (#2)
is what stops that recurring, since every future image comes from the
committed file.
Two things were wrong, and the second actively misled.
The publish instructions described pushing to the Gitea registry by hand.
Publishing now runs on GitHub Actions to ghcr.io, because Cloudflare's
request-body cap returns 413 on a registry layer and our own runners cannot
build images (host mode, all capabilities dropped). The example tag had also
been mangled to "12.11.1-ci".
"Package visibility is PUBLIC, so the runner namespace pulls without a secret"
was false. The Gitea package is private — flux-config's kustomization says so
in as many words — and the runner Deployments have always carried a pull
secret. Reading that line led me to conclude the ghcr package would need no
credential; it does, and forgejo-runner-system now holds
github-registry-credentials for exactly that.
Verified with `docker build --check`: no warnings.
Alpine splits git-http-backend OUT of the `git` package: git alone ships
git-remote-http/https but not the CGI backend. The runner therefore had git
2.52.0 and no git-http-backend, so zegit's internal/forgeprobe smart-HTTP tests
failed at request time with "fork/exec git-http-backend: no such file or
directory" and reported three FAILURES for a missing tool.
That turned Gitea CI red on every zegit master commit while GitHub stayed
green, for hours, with nobody looking — GitHub is the release oracle, which is
exactly why a red mirror is invisible.
Verified against the real base image, not just alpine:3 — on
code.forgejo.org/forgejo/runner:12.11.1 (Alpine 3.23), `apk add git git-daemon`
puts git-http-backend at /usr/libexec/git-core/git-http-backend, and
`git --exec-path` resolves to that directory, which is the lookup the test
harness performs.
Adds the Dockerfile (FROM code.forgejo.org/forgejo/runner:12.11.1, plus
node/git/bash/coreutils/curl/tar/make/ca-certificates) and a README with
the multi-arch build/push workflow. Built and pushed as
gitea.langenoja.se/zegit-zoo/zegit-forgejo-runner:12.11.1-ci; consumed by
flux-config applications/forgejo-runner.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
agent
markus