agent
5427ccbad2
REGRESSION, mine. The committed Dockerfile had drifted from the image actually
in service: the running ci3 image contained these four packages, the Dockerfile
did not, and I rebuilt ci4 from the Dockerfile. The rollout therefore removed
tools zegit's CI depends on.
Proof the deployed ci3 had them: its Gitea runs got PAST `make credentials-check`
— which shells out to python3 — and failed later at cover-check. The ci4 run
fails earlier, at `credentials-coverage: need python3`.
What each is for, so the next rebuild does not drop them again:
python3, py3-yaml the repo's own gates are Python — credentials-coverage.sh,
validate-admission.sh and feature-lint.sh all shell out to
it, and credentials-coverage parses manifests with yaml
gcc, musl-dev cgo, which `go test -race` in the Makefile test target
requires: a C compiler plus libc headers (musl on Alpine)
Verified on the real base image rather than assumed — on
code.forgejo.org/forgejo/runner:12.11.1, `apk add git git-daemon gcc musl-dev
python3 py3-yaml` yields Python 3.12.14, a working `import yaml`, gcc 15.2.0,
and git-http-backend on the git exec path.
The deeper lesson is that a hand-built image can drift from its Dockerfile
silently, and nothing notices until someone rebuilds. Publishing via CI (#2)
is what stops that recurring, since every future image comes from the
committed file.
zegit-forgejo-runner
Custom toolchain image for the cluster's self-hosted Gitea Actions (Forgejo)
runner, which runs in host mode (Docker disabled) — workflow steps execute
inside this image, so it must carry node/git/make/etc. that the stock
code.forgejo.org/forgejo/runner image lacks.
This repo lives only on Gitea (zegit-zoo/zegit-forgejo-runner). The cluster-side
deployment and rationale live in flux-config (runners.md,
applications/forgejo-runner/).
Build & push
The runner needs this image to run any job, so the first build must be done by hand (CI can't bootstrap itself). The cluster is mixed amd64/arm64, so build multi-arch:
docker buildx create --use --name zegit-ci 2>/dev/null || true
docker login gitea.langenoja.se # username + a Gitea personal access token
docker buildx build --platform linux/amd64,linux/arm64 \
-t gitea.langenoja.se/zegit-zoo/zegit-forgejo-runner:12.11.1-ci --push .
The Gitea package is public, so the forgejo-runner-system namespace pulls it
without an imagePullSecret.
Wire it into the cluster
After the push succeeds, in flux-config
applications/forgejo-runner/base/deployment-forgejo-runner.yaml set the image:
on both the register initContainer and the forgejo-runner daemon to
gitea.langenoja.se/zegit-zoo/zegit-forgejo-runner:12.11.1-ci, then commit (Flux
reconciles). Do this only after the image exists, or the pod ImagePullBackOffs.
Tag scheme
<base-runner-version>-ci (e.g. 12.11.1-ci). Must satisfy the cluster's
Gatekeeper enforce-semantic-image-tags constraint — never :latest.
Publishing a new image
./push-image.sh 12.11.1-ci4
The registry hostname in the image reference must stay gitea.langenoja.se,
because that is what the cluster pulls. The script therefore keeps the NAME and
changes only the ROUTE: it pins that hostname to the Istio gateway on the LAN
(10.40.0.10) for the duration of the push, then removes the pin.
That matters because gitea.langenoja.se resolves to Cloudflare, which caps
request bodies below a registry layer and returns 413 Payload Too Large. The
gateway serves a *.langenoja.se wildcard cert, so TLS still validates normally
— no insecure-registries, no --tls-verify=false.
If a run is killed before it can clean up, the pin is left behind. It carries a marker comment, so the next run recognises it as its own and removes it — or you can clear it directly:
./push-image.sh --unpin
A pin you added yourself (no marker) is never touched, and the script refuses to run if an unmarked pin points somewhere other than the gateway.
Requirements: a Gitea PAT with write:package (read:package is pull-only),
LAN reachability of the gateway, and qemu/binfmt registered for the arm64 build.
Multi-arch is not optional. forgejo-runner runs on arm64 nodes and
zegit-runner on amd64, neither deployment sets a nodeSelector, and both use
this single tag.
Two traps the script exists to prevent: pushing to the bare MagicDNS name
(gitea-system-gitea-tailscale-http) makes Docker treat it as a Docker Hub
namespace, since a registry host is only recognised when it contains a dot or
colon; and docker push $_ pushes ., because $_ is the last ARGUMENT of the
build command, not its tag.
Automated builds
.github/workflows/build-image.yml builds and publishes to
ghcr.io/zegit-zoo/zegit-forgejo-runner on GitHub.
Run it from the Actions tab (build image → Run workflow) with the tag, e.g.
12.11.1-ci4. A push to master that touches the Dockerfile only VALIDATES
that both architectures still build — it publishes nothing, because guessing a
tag from a branch push would silently mint versions.
It builds on GitHub rather than our own Gitea Actions because the runners are
host mode (ubuntu-latest:host, no job containers) and drop ALL capabilities
with allowPrivilegeEscalation: false; rootless buildah needs user namespaces
for its RUN steps, so building there would mean weakening that posture. It
publishes to ghcr.io rather than the Gitea registry because
gitea.langenoja.se sits behind Cloudflare, whose body cap returns 413 on a
registry layer — and the cluster already pulls zegit's own images from ghcr.io
with an existing github-registry-credentials pull secret.
push-image.sh remains for building by hand when you need it.