fix: restore gcc, musl-dev, python3 and py3-yaml to the image

REGRESSION, mine. The committed Dockerfile had drifted from the image actually
in service: the running ci3 image contained these four packages, the Dockerfile
did not, and I rebuilt ci4 from the Dockerfile. The rollout therefore removed
tools zegit's CI depends on.

Proof the deployed ci3 had them: its Gitea runs got PAST `make credentials-check`
— which shells out to python3 — and failed later at cover-check. The ci4 run
fails earlier, at `credentials-coverage: need python3`.

What each is for, so the next rebuild does not drop them again:
  python3, py3-yaml  the repo's own gates are Python — credentials-coverage.sh,
                     validate-admission.sh and feature-lint.sh all shell out to
                     it, and credentials-coverage parses manifests with yaml
  gcc, musl-dev      cgo, which `go test -race` in the Makefile test target
                     requires: a C compiler plus libc headers (musl on Alpine)

Verified on the real base image rather than assumed — on
code.forgejo.org/forgejo/runner:12.11.1, `apk add git git-daemon gcc musl-dev
python3 py3-yaml` yields Python 3.12.14, a working `import yaml`, gcc 15.2.0,
and git-http-backend on the git exec path.

The deeper lesson is that a hand-built image can drift from its Dockerfile
silently, and nothing notices until someone rebuilds. Publishing via CI (#2)
is what stops that recurring, since every future image comes from the
committed file.

.github/workflows/build-image.yml

@@ -0,0 +1,93 @@
+name: build image
+
+# Publishes the runner toolchain image to ghcr.io.
+#
+# Why GitHub rather than our own Gitea Actions: the runners are host mode
+# (`ubuntu-latest:host`, no job containers) and their container drops ALL
+# capabilities with allowPrivilegeEscalation:false. Rootless buildah needs
+# newuidmap/user namespaces for its RUN steps, so building images there would
+# mean weakening the posture that hardening established. GitHub runners have
+# docker + buildx + QEMU already.
+#
+# Why ghcr.io rather than the Gitea registry: gitea.langenoja.se sits behind
+# Cloudflare, whose request-body cap returns 413 on a registry layer. The
+# cluster already pulls zegit's own images from ghcr.io and the runner
+# deployments already carry a github-registry-credentials pull secret, so this
+# reuses an established, trusted path instead of inventing one.
+
+on:
+  workflow_dispatch:
+    inputs:
+      tag:
+        description: 'Image tag, e.g. 12.11.1-ci4 (upstream runner version + ci revision)'
+        required: true
+        type: string
+  push:
+    branches: [master]
+    paths:
+      - 'Dockerfile'
+      - '.github/workflows/build-image.yml'
+
+permissions:
+  contents: read
+  packages: write
+
+env:
+  IMAGE: ghcr.io/${{ github.repository }}
+
+jobs:
+  build:
+    # GitHub-only: the Gitea mirror has no ghcr.io and no GITHUB_TOKEN.
+    if: ${{ github.server_url == 'https://github.com' }}
+    runs-on: ubuntu-latest
+    timeout-minutes: 45
+    steps:
+      - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
+
+      # arm64 is not optional: forgejo-runner runs on arm64 nodes and
+      # zegit-runner on amd64, neither deployment sets a nodeSelector, and both
+      # use this one tag.
+      - uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3
+      - uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
+
+      - uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
+        with:
+          registry: ghcr.io
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+
+      # A master push VALIDATES the Dockerfile still builds on both arches but
+      # publishes nothing — only an explicit dispatch names a tag and pushes.
+      # Guessing a tag from a branch push would silently mint versions.
+      - id: meta
+        run: |
+          if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
+            echo "push=true" >> "$GITHUB_OUTPUT"
+            echo "tags=${IMAGE}:${{ inputs.tag }}" >> "$GITHUB_OUTPUT"
+          else
+            echo "push=false" >> "$GITHUB_OUTPUT"
+            echo "tags=${IMAGE}:ci-validation" >> "$GITHUB_OUTPUT"
+          fi
+
+      - uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8 # v6
+        with:
+          context: .
+          platforms: linux/amd64,linux/arm64
+          push: ${{ steps.meta.outputs.push }}
+          tags: ${{ steps.meta.outputs.tags }}
+          provenance: false
+          sbom: false
+
+      # Read the published index back rather than trusting the push's exit
+      # code, and fail if either architecture is absent — a single-arch image
+      # under this tag would CrashLoop on half the nodes.
+      - name: Verify both architectures were published
+        if: ${{ steps.meta.outputs.push == 'true' }}
+        run: |
+          set -euo pipefail
+          out=$(docker buildx imagetools inspect "${IMAGE}:${{ inputs.tag }}")
+          echo "$out"
+          for want in amd64 arm64; do
+            echo "$out" | grep -q "linux/$want" || { echo "::error::published image is missing linux/$want"; exit 1; }
+          done
+          echo "✓ ${IMAGE}:${{ inputs.tag }} carries linux/amd64 and linux/arm64"

Dockerfile

@@ -21,23 +21,41 @@ USER root
 
 # nodejs   -> JS actions runtime (actions/checkout, actions/setup-go, ...)
 # git      -> checkout
+# git-daemon -> provides git-http-backend, which Alpine splits OUT of the `git`
+#             package (git alone ships git-remote-http/https but NOT the CGI
+#             backend). Without it, internal/forgeprobe's smart-HTTP tests died
+#             with "fork/exec git-http-backend: no such file or directory" and
+#             reported three FAILURES for a missing tool — which turned Gitea CI
+#             red on every zegit master commit while GitHub stayed green, for
+#             hours, unnoticed.
 # bash     -> `shell: bash` steps and many action entrypoints
 # coreutils-> GNU userland (alpine busybox tools trip up some scripts)
 # curl,tar -> setup-go / gitleaks download + extract their own arch-specific binaries
 # make     -> zegit Makefile targets
 # ca-certificates -> trusted roots for HTTPS to gitea/github/goproxy
+# gcc,musl-dev -> C toolchain for cgo; `go test -race` (the Makefile test
+#             target) requires cgo, which needs a C compiler + libc headers
+#             (musl on Alpine)
+# python3,py3-yaml -> the repo's own gates are Python: credentials-coverage.sh,
+#             validate-admission.sh and feature-lint.sh all shell out to
+#             python3, and credentials-coverage parses manifests with yaml
 #
 # Go is intentionally NOT baked in: actions/setup-go fetches the pinned version
 # per-run (node + tar are present), so the image needs no rebuild on Go bumps.
 RUN apk add --no-cache \
       nodejs \
       git \
+      git-daemon \
       bash \
       coreutils \
       curl \
       tar \
       make \
-      ca-certificates
+      ca-certificates \
+      gcc \
+      musl-dev \
+      python3 \
+      py3-yaml
 
 # Match the runner pod's runAsUser (1000); keep the image non-root by default.
 USER 1000

README.md

@@ -37,3 +37,60 @@ reconciles). Do this only *after* the image exists, or the pod ImagePullBackOffs
 
 `<base-runner-version>-ci` (e.g. `12.11.1-ci`). Must satisfy the cluster's
 Gatekeeper `enforce-semantic-image-tags` constraint — never `:latest`.
+
+## Publishing a new image
+
+    ./push-image.sh 12.11.1-ci4
+
+The registry hostname in the image reference must stay `gitea.langenoja.se`,
+because that is what the cluster pulls. The script therefore keeps the NAME and
+changes only the ROUTE: it pins that hostname to the Istio gateway on the LAN
+(`10.40.0.10`) for the duration of the push, then removes the pin.
+
+That matters because `gitea.langenoja.se` resolves to Cloudflare, which caps
+request bodies below a registry layer and returns **413 Payload Too Large**. The
+gateway serves a `*.langenoja.se` wildcard cert, so TLS still validates normally
+— no `insecure-registries`, no `--tls-verify=false`.
+
+If a run is killed before it can clean up, the pin is left behind. It carries a
+marker comment, so the next run recognises it as its own and removes it — or you
+can clear it directly:
+
+    ./push-image.sh --unpin
+
+A pin you added yourself (no marker) is never touched, and the script refuses to
+run if an unmarked pin points somewhere other than the gateway.
+
+Requirements: a Gitea PAT with **`write:package`** (`read:package` is pull-only),
+LAN reachability of the gateway, and qemu/binfmt registered for the arm64 build.
+
+Multi-arch is not optional. `forgejo-runner` runs on arm64 nodes and
+`zegit-runner` on amd64, neither deployment sets a `nodeSelector`, and both use
+this single tag.
+
+Two traps the script exists to prevent: pushing to the bare MagicDNS name
+(`gitea-system-gitea-tailscale-http`) makes Docker treat it as a **Docker Hub
+namespace**, since a registry host is only recognised when it contains a dot or
+colon; and `docker push $_` pushes `.`, because `$_` is the last ARGUMENT of the
+build command, not its tag.
+
+### Automated builds
+
+`.github/workflows/build-image.yml` builds and publishes to
+**ghcr.io/zegit-zoo/zegit-forgejo-runner** on GitHub.
+
+Run it from the Actions tab (**build image** → Run workflow) with the tag, e.g.
+`12.11.1-ci4`. A push to `master` that touches the Dockerfile only VALIDATES
+that both architectures still build — it publishes nothing, because guessing a
+tag from a branch push would silently mint versions.
+
+It builds on GitHub rather than our own Gitea Actions because the runners are
+host mode (`ubuntu-latest:host`, no job containers) and drop ALL capabilities
+with `allowPrivilegeEscalation: false`; rootless buildah needs user namespaces
+for its RUN steps, so building there would mean weakening that posture. It
+publishes to ghcr.io rather than the Gitea registry because
+`gitea.langenoja.se` sits behind Cloudflare, whose body cap returns **413** on a
+registry layer — and the cluster already pulls zegit's own images from ghcr.io
+with an existing `github-registry-credentials` pull secret.
+
+`push-image.sh` remains for building by hand when you need it.

push-image.sh

@@ -0,0 +1,120 @@
+#!/usr/bin/env bash
+# push-image.sh — build and publish the runner toolchain image, multi-arch.
+#
+# Why this script exists: doing it by hand hit four obstacles in a row.
+#
+#   1. 413 Payload Too Large. gitea.langenoja.se is behind Cloudflare, which
+#      caps request bodies well below a registry layer. Fixed by pinning the
+#      hostname to the Istio gateway on the LAN (below) so the push never
+#      traverses Cloudflare.
+#   2. Docker only treats a name as a registry HOST if it contains a dot or a
+#      colon. Pushing to the bare MagicDNS name `gitea-system-gitea-tailscale-http`
+#      was parsed as a Docker Hub NAMESPACE, so Docker Hub refused the auth.
+#   3. The Gitea PAT needs `write:package`. `read:package` is pull-only.
+#   4. Multi-arch is required: forgejo-runner runs on arm64 nodes and
+#      zegit-runner on amd64, neither deployment sets a nodeSelector, and both
+#      use this one tag.
+#
+# The hostname MUST stay gitea.langenoja.se in the image reference, because that
+# is what the cluster pulls. So we keep the name and change only the ROUTE, via
+# an /etc/hosts pin that is removed again on exit.
+#
+# The Istio gateway serves a *.langenoja.se wildcard cert, so TLS validates
+# normally — no insecure-registries, no --tls-verify=false.
+set -euo pipefail
+
+TAG="${1:-}"
+GATEWAY_IP="${GATEWAY_IP:-10.40.0.10}"
+REGISTRY="gitea.langenoja.se"
+REPO="$REGISTRY/zegit-zoo/zegit-forgejo-runner"
+MARKER="# added by push-image.sh"
+HOSTS_LINE="$GATEWAY_IP  $REGISTRY  $MARKER"
+
+die() { echo "error: $*" >&2; exit 1; }
+
+if [ "$TAG" = "--unpin" ]; then
+  MARKER="# added by push-image.sh"
+  if grep -qF "$MARKER" /etc/hosts 2>/dev/null; then
+    sudo sed -i "\|${MARKER}\$|d" /etc/hosts; echo "removed the leftover pin"
+  else
+    echo "no pin of ours in /etc/hosts; nothing to do"
+  fi
+  exit 0
+fi
+[ -n "$TAG" ] || die "usage: $0 <tag> | --unpin   (e.g. $0 12.11.1-ci4)"
+command -v docker >/dev/null || die "docker not on PATH"
+
+# Three cases, and they must not be confused:
+#   OURS      — a line carrying our marker. A LEFTOVER from a run that died
+#               before its trap fired. Reuse it, and DO remove it on exit.
+#   THEIRS    — a line without our marker. Deliberate; never touch it, and
+#               refuse if it points somewhere other than the gateway.
+#   ABSENT    — add ours.
+# The earlier version could not tell OURS from THEIRS, so a leftover pin was
+# treated as deliberate and survived every subsequent run. That is the bug that
+# made an operator revert /etc/hosts by hand.
+PIN_OWNED=0
+if grep -qF "$MARKER" /etc/hosts 2>/dev/null; then
+  PIN_OWNED=1
+  echo "found a leftover pin from an interrupted run; reusing and will clean it up"
+elif grep -qE "^[^#]*[[:space:]]$REGISTRY([[:space:]]|\$)" /etc/hosts 2>/dev/null; then
+  existing=$(grep -E "^[^#]*[[:space:]]$REGISTRY([[:space:]]|\$)" /etc/hosts | head -1)
+  [ "$(echo "$existing" | awk '{print $1}')" = "$GATEWAY_IP" ] \
+    || die "/etc/hosts already pins $REGISTRY elsewhere (not ours, not the gateway): $existing"
+  echo "an unmarked pin to the gateway already exists; leaving it exactly as found"
+fi
+
+# Removes ONLY lines carrying our marker, so a deliberate pin is never touched.
+unpin() {
+  if grep -qF "$MARKER" /etc/hosts 2>/dev/null; then
+    sudo sed -i "\|${MARKER}\$|d" /etc/hosts
+    echo "removed the /etc/hosts pin"
+  fi
+}
+cleanup() { [ "$PIN_OWNED" = "1" ] && unpin; }
+# INT and TERM too: EXIT alone leaves the pin behind on Ctrl-C in some shells.
+trap cleanup EXIT INT TERM
+
+if [ "$PIN_OWNED" = "0" ] && ! grep -qE "^[^#]*[[:space:]]$REGISTRY([[:space:]]|\$)" /etc/hosts 2>/dev/null; then
+  echo "pinning $REGISTRY -> $GATEWAY_IP (bypasses Cloudflare; removed on exit)"
+  echo "$HOSTS_LINE" | sudo tee -a /etc/hosts >/dev/null
+  PIN_OWNED=1
+fi
+
+# Prove the route before building anything: 401 means the registry answered and
+# wants auth, which is success here. A timeout means the pin or the LAN is wrong.
+code=$(curl -sS -o /dev/null -w '%{http_code}' --max-time 20 "https://$REGISTRY/v2/" || true)
+case "$code" in
+  401|200) echo "registry reachable via the pin (HTTP $code)" ;;
+  *) die "registry not reachable via $GATEWAY_IP (HTTP ${code:-timeout}). Is this host on the LAN?" ;;
+esac
+
+docker login "$REGISTRY" || die "login failed — the PAT needs write:package, not read:package"
+
+for arch in amd64 arm64; do
+  echo "building linux/$arch"
+  docker build --platform "linux/$arch" -t "$REPO:$TAG-$arch" . \
+    || die "build failed for $arch (arm64 needs qemu/binfmt registered)"
+  # Explicit tag, never $_ — in zsh $_ is the last ARGUMENT, which here is '.'
+  docker push "$REPO:$TAG-$arch" || die "push failed for $arch"
+done
+
+docker manifest rm "$REPO:$TAG" >/dev/null 2>&1 || true
+docker manifest create "$REPO:$TAG" --amend "$REPO:$TAG-amd64" --amend "$REPO:$TAG-arm64"
+docker manifest push "$REPO:$TAG"
+
+# Verify by reading it back, rather than trusting the push's exit code.
+echo "verifying the published manifest lists both architectures"
+got=$(docker manifest inspect "$REPO:$TAG" | grep -o '"architecture": *"[a-z0-9]*"' | sort -u | wc -l)
+[ "$got" -ge 2 ] || die "published manifest does not list two architectures"
+docker manifest inspect "$REPO:$TAG" | grep -o '"architecture": *"[a-z0-9]*"' | sort -u
+
+cat <<EOF
+
+published $REPO:$TAG
+
+Next, in flux-config, bump BOTH deployments that use this image:
+  applications/forgejo-runner/base/deployment-zegit-runner.yaml
+  applications/forgejo-runner/base/deployment-forgejo-runner.yaml
+(markus-runner uses markus/gitea-runner and is unaffected.)
+EOF