docs: correct the registry notes in the Dockerfile header

Two things were wrong, and the second actively misled.

The publish instructions described pushing to the Gitea registry by hand.
Publishing now runs on GitHub Actions to ghcr.io, because Cloudflare's
request-body cap returns 413 on a registry layer and our own runners cannot
build images (host mode, all capabilities dropped). The example tag had also
been mangled to "12.11.1-ci".

"Package visibility is PUBLIC, so the runner namespace pulls without a secret"
was false. The Gitea package is private — flux-config's kustomization says so
in as many words — and the runner Deployments have always carried a pull
secret. Reading that line led me to conclude the ghcr package would need no
credential; it does, and forgejo-runner-system now holds
github-registry-credentials for exactly that.

Verified with `docker build --check`: no warnings.

.github/workflows/build-image.yml

@@ -0,0 +1,93 @@
+name: build image
+
+# Publishes the runner toolchain image to ghcr.io.
+#
+# Why GitHub rather than our own Gitea Actions: the runners are host mode
+# (`ubuntu-latest:host`, no job containers) and their container drops ALL
+# capabilities with allowPrivilegeEscalation:false. Rootless buildah needs
+# newuidmap/user namespaces for its RUN steps, so building images there would
+# mean weakening the posture that hardening established. GitHub runners have
+# docker + buildx + QEMU already.
+#
+# Why ghcr.io rather than the Gitea registry: gitea.langenoja.se sits behind
+# Cloudflare, whose request-body cap returns 413 on a registry layer. The
+# cluster already pulls zegit's own images from ghcr.io and the runner
+# deployments already carry a github-registry-credentials pull secret, so this
+# reuses an established, trusted path instead of inventing one.
+
+on:
+  workflow_dispatch:
+    inputs:
+      tag:
+        description: 'Image tag, e.g. 12.11.1-ci4 (upstream runner version + ci revision)'
+        required: true
+        type: string
+  push:
+    branches: [master]
+    paths:
+      - 'Dockerfile'
+      - '.github/workflows/build-image.yml'
+
+permissions:
+  contents: read
+  packages: write
+
+env:
+  IMAGE: ghcr.io/${{ github.repository }}
+
+jobs:
+  build:
+    # GitHub-only: the Gitea mirror has no ghcr.io and no GITHUB_TOKEN.
+    if: ${{ github.server_url == 'https://github.com' }}
+    runs-on: ubuntu-latest
+    timeout-minutes: 45
+    steps:
+      - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
+
+      # arm64 is not optional: forgejo-runner runs on arm64 nodes and
+      # zegit-runner on amd64, neither deployment sets a nodeSelector, and both
+      # use this one tag.
+      - uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3
+      - uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
+
+      - uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
+        with:
+          registry: ghcr.io
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+
+      # A master push VALIDATES the Dockerfile still builds on both arches but
+      # publishes nothing — only an explicit dispatch names a tag and pushes.
+      # Guessing a tag from a branch push would silently mint versions.
+      - id: meta
+        run: |
+          if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
+            echo "push=true" >> "$GITHUB_OUTPUT"
+            echo "tags=${IMAGE}:${{ inputs.tag }}" >> "$GITHUB_OUTPUT"
+          else
+            echo "push=false" >> "$GITHUB_OUTPUT"
+            echo "tags=${IMAGE}:ci-validation" >> "$GITHUB_OUTPUT"
+          fi
+
+      - uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8 # v6
+        with:
+          context: .
+          platforms: linux/amd64,linux/arm64
+          push: ${{ steps.meta.outputs.push }}
+          tags: ${{ steps.meta.outputs.tags }}
+          provenance: false
+          sbom: false
+
+      # Read the published index back rather than trusting the push's exit
+      # code, and fail if either architecture is absent — a single-arch image
+      # under this tag would CrashLoop on half the nodes.
+      - name: Verify both architectures were published
+        if: ${{ steps.meta.outputs.push == 'true' }}
+        run: |
+          set -euo pipefail
+          out=$(docker buildx imagetools inspect "${IMAGE}:${{ inputs.tag }}")
+          echo "$out"
+          for want in amd64 arm64; do
+            echo "$out" | grep -q "linux/$want" || { echo "::error::published image is missing linux/$want"; exit 1; }
+          done
+          echo "✓ ${IMAGE}:${{ inputs.tag }} carries linux/amd64 and linux/arm64"

Dockerfile

@@ -6,21 +6,33 @@
 # The runner pod is non-root with all caps dropped, so it cannot apt/apk-add at
 # runtime -- the tools must be baked in here.
 #
-# Build multi-arch (cluster is mixed amd64/arm64) and push to the Gitea registry:
+# PUBLISHING IS AUTOMATED. Run the "build image" workflow on GitHub with a tag
+# (e.g. 12.11.1-ci4); it builds both architectures and pushes to
+# ghcr.io/zegit-zoo/zegit-forgejo-runner. ./push-image.sh remains for building
+# by hand. README.md explains why the registry moved off Gitea (Cloudflare's
+# request-body cap returns 413 on a registry layer) and why the build cannot run
+# on our own runners (host mode, all capabilities dropped).
 #
-#   docker buildx create --use --name zegit-ci 2>/dev/null || true
-#   docker login gitea.langenoja.se        # username + a Gitea personal access token
-#   docker buildx build --platform linux/amd64,linux/arm64 \
-#     -t gitea.langenoja.se/zegit-zoo/zegit-forgejo-runner:12.11.1-ci --push .
+# Then in flux-config set the image on BOTH deployment-zegit-runner and
+# deployment-forgejo-runner, digest-pinned.
 #
-# Then in flux-config set both containers' image: to the tag above and commit.
-# Package visibility is PUBLIC, so the runner namespace pulls without a secret.
+# The image is NOT public. Pulls need a registry credential in
+# forgejo-runner-system -- github-registry-credentials for ghcr.io. An earlier
+# version of this comment claimed the package was public and needed no secret;
+# that was wrong for the Gitea registry too, which is private.
 FROM code.forgejo.org/forgejo/runner:12.11.1
 
 USER root
 
 # nodejs   -> JS actions runtime (actions/checkout, actions/setup-go, ...)
 # git      -> checkout
+# git-daemon -> provides git-http-backend, which Alpine splits OUT of the `git`
+#             package (git alone ships git-remote-http/https but NOT the CGI
+#             backend). Without it, internal/forgeprobe's smart-HTTP tests died
+#             with "fork/exec git-http-backend: no such file or directory" and
+#             reported three FAILURES for a missing tool — which turned Gitea CI
+#             red on every zegit master commit while GitHub stayed green, for
+#             hours, unnoticed.
 # bash     -> `shell: bash` steps and many action entrypoints
 # coreutils-> GNU userland (alpine busybox tools trip up some scripts)
 # curl,tar -> setup-go / gitleaks download + extract their own arch-specific binaries
@@ -32,6 +44,7 @@ USER root
 RUN apk add --no-cache \
       nodejs \
       git \
+      git-daemon \
       bash \
       coreutils \
       curl \

README.md

@@ -37,3 +37,60 @@ reconciles). Do this only *after* the image exists, or the pod ImagePullBackOffs
 
 `<base-runner-version>-ci` (e.g. `12.11.1-ci`). Must satisfy the cluster's
 Gatekeeper `enforce-semantic-image-tags` constraint — never `:latest`.
+
+## Publishing a new image
+
+    ./push-image.sh 12.11.1-ci4
+
+The registry hostname in the image reference must stay `gitea.langenoja.se`,
+because that is what the cluster pulls. The script therefore keeps the NAME and
+changes only the ROUTE: it pins that hostname to the Istio gateway on the LAN
+(`10.40.0.10`) for the duration of the push, then removes the pin.
+
+That matters because `gitea.langenoja.se` resolves to Cloudflare, which caps
+request bodies below a registry layer and returns **413 Payload Too Large**. The
+gateway serves a `*.langenoja.se` wildcard cert, so TLS still validates normally
+— no `insecure-registries`, no `--tls-verify=false`.
+
+If a run is killed before it can clean up, the pin is left behind. It carries a
+marker comment, so the next run recognises it as its own and removes it — or you
+can clear it directly:
+
+    ./push-image.sh --unpin
+
+A pin you added yourself (no marker) is never touched, and the script refuses to
+run if an unmarked pin points somewhere other than the gateway.
+
+Requirements: a Gitea PAT with **`write:package`** (`read:package` is pull-only),
+LAN reachability of the gateway, and qemu/binfmt registered for the arm64 build.
+
+Multi-arch is not optional. `forgejo-runner` runs on arm64 nodes and
+`zegit-runner` on amd64, neither deployment sets a `nodeSelector`, and both use
+this single tag.
+
+Two traps the script exists to prevent: pushing to the bare MagicDNS name
+(`gitea-system-gitea-tailscale-http`) makes Docker treat it as a **Docker Hub
+namespace**, since a registry host is only recognised when it contains a dot or
+colon; and `docker push $_` pushes `.`, because `$_` is the last ARGUMENT of the
+build command, not its tag.
+
+### Automated builds
+
+`.github/workflows/build-image.yml` builds and publishes to
+**ghcr.io/zegit-zoo/zegit-forgejo-runner** on GitHub.
+
+Run it from the Actions tab (**build image** → Run workflow) with the tag, e.g.
+`12.11.1-ci4`. A push to `master` that touches the Dockerfile only VALIDATES
+that both architectures still build — it publishes nothing, because guessing a
+tag from a branch push would silently mint versions.
+
+It builds on GitHub rather than our own Gitea Actions because the runners are
+host mode (`ubuntu-latest:host`, no job containers) and drop ALL capabilities
+with `allowPrivilegeEscalation: false`; rootless buildah needs user namespaces
+for its RUN steps, so building there would mean weakening that posture. It
+publishes to ghcr.io rather than the Gitea registry because
+`gitea.langenoja.se` sits behind Cloudflare, whose body cap returns **413** on a
+registry layer — and the cluster already pulls zegit's own images from ghcr.io
+with an existing `github-registry-credentials` pull secret.
+
+`push-image.sh` remains for building by hand when you need it.

push-image.sh

@@ -0,0 +1,120 @@
+#!/usr/bin/env bash
+# push-image.sh — build and publish the runner toolchain image, multi-arch.
+#
+# Why this script exists: doing it by hand hit four obstacles in a row.
+#
+#   1. 413 Payload Too Large. gitea.langenoja.se is behind Cloudflare, which
+#      caps request bodies well below a registry layer. Fixed by pinning the
+#      hostname to the Istio gateway on the LAN (below) so the push never
+#      traverses Cloudflare.
+#   2. Docker only treats a name as a registry HOST if it contains a dot or a
+#      colon. Pushing to the bare MagicDNS name `gitea-system-gitea-tailscale-http`
+#      was parsed as a Docker Hub NAMESPACE, so Docker Hub refused the auth.
+#   3. The Gitea PAT needs `write:package`. `read:package` is pull-only.
+#   4. Multi-arch is required: forgejo-runner runs on arm64 nodes and
+#      zegit-runner on amd64, neither deployment sets a nodeSelector, and both
+#      use this one tag.
+#
+# The hostname MUST stay gitea.langenoja.se in the image reference, because that
+# is what the cluster pulls. So we keep the name and change only the ROUTE, via
+# an /etc/hosts pin that is removed again on exit.
+#
+# The Istio gateway serves a *.langenoja.se wildcard cert, so TLS validates
+# normally — no insecure-registries, no --tls-verify=false.
+set -euo pipefail
+
+TAG="${1:-}"
+GATEWAY_IP="${GATEWAY_IP:-10.40.0.10}"
+REGISTRY="gitea.langenoja.se"
+REPO="$REGISTRY/zegit-zoo/zegit-forgejo-runner"
+MARKER="# added by push-image.sh"
+HOSTS_LINE="$GATEWAY_IP  $REGISTRY  $MARKER"
+
+die() { echo "error: $*" >&2; exit 1; }
+
+if [ "$TAG" = "--unpin" ]; then
+  MARKER="# added by push-image.sh"
+  if grep -qF "$MARKER" /etc/hosts 2>/dev/null; then
+    sudo sed -i "\|${MARKER}\$|d" /etc/hosts; echo "removed the leftover pin"
+  else
+    echo "no pin of ours in /etc/hosts; nothing to do"
+  fi
+  exit 0
+fi
+[ -n "$TAG" ] || die "usage: $0 <tag> | --unpin   (e.g. $0 12.11.1-ci4)"
+command -v docker >/dev/null || die "docker not on PATH"
+
+# Three cases, and they must not be confused:
+#   OURS      — a line carrying our marker. A LEFTOVER from a run that died
+#               before its trap fired. Reuse it, and DO remove it on exit.
+#   THEIRS    — a line without our marker. Deliberate; never touch it, and
+#               refuse if it points somewhere other than the gateway.
+#   ABSENT    — add ours.
+# The earlier version could not tell OURS from THEIRS, so a leftover pin was
+# treated as deliberate and survived every subsequent run. That is the bug that
+# made an operator revert /etc/hosts by hand.
+PIN_OWNED=0
+if grep -qF "$MARKER" /etc/hosts 2>/dev/null; then
+  PIN_OWNED=1
+  echo "found a leftover pin from an interrupted run; reusing and will clean it up"
+elif grep -qE "^[^#]*[[:space:]]$REGISTRY([[:space:]]|\$)" /etc/hosts 2>/dev/null; then
+  existing=$(grep -E "^[^#]*[[:space:]]$REGISTRY([[:space:]]|\$)" /etc/hosts | head -1)
+  [ "$(echo "$existing" | awk '{print $1}')" = "$GATEWAY_IP" ] \
+    || die "/etc/hosts already pins $REGISTRY elsewhere (not ours, not the gateway): $existing"
+  echo "an unmarked pin to the gateway already exists; leaving it exactly as found"
+fi
+
+# Removes ONLY lines carrying our marker, so a deliberate pin is never touched.
+unpin() {
+  if grep -qF "$MARKER" /etc/hosts 2>/dev/null; then
+    sudo sed -i "\|${MARKER}\$|d" /etc/hosts
+    echo "removed the /etc/hosts pin"
+  fi
+}
+cleanup() { [ "$PIN_OWNED" = "1" ] && unpin; }
+# INT and TERM too: EXIT alone leaves the pin behind on Ctrl-C in some shells.
+trap cleanup EXIT INT TERM
+
+if [ "$PIN_OWNED" = "0" ] && ! grep -qE "^[^#]*[[:space:]]$REGISTRY([[:space:]]|\$)" /etc/hosts 2>/dev/null; then
+  echo "pinning $REGISTRY -> $GATEWAY_IP (bypasses Cloudflare; removed on exit)"
+  echo "$HOSTS_LINE" | sudo tee -a /etc/hosts >/dev/null
+  PIN_OWNED=1
+fi
+
+# Prove the route before building anything: 401 means the registry answered and
+# wants auth, which is success here. A timeout means the pin or the LAN is wrong.
+code=$(curl -sS -o /dev/null -w '%{http_code}' --max-time 20 "https://$REGISTRY/v2/" || true)
+case "$code" in
+  401|200) echo "registry reachable via the pin (HTTP $code)" ;;
+  *) die "registry not reachable via $GATEWAY_IP (HTTP ${code:-timeout}). Is this host on the LAN?" ;;
+esac
+
+docker login "$REGISTRY" || die "login failed — the PAT needs write:package, not read:package"
+
+for arch in amd64 arm64; do
+  echo "building linux/$arch"
+  docker build --platform "linux/$arch" -t "$REPO:$TAG-$arch" . \
+    || die "build failed for $arch (arm64 needs qemu/binfmt registered)"
+  # Explicit tag, never $_ — in zsh $_ is the last ARGUMENT, which here is '.'
+  docker push "$REPO:$TAG-$arch" || die "push failed for $arch"
+done
+
+docker manifest rm "$REPO:$TAG" >/dev/null 2>&1 || true
+docker manifest create "$REPO:$TAG" --amend "$REPO:$TAG-amd64" --amend "$REPO:$TAG-arm64"
+docker manifest push "$REPO:$TAG"
+
+# Verify by reading it back, rather than trusting the push's exit code.
+echo "verifying the published manifest lists both architectures"
+got=$(docker manifest inspect "$REPO:$TAG" | grep -o '"architecture": *"[a-z0-9]*"' | sort -u | wc -l)
+[ "$got" -ge 2 ] || die "published manifest does not list two architectures"
+docker manifest inspect "$REPO:$TAG" | grep -o '"architecture": *"[a-z0-9]*"' | sort -u
+
+cat <<EOF
+
+published $REPO:$TAG
+
+Next, in flux-config, bump BOTH deployments that use this image:
+  applications/forgejo-runner/base/deployment-zegit-runner.yaml
+  applications/forgejo-runner/base/deployment-forgejo-runner.yaml
+(markus-runner uses markus/gitea-runner and is unaffected.)
+EOF