README.md

@@ -37,3 +37,39 @@ reconciles). Do this only *after* the image exists, or the pod ImagePullBackOffs
 
 `<base-runner-version>-ci` (e.g. `12.11.1-ci`). Must satisfy the cluster's
 Gatekeeper `enforce-semantic-image-tags` constraint — never `:latest`.
+
+## Publishing a new image
+
+    ./push-image.sh 12.11.1-ci4
+
+The registry hostname in the image reference must stay `gitea.langenoja.se`,
+because that is what the cluster pulls. The script therefore keeps the NAME and
+changes only the ROUTE: it pins that hostname to the Istio gateway on the LAN
+(`10.40.0.10`) for the duration of the push, then removes the pin.
+
+That matters because `gitea.langenoja.se` resolves to Cloudflare, which caps
+request bodies below a registry layer and returns **413 Payload Too Large**. The
+gateway serves a `*.langenoja.se` wildcard cert, so TLS still validates normally
+— no `insecure-registries`, no `--tls-verify=false`.
+
+If a run is killed before it can clean up, the pin is left behind. It carries a
+marker comment, so the next run recognises it as its own and removes it — or you
+can clear it directly:
+
+    ./push-image.sh --unpin
+
+A pin you added yourself (no marker) is never touched, and the script refuses to
+run if an unmarked pin points somewhere other than the gateway.
+
+Requirements: a Gitea PAT with **`write:package`** (`read:package` is pull-only),
+LAN reachability of the gateway, and qemu/binfmt registered for the arm64 build.
+
+Multi-arch is not optional. `forgejo-runner` runs on arm64 nodes and
+`zegit-runner` on amd64, neither deployment sets a `nodeSelector`, and both use
+this single tag.
+
+Two traps the script exists to prevent: pushing to the bare MagicDNS name
+(`gitea-system-gitea-tailscale-http`) makes Docker treat it as a **Docker Hub
+namespace**, since a registry host is only recognised when it contains a dot or
+colon; and `docker push $_` pushes `.`, because `$_` is the last ARGUMENT of the
+build command, not its tag.

push-image.sh

@@ -0,0 +1,120 @@
+#!/usr/bin/env bash
+# push-image.sh — build and publish the runner toolchain image, multi-arch.
+#
+# Why this script exists: doing it by hand hit four obstacles in a row.
+#
+#   1. 413 Payload Too Large. gitea.langenoja.se is behind Cloudflare, which
+#      caps request bodies well below a registry layer. Fixed by pinning the
+#      hostname to the Istio gateway on the LAN (below) so the push never
+#      traverses Cloudflare.
+#   2. Docker only treats a name as a registry HOST if it contains a dot or a
+#      colon. Pushing to the bare MagicDNS name `gitea-system-gitea-tailscale-http`
+#      was parsed as a Docker Hub NAMESPACE, so Docker Hub refused the auth.
+#   3. The Gitea PAT needs `write:package`. `read:package` is pull-only.
+#   4. Multi-arch is required: forgejo-runner runs on arm64 nodes and
+#      zegit-runner on amd64, neither deployment sets a nodeSelector, and both
+#      use this one tag.
+#
+# The hostname MUST stay gitea.langenoja.se in the image reference, because that
+# is what the cluster pulls. So we keep the name and change only the ROUTE, via
+# an /etc/hosts pin that is removed again on exit.
+#
+# The Istio gateway serves a *.langenoja.se wildcard cert, so TLS validates
+# normally — no insecure-registries, no --tls-verify=false.
+set -euo pipefail
+
+TAG="${1:-}"
+GATEWAY_IP="${GATEWAY_IP:-10.40.0.10}"
+REGISTRY="gitea.langenoja.se"
+REPO="$REGISTRY/zegit-zoo/zegit-forgejo-runner"
+MARKER="# added by push-image.sh"
+HOSTS_LINE="$GATEWAY_IP  $REGISTRY  $MARKER"
+
+die() { echo "error: $*" >&2; exit 1; }
+
+if [ "$TAG" = "--unpin" ]; then
+  MARKER="# added by push-image.sh"
+  if grep -qF "$MARKER" /etc/hosts 2>/dev/null; then
+    sudo sed -i "\|${MARKER}\$|d" /etc/hosts; echo "removed the leftover pin"
+  else
+    echo "no pin of ours in /etc/hosts; nothing to do"
+  fi
+  exit 0
+fi
+[ -n "$TAG" ] || die "usage: $0 <tag> | --unpin   (e.g. $0 12.11.1-ci4)"
+command -v docker >/dev/null || die "docker not on PATH"
+
+# Three cases, and they must not be confused:
+#   OURS      — a line carrying our marker. A LEFTOVER from a run that died
+#               before its trap fired. Reuse it, and DO remove it on exit.
+#   THEIRS    — a line without our marker. Deliberate; never touch it, and
+#               refuse if it points somewhere other than the gateway.
+#   ABSENT    — add ours.
+# The earlier version could not tell OURS from THEIRS, so a leftover pin was
+# treated as deliberate and survived every subsequent run. That is the bug that
+# made an operator revert /etc/hosts by hand.
+PIN_OWNED=0
+if grep -qF "$MARKER" /etc/hosts 2>/dev/null; then
+  PIN_OWNED=1
+  echo "found a leftover pin from an interrupted run; reusing and will clean it up"
+elif grep -qE "^[^#]*[[:space:]]$REGISTRY([[:space:]]|\$)" /etc/hosts 2>/dev/null; then
+  existing=$(grep -E "^[^#]*[[:space:]]$REGISTRY([[:space:]]|\$)" /etc/hosts | head -1)
+  [ "$(echo "$existing" | awk '{print $1}')" = "$GATEWAY_IP" ] \
+    || die "/etc/hosts already pins $REGISTRY elsewhere (not ours, not the gateway): $existing"
+  echo "an unmarked pin to the gateway already exists; leaving it exactly as found"
+fi
+
+# Removes ONLY lines carrying our marker, so a deliberate pin is never touched.
+unpin() {
+  if grep -qF "$MARKER" /etc/hosts 2>/dev/null; then
+    sudo sed -i "\|${MARKER}\$|d" /etc/hosts
+    echo "removed the /etc/hosts pin"
+  fi
+}
+cleanup() { [ "$PIN_OWNED" = "1" ] && unpin; }
+# INT and TERM too: EXIT alone leaves the pin behind on Ctrl-C in some shells.
+trap cleanup EXIT INT TERM
+
+if [ "$PIN_OWNED" = "0" ] && ! grep -qE "^[^#]*[[:space:]]$REGISTRY([[:space:]]|\$)" /etc/hosts 2>/dev/null; then
+  echo "pinning $REGISTRY -> $GATEWAY_IP (bypasses Cloudflare; removed on exit)"
+  echo "$HOSTS_LINE" | sudo tee -a /etc/hosts >/dev/null
+  PIN_OWNED=1
+fi
+
+# Prove the route before building anything: 401 means the registry answered and
+# wants auth, which is success here. A timeout means the pin or the LAN is wrong.
+code=$(curl -sS -o /dev/null -w '%{http_code}' --max-time 20 "https://$REGISTRY/v2/" || true)
+case "$code" in
+  401|200) echo "registry reachable via the pin (HTTP $code)" ;;
+  *) die "registry not reachable via $GATEWAY_IP (HTTP ${code:-timeout}). Is this host on the LAN?" ;;
+esac
+
+docker login "$REGISTRY" || die "login failed — the PAT needs write:package, not read:package"
+
+for arch in amd64 arm64; do
+  echo "building linux/$arch"
+  docker build --platform "linux/$arch" -t "$REPO:$TAG-$arch" . \
+    || die "build failed for $arch (arm64 needs qemu/binfmt registered)"
+  # Explicit tag, never $_ — in zsh $_ is the last ARGUMENT, which here is '.'
+  docker push "$REPO:$TAG-$arch" || die "push failed for $arch"
+done
+
+docker manifest rm "$REPO:$TAG" >/dev/null 2>&1 || true
+docker manifest create "$REPO:$TAG" --amend "$REPO:$TAG-amd64" --amend "$REPO:$TAG-arm64"
+docker manifest push "$REPO:$TAG"
+
+# Verify by reading it back, rather than trusting the push's exit code.
+echo "verifying the published manifest lists both architectures"
+got=$(docker manifest inspect "$REPO:$TAG" | grep -o '"architecture": *"[a-z0-9]*"' | sort -u | wc -l)
+[ "$got" -ge 2 ] || die "published manifest does not list two architectures"
+docker manifest inspect "$REPO:$TAG" | grep -o '"architecture": *"[a-z0-9]*"' | sort -u
+
+cat <<EOF
+
+published $REPO:$TAG
+
+Next, in flux-config, bump BOTH deployments that use this image:
+  applications/forgejo-runner/base/deployment-zegit-runner.yaml
+  applications/forgejo-runner/base/deployment-forgejo-runner.yaml
+(markus-runner uses markus/gitea-runner and is unaffected.)
+EOF