fix: compose the manifest with imagetools, not docker manifest

A per-arch push came back as "is a manifest list". Since Buildx 0.10, provenance and SBOM attestations are ON by default, so even a SINGLE-platform build publishes an OCI index (image + attestation) rather than a plain manifest — and `docker manifest create` refuses to nest an index inside an index. Two changes, either of which alone would fix it, kept together because they fail in different directions: --provenance=false --sbom=false keeps each per-arch push a plain manifest. Probed for support first, so an older docker that lacks the flags still works. imagetools create composes correctly whether the sources are plain manifests OR indexes, so attestations reappearing later cannot break it again. Falls back to docker manifest when buildx is unavailable, with the actionable hint in the failure message. Verification now accepts either inspector's output and asserts both platforms are present by name.
2026-08-20 14:28:18 +02:00
5 changed files with 45 additions and 436 deletions
@@ -1,174 +0,0 @@
-name: build image (gitea)
-
-# Publishes the runner toolchain image from the SELF-HOSTED Gitea runners, via
-# the in-cluster rootless buildkitd — the same path zegit's own images already
-# use (zegit .github/workflows/image-gitea.yml).
-#
-# Why this exists alongside build-image.yml: that workflow is GitHub-only, so
-# the image the Gitea runners are BUILT FROM could only be produced by GitHub.
-# When GitHub Actions became unavailable on 2026-08-29 that circularity bit —
-# the fix for a missing tool in this very image (openssl/jq, whose absence
-# blocked v0.0.134 and v0.0.135) could not be published on the forge the project
-# is moving to. This removes the dependency.
-#
-# build-image.yml's header says the runners cannot build images because rootless
-# buildah needs user namespaces the pod does not have. That is true of buildah
-# IN THE JOB, and irrelevant here: buildkitd is its own hardened workload in
-# buildkit-system, and this job only speaks to it over the network with
-# buildctl. Nothing is built inside the runner container.
-
-on:
-  workflow_dispatch:
-    inputs:
-      tag:
-        description: 'Image tag, e.g. 12.11.1-ci6 (upstream runner version + ci revision)'
-        required: true
-        type: string
-  push:
-    branches: [master]
-    paths:
-      - 'Dockerfile'
-      - '.github/workflows/build-image-gitea.yml'
-
-env:
-  BUILDKIT_HOST: tcp://buildkitd.buildkit-system.svc.cluster.local:1234
-  BUILDKIT_VERSION: v0.32.2
-  REGISTRY: gitea.langenoja.se
-  IMAGE_PATH: zegit-zoo/zegit-forgejo-runner
-
-jobs:
-  build:
-    # Gitea-only: buildkitd is an in-cluster service that GitHub runners cannot
-    # reach. build-image.yml still covers the GitHub/ghcr.io side.
-    if: ${{ github.server_url != 'https://github.com' }}
-    runs-on: ubuntu-latest
-    timeout-minutes: 45
-    steps:
-      - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
-        with:
-          persist-credentials: false
-
-      # Workspace-local because the runner is non-root and cannot write to
-      # /usr/local/bin. The smoke test uses the FULL PATH deliberately:
-      # $GITHUB_PATH only takes effect for SUBSEQUENT steps, so a bare
-      # `buildctl --version` here would be command-not-found.
-      - name: Install buildctl
-        run: |
-          set -euo pipefail
-          case "$(uname -m)" in
-            x86_64) arch=amd64;;
-            aarch64) arch=arm64;;
-            *) echo "::error::unsupported arch $(uname -m)"; exit 1;;
-          esac
-          mkdir -p "$GITHUB_WORKSPACE/.bin"
-          curl -fsSL "https://github.com/moby/buildkit/releases/download/${BUILDKIT_VERSION}/buildkit-${BUILDKIT_VERSION}.linux-${arch}.tar.gz" \
-            | tar -xz -C "$GITHUB_WORKSPACE/.bin" --strip-components=1 bin/buildctl
-          echo "$GITHUB_WORKSPACE/.bin" >> "$GITHUB_PATH"
-          "$GITHUB_WORKSPACE/.bin/buildctl" --version
-
-      - name: buildkitd reachable
-        run: buildctl debug workers
-
-      # A master push VALIDATES that the Dockerfile still builds on both arches
-      # but publishes nothing; only an explicit dispatch names a tag and pushes.
-      # Guessing a tag from a branch push would silently mint versions.
-      - id: meta
-        run: |
-          set -euo pipefail
-          if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
-            echo "push=true"  >> "$GITHUB_OUTPUT"
-            echo "tag=${{ inputs.tag }}" >> "$GITHUB_OUTPUT"
-          else
-            echo "push=false" >> "$GITHUB_OUTPUT"
-            echo "tag=ci-validation" >> "$GITHUB_OUTPUT"
-          fi
-
-      # Refuse a publish that cannot complete, rather than reporting success
-      # having pushed nothing. A validation run needs no credentials.
-      # The Gitea Actions token cannot write packages; this needs an explicit
-      # package-write PAT (read:package is pull-only).
-      - name: Registry login
-        if: ${{ steps.meta.outputs.push == 'true' }}
-        env:
-          TOKEN: ${{ secrets.ZEGIT_CI_PACKAGE_TOKEN }}
-        run: |
-          set -euo pipefail
-          if [ -z "${TOKEN:-}" ]; then
-            echo "::error::ZEGIT_CI_PACKAGE_TOKEN is not visible to this run. It is the org-scoped package-write token every zegit-zoo image workflow uses (docs: flux-config docs/rotation/forge-secrets.yaml); this repository must be able to read the org secret, and the name must match."
-            exit 1
-          fi
-          mkdir -p "$HOME/.docker"
-          printf '{"auths":{"%s":{"auth":"%s"}}}\n' \
-            "$REGISTRY" "$(printf 'zegit-zoo:%s' "$TOKEN" | base64 -w0)" \
-            > "$HOME/.docker/config.json"
-
-      - name: Build multi-arch
-        run: |
-          set -euo pipefail
-          REF="${REGISTRY}/${IMAGE_PATH}:${{ steps.meta.outputs.tag }}"
-          # No --output on a validation run: buildctl v0.32 has no "cacheonly"
-          # exporter (run 16, 2026-09-08: 'exporter "cacheonly" could not be
-          # found'), so every push-triggered validation since this workflow
-          # was added had failed — silently, since nothing depended on it. A
-          # build without an exporter still builds every stage on both arches.
-          if [ "${{ steps.meta.outputs.push }}" = "true" ]; then
-            buildctl build --progress=plain \
-              --frontend dockerfile.v0 \
-              --local context=. \
-              --local dockerfile=. \
-              --opt platform=linux/amd64,linux/arm64 \
-              --output "type=image,name=${REF},push=true"
-          else
-            buildctl build --progress=plain \
-              --frontend dockerfile.v0 \
-              --local context=. \
-              --local dockerfile=. \
-              --opt platform=linux/amd64,linux/arm64
-          fi
-
-      # The image is a TOOLCHAIN, so "it built" is not the property that matters
-      # — "it still contains what the gates shell out to" is. build-image.yml
-      # asserts that with `docker run`, which does not exist on these runners;
-      # buildkit can do it directly by building a stage FROM the published image
-      # whose RUN fails when a tool is absent. That is strictly better: buildkit
-      # runs it natively per platform, so a package that resolves on amd64 but
-      # not on arm64 cannot reach half the nodes unnoticed.
-      #
-      # openssl and jq are on this list because their absence blocked v0.0.134
-      # and v0.0.135: the release tag gate shells out to both to verify the
-      # Control Plane's signed quorum verdict, and a missing tool was reported
-      # as "verdict not signed" and "signature FAILED verification" — a tooling
-      # gap narrated as a security verdict. Extend this list whenever a gate
-      # grows a shell dependency; it only covers what someone remembered to add.
-      - name: Assert the published image carries the toolchain the gates need
-        if: ${{ steps.meta.outputs.push == 'true' }}
-        run: |
-          set -euo pipefail
-          REF="${REGISTRY}/${IMAGE_PATH}:${{ steps.meta.outputs.tag }}"
-          work=$(mktemp -d)
-          {
-            echo "FROM ${REF}"
-            echo 'RUN set -eu; \'
-            echo '    python3 -c "import yaml, sys; print(\"python3+yaml\", sys.version.split()[0])"; \'
-            echo '    for t in gcc node bash make curl tar openssl jq chromium-browser psql; do \'
-            echo '      command -v "$t" >/dev/null || { echo "MISSING: $t" >&2; exit 1; }; \'
-            echo '      echo "ok: $t"; \'
-            echo '    done; \'
-            echo '    test -x "$(git --exec-path)/git-http-backend" || { echo "MISSING: git-http-backend" >&2; exit 1; }; \'
-            echo '    echo "ok: git-http-backend"'
-          } > "$work/Dockerfile"
-          cat "$work/Dockerfile"
-          buildctl build --progress=plain \
-            --frontend dockerfile.v0 \
-            --local context="$work" \
-            --local dockerfile="$work" \
-            --opt platform=linux/amd64,linux/arm64
-          echo "✓ toolchain intact on linux/amd64 and linux/arm64"
-
-      - name: Summary
-        run: |
-          if [ "${{ steps.meta.outputs.push }}" = "true" ]; then
-            echo "published ${REGISTRY}/${IMAGE_PATH}:${{ steps.meta.outputs.tag }} (amd64+arm64), toolchain asserted on both"
-          else
-            echo "validation only — the Dockerfile builds on both arches; nothing published"
-          fi
@@ -1,168 +0,0 @@
-name: build image
-
-# Publishes the runner toolchain image to ghcr.io.
-#
-# Why GitHub rather than our own Gitea Actions: the runners are host mode
-# (`ubuntu-latest:host`, no job containers) and their container drops ALL
-# capabilities with allowPrivilegeEscalation:false. Rootless buildah needs
-# newuidmap/user namespaces for its RUN steps, so building images there would
-# mean weakening the posture that hardening established. GitHub runners have
-# docker + buildx + QEMU already.
-#
-# Why ghcr.io rather than the Gitea registry: gitea.langenoja.se sits behind
-# Cloudflare, whose request-body cap returns 413 on a registry layer. The
-# cluster already pulls zegit's own images from ghcr.io and the runner
-# deployments already carry a github-registry-credentials pull secret, so this
-# reuses an established, trusted path instead of inventing one.
-
-on:
-  workflow_dispatch:
-    inputs:
-      tag:
-        description: 'Image tag, e.g. 12.11.1-ci4 (upstream runner version + ci revision)'
-        required: true
-        type: string
-  push:
-    branches: [master]
-    paths:
-      - 'Dockerfile'
-      - '.github/workflows/build-image.yml'
-
-permissions:
-  contents: read
-  packages: write
-
-env:
-  IMAGE: ghcr.io/${{ github.repository }}
-
-jobs:
-  build:
-    # GitHub-only: the Gitea mirror has no ghcr.io and no GITHUB_TOKEN.
-    if: ${{ github.server_url == 'https://github.com' }}
-    runs-on: ubuntu-latest
-    timeout-minutes: 45
-    steps:
-      - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
-
-      # arm64 is not optional: forgejo-runner runs on arm64 nodes and
-      # zegit-runner on amd64, neither deployment sets a nodeSelector, and both
-      # use this one tag.
-      - uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3
-      - uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
-
-      - uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
-        with:
-          registry: ghcr.io
-          username: ${{ github.actor }}
-          password: ${{ secrets.GITHUB_TOKEN }}
-
-      # A master push VALIDATES the Dockerfile still builds on both arches but
-      # publishes nothing — only an explicit dispatch names a tag and pushes.
-      # Guessing a tag from a branch push would silently mint versions.
-      - id: meta
-        run: |
-          if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
-            echo "push=true" >> "$GITHUB_OUTPUT"
-            echo "tags=${IMAGE}:${{ inputs.tag }}" >> "$GITHUB_OUTPUT"
-          else
-            echo "push=false" >> "$GITHUB_OUTPUT"
-            echo "tags=${IMAGE}:ci-validation" >> "$GITHUB_OUTPUT"
-          fi
-
-      - uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8 # v6
-        with:
-          context: .
-          platforms: linux/amd64,linux/arm64
-          push: ${{ steps.meta.outputs.push }}
-          tags: ${{ steps.meta.outputs.tags }}
-          provenance: false
-          sbom: false
-
-      # Read the published index back rather than trusting the push's exit
-      # code, and fail if either architecture is absent — a single-arch image
-      # under this tag would CrashLoop on half the nodes.
-      - name: Verify both architectures were published
-        if: ${{ steps.meta.outputs.push == 'true' }}
-        run: |
-          set -euo pipefail
-          out=$(docker buildx imagetools inspect "${IMAGE}:${{ inputs.tag }}")
-          echo "$out"
-          for want in amd64 arm64; do
-            echo "$out" | grep -q "linux/$want" || { echo "::error::published image is missing linux/$want"; exit 1; }
-          done
-          echo "✓ ${IMAGE}:${{ inputs.tag }} carries linux/amd64 and linux/arm64"
-
-      # The image is a TOOLCHAIN, so "it built" is not the property that matters
-      # — "it still contains what the gates shell out to" is. This exists because
-      # a hand-built image had drifted from this Dockerfile: the deployed ci3
-      # carried gcc, musl-dev, python3 and py3-yaml, the Dockerfile did not, and
-      # rebuilding from the Dockerfile silently dropped them. CI runs then failed
-      # at "credentials-coverage: need python3" instead of running the tests.
-      #
-      # Asserted on BOTH architectures, because a package that resolves on amd64
-      # and not on arm64 would otherwise reach half the nodes unnoticed.
-      #
-      # openssl and jq were added after this check already existed and still
-      # missed them: the release tag gate shells out to both to verify the
-      # Control Plane's signed quorum verdict, and their absence surfaced as
-      # "verdict not signed" / "signature FAILED verification" rather than as a
-      # missing tool, blocking v0.0.134 and v0.0.135. The list only covers what
-      # someone remembered to add, so add to it whenever a gate grows a new
-      # shell dependency.
-      #
-      # DERIVED once, 2026-08-31, rather than remembered: every command invoked
-      # from a `run:` block across zegit's Gitea-executed workflows and its
-      # composite actions was extracted and checked against this list. Result:
-      # no missing apk package. Everything resolved to one of three sources —
-      # baked in here; provided by busybox/coreutils; or downloaded per-run and
-      # immediately version-checked (go via setup-go, buildctl, flux, cosign,
-      # syft, goreleaser). The download path needs nothing from this image
-      # beyond curl and tar, both already asserted.
-      #
-      # What the audit DID find is that the busybox/coreutils layer was entirely
-      # unasserted, so dropping `coreutils` from the apk line above would pass
-      # this check while base64, mktemp, seq and sort silently vanished — and by
-      # the history above, their absence would be narrated as whatever the gate
-      # using them happened to be doing. They are asserted below now. They are
-      # cheap to check and the failure they prevent is expensive to read.
-      - name: Verify the image still contains the toolchain the gates need
-        if: ${{ steps.meta.outputs.push == 'true' }}
-        run: |
-          set -euo pipefail
-          REF="${IMAGE}:${{ inputs.tag }}"
-          fail=0
-          for plat in linux/amd64 linux/arm64; do
-            echo "── $plat"
-            out=$(docker run --rm --platform "$plat" --entrypoint sh "$REF" -c '
-              set -u
-              py=$(python3 -c "import yaml,sys;print(sys.version.split()[0])" 2>/dev/null) \
-                && echo "python3+yaml=$py" || echo "python3+yaml=MISSING"
-              command -v gcc      >/dev/null && echo "gcc=ok"      || echo "gcc=MISSING"
-              command -v node     >/dev/null && echo "node=ok"     || echo "node=MISSING"
-              command -v bash     >/dev/null && echo "bash=ok"     || echo "bash=MISSING"
-              command -v make     >/dev/null && echo "make=ok"     || echo "make=MISSING"
-              command -v curl     >/dev/null && echo "curl=ok"     || echo "curl=MISSING"
-              command -v tar      >/dev/null && echo "tar=ok"      || echo "tar=MISSING"
-              command -v openssl  >/dev/null && echo "openssl=ok"  || echo "openssl=MISSING"
-              command -v jq       >/dev/null && echo "jq=ok"       || echo "jq=MISSING"
-              # busybox/coreutils layer: used across zegit's gates for base64
-              # decoding of DSSE payloads, temp dirs, poll loops and sorting.
-              for t in base64 mktemp seq sort sed grep cut head wc uname date; do
-                command -v "$t" >/dev/null && echo "$t=ok" || echo "$t=MISSING"
-              done
-              [ -x "$(git --exec-path)/git-http-backend" ] \
-                && echo "git-http-backend=ok" || echo "git-http-backend=MISSING"
-            ' 2>&1) || { echo "::error::could not run $REF on $plat"; fail=1; continue; }
-            echo "$out"
-            if echo "$out" | grep -q MISSING; then
-              echo "$out" | grep MISSING | while read -r l; do
-                echo "::error::$plat is missing a required tool: $l"
-              done
-              fail=1
-            fi
-          done
-          [ "$fail" = 0 ] || {
-            echo "::error::The published image lost a tool the repo's gates depend on. Restore it in the Dockerfile — do not weaken this check."
-            exit 1
-          }
-          echo "✓ toolchain intact on both architectures"
@@ -6,20 +6,15 @@
 # The runner pod is non-root with all caps dropped, so it cannot apt/apk-add at
 # runtime -- the tools must be baked in here.
 #
-# PUBLISHING IS AUTOMATED. Run the "build image" workflow on GitHub with a tag
-# (e.g. 12.11.1-ci4); it builds both architectures and pushes to
-# ghcr.io/zegit-zoo/zegit-forgejo-runner. ./push-image.sh remains for building
-# by hand. README.md explains why the registry moved off Gitea (Cloudflare's
-# request-body cap returns 413 on a registry layer) and why the build cannot run
-# on our own runners (host mode, all capabilities dropped).
+# Build multi-arch (cluster is mixed amd64/arm64) and push to the Gitea registry:
 #
-# Then in flux-config set the image on BOTH deployment-zegit-runner and
-# deployment-forgejo-runner, digest-pinned.
+#   docker buildx create --use --name zegit-ci 2>/dev/null || true
+#   docker login gitea.langenoja.se        # username + a Gitea personal access token
+#   docker buildx build --platform linux/amd64,linux/arm64 \
+#     -t gitea.langenoja.se/zegit-zoo/zegit-forgejo-runner:12.11.1-ci --push .
 #
-# The image is NOT public. Pulls need a registry credential in
-# forgejo-runner-system -- github-registry-credentials for ghcr.io. An earlier
-# version of this comment claimed the package was public and needed no secret;
-# that was wrong for the Gitea registry too, which is private.
+# Then in flux-config set both containers' image: to the tag above and commit.
+# Package visibility is PUBLIC, so the runner namespace pulls without a secret.
 FROM code.forgejo.org/forgejo/runner:12.11.1

 USER root
@@ -38,38 +33,6 @@ USER root
 # curl,tar -> setup-go / gitleaks download + extract their own arch-specific binaries
 # make     -> zegit Makefile targets
 # ca-certificates -> trusted roots for HTTPS to gitea/github/goproxy
-# openssl  -> the release tag gate verifies the Control Plane's SIGNED quorum
-#             verdict (Ed25519 DSSE) with `openssl pkeyutl -verify -rawin`.
-#             Absent, that command failed "command not found" and the gate's
-#             `|| { echo ...; exit 1; }` reported it as "Quorum verdict
-#             signature FAILED verification" -- a TOOLING gap narrated as a
-#             SECURITY verdict, pointing the investigation at a perfectly
-#             valid signed approval instead of at this image. It blocked
-#             v0.0.134 and v0.0.135 on 2026-08-31.
-# jq       -> the same story one step earlier: the gate parsed that envelope
-#             with `jq -e`, and a missing jq was reported as "not a signed
-#             DSSE envelope". jq is the reflex tool for JSON in shell steps,
-#             so its absence keeps resurfacing as a misleading failure in
-#             whichever step reaches for it next.
-# gcc,musl-dev -> C toolchain for cgo; `go test -race` (the Makefile test
-#             target) requires cgo, which needs a C compiler + libc headers
-#             (musl on Alpine)
-# python3,py3-yaml -> the repo's own gates are Python: credentials-coverage.sh,
-#             validate-admission.sh and feature-lint.sh all shell out to
-#             python3, and credentials-coverage parses manifests with yaml
-#
-# chromium, font-noto -> the browser gates (2026-09-08): zegit's `browser
-#             tests` (chromedp + a CDP virtual authenticator over the WebAuthn
-#             ceremony pages) and `lighthouse` (treosh/lighthouse-ci-action
-#             via chrome-launcher) were github.com-only jobs, so when `ci`
-#             was routed to this runner on 2026-08-31 they stopped running
-#             anywhere. chromedp finds `chromium`/`chromium-browser` on PATH;
-#             chrome-launcher honours CHROME_PATH. font-noto so text renders
-#             (an empty font set makes layout/contrast audits meaningless).
-# postgresql-client -> `psql`, so the lighthouse job can CREATE/DROP its own
-#             database on the runner's ci-postgres (ZEGIT_TESTPG_DSN) instead
-#             of migrating the shared maintenance database from under the
-#             integration suite's template.
 #
 # Go is intentionally NOT baked in: actions/setup-go fetches the pinned version
 # per-run (node + tar are present), so the image needs no rebuild on Go bumps.
@@ -82,16 +45,7 @@ RUN apk add --no-cache \
      curl \
      tar \
      make \
-      ca-certificates \
-      gcc \
-      musl-dev \
-      python3 \
-      py3-yaml \
-      openssl \
-      jq \
-      chromium \
-      font-noto \
-      postgresql-client
+      ca-certificates

 # Match the runner pod's runAsUser (1000); keep the image non-root by default.
 USER 1000
@@ -61,6 +61,13 @@ can clear it directly:
 A pin you added yourself (no marker) is never touched, and the script refuses to
 run if an unmarked pin points somewhere other than the gateway.

+If a per-arch push reports *"is a manifest list"*, that is BuildKit's provenance
+attestation: since Buildx 0.10 it is on by default, so even a single-platform
+build publishes an OCI **index** rather than a plain manifest, and
+`docker manifest create` refuses to nest one. The script builds with
+`--provenance=false --sbom=false` and composes with `docker buildx imagetools
+create`, which handles either shape.
+
 Requirements: a Gitea PAT with **`write:package`** (`read:package` is pull-only),
 LAN reachability of the gateway, and qemu/binfmt registered for the arm64 build.

@@ -73,24 +80,3 @@ Two traps the script exists to prevent: pushing to the bare MagicDNS name
 namespace**, since a registry host is only recognised when it contains a dot or
 colon; and `docker push $_` pushes `.`, because `$_` is the last ARGUMENT of the
 build command, not its tag.
-
-### Automated builds
-
-`.github/workflows/build-image.yml` builds and publishes to
-**ghcr.io/zegit-zoo/zegit-forgejo-runner** on GitHub.
-
-Run it from the Actions tab (**build image** → Run workflow) with the tag, e.g.
-`12.11.1-ci4`. A push to `master` that touches the Dockerfile only VALIDATES
-that both architectures still build — it publishes nothing, because guessing a
-tag from a branch push would silently mint versions.
-
-It builds on GitHub rather than our own Gitea Actions because the runners are
-host mode (`ubuntu-latest:host`, no job containers) and drop ALL capabilities
-with `allowPrivilegeEscalation: false`; rootless buildah needs user namespaces
-for its RUN steps, so building there would mean weakening that posture. It
-publishes to ghcr.io rather than the Gitea registry because
-`gitea.langenoja.se` sits behind Cloudflare, whose body cap returns **413** on a
-registry layer — and the cluster already pulls zegit's own images from ghcr.io
-with an existing `github-registry-credentials` pull secret.
-
-`push-image.sh` remains for building by hand when you need it.
@@ -91,35 +91,46 @@ esac

 docker login "$REGISTRY" || die "login failed — the PAT needs write:package, not read:package"

+# --provenance=false --sbom=false: since Buildx 0.10 attestations are ON by
+# default, so even a SINGLE-platform build publishes an OCI INDEX (image +
+# attestation) instead of a plain manifest. `docker manifest create` then
+# refuses it with "is a manifest list". Suppressing them keeps each arch a
+# plain manifest. Harmless if the flags are unsupported — we probe first.
+PROV_FLAGS=""
+if docker build --help 2>/dev/null | grep -q -- '--provenance'; then
+  PROV_FLAGS="--provenance=false --sbom=false"
+fi
+
 for arch in amd64 arm64; do
  echo "building linux/$arch"
-  docker build --platform "linux/$arch" -t "$REPO:$TAG-$arch" . \
-    || die "build failed for $arch (arm64 needs qemu/binfmt registered)"
+  # shellcheck disable=SC2086  # PROV_FLAGS is deliberately word-split
+  docker build $PROV_FLAGS --platform "linux/$arch" -t "$REPO:$TAG-$arch" . \
+    || die "build failed for $arch (arm64 needs qemu/binfmt: docker run --privileged --rm tonistiigi/binfmt --install arm64)"
  # Explicit tag, never $_ — in zsh $_ is the last ARGUMENT, which here is '.'
  docker push "$REPO:$TAG-$arch" || die "push failed for $arch"
 done

-# `docker manifest create` REFUSES a source that is itself a manifest list:
-#   ERROR: gitea.langenoja.se/...:<tag>-amd64 is a manifest list
-# With Docker's containerd image store, `docker build --platform linux/amd64`
-# produces an OCI index rather than a plain manifest, so both per-arch tags are
-# lists and the combine step fails after the (slow, large) pushes have already
-# succeeded — leaving the per-arch tags published and the plain tag absent,
-# which is the one the Deployments reference. Seen on 12.11.1-ci6, 2026-08-31.
-#
-# `buildx imagetools create` accepts indexes as sources and does the combine
-# registry-side, so it works with either image store.
+# `docker buildx imagetools create` rather than `docker manifest create`: it
+# composes correctly whether the sources are plain manifests OR indexes, so it
+# does not fail with "is a manifest list" when attestations sneak back in. It
+# is a client-side registry operation, so it honours the /etc/hosts pin too.
+if docker buildx imagetools create --help >/dev/null 2>&1; then
  docker buildx imagetools create -t "$REPO:$TAG" "$REPO:$TAG-amd64" "$REPO:$TAG-arm64" \
-  || die "failed to combine the per-arch tags into $REPO:$TAG"
+    || die "imagetools create failed"
+else
+  docker manifest rm "$REPO:$TAG" >/dev/null 2>&1 || true
+  docker manifest create "$REPO:$TAG" --amend "$REPO:$TAG-amd64" --amend "$REPO:$TAG-arm64" \
+    || die "manifest create failed — if it says \"is a manifest list\", rebuild with --provenance=false"
+  docker manifest push "$REPO:$TAG"
+fi

 # Verify by reading it back, rather than trusting the push's exit code.
 echo "verifying the published manifest lists both architectures"
-inspect=$(docker buildx imagetools inspect "$REPO:$TAG") \
-  || die "published $REPO:$TAG cannot be read back"
-got=$(echo "$inspect" | grep -o 'linux/[a-z0-9]*' | sort -u | grep -cE 'linux/(amd64|arm64)')
-[ "$got" -ge 2 ] || die "published manifest does not list both architectures:
-$inspect"
-echo "$inspect" | grep -o 'linux/[a-z0-9]*' | sort -u
+inspect=$(docker buildx imagetools inspect "$REPO:$TAG" 2>/dev/null || docker manifest inspect "$REPO:$TAG")
+for want in amd64 arm64; do
+  echo "$inspect" | grep -q "$want" || die "published manifest is missing linux/$want"
+done
+echo "$inspect" | grep -iE 'platform|architecture' | sort -u | head -6

 cat <<EOF