CI: Kaniko image push fails — 7 consecutive deploy failures since pipeline #415 #199

New issue

Closed

opened 2026-06-12 03:11:38 +00:00 by ldraney · 0 comments

ldraney commented

2026-06-12 03:11:38 +00:00

Owner

Type

Bug

Lineage

Standalone — discovered during production incident triage (stale deploy, 8 commits behind HEAD).

Repo

ldraney/landscaping-assistant

What Broke

Every push-to-main pipeline since #415 fails at the build-and-push step. The Kaniko Docker build completes successfully (all Dockerfile commands execute, cache layers push fine), but the final image push to Harbor fails with HTTP/2 stream errors. Prod is stuck on commit 683c643 while HEAD is at 9ef0184.

Harbor registry logs show: "client disconnected during blob PATCH" error="unexpected EOF" — the Kaniko pod is terminated mid-upload. Two large blobs (66MB, 279MB) get only a few MB uploaded before disconnect.

Harbor core confirms: http: proxy error: readfrom tcp ... unexpected EOF

Contributing dead config: PR #188 added backend_options.kubernetes.volumeMounts referencing a volume bundle-cache, but WOODPECKER_BACKEND_K8S_VOLUMES was never configured on the Woodpecker agent. PVC woodpecker-bundle-cache has been Pending 33+ hours. Silently ignored but should be removed.

Repro Steps

Merge any PR to main
Push-to-main pipeline triggers, runs lint + test successfully
build-and-push step starts Kaniko build — all Dockerfile steps complete (~228s)
Kaniko attempts to push final image to harbor.harbor.svc.cluster.local
Observe: push fails with stream error: stream ID N; INTERNAL_ERROR; received from peer

Expected Behavior

build-and-push step pushes the image to Harbor. ArgoCD image updater detects the new tag and rolls out the deployment. Prod runs the latest main commit.

Environment

Cluster/namespace: prod / landscaping-assistant
Service version/commit: deployed 683c643, HEAD 9ef0184
Woodpecker pipelines affected: #430, #431, #434, #443, #446, #449, #450
Last successful pipeline: #415
Kaniko version: woodpeckerci/plugin-kaniko:2.3.0

Acceptance Criteria

Push-to-main pipeline succeeds end-to-end (build + push + deploy)
ArgoCD image updater picks up new tag and deploys latest main
Dead backend_options volume mount and PVC cleaned up
Pipeline still runs tests against Postgres service
Single-stage Dockerfile optimization preserved (not the cause)

landscaping-assistant project
PR #188 — pipeline optimization that added the dead volume config

### Type Bug ### Lineage Standalone — discovered during production incident triage (stale deploy, 8 commits behind HEAD). ### Repo `ldraney/landscaping-assistant` ### What Broke Every push-to-main pipeline since #415 fails at the `build-and-push` step. The Kaniko Docker build completes successfully (all Dockerfile commands execute, cache layers push fine), but the **final image push to Harbor fails** with HTTP/2 stream errors. Prod is stuck on commit `683c643` while HEAD is at `9ef0184`. Harbor registry logs show: `"client disconnected during blob PATCH" error="unexpected EOF"` — the Kaniko pod is terminated mid-upload. Two large blobs (66MB, 279MB) get only a few MB uploaded before disconnect. Harbor core confirms: `http: proxy error: readfrom tcp ... unexpected EOF` Contributing dead config: PR #188 added `backend_options.kubernetes.volumeMounts` referencing a volume `bundle-cache`, but `WOODPECKER_BACKEND_K8S_VOLUMES` was never configured on the Woodpecker agent. PVC `woodpecker-bundle-cache` has been Pending 33+ hours. Silently ignored but should be removed. ### Repro Steps 1. Merge any PR to main 2. Push-to-main pipeline triggers, runs lint + test successfully 3. `build-and-push` step starts Kaniko build — all Dockerfile steps complete (~228s) 4. Kaniko attempts to push final image to `harbor.harbor.svc.cluster.local` 5. Observe: push fails with `stream error: stream ID N; INTERNAL_ERROR; received from peer` ### Expected Behavior `build-and-push` step pushes the image to Harbor. ArgoCD image updater detects the new tag and rolls out the deployment. Prod runs the latest main commit. ### Environment - Cluster/namespace: prod / `landscaping-assistant` - Service version/commit: deployed `683c643`, HEAD `9ef0184` - Woodpecker pipelines affected: #430, #431, #434, #443, #446, #449, #450 - Last successful pipeline: #415 - Kaniko version: `woodpeckerci/plugin-kaniko:2.3.0` ### Acceptance Criteria - [ ] Push-to-main pipeline succeeds end-to-end (build + push + deploy) - [ ] ArgoCD image updater picks up new tag and deploys latest main - [ ] Dead `backend_options` volume mount and PVC cleaned up - [ ] Pipeline still runs tests against Postgres service - [ ] Single-stage Dockerfile optimization preserved (not the cause) ### Related - `landscaping-assistant` project - PR #188 — pipeline optimization that added the dead volume config