#204 - Bug: image tag automation not firing — manual deploys required - forgejo_admin/pal-e-platform

forgejo_admin commented

2026-03-27 06:36:10 +00:00

Owner

Type

Bug

Lineage

Standalone — discovered during westside-app and basketball-api deploys. Neither auto-updated kustomize overlay tags.

Repo

forgejo_admin/pal-e-platform

What Broke

After CI pushes a new image to Harbor, the kustomize overlay newTag in pal-e-deployments is not automatically updated. Every deploy requires:

Manual PR to update newTag in kustomization.yaml
Manual kubectl set image as a workaround

Issue #148 (done) was supposed to automate this but the automation isn't triggering.

Repro Steps

Merge a PR to westside-app (triggers CI build + push to Harbor)
Check pal-e-deployments/overlays/westsidekingsandqueens/prod/kustomization.yaml
Observe: newTag is still the old SHA
Image is in Harbor but never deployed

Expected Behavior

After CI pushes a new image, the kustomize overlay newTag should be automatically updated (either via Woodpecker post-build step, ArgoCD Image Updater, or equivalent automation).

Environment

CI: Woodpecker
Registry: Harbor (internal)
Deployment: ArgoCD + kustomize overlays in pal-e-deployments

User Story

As a superadmin
I want image tags to auto-update in kustomize overlays after CI pushes
So that ArgoCD deploys new builds without manual intervention

Context

Issue #148 ("Automate image tag updates in pal-e-deployments") is marked done on the board but the automation isn't working. Need to investigate what was built, why it stopped, and fix or rebuild it.

File Targets

Files the agent should investigate:

.woodpecker.yaml in each app repo — check for post-build update step
pal-e-deployments/ — check for any automation scripts
ArgoCD Image Updater config (if installed)
Woodpecker pipeline logs for update steps

Files the agent should NOT touch:

Individual kustomization.yaml files — the automation should handle these

Acceptance Criteria

After CI build succeeds, kustomize overlay newTag is updated automatically
ArgoCD syncs the new tag without manual intervention
Works for all services with CI pipelines

Test Expectations

Manual: push a no-op commit to westside-app, verify tag auto-updates in pal-e-deployments
Manual: verify ArgoCD picks up the new tag and syncs
Run command: check Woodpecker pipeline logs for update step output

Constraints

Must not require manual PRs for routine image tag updates
Automation must use internal Forgejo URLs (no hairpin)
Depends on ArgoCD source URL fix (sibling issue) for full end-to-end flow

Checklist

PR opened
Tests pass
No unrelated changes

project-pal-e-platform
forgejo_admin/pal-e-platform#148 — original automation issue (marked done but not working)

### Type Bug ### Lineage Standalone — discovered during westside-app and basketball-api deploys. Neither auto-updated kustomize overlay tags. ### Repo `forgejo_admin/pal-e-platform` ### What Broke After CI pushes a new image to Harbor, the kustomize overlay `newTag` in `pal-e-deployments` is not automatically updated. Every deploy requires: 1. Manual PR to update `newTag` in kustomization.yaml 2. Manual `kubectl set image` as a workaround Issue #148 (done) was supposed to automate this but the automation isn't triggering. ### Repro Steps 1. Merge a PR to westside-app (triggers CI build + push to Harbor) 2. Check `pal-e-deployments/overlays/westsidekingsandqueens/prod/kustomization.yaml` 3. Observe: `newTag` is still the old SHA 4. Image is in Harbor but never deployed ### Expected Behavior After CI pushes a new image, the kustomize overlay `newTag` should be automatically updated (either via Woodpecker post-build step, ArgoCD Image Updater, or equivalent automation). ### Environment - CI: Woodpecker - Registry: Harbor (internal) - Deployment: ArgoCD + kustomize overlays in pal-e-deployments ### User Story As a superadmin I want image tags to auto-update in kustomize overlays after CI pushes So that ArgoCD deploys new builds without manual intervention ### Context Issue #148 ("Automate image tag updates in pal-e-deployments") is marked done on the board but the automation isn't working. Need to investigate what was built, why it stopped, and fix or rebuild it. ### File Targets Files the agent should investigate: - `.woodpecker.yaml` in each app repo — check for post-build update step - `pal-e-deployments/` — check for any automation scripts - ArgoCD Image Updater config (if installed) - Woodpecker pipeline logs for update steps Files the agent should NOT touch: - Individual kustomization.yaml files — the automation should handle these ### Acceptance Criteria - [ ] After CI build succeeds, kustomize overlay `newTag` is updated automatically - [ ] ArgoCD syncs the new tag without manual intervention - [ ] Works for all services with CI pipelines ### Test Expectations - [ ] Manual: push a no-op commit to westside-app, verify tag auto-updates in pal-e-deployments - [ ] Manual: verify ArgoCD picks up the new tag and syncs - Run command: check Woodpecker pipeline logs for update step output ### Constraints - Must not require manual PRs for routine image tag updates - Automation must use internal Forgejo URLs (no hairpin) - Depends on ArgoCD source URL fix (sibling issue) for full end-to-end flow ### Checklist - [ ] PR opened - [ ] Tests pass - [ ] No unrelated changes ### Related - `project-pal-e-platform` - `forgejo_admin/pal-e-platform#148` — original automation issue (marked done but not working)

forgejo_admin commented

2026-03-27 06:39:45 +00:00

Author

Owner

Scope Review: NEEDS_REFINEMENT

Review note: review-453-2026-03-26
Ticket has all required bug template sections and full traceability, but needs refinement before moving to next_up.

Solution approach not selected (3 options listed, none chosen) -- consider a Spike first
Predecessor issue #148 is closed but its automation was never built (false-done) -- needs explicit acknowledgment
ArgoCD Image Updater is not installed -- ticket should state this as fact, not "check if installed"
Multi-repo scope unclear -- if per-repo pipeline changes needed, child issues required
Acceptance criteria lack specific verification commands for agent execution

## Scope Review: NEEDS_REFINEMENT Review note: `review-453-2026-03-26` Ticket has all required bug template sections and full traceability, but needs refinement before moving to next_up. - Solution approach not selected (3 options listed, none chosen) -- consider a Spike first - Predecessor issue #148 is closed but its automation was never built (false-done) -- needs explicit acknowledgment - ArgoCD Image Updater is not installed -- ticket should state this as fact, not "check if installed" - Multi-repo scope unclear -- if per-repo pipeline changes needed, child issues required - Acceptance criteria lack specific verification commands for agent execution

forgejo_admin commented

2026-03-27 06:50:50 +00:00

Author

Owner

Refinement (post review-453)

Addressed findings:

Solution approach selected: Woodpecker post-build step. Each repo's .woodpecker.yaml gets a update-tag step after build-and-push that:
- Clones pal-e-deployments
- Updates newTag in the service's kustomization.yaml
- Commits and pushes
  This is the simplest approach — no new infrastructure (ArgoCD Image Updater), just a CI step.
#148 is false-done. Marking #148 as superseded by this issue. The automation was never built.
ArgoCD Image Updater is NOT installed anywhere in the platform. Not pursuing it — Woodpecker post-build step is simpler and uses existing infra.
Multi-repo scope: 7 app repos need the post-build step added to .woodpecker.yaml. This is a cross-repo rollout, same pattern as #193 (Kaniko fix). Can be done in parallel per repo.
Verification commands added:
- Push no-op commit to westside-app
- Verify pal-e-deployments gets a commit updating the tag
- Verify ArgoCD syncs (depends on #203 being resolved first)

Dependencies

Depends on #203 (ArgoCD source URLs) for full end-to-end verification
Can be built and merged independently — the tag update commits will be ready for ArgoCD once #203 lands

Status: READY

## Refinement (post review-453) ### Addressed findings: 1. **Solution approach selected: Woodpecker post-build step.** Each repo's `.woodpecker.yaml` gets a `update-tag` step after `build-and-push` that: - Clones pal-e-deployments - Updates `newTag` in the service's kustomization.yaml - Commits and pushes This is the simplest approach — no new infrastructure (ArgoCD Image Updater), just a CI step. 2. **#148 is false-done.** Marking #148 as superseded by this issue. The automation was never built. 3. **ArgoCD Image Updater is NOT installed** anywhere in the platform. Not pursuing it — Woodpecker post-build step is simpler and uses existing infra. 4. **Multi-repo scope:** 7 app repos need the post-build step added to `.woodpecker.yaml`. This is a cross-repo rollout, same pattern as #193 (Kaniko fix). Can be done in parallel per repo. 5. **Verification commands added:** - Push no-op commit to westside-app - Verify pal-e-deployments gets a commit updating the tag - Verify ArgoCD syncs (depends on #203 being resolved first) ### Dependencies - Depends on #203 (ArgoCD source URLs) for full end-to-end verification - Can be built and merged independently — the tag update commits will be ready for ArgoCD once #203 lands ### Status: READY