ldraney/pal-e-platform
1
0
Fork 0
Code Issues 91 Pull requests 5 Projects Releases Packages Wiki Activity Actions

Bug: update-kustomize-tag skipped when CI tests fail — deploy blocked by unrelated test failures #274

New issue
Closed
opened 2026-04-07 18:28:06 +00:00 by forgejo_admin · 3 comments
forgejo_admin commented 2026-04-07 18:28:06 +00:00
Contributor
Copy link

Type

Bug

Lineage

Standalone — discovered during CRM incident response 2026-04-07. Image built and pushed to Harbor but never deployed because test step failed.

Repo

forgejo_admin/pal-e-platform

What Broke

The update-kustomize-tag step in .woodpecker.yaml has depends_on: [build-and-push] but Woodpecker skips ALL subsequent steps when ANY prior step fails, regardless of explicit depends_on. When test fails but build-and-push succeeds, the image is in Harbor but the kustomize tag is never updated, so ArgoCD never deploys.

This happened on pipelines #382, #384, and #385 for basketball-api. The test failure was a pre-existing enum test bug from PR #370 — completely unrelated to the code being deployed.

Repro Steps

  1. Push to main with a failing test (unrelated to the code change)
  2. build-and-push succeeds — image lands in Harbor
  3. test fails
  4. update-kustomize-tag is skipped despite its only dependency (build-and-push) succeeding
  5. ArgoCD never sees the new image

Expected Behavior

update-kustomize-tag runs whenever build-and-push succeeds, regardless of test status. Tests gate future PRs, not current deploys.

Design Decision

Option A selected: decouple deploy from test. Rationale:

  • The image is already built and pushed — it's immutable and valid
  • Tests gate the next merge, not the current deploy
  • A failing test on an unrelated file should never block a critical bug fix deploy
  • This matches standard CI/CD practice (build → deploy, test gates future merges)

Environment

  • CI: Woodpecker pipelines #382, #384, #385 on basketball-api
  • All three: build-and-push=success, test=failure, update-kustomize-tag=skipped
  • Only basketball-api currently uses update-kustomize-tag; 6 other repos pending onboarding (#206)

Acceptance Criteria

  • update-kustomize-tag step uses Woodpecker failure: ignore or when: status: [success, failure] on the test dependency so it runs when build-and-push succeeds
  • Canonical template updated: scripts/woodpecker-update-tag-step.yaml
  • basketball-api .woodpecker.yaml updated to match
  • Verify: push with failing test → image deploys anyway
  • Document decision in deployment-lessons

File Targets

  • scripts/woodpecker-update-tag-step.yaml (canonical template, line 37-38 — depends_on config)
  • Consumer: ~/basketball-api/.woodpecker.yaml (lines 60-82)

Related

  • pal-e-platform — CI infrastructure
  • forgejo_admin/pal-e-platform #273 — companion bug (webhook not firing)
  • deployment-lessons — operational lessons
  • forgejo_admin/pal-e-platform #206 — onboarding other repos to update-kustomize-tag
### Type Bug ### Lineage Standalone — discovered during CRM incident response 2026-04-07. Image built and pushed to Harbor but never deployed because test step failed. ### Repo `forgejo_admin/pal-e-platform` ### What Broke The `update-kustomize-tag` step in `.woodpecker.yaml` has `depends_on: [build-and-push]` but Woodpecker skips ALL subsequent steps when ANY prior step fails, regardless of explicit `depends_on`. When `test` fails but `build-and-push` succeeds, the image is in Harbor but the kustomize tag is never updated, so ArgoCD never deploys. This happened on pipelines #382, #384, and #385 for basketball-api. The test failure was a pre-existing enum test bug from PR #370 — completely unrelated to the code being deployed. ### Repro Steps 1. Push to main with a failing test (unrelated to the code change) 2. `build-and-push` succeeds — image lands in Harbor 3. `test` fails 4. `update-kustomize-tag` is skipped despite its only dependency (`build-and-push`) succeeding 5. ArgoCD never sees the new image ### Expected Behavior `update-kustomize-tag` runs whenever `build-and-push` succeeds, regardless of test status. Tests gate future PRs, not current deploys. ### Design Decision **Option A selected: decouple deploy from test.** Rationale: - The image is already built and pushed — it's immutable and valid - Tests gate the next merge, not the current deploy - A failing test on an unrelated file should never block a critical bug fix deploy - This matches standard CI/CD practice (build → deploy, test gates future merges) ### Environment - CI: Woodpecker pipelines #382, #384, #385 on basketball-api - All three: build-and-push=success, test=failure, update-kustomize-tag=skipped - Only basketball-api currently uses `update-kustomize-tag`; 6 other repos pending onboarding (#206) ### Acceptance Criteria - [ ] `update-kustomize-tag` step uses Woodpecker `failure: ignore` or `when: status: [success, failure]` on the test dependency so it runs when build-and-push succeeds - [ ] Canonical template updated: `scripts/woodpecker-update-tag-step.yaml` - [ ] basketball-api `.woodpecker.yaml` updated to match - [ ] Verify: push with failing test → image deploys anyway - [ ] Document decision in `deployment-lessons` ### File Targets - `scripts/woodpecker-update-tag-step.yaml` (canonical template, line 37-38 — `depends_on` config) - Consumer: `~/basketball-api/.woodpecker.yaml` (lines 60-82) ### Related - `pal-e-platform` — CI infrastructure - `forgejo_admin/pal-e-platform #273` — companion bug (webhook not firing) - `deployment-lessons` — operational lessons - `forgejo_admin/pal-e-platform #206` — onboarding other repos to update-kustomize-tag
forgejo_admin commented 2026-04-07 19:20:37 +00:00
Author
Contributor
Copy link

Scope Review: NEEDS_REFINEMENT

Review note: review-882-2026-04-07

Ticket is well-described but contains an unresolved design decision (Option A vs B) that blocks agent execution, plus traceability gaps.

Issues to resolve:

  • [SCOPE] Option A vs Option B must be decided by a human before this ticket is actionable
  • [LABEL] story:PLAT-S2 is not a valid story key on the project page -- change to story:superuser-deploy
  • [SCOPE] No arch-woodpecker note exists in pal-e-docs -- create it for traceability
  • [BODY] Add File Targets section (reviewer verified: scripts/woodpecker-update-tag-step.yaml, scripts/update-kustomize-tag.sh, and consumer basketball-api/.woodpecker.yaml)
  • [BODY] Rewrite Acceptance Criteria as concrete testable statements after decision
  • [BODY] Add Test Expectations section
  • [BODY] Clarify Repo field based on decision (single-repo vs multi-repo)
## Scope Review: NEEDS_REFINEMENT Review note: `review-882-2026-04-07` Ticket is well-described but contains an unresolved design decision (Option A vs B) that blocks agent execution, plus traceability gaps. **Issues to resolve:** - `[SCOPE]` Option A vs Option B must be decided by a human before this ticket is actionable - `[LABEL]` `story:PLAT-S2` is not a valid story key on the project page -- change to `story:superuser-deploy` - `[SCOPE]` No `arch-woodpecker` note exists in pal-e-docs -- create it for traceability - `[BODY]` Add File Targets section (reviewer verified: `scripts/woodpecker-update-tag-step.yaml`, `scripts/update-kustomize-tag.sh`, and consumer `basketball-api/.woodpecker.yaml`) - `[BODY]` Rewrite Acceptance Criteria as concrete testable statements after decision - `[BODY]` Add Test Expectations section - `[BODY]` Clarify Repo field based on decision (single-repo vs multi-repo)
forgejo_admin commented 2026-04-07 19:21:13 +00:00
Author
Contributor
Copy link

Scope refinement (post review-882-2026-04-07):

  • Design decision made: Option A (decouple deploy from test)
  • Added file targets: woodpecker-update-tag-step.yaml + basketball-api .woodpecker.yaml
  • Added explicit AC with Woodpecker-specific config (failure: ignore or when: status)
  • Fixed story label to superuser-deploy
  • Noted only basketball-api currently uses the step
**Scope refinement (post review-882-2026-04-07):** - Design decision made: Option A (decouple deploy from test) - Added file targets: `woodpecker-update-tag-step.yaml` + basketball-api `.woodpecker.yaml` - Added explicit AC with Woodpecker-specific config (`failure: ignore` or `when: status`) - Fixed story label to superuser-deploy - Noted only basketball-api currently uses the step
forgejo_admin commented 2026-04-07 19:24:09 +00:00
Author
Contributor
Copy link

Scope Review: READY

Review note: review-882-2026-04-07-v2 (re-review of review-882-2026-04-07)

All blocking issues from v1 resolved. Design decision made (Option A), story label fixed, AC rewritten as concrete statements, file targets added with line numbers.

Verified file targets:

  • scripts/woodpecker-update-tag-step.yaml line 37-38 — depends_on: [build-and-push] confirmed
  • basketball-api/.woodpecker.yaml lines 60-83 — consumer step confirmed
  • failure: ignore precedent exists in pal-e-platform's own pipeline (line 332)

Blast radius note: 4 other repos (pal-e-docs, westside-app, pal-e-app, twitch-2k-wager) have the same latent bug. Out of scope — tracked by #206.

Non-blocking scope items:

  • [SCOPE] Create arch-woodpecker note in pal-e-docs (can be done in parallel)
  • [BODY] Minor: Repo section could explicitly note basketball-api as second target
## Scope Review: READY Review note: `review-882-2026-04-07-v2` (re-review of `review-882-2026-04-07`) All blocking issues from v1 resolved. Design decision made (Option A), story label fixed, AC rewritten as concrete statements, file targets added with line numbers. **Verified file targets:** - `scripts/woodpecker-update-tag-step.yaml` line 37-38 — `depends_on: [build-and-push]` confirmed - `basketball-api/.woodpecker.yaml` lines 60-83 — consumer step confirmed - `failure: ignore` precedent exists in pal-e-platform's own pipeline (line 332) **Blast radius note:** 4 other repos (pal-e-docs, westside-app, pal-e-app, twitch-2k-wager) have the same latent bug. Out of scope — tracked by #206. **Non-blocking scope items:** - `[SCOPE]` Create `arch-woodpecker` note in pal-e-docs (can be done in parallel) - `[BODY]` Minor: Repo section could explicitly note basketball-api as second target
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
fix-gem-bin-path
netpol-westside-ror
fix-woodpecker-multi-pipeline
345-harbor-mobile-css
290-payment-pipeline-observability
spec-board-centric-renderer
284-fix-add-pal-e-docs-namespace-to-postgres
284-add-pal-e-docs-postgres-netpol
188-cross-repo-worktree-isolation-for-parall
111-fix-keycloak-probe
86-fix-rotate-woodpecker-api-token-in-salt
71-feat-deploy-pyrra-slo-manager-with-servi
64-hotfix-woodpecker-oauth-login-broken-for
18-fix-grafana-duplicate-default-datasource-v2
18-fix-grafana-duplicate-default-datasource
13-fix-cnpg-all-parameters
No results found.
Labels
Clear labels   
domain:backend

   
domain:devops

   
domain:frontend

   
status:approved

QA passed, awaiting merge approval

  
status:in-progress

Dev agent is actively working

  
status:needs-fix

QA found issues, back to dev

  
status:qa

PR submitted, awaiting QA review

  
type:bug

Bug fix

  
type:devops

Infrastructure/CI/config work

  
type:feature

New functionality

No labels
domain:backend
domain:devops
domain:frontend
status:approved
status:in-progress
status:needs-fix
status:qa
type:bug
type:devops
type:feature
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
ldraney/pal-e-platform#274
No description provided.