Automate image tag updates in pal-e-deployments #148

New issue

Closed

opened 2026-03-24 08:39:14 +00:00 by forgejo_admin · 1 comment

forgejo_admin commented

2026-03-24 08:39:14 +00:00

Owner

Type

Feature

Lineage

plan-pal-e-platform → discovered scope from westside-contracts deploy

Repo

forgejo_admin/pal-e-platform, forgejo_admin/pal-e-deployments

User Story

As a platform operator
I want image tags in pal-e-deployments to update automatically when CI pushes a new image
So that deployments don't require manual tag commits and ArgoCD rolls out automatically

Context

Currently pal-e-deployments hardcodes image tags (e.g. newTag: a81d881). Every deploy requires:

CI builds and pushes image to Harbor
Someone manually updates the tag in pal-e-deployments kustomization.yaml
Commits and pushes
ArgoCD syncs

This broke the basketball-api deploy — CI pipeline #117 succeeded but the pod never updated because the tag in pal-e-deployments was stale. The GroupMe migration (PR #158) sat undeployed despite being merged and built.

File Targets

pal-e-deployments/overlays/*/prod/kustomization.yaml — all hardcoded newTag values
Woodpecker CI pipelines — need a step to update pal-e-deployments after image push

Acceptance Criteria

CI push to main → image built → tag auto-updated in pal-e-deployments → ArgoCD syncs
No manual commits to pal-e-deployments for routine deploys
Rollback is still possible (tag history in git)

Test Expectations

Push to basketball-api main → verify new pod rolls out without manual intervention
Run command: kubectl get deploy -n basketball-api -o jsonpath='{.items[0].spec.template.spec.containers[0].image}'

Constraints

Options: ArgoCD Image Updater, CI writes to pal-e-deployments, or Woodpecker post-build step
Must preserve rollback capability (no latest tag in prod)
Must work for all repos, not just basketball-api

Checklist

Solution selected
Implemented
Tested across multiple repos

project-pal-e-platform — platform project
plan-pal-e-platform — platform hardening plan
Discovered during westside-contracts deploy session 2026-03-24

### Type Feature ### Lineage `plan-pal-e-platform` → discovered scope from westside-contracts deploy ### Repo `forgejo_admin/pal-e-platform`, `forgejo_admin/pal-e-deployments` ### User Story As a platform operator I want image tags in pal-e-deployments to update automatically when CI pushes a new image So that deployments don't require manual tag commits and ArgoCD rolls out automatically ### Context Currently pal-e-deployments hardcodes image tags (e.g. `newTag: a81d881`). Every deploy requires: 1. CI builds and pushes image to Harbor 2. Someone manually updates the tag in pal-e-deployments kustomization.yaml 3. Commits and pushes 4. ArgoCD syncs This broke the basketball-api deploy — CI pipeline #117 succeeded but the pod never updated because the tag in pal-e-deployments was stale. The GroupMe migration (PR #158) sat undeployed despite being merged and built. ### File Targets - `pal-e-deployments/overlays/*/prod/kustomization.yaml` — all hardcoded `newTag` values - Woodpecker CI pipelines — need a step to update pal-e-deployments after image push ### Acceptance Criteria - [ ] CI push to main → image built → tag auto-updated in pal-e-deployments → ArgoCD syncs - [ ] No manual commits to pal-e-deployments for routine deploys - [ ] Rollback is still possible (tag history in git) ### Test Expectations - [ ] Push to basketball-api main → verify new pod rolls out without manual intervention - Run command: `kubectl get deploy -n basketball-api -o jsonpath='{.items[0].spec.template.spec.containers[0].image}'` ### Constraints - Options: ArgoCD Image Updater, CI writes to pal-e-deployments, or Woodpecker post-build step - Must preserve rollback capability (no `latest` tag in prod) - Must work for all repos, not just basketball-api ### Checklist - [ ] Solution selected - [ ] Implemented - [ ] Tested across multiple repos ### Related - `project-pal-e-platform` — platform project - `plan-pal-e-platform` — platform hardening plan - Discovered during westside-contracts deploy session 2026-03-24

forgejo_admin commented

2026-03-24 20:59:15 +00:00

Author

Owner

Scope Investigation (2026-03-24)

ArgoCD Image Updater IS deployed (pal-e-services Helm chart v0.12.0). Infrastructure is in place but configuration is incomplete.

Current State

8 of 9 services have source_repo set → get write-back annotation
pal-e-mail has NO source_repo → no write-back, tags are static
All 9 overlays in pal-e-deployments have hardcoded newTag entries

The Gap

Image Updater writes to .argocd-source-{service}.yaml files in the repo, but kustomization.yaml still has static newTag entries. Either:

Image Updater isn't actually committing (git permissions issue)
Or the .argocd-source-* files aren't being read by Kustomize

Fix Steps

Verify write-back — check if .argocd-source-*.yaml files exist in pal-e-deployments after a Harbor push. If they don't exist, the git-creds token may lack write access.
Add pal-e-mail to source_repo config in k3s.tfvars
Remove hardcoded newTag from kustomization.yaml files once write-back is confirmed working
Check Forgejo token permissions — forgejo-argocd-token needs write access to pal-e-deployments

Sizing

This is a 3-point investigation (verify write-back, fix perms) followed by a 2-point cleanup (remove hardcoded tags, add pal-e-mail). Total: ~5 points as scoped.

## Scope Investigation (2026-03-24) **ArgoCD Image Updater IS deployed** (pal-e-services Helm chart v0.12.0). Infrastructure is in place but configuration is incomplete. ### Current State - **8 of 9 services** have `source_repo` set → get write-back annotation - **pal-e-mail** has NO `source_repo` → no write-back, tags are static - All 9 overlays in pal-e-deployments have **hardcoded `newTag`** entries ### The Gap Image Updater writes to `.argocd-source-{service}.yaml` files in the repo, but kustomization.yaml still has static `newTag` entries. Either: 1. Image Updater isn't actually committing (git permissions issue) 2. Or the `.argocd-source-*` files aren't being read by Kustomize ### Fix Steps 1. **Verify write-back** — check if `.argocd-source-*.yaml` files exist in pal-e-deployments after a Harbor push. If they don't exist, the git-creds token may lack write access. 2. **Add pal-e-mail** to `source_repo` config in k3s.tfvars 3. **Remove hardcoded newTag** from kustomization.yaml files once write-back is confirmed working 4. **Check Forgejo token permissions** — `forgejo-argocd-token` needs write access to pal-e-deployments ### Sizing This is a **3-point investigation** (verify write-back, fix perms) followed by a **2-point cleanup** (remove hardcoded tags, add pal-e-mail). Total: ~5 points as scoped.