forgejo_admin/pal-e-platform
1
0
Fork 0
Code Issues 30 Pull requests Projects Releases Packages Wiki Activity Actions

Bug: ArgoCD apps point to wrong source repos + external Forgejo URLs #203

New issue
Closed
opened 2026-03-27 06:35:55 +00:00 by forgejo_admin · 4 comments
forgejo_admin commented 2026-03-27 06:35:55 +00:00
Owner
Copy link

Type

Bug

Lineage

Standalone — discovered during westside-app deployment. All ArgoCD apps have the same issue.

Repo

forgejo_admin/pal-e-platform

What Broke

All ArgoCD applications point to individual app repos (e.g. westside-app.git/k8s/) instead of pal-e-deployments.git/overlays/*/prod/. They also use the external Forgejo URL (forgejo.tail5b443a.ts.net) which hits a Tailscale hairpin EOF from inside the cluster.

Result: ArgoCD can never auto-sync. Every deploy requires manual kubectl set image.

Repro Steps

  1. kubectl get application westsidekingsandqueens -n argocd -o jsonpath='{.spec.source.repoURL}'
  2. Observe: https://forgejo.tail5b443a.ts.net/forgejo_admin/westside-app.git (wrong repo, external URL)
  3. kubectl get application westsidekingsandqueens -n argocd -o jsonpath='{.status.operationState.message}'
  4. Observe: ComparisonError: failed to list refs: EOF

Expected Behavior

ArgoCD apps should point to:

  • Repo: http://forgejo-http.forgejo.svc.cluster.local:80/forgejo_admin/pal-e-deployments.git (internal URL)
  • Path: overlays/{service}/prod

Environment

  • Cluster/namespace: argocd
  • Affected apps: all (westsidekingsandqueens, basketball-api, pal-e-docs, pal-e-app, etc.)

User Story

As a superadmin
I want ArgoCD to auto-sync from pal-e-deployments using internal Forgejo URLs
So that image tag updates in kustomize overlays deploy automatically without manual kubectl intervention

Context

pal-e-services/k3s.tfvars already has correct source_repo and source_path for each service. The services.tf creates ArgoCD apps from these values. But tofu apply hasn't been run to update the ArgoCD apps since the migration to pal-e-deployments. The apps were created before the overlay migration and never updated.

Issue #143 (done) was supposed to fix internal URLs but the ArgoCD apps weren't updated in that pass.

File Targets

Files the agent should modify:

  • pal-e-services/terraform/services.tf — verify ArgoCD app resource uses internal Forgejo URL
  • pal-e-services/terraform/k3s.tfvars — verify source_repo/source_path are correct per service

Files the agent should NOT touch:

  • Individual app repos — no changes needed
  • pal-e-platform/terraform/ — ArgoCD is managed by pal-e-services

Acceptance Criteria

  • All ArgoCD apps point to pal-e-deployments repo with correct overlay path
  • All ArgoCD apps use internal Forgejo URL (no hairpin EOF)
  • kubectl get application -n argocd shows Synced for all apps
  • Image tag update in pal-e-deployments triggers automatic ArgoCD sync

Test Expectations

  • Manual: tofu plan -lock=false shows ArgoCD app source changes
  • Manual: after apply, kubectl get application -n argocd shows no ComparisonError
  • Run command: kubectl get application -n argocd -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.sync.status}{"\n"}{end}'

Constraints

  • Must run tofu plan -lock=false before apply
  • ArgoCD app updates are non-destructive (source URL change, not app recreation)
  • Internal URL format: http://forgejo-http.forgejo.svc.cluster.local:80/{owner}/{repo}.git

Checklist

  • PR opened
  • Tests pass
  • No unrelated changes

Related

  • project-pal-e-platform
  • forgejo_admin/pal-e-platform#143 — original internal URL migration (incomplete)

Notes

Blocked by pal-e-services#36 (:80 port fix). Code may already be correct — verify before implementing. May be a tofu-apply-only ticket, not a PR ticket.

### Type Bug ### Lineage Standalone — discovered during westside-app deployment. All ArgoCD apps have the same issue. ### Repo `forgejo_admin/pal-e-platform` ### What Broke All ArgoCD applications point to individual app repos (e.g. `westside-app.git/k8s/`) instead of `pal-e-deployments.git/overlays/*/prod/`. They also use the external Forgejo URL (`forgejo.tail5b443a.ts.net`) which hits a Tailscale hairpin EOF from inside the cluster. Result: ArgoCD can never auto-sync. Every deploy requires manual `kubectl set image`. ### Repro Steps 1. `kubectl get application westsidekingsandqueens -n argocd -o jsonpath='{.spec.source.repoURL}'` 2. Observe: `https://forgejo.tail5b443a.ts.net/forgejo_admin/westside-app.git` (wrong repo, external URL) 3. `kubectl get application westsidekingsandqueens -n argocd -o jsonpath='{.status.operationState.message}'` 4. Observe: `ComparisonError: failed to list refs: EOF` ### Expected Behavior ArgoCD apps should point to: - Repo: `http://forgejo-http.forgejo.svc.cluster.local:80/forgejo_admin/pal-e-deployments.git` (internal URL) - Path: `overlays/{service}/prod` ### Environment - Cluster/namespace: argocd - Affected apps: all (westsidekingsandqueens, basketball-api, pal-e-docs, pal-e-app, etc.) ### User Story As a superadmin I want ArgoCD to auto-sync from pal-e-deployments using internal Forgejo URLs So that image tag updates in kustomize overlays deploy automatically without manual kubectl intervention ### Context `pal-e-services/k3s.tfvars` already has correct `source_repo` and `source_path` for each service. The `services.tf` creates ArgoCD apps from these values. But `tofu apply` hasn't been run to update the ArgoCD apps since the migration to pal-e-deployments. The apps were created before the overlay migration and never updated. Issue #143 (done) was supposed to fix internal URLs but the ArgoCD apps weren't updated in that pass. ### File Targets Files the agent should modify: - `pal-e-services/terraform/services.tf` — verify ArgoCD app resource uses internal Forgejo URL - `pal-e-services/terraform/k3s.tfvars` — verify source_repo/source_path are correct per service Files the agent should NOT touch: - Individual app repos — no changes needed - `pal-e-platform/terraform/` — ArgoCD is managed by pal-e-services ### Acceptance Criteria - [ ] All ArgoCD apps point to `pal-e-deployments` repo with correct overlay path - [ ] All ArgoCD apps use internal Forgejo URL (no hairpin EOF) - [ ] `kubectl get application -n argocd` shows Synced for all apps - [ ] Image tag update in pal-e-deployments triggers automatic ArgoCD sync ### Test Expectations - [ ] Manual: `tofu plan -lock=false` shows ArgoCD app source changes - [ ] Manual: after apply, `kubectl get application -n argocd` shows no ComparisonError - Run command: `kubectl get application -n argocd -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.sync.status}{"\n"}{end}'` ### Constraints - Must run `tofu plan -lock=false` before apply - ArgoCD app updates are non-destructive (source URL change, not app recreation) - Internal URL format: `http://forgejo-http.forgejo.svc.cluster.local:80/{owner}/{repo}.git` ### Checklist - [ ] PR opened - [ ] Tests pass - [ ] No unrelated changes ### Related - `project-pal-e-platform` - `forgejo_admin/pal-e-platform#143` — original internal URL migration (incomplete) ### Notes > **Blocked by pal-e-services#36** (:80 port fix). Code may already be correct — verify before implementing. May be a tofu-apply-only ticket, not a PR ticket.
forgejo_admin commented 2026-03-27 06:40:08 +00:00
Author
Owner
Copy link

Scope Review: NEEDS_REFINEMENT

Review note: review-452-2026-03-26

The Terraform code already contains the correct values -- both services.tf (internal URL on line 148) and k3s.tfvars (all 9 services point to pal-e-deployments). The actual work is tofu apply, not code modification.

  • File targets misleading -- issue says "modify" but nothing needs modifying. Rephrase to "verify and apply."
  • Undocumented blocker: #196 -- tofu apply is blocked by MinIO provider refresh. If apply is the fix, #196 blocks this ticket.
  • Undocumented dependency: #197 -- Terraform state splitting is actively in progress. Apply on monolithic state risks conflict.
  • Possible duplicate of #200 (closed) -- network policy fix may have already resolved the EOF errors. Repro steps should be re-run to confirm bug still exists.
  • Repo mismatch -- filed on pal-e-platform but all file targets are in pal-e-services. Agent will look in the wrong repo.
## Scope Review: NEEDS_REFINEMENT Review note: `review-452-2026-03-26` The Terraform code already contains the correct values -- both `services.tf` (internal URL on line 148) and `k3s.tfvars` (all 9 services point to `pal-e-deployments`). The actual work is `tofu apply`, not code modification. - **File targets misleading** -- issue says "modify" but nothing needs modifying. Rephrase to "verify and apply." - **Undocumented blocker: #196** -- tofu apply is blocked by MinIO provider refresh. If apply is the fix, #196 blocks this ticket. - **Undocumented dependency: #197** -- Terraform state splitting is actively in progress. Apply on monolithic state risks conflict. - **Possible duplicate of #200** (closed) -- network policy fix may have already resolved the EOF errors. Repro steps should be re-run to confirm bug still exists. - **Repo mismatch** -- filed on pal-e-platform but all file targets are in pal-e-services. Agent will look in the wrong repo.
forgejo_admin commented 2026-03-27 06:50:40 +00:00
Author
Owner
Copy link

Refinement (post review-452)

Addressed findings:

  1. Code is already correct — this is a tofu apply, not a code change. Reclassifying. The fix is running tofu apply in pal-e-services, not writing code.

  2. Blocked by #196 (MinIO provider) and #197 (state splitting). This ticket cannot move to in_progress until those resolve. Adding depends:#196,#197 to labels.

  3. May already be fixed by #200. Need to re-run repro steps after #200's changes. Adding verification step: check if ArgoCD can reach internal Forgejo before doing anything.

  4. Repo mismatch. File targets are in pal-e-services, not pal-e-platform. Agent must be dispatched to pal-e-services. Noting in constraints.

Updated scope

This is NOT a code PR. It's a tofu apply on pal-e-services after #196 and #197 unblock. The agent should:

  1. Verify #200 didn't already fix the EOF error
  2. Run tofu plan -lock=false in pal-e-services
  3. Confirm only ArgoCD app source changes appear
  4. Apply

Status: READY (but blocked by #196, #197)

## Refinement (post review-452) ### Addressed findings: 1. **Code is already correct — this is a `tofu apply`, not a code change.** Reclassifying. The fix is running `tofu apply` in pal-e-services, not writing code. 2. **Blocked by #196 (MinIO provider) and #197 (state splitting).** This ticket cannot move to in_progress until those resolve. Adding `depends:#196,#197` to labels. 3. **May already be fixed by #200.** Need to re-run repro steps after #200's changes. Adding verification step: check if ArgoCD can reach internal Forgejo before doing anything. 4. **Repo mismatch.** File targets are in `pal-e-services`, not `pal-e-platform`. Agent must be dispatched to pal-e-services. Noting in constraints. ### Updated scope This is NOT a code PR. It's a `tofu apply` on pal-e-services after #196 and #197 unblock. The agent should: 1. Verify #200 didn't already fix the EOF error 2. Run `tofu plan -lock=false` in pal-e-services 3. Confirm only ArgoCD app source changes appear 4. Apply ### Status: READY (but blocked by #196, #197)
forgejo_admin commented 2026-03-27 21:08:24 +00:00
Author
Owner
Copy link

Scope Review: NEEDS_REFINEMENT

Review note: review-452-2026-03-27

The Terraform code (services.tf + k3s.tfvars) is already correct — all 9 services point to pal-e-deployments with internal URLs. An agent dispatched for this ticket would find nothing to change.

Issues found:

  • Repo mismatch: Issue filed on pal-e-platform but file targets are in pal-e-services
  • Blocked by pal-e-services#36: The :80 port in services.tf line 148 and main.tf line 320 must be removed before tofu apply will succeed (SOPS CMP EOF failure). That fix is tracked separately in pal-e-services#36 (board item #460)
  • Unclear deliverable: If the fix is "run tofu apply," that's an operational task, not a code PR. If it includes the :80 fix, it overlaps with #36
  • Undocumented dependency: argocd_repository_credentials.forgejo in main.tf also has the :80 issue — not mentioned in this ticket

Recommendation: Merge with or mark as blocked-by pal-e-services#36, refile on correct repo, and clarify whether this produces a PR or is an operational apply.

## Scope Review: NEEDS_REFINEMENT Review note: `review-452-2026-03-27` The Terraform code (services.tf + k3s.tfvars) is already correct — all 9 services point to pal-e-deployments with internal URLs. An agent dispatched for this ticket would find nothing to change. **Issues found:** - **Repo mismatch**: Issue filed on pal-e-platform but file targets are in pal-e-services - **Blocked by pal-e-services#36**: The `:80` port in services.tf line 148 and main.tf line 320 must be removed before `tofu apply` will succeed (SOPS CMP EOF failure). That fix is tracked separately in pal-e-services#36 (board item #460) - **Unclear deliverable**: If the fix is "run tofu apply," that's an operational task, not a code PR. If it includes the `:80` fix, it overlaps with #36 - **Undocumented dependency**: `argocd_repository_credentials.forgejo` in main.tf also has the `:80` issue — not mentioned in this ticket **Recommendation**: Merge with or mark as blocked-by pal-e-services#36, refile on correct repo, and clarify whether this produces a PR or is an operational apply.
forgejo_admin commented 2026-03-27 22:04:29 +00:00
Author
Owner
Copy link

Issue body updated per scope review corrections.

Issue body updated per scope review corrections.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
188-cross-repo-worktree-isolation-for-parall
111-fix-keycloak-probe
86-fix-rotate-woodpecker-api-token-in-salt
71-feat-deploy-pyrra-slo-manager-with-servi
64-hotfix-woodpecker-oauth-login-broken-for
18-fix-grafana-duplicate-default-datasource-v2
18-fix-grafana-duplicate-default-datasource
13-fix-cnpg-all-parameters
No results found.
Labels
Clear labels   
domain:backend

   
domain:devops

   
domain:frontend

   
status:approved

QA passed, awaiting merge approval

  
status:in-progress

Dev agent is actively working

  
status:needs-fix

QA found issues, back to dev

  
status:qa

PR submitted, awaiting QA review

  
type:bug

Bug fix

  
type:devops

Infrastructure/CI/config work

  
type:feature

New functionality

No labels
domain:backend
domain:devops
domain:frontend
status:approved
status:in-progress
status:needs-fix
status:qa
type:bug
type:devops
type:feature
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
forgejo_admin/pal-e-platform#203
No description provided.