ArgoCD: switch all apps to internal Forgejo URL (DERP hairpin fix) #143

New issue

Closed

opened 2026-03-22 15:42:37 +00:00 by forgejo_admin · 0 comments

forgejo_admin commented

2026-03-22 15:42:37 +00:00

Owner

Type

Bug

Lineage

Discovered during plan-pal-e-mail → Phase 1 deployment

Repo

forgejo_admin/pal-e-platform

What Broke

tofu apply fails when creating new ArgoCD Applications. ArgoCD repo-server cannot reach forgejo.tail5b443a.ts.net from inside the cluster — TLS handshake fails with EOF. DNS resolves to DERP relay IPs (2607:f740:0:3f::2f0, 208.111.35.209), IPv6 unreachable, IPv4 TLS terminates prematurely.

Error: rpc error: code = Unknown desc = failed to list refs: Get "https://forgejo.tail5b443a.ts.net/forgejo_admin/pal-e-deployments.git/info/refs?service=git-upload-pack": EOF

Existing apps still sync from cache, but new app creation or cache expiry will fail.

Repro Steps

Add a new service to k3s.tfvars with source_repo = "forgejo_admin/pal-e-deployments"
Run tofu apply -lock=false -var-file=k3s.tfvars
Observe: ArgoCD Application creation fails with EOF on git ls-remote to external Forgejo URL

Expected Behavior

tofu apply should create ArgoCD Applications successfully. ArgoCD should use the internal Forgejo service URL (http://forgejo-http.forgejo.svc.cluster.local) to avoid the Tailscale DERP hairpin path entirely.

Environment

Cluster: prod (k3s)
ArgoCD version: check kubectl get deploy -n argocd argocd-server -o jsonpath='{.spec.template.spec.containers[0].image}'
Woodpecker CI clone step already uses internal URL (fixed in prior CI pipeline work)
Also missing: forgejo NetworkPolicy does not allow argocd namespace ingress (patched via kubectl, terraform fix pending)

Acceptance Criteria

All ArgoCD Applications use http://forgejo-http.forgejo.svc.cluster.local as repoURL
tofu apply for new services succeeds without DERP failures
terraform/network-policies.tf allows argocd → forgejo ingress
Image Updater write-back still works after URL migration
service-onboarding-sop updated to document internal URL

plan-pal-e-mail — discovered during Phase 1
feedback_ci_pipeline_lessons.md — same DERP hairpin class of issue
sop-ci-pipeline-recovery — CI recovery procedures

### Type Bug ### Lineage Discovered during `plan-pal-e-mail` → Phase 1 deployment ### Repo `forgejo_admin/pal-e-platform` ### What Broke `tofu apply` fails when creating new ArgoCD Applications. ArgoCD repo-server cannot reach `forgejo.tail5b443a.ts.net` from inside the cluster — TLS handshake fails with EOF. DNS resolves to DERP relay IPs (2607:f740:0:3f::2f0, 208.111.35.209), IPv6 unreachable, IPv4 TLS terminates prematurely. Error: `rpc error: code = Unknown desc = failed to list refs: Get "https://forgejo.tail5b443a.ts.net/forgejo_admin/pal-e-deployments.git/info/refs?service=git-upload-pack": EOF` Existing apps still sync from cache, but new app creation or cache expiry will fail. ### Repro Steps 1. Add a new service to `k3s.tfvars` with `source_repo = "forgejo_admin/pal-e-deployments"` 2. Run `tofu apply -lock=false -var-file=k3s.tfvars` 3. Observe: ArgoCD Application creation fails with EOF on git ls-remote to external Forgejo URL ### Expected Behavior `tofu apply` should create ArgoCD Applications successfully. ArgoCD should use the internal Forgejo service URL (`http://forgejo-http.forgejo.svc.cluster.local`) to avoid the Tailscale DERP hairpin path entirely. ### Environment - Cluster: prod (k3s) - ArgoCD version: check `kubectl get deploy -n argocd argocd-server -o jsonpath='{.spec.template.spec.containers[0].image}'` - Woodpecker CI clone step already uses internal URL (fixed in prior CI pipeline work) - Also missing: forgejo NetworkPolicy does not allow argocd namespace ingress (patched via kubectl, terraform fix pending) ### Acceptance Criteria - [ ] All ArgoCD Applications use `http://forgejo-http.forgejo.svc.cluster.local` as repoURL - [ ] `tofu apply` for new services succeeds without DERP failures - [ ] `terraform/network-policies.tf` allows argocd → forgejo ingress - [ ] Image Updater write-back still works after URL migration - [ ] `service-onboarding-sop` updated to document internal URL ### Related - `plan-pal-e-mail` — discovered during Phase 1 - `feedback_ci_pipeline_lessons.md` — same DERP hairpin class of issue - `sop-ci-pipeline-recovery` — CI recovery procedures