Bug: Switch Keycloak URL to cluster-internal service — 50% TLS failure rate via funnel #149

New issue

Closed

opened 2026-03-21 22:14:34 +00:00 by forgejo_admin · 0 comments

forgejo_admin commented

2026-03-21 22:14:34 +00:00

Owner

Type

Bug

Lineage

plan-wkq → Phase 11 (Girls Tryout — March 24)
Discovered during E2E registration validation

Repo

forgejo_admin/basketball-api

What Broke

basketball-api connects to Keycloak via the external Tailscale funnel URL (https://keycloak.tail5b443a.ts.net). TLS connections through funnels drop ~50% from within the cluster — SSL: UNEXPECTED_EOF_WHILE_READING. This causes Keycloak account creation to fail intermittently during registration.

Proof — 10 requests from the basketball-api pod:

External URL (https://keycloak.tail5b443a.ts.net): 5/10 failed
Internal service (http://keycloak.keycloak.svc.cluster.local): 10/10 succeeded

Repro Steps

Register a new player on westsidekingsandqueens.tail5b443a.ts.net/register
~50% of the time, Keycloak account creation fails silently
Log shows: "Failed to get Keycloak admin token"
Player is registered in DB but has no Keycloak account or credentials

Expected Behavior

Keycloak account creation succeeds 100% of the time. Pod-to-pod traffic stays inside the cluster.

Environment

Cluster/namespace: basketball-api → keycloak
Internal service: keycloak.keycloak.svc.cluster.local:80 (HTTP, works 100%)
External URL: keycloak.tail5b443a.ts.net (HTTPS via Tailscale funnel, 50% failure)

Acceptance Criteria

keycloak_base_url in config uses internal service URL for admin API calls
keycloak_realm_url stays external for OIDC token validation (tokens are issued with external issuer URL)
Registration creates Keycloak accounts reliably (10/10)
Password reset flow works reliably

File Targets

src/basketball_api/config.py — add keycloak_admin_url config pointing to internal service
src/basketball_api/services/keycloak.py — use keycloak_admin_url for admin API calls
src/basketball_api/routes/password_reset.py — use keycloak_admin_url for password reset
k8s/deployment.yaml — add BASKETBALL_KEYCLOAK_ADMIN_URL env var

Constraints

OIDC token validation must still use the external URL (token issuer claim matches external URL)
Admin API calls (account creation, password reset) should use internal URL
Same pattern as Forgejo internal URL fix (pal-e-platform PR #120)

Checklist

PR opened
Tests pass
No unrelated changes

pal-e-platform #120 — same pattern (internal URL for pod-to-pod)
basketball-api #132 — password reset (affected by this)
basketball-api #136 — Gmail OAuth fix (related infra)

### Type Bug ### Lineage `plan-wkq` → Phase 11 (Girls Tryout — March 24) Discovered during E2E registration validation ### Repo `forgejo_admin/basketball-api` ### What Broke basketball-api connects to Keycloak via the external Tailscale funnel URL (`https://keycloak.tail5b443a.ts.net`). TLS connections through funnels drop ~50% from within the cluster — `SSL: UNEXPECTED_EOF_WHILE_READING`. This causes Keycloak account creation to fail intermittently during registration. Proof — 10 requests from the basketball-api pod: - External URL (`https://keycloak.tail5b443a.ts.net`): **5/10 failed** - Internal service (`http://keycloak.keycloak.svc.cluster.local`): **10/10 succeeded** ### Repro Steps 1. Register a new player on `westsidekingsandqueens.tail5b443a.ts.net/register` 2. ~50% of the time, Keycloak account creation fails silently 3. Log shows: `"Failed to get Keycloak admin token"` 4. Player is registered in DB but has no Keycloak account or credentials ### Expected Behavior Keycloak account creation succeeds 100% of the time. Pod-to-pod traffic stays inside the cluster. ### Environment - Cluster/namespace: basketball-api → keycloak - Internal service: `keycloak.keycloak.svc.cluster.local:80` (HTTP, works 100%) - External URL: `keycloak.tail5b443a.ts.net` (HTTPS via Tailscale funnel, 50% failure) ### Acceptance Criteria - [ ] `keycloak_base_url` in config uses internal service URL for admin API calls - [ ] `keycloak_realm_url` stays external for OIDC token validation (tokens are issued with external issuer URL) - [ ] Registration creates Keycloak accounts reliably (10/10) - [ ] Password reset flow works reliably ### File Targets - `src/basketball_api/config.py` — add `keycloak_admin_url` config pointing to internal service - `src/basketball_api/services/keycloak.py` — use `keycloak_admin_url` for admin API calls - `src/basketball_api/routes/password_reset.py` — use `keycloak_admin_url` for password reset - `k8s/deployment.yaml` — add `BASKETBALL_KEYCLOAK_ADMIN_URL` env var ### Constraints - OIDC token validation must still use the external URL (token issuer claim matches external URL) - Admin API calls (account creation, password reset) should use internal URL - Same pattern as Forgejo internal URL fix (pal-e-platform PR #120) ### Checklist - [ ] PR opened - [ ] Tests pass - [ ] No unrelated changes ### Related - pal-e-platform #120 — same pattern (internal URL for pod-to-pod) - basketball-api #132 — password reset (affected by this) - basketball-api #136 — Gmail OAuth fix (related infra)