fix: persistent WOODPECKER_AGENT_SECRET + probe URL fixes (Phase 14a) #68

Merged

forgejo_admin merged 1 commit from 66-hotfix-woodpecker-agent-secret into main

2026-03-14 21:51:03 +00:00

forgejo_admin commented

2026-03-14 21:50:21 +00:00

Owner

Summary

Incident hotfix: incident-2026-03-14-woodpecker-webhook-signatures

Root cause: WOODPECKER_AGENT_SECRET was not set — Woodpecker generated a random JWT signing key at every pod restart, silently invalidating all webhook tokens, API tokens, and agent auth. Merge=deploy broken across all 28 repos.

Changes

terraform/main.tf — Add WOODPECKER_AGENT_SECRET env var to server + agent via set_sensitive. Fix 4 Blackbox Exporter probe URLs: Forgejo port 3000→80, Keycloak 8080→9000, pal-e-docs /api/health→/healthz, basketball-api /api/health→/
terraform/variables.tf — Add woodpecker_agent_secret variable (sensitive)
terraform/.terraform.lock.hcl — Provider hash updates from tofu init

Test Plan

tofu validate passes
tofu apply already run — Woodpecker restarted with persistent key
28 repos deactivated/re-activated with fresh webhooks
API token regenerated and verified (returns user data, not 401)
Woodpecker server logs clean — no "token signature is invalid" errors
THIS PR IS THE TEST — if Woodpecker pipeline triggers from this push, the incident is resolved

Review Checklist

No secrets committed (agent secret is in k3s.tfvars, gitignored)
No unrelated changes
Already applied to cluster

incident-2026-03-14-woodpecker-webhook-signatures — the incident this fixes
phase-pal-e-platform-14a-webhook-fix — the phase
Closes #66

## Summary Incident hotfix: `incident-2026-03-14-woodpecker-webhook-signatures` Root cause: `WOODPECKER_AGENT_SECRET` was not set — Woodpecker generated a random JWT signing key at every pod restart, silently invalidating all webhook tokens, API tokens, and agent auth. Merge=deploy broken across all 28 repos. ## Changes - `terraform/main.tf` — Add `WOODPECKER_AGENT_SECRET` env var to server + agent via `set_sensitive`. Fix 4 Blackbox Exporter probe URLs: Forgejo port 3000→80, Keycloak 8080→9000, pal-e-docs `/api/health`→`/healthz`, basketball-api `/api/health`→`/` - `terraform/variables.tf` — Add `woodpecker_agent_secret` variable (sensitive) - `terraform/.terraform.lock.hcl` — Provider hash updates from `tofu init` ## Test Plan - [x] `tofu validate` passes - [x] `tofu apply` already run — Woodpecker restarted with persistent key - [x] 28 repos deactivated/re-activated with fresh webhooks - [x] API token regenerated and verified (returns user data, not 401) - [x] Woodpecker server logs clean — no "token signature is invalid" errors - [ ] **THIS PR IS THE TEST** — if Woodpecker pipeline triggers from this push, the incident is resolved ## Review Checklist - [x] No secrets committed (agent secret is in k3s.tfvars, gitignored) - [x] No unrelated changes - [x] Already applied to cluster ## Related - `incident-2026-03-14-woodpecker-webhook-signatures` — the incident this fixes - `phase-pal-e-platform-14a-webhook-fix` — the phase - Closes #66

forgejo_admin added 1 commit

2026-03-14 21:50:21 +00:00

fix: add persistent WOODPECKER_AGENT_SECRET + fix probe URLs

ci/woodpecker/push/woodpecker Pipeline was successful

Details

ci/woodpecker/pr/woodpecker Pipeline was successful

Details

ci/woodpecker/pull_request_closed/woodpecker Pipeline was successful

Details

e063530ee5

Incident: incident-2026-03-14-woodpecker-webhook-signatures

Root cause: WOODPECKER_AGENT_SECRET was not set, causing Woodpecker
to generate a random JWT signing key at every pod restart. This
silently invalidated all webhook tokens, API tokens, and agent auth
on every restart — breaking merge=deploy automation.

Fix:
- Add WOODPECKER_AGENT_SECRET env var to server + agent Helm values
- Add woodpecker_agent_secret variable to variables.tf
- Fix Blackbox Exporter probe URLs: Forgejo port 3000→80,
  Keycloak 8080→9000, pal-e-docs /api/health→/healthz,
  basketball-api /api/health→/

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>