Persist new Stripe webhook secret in sealed-secrets pipeline #348

New issue

Open

opened 2026-04-06 14:42:29 +00:00 by forgejo_admin · 0 comments

forgejo_admin commented

2026-04-06 14:42:29 +00:00

Owner

Type

Bug

Lineage

Follow-up to forgejo_admin/basketball-api#343. The webhook secret mismatch was fixed manually via kubectl patch on 2026-04-06 but will be overwritten on next sealed-secret redeployment.

Repo

ldraney/pal-e-services (sealed-secrets source for basketball-api-secrets)

What Broke

The Stripe webhook signing secret was manually updated in the k8s secret basketball-api-secrets (namespace: basketball-api) on 2026-04-06. The old value whsec_iHDx... never matched Stripe's signing key — every webhook delivery got 400 Invalid signature since the endpoint was created. A new Stripe endpoint was created with correct secret whsec_0DKZ... and patched into k8s.

This manual fix will be overwritten when pal-e-services re-applies the sealed-secret, reverting to the old broken value.

Repro Steps

Run tofu apply in pal-e-services (or ArgoCD syncs sealed-secrets)
The sealed-secret overwrites basketball-api-secrets with the old whsec_iHDx... value
Pod restart picks up old secret
Stripe webhook signature verification fails again
All payment webhooks stop working

Expected Behavior

The sealed-secret source in pal-e-services should contain the new webhook secret (whsec_0DKZ...) so that redeployments preserve the working configuration.

Environment

Cluster/namespace: prod / basketball-api
K8s secret: basketball-api-secrets key stripe-webhook-secret
New Stripe endpoint: we_1TJDmsR9SdzWqVXMibmm6ytY
New secret value: stored in ~/secrets/ (do NOT commit plaintext)
Sealed-secrets source: pal-e-services/terraform/ (encrypted secret definition)

Acceptance Criteria

New webhook secret encrypted and committed to pal-e-services sealed-secret source
tofu apply in pal-e-services produces no diff on basketball-api-secrets
Pod restart after apply still has working webhook signature verification
Document the new Stripe webhook endpoint ID in the secrets inventory

project-westside-basketball — project this affects
forgejo_admin/basketball-api#343 — original investigation
sop-network-security — secrets management procedures
Key files: pal-e-services/terraform/ (sealed-secrets for basketball-api namespace)

### Type Bug ### Lineage Follow-up to forgejo_admin/basketball-api#343. The webhook secret mismatch was fixed manually via `kubectl patch` on 2026-04-06 but will be overwritten on next sealed-secret redeployment. ### Repo `ldraney/pal-e-services` (sealed-secrets source for basketball-api-secrets) ### What Broke The Stripe webhook signing secret was manually updated in the k8s secret `basketball-api-secrets` (namespace: basketball-api) on 2026-04-06. The old value `whsec_iHDx...` never matched Stripe's signing key — every webhook delivery got `400 Invalid signature` since the endpoint was created. A new Stripe endpoint was created with correct secret `whsec_0DKZ...` and patched into k8s. This manual fix will be **overwritten** when pal-e-services re-applies the sealed-secret, reverting to the old broken value. ### Repro Steps 1. Run `tofu apply` in pal-e-services (or ArgoCD syncs sealed-secrets) 2. The sealed-secret overwrites `basketball-api-secrets` with the old `whsec_iHDx...` value 3. Pod restart picks up old secret 4. Stripe webhook signature verification fails again 5. All payment webhooks stop working ### Expected Behavior The sealed-secret source in pal-e-services should contain the new webhook secret (`whsec_0DKZ...`) so that redeployments preserve the working configuration. ### Environment - Cluster/namespace: prod / basketball-api - K8s secret: `basketball-api-secrets` key `stripe-webhook-secret` - New Stripe endpoint: `we_1TJDmsR9SdzWqVXMibmm6ytY` - New secret value: stored in `~/secrets/` (do NOT commit plaintext) - Sealed-secrets source: `pal-e-services/terraform/` (encrypted secret definition) ### Acceptance Criteria - [ ] New webhook secret encrypted and committed to pal-e-services sealed-secret source - [ ] `tofu apply` in pal-e-services produces no diff on `basketball-api-secrets` - [ ] Pod restart after apply still has working webhook signature verification - [ ] Document the new Stripe webhook endpoint ID in the secrets inventory ### Related - `project-westside-basketball` — project this affects - `forgejo_admin/basketball-api#343` — original investigation - `sop-network-security` — secrets management procedures - Key files: `pal-e-services/terraform/` (sealed-secrets for basketball-api namespace)