Investigate Gmail OAuth alerts — token is PERMANENT but alerts firing #373

New issue

Open

opened 2026-05-19 02:18:32 +00:00 by ldraney · 0 comments

ldraney commented

2026-05-19 02:18:32 +00:00

Owner

Type

Bug

Lineage

Standalone — discovered during platform health audit 2026-05-18.

Repo

ldraney/pal-e-platform

What Broke

Two alerts firing: GmailOAuthTokenExpired (CRITICAL) and GmailOAuthTokenExpiringSoon (WARNING). However, the local token at ~/secrets/google-oauth/gmail-westsidebasketball.json reports refresh_token_expires_in: PERMANENT. Token is not actually expired.

Possible causes: k8s secrets out of sync with local file, stale Prometheus metrics, or pod mounting an older secret version.

Repro Steps

Check local token: reports PERMANENT (not expired)
Check Alertmanager — GmailOAuthTokenExpired is CRITICAL
Compare k8s secret with local file — suspected mismatch

Expected Behavior

If the token is PERMANENT, no OAuth expiry alerts should fire.

Environment

Cluster/namespace: prod / basketball-api
SOP: sop-gmail-oauth (step 1 done — token is PERMANENT)
Related alerts: GmailOAuthTokenExpired, GmailOAuthTokenExpiringSoon

Acceptance Criteria

Root cause identified (stale k8s secret vs stale metrics vs other)
k8s secrets synced with local file if needed (per sop-gmail-oauth step 3)
Both alerts clear in Prometheus/Alertmanager
Email send tested end-to-end

project-pal-e-platform — platform health
sop-gmail-oauth — token management SOP

### Type Bug ### Lineage Standalone — discovered during platform health audit 2026-05-18. ### Repo `ldraney/pal-e-platform` ### What Broke Two alerts firing: `GmailOAuthTokenExpired` (CRITICAL) and `GmailOAuthTokenExpiringSoon` (WARNING). However, the local token at `~/secrets/google-oauth/gmail-westsidebasketball.json` reports `refresh_token_expires_in: PERMANENT`. Token is not actually expired. Possible causes: k8s secrets out of sync with local file, stale Prometheus metrics, or pod mounting an older secret version. ### Repro Steps 1. Check local token: reports PERMANENT (not expired) 2. Check Alertmanager — GmailOAuthTokenExpired is CRITICAL 3. Compare k8s secret with local file — suspected mismatch ### Expected Behavior If the token is PERMANENT, no OAuth expiry alerts should fire. ### Environment - Cluster/namespace: prod / basketball-api - SOP: `sop-gmail-oauth` (step 1 done — token is PERMANENT) - Related alerts: GmailOAuthTokenExpired, GmailOAuthTokenExpiringSoon ### Acceptance Criteria - [ ] Root cause identified (stale k8s secret vs stale metrics vs other) - [ ] k8s secrets synced with local file if needed (per sop-gmail-oauth step 3) - [ ] Both alerts clear in Prometheus/Alertmanager - [ ] Email send tested end-to-end ### Related - `project-pal-e-platform` — platform health - `sop-gmail-oauth` — token management SOP