Automate Gmail OAuth re-auth lifecycle (7-day token expiry)

forgejo_admin commented

2026-03-26 03:03:11 +00:00

Owner

Type

Feature

Lineage

Standalone — discovered during repeated manual Gmail OAuth re-auth cycles (7-day token expiry).
Prerequisite forgejo_admin/gmail-mcp#6 (SSH-compatible reauth tool, board item #361) is done.

Repo

forgejo_admin/pal-e-platform

User Story

As a platform operator
I want a single script that re-authenticates Gmail OAuth and syncs the refreshed token to all consuming k8s workloads
So that the 7-day token expiry cycle is a 1-minute ops task instead of a manual multi-step firefight

Context

Gmail OAuth tokens for westsidebasketball@gmail.com expire every 7 days (Google's policy for unverified apps). Today, re-auth is a manual, error-prone process: run gmail-mcp's reauth tool via SSH, then manually copy the refreshed token file into the correct k8s secrets/PVCs for each consuming service.

Consumers of the Gmail OAuth token:

basketball-api — reads from k8s secret gmail-oauth-token in basketball-api namespace, mounted at /secrets/google-oauth via initContainer copy pattern.
pal-e-mail — reads from PVC gmail-oauth in pal-e-mail namespace, mounted at /secrets/gmail. This is a PersistentVolumeClaim, NOT a k8s secret.
gmail-mcp — reads the token file directly from the local filesystem at ~/secrets/google-oauth/gmail-westsidebasktball.json.

Prerequisite satisfied: gmail-mcp#6 added an SSH-compatible reauth CLI tool that can perform the OAuth browser flow and write the refreshed token to the local file. This ticket wraps that tool in an end-to-end automation script.

File Targets

Files the agent should create or modify:

scripts/gmail-reauth.sh (new) — orchestration script that: (1) calls gmail-mcp reauth, (2) updates k8s secret gmail-oauth-token in basketball-api namespace, (3) restarts affected deployments to pick up new token
salt/pillar/secrets_registry.sls — add Gmail OAuth token entry under platform: section with rotation_days: 7, origin external, provider Google OAuth (westsidebasketball@gmail.com)
terraform/modules/monitoring/main.tf — add PrometheusRule for gmail-oauth-token day-6 expiry alerting

Files the agent should NOT touch:

~/gmail-sdk/src/gmail_sdk/auth.py — upstream SDK, not in this repo's scope
~/gmail-mcp/ — already has SSH reauth tool (done in #6), no changes needed
~/pal-e-deployments/ — deployment patches are owned by pal-e-deployments repo

Acceptance Criteria

Running scripts/gmail-reauth.sh performs end-to-end reauth: invokes gmail-mcp reauth, reads the refreshed local token, updates k8s secret gmail-oauth-token in basketball-api namespace, and triggers a rollout restart of basketball-api
salt/pillar/secrets_registry.sls has a new gmail_oauth_token entry under platform: with rotation_days: 7 and correct metadata
After script runs, kubectl get secret gmail-oauth-token -n basketball-api -o jsonpath='{.data.gmail-westsidebasketball\.json}' returns the updated token (base64-decoded matches the local file)
Script validates the refreshed token has the required 4 scopes: gmail.send, gmail.readonly, gmail.modify, gmail.labels
Script handles the local filename typo transparently — reads from ~/secrets/google-oauth/gmail-westsidebasktball.json (missing 'e') and writes to k8s secret key gmail-westsidebasketball.json (correct spelling), matching the existing k8s convention
PrometheusRule for gmail-oauth-expiry exists in terraform/modules/monitoring/main.tf and fires when the gmail-oauth-token secret is older than 6 days
Stale gmail-oauth-westsidebasketball secret deleted from basketball-api namespace (only gmail-oauth-token should remain)

Test Expectations

Manual test: run scripts/gmail-reauth.sh and verify basketball-api pod restarts with new token
Dry-run mode: script supports --dry-run flag that prints what it would do without mutating k8s state
Error handling: script exits non-zero with clear message if gmail-mcp reauth fails, if kubectl is unavailable, or if token file is missing

Constraints

Token filename typo: The local file is gmail-westsidebasktball.json (missing 'e' in basketball). The k8s secret key is gmail-westsidebasketball.json (correct spelling). The script MUST bridge this naming mismatch explicitly. Do not rename the local file (gmail-mcp writes to it).
basketball-api mount pattern: Uses an initContainer (copy-gmail-oauth) that copies from the read-only secret volume to a writable emptyDir. A rollout restart is required after secret update for the pod to pick up new data.
Script style: Follow existing scripts/update-kustomize-tag.sh conventions (bash, set -euo pipefail, usage function, color output).
No gmail-sdk/gmail-mcp changes: Those repos are out of scope.

Resolved: Convert pal-e-mail to use a k8s secret instead of a PVC for consistency. Separate ticket in pal-e-deployments to handle the migration. This issue covers basketball-api secret update only.

Resolved: Clean up stale gmail-oauth-westsidebasketball secret. The deployment only references gmail-oauth-token. Delete the stale secret as part of this ticket to reduce confusion.

Resolved: Prometheus alerting rule for day-6 token expiry. Add a PrometheusRule resource that fires when the gmail-oauth-token secret is older than 6 days. Ships with this ticket as an additional file target.

Checklist

PR opened
Tests pass (manual + dry-run)
No unrelated changes
secrets_registry.sls entry added
Script tested against live basketball-api namespace

pal-e-platform — project this affects
forgejo_admin/gmail-mcp#6 — SSH reauth tool (prerequisite, done)
Board item #359 — arch:google-oauth,type:feature,scope:discovered — NOTE: needs story:platform-reliability label added (Betty Sue action)
Board item #361 — gmail-mcp SSH reauth (done)
review-359-2026-03-27-v2 — QA review that identified the broken issue body

### Type Feature ### Lineage Standalone — discovered during repeated manual Gmail OAuth re-auth cycles (7-day token expiry). Prerequisite `forgejo_admin/gmail-mcp#6` (SSH-compatible reauth tool, board item #361) is done. ### Repo `forgejo_admin/pal-e-platform` ### User Story As a platform operator I want a single script that re-authenticates Gmail OAuth and syncs the refreshed token to all consuming k8s workloads So that the 7-day token expiry cycle is a 1-minute ops task instead of a manual multi-step firefight ### Context Gmail OAuth tokens for `westsidebasketball@gmail.com` expire every 7 days (Google's policy for unverified apps). Today, re-auth is a manual, error-prone process: run gmail-mcp's `reauth` tool via SSH, then manually copy the refreshed token file into the correct k8s secrets/PVCs for each consuming service. **Consumers of the Gmail OAuth token:** 1. **basketball-api** — reads from k8s secret `gmail-oauth-token` in `basketball-api` namespace, mounted at `/secrets/google-oauth` via initContainer copy pattern. 2. **pal-e-mail** — reads from PVC `gmail-oauth` in `pal-e-mail` namespace, mounted at `/secrets/gmail`. This is a PersistentVolumeClaim, NOT a k8s secret. 3. **gmail-mcp** — reads the token file directly from the local filesystem at `~/secrets/google-oauth/gmail-westsidebasktball.json`. **Prerequisite satisfied:** `gmail-mcp#6` added an SSH-compatible `reauth` CLI tool that can perform the OAuth browser flow and write the refreshed token to the local file. This ticket wraps that tool in an end-to-end automation script. ### File Targets Files the agent should create or modify: - `scripts/gmail-reauth.sh` (new) — orchestration script that: (1) calls gmail-mcp reauth, (2) updates k8s secret `gmail-oauth-token` in `basketball-api` namespace, (3) restarts affected deployments to pick up new token - `salt/pillar/secrets_registry.sls` — add Gmail OAuth token entry under `platform:` section with `rotation_days: 7`, origin `external`, provider `Google OAuth (westsidebasketball@gmail.com)` - `terraform/modules/monitoring/main.tf` — add PrometheusRule for gmail-oauth-token day-6 expiry alerting Files the agent should NOT touch: - `~/gmail-sdk/src/gmail_sdk/auth.py` — upstream SDK, not in this repo's scope - `~/gmail-mcp/` — already has SSH reauth tool (done in #6), no changes needed - `~/pal-e-deployments/` — deployment patches are owned by pal-e-deployments repo ### Acceptance Criteria - [ ] Running `scripts/gmail-reauth.sh` performs end-to-end reauth: invokes gmail-mcp reauth, reads the refreshed local token, updates k8s secret `gmail-oauth-token` in `basketball-api` namespace, and triggers a rollout restart of basketball-api - [ ] `salt/pillar/secrets_registry.sls` has a new `gmail_oauth_token` entry under `platform:` with `rotation_days: 7` and correct metadata - [ ] After script runs, `kubectl get secret gmail-oauth-token -n basketball-api -o jsonpath='{.data.gmail-westsidebasketball\.json}'` returns the updated token (base64-decoded matches the local file) - [ ] Script validates the refreshed token has the required 4 scopes: `gmail.send`, `gmail.readonly`, `gmail.modify`, `gmail.labels` - [ ] Script handles the local filename typo transparently — reads from `~/secrets/google-oauth/gmail-westsidebasktball.json` (missing 'e') and writes to k8s secret key `gmail-westsidebasketball.json` (correct spelling), matching the existing k8s convention - [ ] PrometheusRule for `gmail-oauth-expiry` exists in `terraform/modules/monitoring/main.tf` and fires when the `gmail-oauth-token` secret is older than 6 days - [ ] Stale `gmail-oauth-westsidebasketball` secret deleted from `basketball-api` namespace (only `gmail-oauth-token` should remain) ### Test Expectations - [ ] Manual test: run `scripts/gmail-reauth.sh` and verify basketball-api pod restarts with new token - [ ] Dry-run mode: script supports `--dry-run` flag that prints what it would do without mutating k8s state - [ ] Error handling: script exits non-zero with clear message if gmail-mcp reauth fails, if kubectl is unavailable, or if token file is missing ### Constraints - **Token filename typo:** The local file is `gmail-westsidebasktball.json` (missing 'e' in basketball). The k8s secret key is `gmail-westsidebasketball.json` (correct spelling). The script MUST bridge this naming mismatch explicitly. Do not rename the local file (gmail-mcp writes to it). - **basketball-api mount pattern:** Uses an initContainer (`copy-gmail-oauth`) that copies from the read-only secret volume to a writable emptyDir. A rollout restart is required after secret update for the pod to pick up new data. - **Script style:** Follow existing `scripts/update-kustomize-tag.sh` conventions (bash, `set -euo pipefail`, usage function, color output). - **No gmail-sdk/gmail-mcp changes:** Those repos are out of scope. **Resolved:** Convert pal-e-mail to use a k8s secret instead of a PVC for consistency. Separate ticket in pal-e-deployments to handle the migration. This issue covers basketball-api secret update only. **Resolved:** Clean up stale gmail-oauth-westsidebasketball secret. The deployment only references gmail-oauth-token. Delete the stale secret as part of this ticket to reduce confusion. **Resolved:** Prometheus alerting rule for day-6 token expiry. Add a PrometheusRule resource that fires when the gmail-oauth-token secret is older than 6 days. Ships with this ticket as an additional file target. ### Checklist - [ ] PR opened - [ ] Tests pass (manual + dry-run) - [ ] No unrelated changes - [ ] secrets_registry.sls entry added - [ ] Script tested against live basketball-api namespace ### Related - `pal-e-platform` — project this affects - `forgejo_admin/gmail-mcp#6` — SSH reauth tool (prerequisite, done) - Board item #359 — `arch:google-oauth,type:feature,scope:discovered` — **NOTE: needs `story:platform-reliability` label added (Betty Sue action)** - Board item #361 — gmail-mcp SSH reauth (done) - `review-359-2026-03-27-v2` — QA review that identified the broken issue body

forgejo_admin commented

2026-03-27 21:08:23 +00:00

Author

Owner

Scope Review: NEEDS_REFINEMENT

Review note: review-359-2026-03-27
Template is complete but four issues must be resolved before this ticket is agent-ready.

Missing story label — board item #359 has no story:X label. Suggest story:platform-reliability.
Alerting mechanism unspecified — "Cron/scheduled agent config — location TBD" gives the agent no file target. Pick Prometheus alert rule, CronJob, or Salt job and add the path.
pal-e-mail sync path ambiguous — pal-e-mail mounts a PVC (gmail-oauth), not a k8s secret. AC #3 says "pal-e-mail secrets if applicable" but doesn't resolve whether the PVC is in scope or needs a separate ticket.
Token file naming mismatch undocumented — local file gmail-westsidebasktball.json (typo, no 'e') vs k8s secret gmail-oauth-westsidebasketball (correct spelling). Must be explicitly documented in Constraints so the agent handles the mapping correctly.

Additionally: two k8s secrets exist in basketball-api (gmail-oauth-token and gmail-oauth-westsidebasketball) — clarify whether both are active or one is stale.

## Scope Review: NEEDS_REFINEMENT Review note: `review-359-2026-03-27` Template is complete but four issues must be resolved before this ticket is agent-ready. - **Missing story label** — board item #359 has no `story:X` label. Suggest `story:platform-reliability`. - **Alerting mechanism unspecified** — "Cron/scheduled agent config — location TBD" gives the agent no file target. Pick Prometheus alert rule, CronJob, or Salt job and add the path. - **pal-e-mail sync path ambiguous** — pal-e-mail mounts a PVC (`gmail-oauth`), not a k8s secret. AC #3 says "pal-e-mail secrets if applicable" but doesn't resolve whether the PVC is in scope or needs a separate ticket. - **Token file naming mismatch undocumented** — local file `gmail-westsidebasktball.json` (typo, no 'e') vs k8s secret `gmail-oauth-westsidebasketball` (correct spelling). Must be explicitly documented in Constraints so the agent handles the mapping correctly. Additionally: two k8s secrets exist in basketball-api (`gmail-oauth-token` and `gmail-oauth-westsidebasketball`) — clarify whether both are active or one is stale.

forgejo_admin commented

2026-03-27 22:14:04 +00:00

Author

Owner

Issue body updated per scope review corrections.

forgejo_admin commented

2026-03-28 05:42:05 +00:00

Author

Owner

Scope Review: BLOCK

Review note: review-359-2026-03-27-v2
Issue body is the literal string $NEW_BODY — a shell variable that was never interpolated. No spec exists for an agent to work from.

Prior review (review-359-2026-03-27) found NEEDS_REFINEMENT with 4 issues. A subsequent "Issue body updated" comment did not actually update the body — the broken $NEW_BODY string remains. All original spec content has been lost.

CRITICAL: Issue body must be rewritten from scratch using template-issue-feature. Prior review documents what the original content covered.
[LABEL] Add story:platform-reliability label to board item #359
[BODY] Specify day-6 alerting mechanism (Prometheus rule, CronJob, or Salt job) with file target path
[BODY] Clarify pal-e-mail sync path — uses PVC gmail-oauth, not a k8s secret
[BODY] Document gmail-westsidebasktball.json (typo) vs gmail-oauth-westsidebasketball (correct) naming mismatch in Constraints
[SCOPE] Clarify whether both basketball-api secrets (gmail-oauth-token + gmail-oauth-westsidebasketball) are active or one is stale

## Scope Review: BLOCK Review note: `review-359-2026-03-27-v2` Issue body is the literal string `$NEW_BODY` — a shell variable that was never interpolated. No spec exists for an agent to work from. Prior review (`review-359-2026-03-27`) found NEEDS_REFINEMENT with 4 issues. A subsequent "Issue body updated" comment did not actually update the body — the broken `$NEW_BODY` string remains. All original spec content has been lost. - **CRITICAL: Issue body must be rewritten from scratch** using `template-issue-feature`. Prior review documents what the original content covered. - `[LABEL]` Add `story:platform-reliability` label to board item #359 - `[BODY]` Specify day-6 alerting mechanism (Prometheus rule, CronJob, or Salt job) with file target path - `[BODY]` Clarify pal-e-mail sync path — uses PVC `gmail-oauth`, not a k8s secret - `[BODY]` Document `gmail-westsidebasktball.json` (typo) vs `gmail-oauth-westsidebasketball` (correct) naming mismatch in Constraints - `[SCOPE]` Clarify whether both basketball-api secrets (`gmail-oauth-token` + `gmail-oauth-westsidebasketball`) are active or one is stale

forgejo_admin commented

2026-03-28 11:41:17 +00:00

Author

Owner

Scope Review: NEEDS_REFINEMENT

Review note: review-359-2026-03-27-v3
Body rewrite is solid — all template sections present, 3 prior decisions resolved, file targets verified. Three mechanical fixes remain before READY:

[LABEL] Add story:platform-reliability label to board item #359 (carried from v1+v2, still missing)
[BODY] Add terraform/modules/monitoring/main.tf to File Targets — the "Resolved" section commits to a PrometheusRule but the File Targets section omits it
[BODY] Add 2 missing ACs: (a) PrometheusRule gmail-oauth-expiry exists in monitoring namespace; (b) stale secret gmail-oauth-westsidebasketball deleted from basketball-api namespace

No human decisions needed — all three are mechanical. Once applied, this ticket is READY.

## Scope Review: NEEDS_REFINEMENT Review note: `review-359-2026-03-27-v3` Body rewrite is solid — all template sections present, 3 prior decisions resolved, file targets verified. Three mechanical fixes remain before READY: - `[LABEL]` Add `story:platform-reliability` label to board item #359 (carried from v1+v2, still missing) - `[BODY]` Add `terraform/modules/monitoring/main.tf` to File Targets — the "Resolved" section commits to a PrometheusRule but the File Targets section omits it - `[BODY]` Add 2 missing ACs: (a) PrometheusRule `gmail-oauth-expiry` exists in monitoring namespace; (b) stale secret `gmail-oauth-westsidebasketball` deleted from basketball-api namespace No human decisions needed — all three are mechanical. Once applied, this ticket is READY.

forgejo_admin commented

2026-03-28 11:46:10 +00:00

Author

Owner

Scope Review: READY (v4)

Review note: review-359-2026-03-27-v4

All 3 v3 mechanical fixes confirmed applied: monitoring/main.tf in File Targets, PrometheusRule AC, stale secret deletion AC. All 7 ACs are concrete and agent-executable. 3 file targets verified against live codebase and k8s cluster.

[LABEL] (non-blocking) Add story:platform-reliability label to board item #359 before moving to next_up. Issue body already flags this as a Betty Sue action.

This ticket is ready for agent execution.

## Scope Review: READY (v4) Review note: `review-359-2026-03-27-v4` All 3 v3 mechanical fixes confirmed applied: monitoring/main.tf in File Targets, PrometheusRule AC, stale secret deletion AC. All 7 ACs are concrete and agent-executable. 3 file targets verified against live codebase and k8s cluster. - `[LABEL]` (non-blocking) Add `story:platform-reliability` label to board item #359 before moving to `next_up`. Issue body already flags this as a Betty Sue action. This ticket is ready for agent execution.