fix: writable emptyDir for gmail oauth token refresh

forgejo_admin commented

2026-03-22 17:16:23 +00:00

Owner

Summary

Gmail OAuth token refresh crashes with OSError: Read-only file system because the gmail_sdk writes refreshed tokens to a k8s secret volume mounted read-only
Adds init container + emptyDir pattern: init container copies secret to writable emptyDir, app mounts the writable volume
Unblocks all email sending (password reset, registration, roster exports, tryout announcements)

Changes

k8s/deployment.yaml: added copy-gmail-oauth init container (busybox:1.36) that copies secret data from read-only mount to writable emptyDir. Changed app container volumeMount from read-only gmail-oauth to writable gmail-oauth-writable. Added gmail-oauth-writable emptyDir volume.

Test Plan

Deploy and wait >60 min for access token to expire
Trigger email-sending endpoint (e.g. POST /api/password-reset/request)
Verify email sends successfully (no OSError)
Restart pod and verify email still works (init container re-copies from secret)
Companion PR on pal-e-deployments carries the production kustomize overlay fix

Review Checklist

Passed automated review-fix loop
No secrets committed
No unnecessary file changes
Commit messages are descriptive

Closes #136
Companion PR: pal-e-deployments (same branch name, production kustomize overlay)
Blocked: basketball-api #132 (password reset flow)

## Summary - Gmail OAuth token refresh crashes with `OSError: Read-only file system` because the gmail_sdk writes refreshed tokens to a k8s secret volume mounted read-only - Adds init container + emptyDir pattern: init container copies secret to writable emptyDir, app mounts the writable volume - Unblocks all email sending (password reset, registration, roster exports, tryout announcements) ## Changes - `k8s/deployment.yaml`: added `copy-gmail-oauth` init container (busybox:1.36) that copies secret data from read-only mount to writable emptyDir. Changed app container volumeMount from read-only `gmail-oauth` to writable `gmail-oauth-writable`. Added `gmail-oauth-writable` emptyDir volume. ## Test Plan - [ ] Deploy and wait >60 min for access token to expire - [ ] Trigger email-sending endpoint (e.g. `POST /api/password-reset/request`) - [ ] Verify email sends successfully (no OSError) - [ ] Restart pod and verify email still works (init container re-copies from secret) - [ ] Companion PR on pal-e-deployments carries the production kustomize overlay fix ## Review Checklist - [ ] Passed automated review-fix loop - [ ] No secrets committed - [ ] No unnecessary file changes - [ ] Commit messages are descriptive ## Related - Closes #136 - Companion PR: pal-e-deployments (same branch name, production kustomize overlay) - Blocked: basketball-api #132 (password reset flow)

forgejo_admin added 1 commit

2026-03-22 17:16:23 +00:00

fix: emptyDir + init container for writable gmail oauth volume

ci/woodpecker/pr/woodpecker Pipeline was successful

Details

8733a5863b

k8s secret volumes are read-only. The gmail_sdk refreshes access tokens
and writes them back to disk, crashing with OSError on the read-only
mount. Init container copies secret to writable emptyDir at pod start.

Fixes #136

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

forgejo_admin commented

2026-03-22 17:16:54 +00:00

Author

Owner

Review: PASS -- no issues found.

Reviewed the diff (+13/-2, single file):

Init container (copy-gmail-oauth, busybox:1.36) correctly copies from read-only secret mount to writable emptyDir
App container volumeMount changed from gmail-oauth (readOnly) to gmail-oauth-writable (writable)
BASKETBALL_GMAIL_SECRETS_DIR env var unchanged -- path consistency maintained at /secrets/google-oauth
emptyDir volume added alongside existing secret volume
On pod restart, init container re-copies from immutable secret -- refresh token preserved

This is the in-repo k8s manifest. Companion PR pal-e-deployments#43 carries the production kustomize overlay fix.

**Review: PASS -- no issues found.** Reviewed the diff (+13/-2, single file): - Init container (`copy-gmail-oauth`, busybox:1.36) correctly copies from read-only secret mount to writable emptyDir - App container volumeMount changed from `gmail-oauth` (readOnly) to `gmail-oauth-writable` (writable) - `BASKETBALL_GMAIL_SECRETS_DIR` env var unchanged -- path consistency maintained at `/secrets/google-oauth` - emptyDir volume added alongside existing secret volume - On pod restart, init container re-copies from immutable secret -- refresh token preserved This is the in-repo k8s manifest. Companion PR pal-e-deployments#43 carries the production kustomize overlay fix.

forgejo_admin commented

2026-03-22 17:25:05 +00:00

Author

Owner

PR #151 Review

DOMAIN REVIEW

Tech stack: Kubernetes deployment YAML (kustomize base manifest). No application code changed.

Pattern assessment: The init container + emptyDir pattern is a well-established Kubernetes idiom for making secret-mounted files writable. The design is correct:

Init container (copy-gmail-oauth) mounts the gmail-oauth secret volume read-only at /secrets-ro/google-oauth, copies all files to emptyDir at /secrets/google-oauth.
App container mounts the writable emptyDir at /secrets/google-oauth -- same path the app expects via BASKETBALL_GMAIL_SECRETS_DIR=/secrets/google-oauth.
Pod restart safety: Init container runs on every pod start, re-copying from the canonical secret. emptyDir is ephemeral by design -- correct behavior here.
Volume definitions: Both gmail-oauth (secret) and gmail-oauth-writable (emptyDir) are declared in the volumes section. The secret volume is still needed for the init container source.

Verified: The BASKETBALL_GMAIL_SECRETS_DIR env var (line 58 in deployment.yaml) points to /secrets/google-oauth, which matches the emptyDir mountPath on the app container. Path alignment is correct.

Verified: The gmail_sdk.GmailClient in src/basketball_api/services/email.py uses settings.gmail_secrets_dir for token read/write. The writable emptyDir mount at that path resolves the OSError: Read-only file system crash.

BLOCKERS

None.

This is a pure infrastructure fix -- a single k8s manifest change with no application code modifications. The BLOCKER criteria do not apply:

Test coverage: No new application functionality introduced. The change is a deployment manifest fix. The existing CI pipeline (pytest, ruff check, ruff format) validates application code, not k8s manifests. The Test Plan appropriately describes manual validation steps (deploy, wait for token expiry, trigger email endpoint, verify).
No user input changes: No new attack surface.
No secrets in code: The secret reference (gmail-oauth-token) is a k8s secret name, not a credential.
No auth path changes: Auth logic is untouched.

NITS

busybox image tag: busybox:1.36 is a minor-version tag that will float across patch releases. For reproducibility, consider pinning to a digest or patch version (e.g., busybox:1.36.1). Low risk given the trivial cp command, but worth noting for supply-chain hygiene.
Glob safety on cp command: The command cp /secrets-ro/google-oauth/* /secrets/google-oauth/ relies on shell globbing. If the secret has no keys (empty secret), the glob will fail with "No match" on busybox sh. This is an unlikely edge case (the secret must have keys to function), but a defensive alternative would be: cp -r /secrets-ro/google-oauth/. /secrets/google-oauth/ which avoids glob expansion entirely.
No resource limits on init container: The init container has no resources block. For a cp of a few small OAuth token files this is fine in practice, but adding minimal limits (e.g., cpu: 10m, memory: 16Mi) would be consistent with the app container's resource discipline.

SOP COMPLIANCE

Branch named after issue (136-fix-gmail-oauth-readonly-mount references #136)
PR body has Summary, Changes, Test Plan, Related sections
Related section references plan slug -- PR body does not mention plan-wkq. It references Closes #136 and companion PR but no plan slug.
No secrets committed (secret is referenced by k8s secret name only)
No unnecessary file changes (single file, tightly scoped)
Commit messages are descriptive (title: fix: writable emptyDir for gmail oauth token refresh)

PROCESS OBSERVATIONS

DORA / Change Failure Risk: Low. This is a well-understood k8s pattern applied to a single deployment. The blast radius is limited to the basketball-api pod. Failure mode is benign -- if the init container fails, the pod does not start, and the previous revision remains healthy via ArgoCD rollback.
Companion PR: The PR body mentions a companion change in pal-e-deployments for the production kustomize overlay. Reviewers should verify the companion PR carries the same emptyDir + init container pattern before merging either PR.
Unblocks: This fix unblocks all email-sending functionality (password reset #132, registration confirmation, roster exports, tryout announcements). High business value for a small change.
Issue #130 (Spike: Gmail OAuth token persistence in Postgres): The PR body does not reference this, but the emptyDir approach is a tactical fix. Issue #130 describes the strategic solution (persisting tokens in Postgres instead of the filesystem). This PR correctly solves the immediate problem without scope-creeping into the longer-term migration.

VERDICT: APPROVED

## PR #151 Review ### DOMAIN REVIEW **Tech stack**: Kubernetes deployment YAML (kustomize base manifest). No application code changed. **Pattern assessment**: The init container + emptyDir pattern is a well-established Kubernetes idiom for making secret-mounted files writable. The design is correct: 1. **Init container** (`copy-gmail-oauth`) mounts the `gmail-oauth` secret volume read-only at `/secrets-ro/google-oauth`, copies all files to emptyDir at `/secrets/google-oauth`. 2. **App container** mounts the writable emptyDir at `/secrets/google-oauth` -- same path the app expects via `BASKETBALL_GMAIL_SECRETS_DIR=/secrets/google-oauth`. 3. **Pod restart safety**: Init container runs on every pod start, re-copying from the canonical secret. emptyDir is ephemeral by design -- correct behavior here. 4. **Volume definitions**: Both `gmail-oauth` (secret) and `gmail-oauth-writable` (emptyDir) are declared in the volumes section. The secret volume is still needed for the init container source. **Verified**: The `BASKETBALL_GMAIL_SECRETS_DIR` env var (line 58 in deployment.yaml) points to `/secrets/google-oauth`, which matches the emptyDir mountPath on the app container. Path alignment is correct. **Verified**: The `gmail_sdk.GmailClient` in `src/basketball_api/services/email.py` uses `settings.gmail_secrets_dir` for token read/write. The writable emptyDir mount at that path resolves the `OSError: Read-only file system` crash. ### BLOCKERS None. This is a pure infrastructure fix -- a single k8s manifest change with no application code modifications. The BLOCKER criteria do not apply: - **Test coverage**: No new application functionality introduced. The change is a deployment manifest fix. The existing CI pipeline (`pytest`, `ruff check`, `ruff format`) validates application code, not k8s manifests. The Test Plan appropriately describes manual validation steps (deploy, wait for token expiry, trigger email endpoint, verify). - **No user input changes**: No new attack surface. - **No secrets in code**: The secret reference (`gmail-oauth-token`) is a k8s secret name, not a credential. - **No auth path changes**: Auth logic is untouched. ### NITS 1. **busybox image tag**: `busybox:1.36` is a minor-version tag that will float across patch releases. For reproducibility, consider pinning to a digest or patch version (e.g., `busybox:1.36.1`). Low risk given the trivial `cp` command, but worth noting for supply-chain hygiene. 2. **Glob safety on cp command**: The command `cp /secrets-ro/google-oauth/* /secrets/google-oauth/` relies on shell globbing. If the secret has no keys (empty secret), the glob will fail with "No match" on busybox sh. This is an unlikely edge case (the secret must have keys to function), but a defensive alternative would be: `cp -r /secrets-ro/google-oauth/. /secrets/google-oauth/` which avoids glob expansion entirely. 3. **No resource limits on init container**: The init container has no `resources` block. For a `cp` of a few small OAuth token files this is fine in practice, but adding minimal limits (e.g., `cpu: 10m`, `memory: 16Mi`) would be consistent with the app container's resource discipline. ### SOP COMPLIANCE - [x] Branch named after issue (`136-fix-gmail-oauth-readonly-mount` references #136) - [x] PR body has Summary, Changes, Test Plan, Related sections - [ ] Related section references plan slug -- PR body does not mention `plan-wkq`. It references `Closes #136` and companion PR but no plan slug. - [x] No secrets committed (secret is referenced by k8s secret name only) - [x] No unnecessary file changes (single file, tightly scoped) - [x] Commit messages are descriptive (title: `fix: writable emptyDir for gmail oauth token refresh`) ### PROCESS OBSERVATIONS - **DORA / Change Failure Risk**: Low. This is a well-understood k8s pattern applied to a single deployment. The blast radius is limited to the basketball-api pod. Failure mode is benign -- if the init container fails, the pod does not start, and the previous revision remains healthy via ArgoCD rollback. - **Companion PR**: The PR body mentions a companion change in `pal-e-deployments` for the production kustomize overlay. Reviewers should verify the companion PR carries the same emptyDir + init container pattern before merging either PR. - **Unblocks**: This fix unblocks all email-sending functionality (password reset #132, registration confirmation, roster exports, tryout announcements). High business value for a small change. - **Issue #130 (Spike: Gmail OAuth token persistence in Postgres)**: The PR body does not reference this, but the emptyDir approach is a tactical fix. Issue #130 describes the strategic solution (persisting tokens in Postgres instead of the filesystem). This PR correctly solves the immediate problem without scope-creeping into the longer-term migration. ### VERDICT: APPROVED