feat(database): provision admin_app Postgres role on basketball-api (#302) #304

Merged
forgejo_admin merged 1 commit from 302-admin-postgres-job into main 2026-04-26 01:13:01 +00:00
Contributor

Summary

Adds a one-shot, idempotent k8s Job (admin-app-user-provision) and companion admin-app-db-url Secret to the database terraform module. Provides westside-admin a least-privilege DML user on the basketball-api Postgres database.

Closes #302.

Labels: story:admin-row-crud, arch:postgres.

Changes

  • terraform/modules/database/main.tf — new data.kubernetes_namespace_v1.basketball_api, kubernetes_job_v1.admin_app_user_provision, kubernetes_secret_v1.admin_app_db_url. Job runs psql against postgres.basketball-api.svc.cluster.local as the basketball superuser, creates/rotates admin_app role, applies DML grants and ALTER DEFAULT PRIVILEGES for forward grants.
  • terraform/modules/database/variables.tf — new admin_app_db_password (sensitive).
  • terraform/main.tf, terraform/variables.tf — wire the new variable through to the module.
  • terraform/secrets.auto.tfvars.example — document the new secret with openssl rand -hex 32 generation hint.
  • Makefile — add admin_app_db_password to TF_SECRET_VARS so tofu-validate-secrets requires it.

Design Decisions

  • k8s Job, not terraform postgres provider. review-1097-2026-04-25 flagged that cyrilgdn/postgresql is not registered in this repo and adding it mid-flight expands provider blast radius. The Job approach reuses already-installed hashicorp/kubernetes and runs alongside the existing kubernetes_cron_job_v1.cnpg_backup_verify precedent in the same module.
  • Job name embeds a SHA256 prefix of the password. k8s Job pod templates are immutable. On password rotation the Job is replaced (not patched), so re-applying after rotating the Salt pillar value just re-runs the SQL with ALTER ROLE.
  • Idempotent SQL. DO $$ ... IF NOT EXISTS $$ for CREATE ROLE; GRANT and ALTER DEFAULT PRIVILEGES are no-ops when already in place. Re-running the apply is safe.
  • Forward grants scoped to FOR ROLE basketball. Drizzle migrations run as the basketball superuser; this guarantees future tables/sequences they create automatically grant DML to admin_app without a re-run.
  • Secret in basketball-api namespace. Matches the database location. Track B (pal-e-deployments#133) is responsible for surfacing it to the westside-admin pod (cross-namespace consumption is out of scope here, per dispatch boundary).
  • Constraints honored. No CREATE/ALTER/DROP/superuser on admin_app — only DML and USAGE.

tofu plan Output

tofu validate passes. tofu plan not included because the secrets pillar is required to render secrets.auto.tfvars; will be regenerated by the operator running make tofu-plan after adding admin_app_db_password to the Salt pillar (salt/pillar/secrets/platform.sls).

$ tofu validate
Success! The configuration is valid.

Test Plan

After make tofu-apply lands and ArgoCD reconciles:

  • Pod admin-app-user-provision-<hash>-* in basketball-api namespace reaches Completed.
  • kubectl -n basketball-api get secret admin-app-db-url -o jsonpath='{.data.DATABASE_URL}' | base64 -d returns the expected URL.
  • From a debug pod: psql "$DATABASE_URL" -c "SELECT 1" succeeds.
  • From a debug pod: psql "$DATABASE_URL" -c "INSERT INTO <some_test_table> ..." succeeds.
  • From a debug pod: psql "$DATABASE_URL" -c "CREATE TABLE foo (id int)" fails with permission denied.
  • As basketball superuser, CREATE TABLE bar (id int); verify admin_app immediately has SELECT/INSERT/UPDATE/DELETE on bar (proves ALTER DEFAULT PRIVILEGES works).
  • Rotate password in pillar, re-apply; verify Job is replaced and connection still works with the new password.

Discovered Scope

  • Salt pillar entryadmin_app_db_password must be added to salt/pillar/secrets/platform.sls (encrypted via existing GPG flow). Out of scope for this PR; operator step before make tofu-apply.
  • Cross-namespace secret surfacing — Track B (pal-e-deployments#133) needs to either re-emit the URL via a kustomize secretGenerator in the westside-admin overlay or stand up a small replicator. Documented in the inline module comment.
  • SOP sop-admin-app-password-rotation — pal-e-docs note creation belongs to main session (agents do not write to pal-e-docs). Surfacing for follow-up.

Review Checklist

  • tofu fmt clean (verified)
  • tofu validate passes (verified)
  • No new terraform providers added
  • Secret values not committed (only CHANGEME placeholders in example file)
  • TF_SECRET_VARS updated in Makefile
  • Job is idempotent (re-runnable without error)
  • Least-privilege grants only (no CREATE/ALTER/DROP/superuser)
  • Inline module documentation explains rationale + scope-revision lineage
  • Story + arch labels noted in PR body
  • review-1097-2026-04-25 — scope investigation that drove path A (k8s Job over cyrilgdn provider)
  • feedback_never_write_prod_db — admin_app is the legitimized DML write user
  • project-westside-admin — consuming project
  • arch-deployment-westside-admin — architecture anchor
  • Forgejo issue: #302
  • Blocks: forgejo_admin/westside-admin#1 (Drizzle), forgejo_admin/pal-e-deployments#133 (overlay secrets)
  • Companion: #301 (westside-admin Keycloak client)
## Summary Adds a one-shot, idempotent k8s Job (`admin-app-user-provision`) and companion `admin-app-db-url` Secret to the `database` terraform module. Provides westside-admin a least-privilege DML user on the basketball-api Postgres database. Closes #302. Labels: `story:admin-row-crud`, `arch:postgres`. ## Changes - `terraform/modules/database/main.tf` — new `data.kubernetes_namespace_v1.basketball_api`, `kubernetes_job_v1.admin_app_user_provision`, `kubernetes_secret_v1.admin_app_db_url`. Job runs `psql` against `postgres.basketball-api.svc.cluster.local` as the basketball superuser, creates/rotates `admin_app` role, applies DML grants and `ALTER DEFAULT PRIVILEGES` for forward grants. - `terraform/modules/database/variables.tf` — new `admin_app_db_password` (sensitive). - `terraform/main.tf`, `terraform/variables.tf` — wire the new variable through to the module. - `terraform/secrets.auto.tfvars.example` — document the new secret with `openssl rand -hex 32` generation hint. - `Makefile` — add `admin_app_db_password` to `TF_SECRET_VARS` so `tofu-validate-secrets` requires it. ## Design Decisions - **k8s Job, not terraform postgres provider.** `review-1097-2026-04-25` flagged that `cyrilgdn/postgresql` is not registered in this repo and adding it mid-flight expands provider blast radius. The Job approach reuses already-installed `hashicorp/kubernetes` and runs alongside the existing `kubernetes_cron_job_v1.cnpg_backup_verify` precedent in the same module. - **Job name embeds a SHA256 prefix of the password.** k8s Job pod templates are immutable. On password rotation the Job is replaced (not patched), so re-applying after rotating the Salt pillar value just re-runs the SQL with `ALTER ROLE`. - **Idempotent SQL.** `DO $$ ... IF NOT EXISTS $$` for CREATE ROLE; GRANT and ALTER DEFAULT PRIVILEGES are no-ops when already in place. Re-running the apply is safe. - **Forward grants scoped to `FOR ROLE basketball`.** Drizzle migrations run as the basketball superuser; this guarantees future tables/sequences they create automatically grant DML to admin_app without a re-run. - **Secret in basketball-api namespace.** Matches the database location. Track B (`pal-e-deployments#133`) is responsible for surfacing it to the westside-admin pod (cross-namespace consumption is out of scope here, per dispatch boundary). - **Constraints honored.** No CREATE/ALTER/DROP/superuser on admin_app — only DML and USAGE. ## tofu plan Output `tofu validate` passes. `tofu plan` not included because the secrets pillar is required to render `secrets.auto.tfvars`; will be regenerated by the operator running `make tofu-plan` after adding `admin_app_db_password` to the Salt pillar (`salt/pillar/secrets/platform.sls`). ``` $ tofu validate Success! The configuration is valid. ``` ## Test Plan After `make tofu-apply` lands and ArgoCD reconciles: - [ ] Pod `admin-app-user-provision-<hash>-*` in `basketball-api` namespace reaches `Completed`. - [ ] `kubectl -n basketball-api get secret admin-app-db-url -o jsonpath='{.data.DATABASE_URL}' | base64 -d` returns the expected URL. - [ ] From a debug pod: `psql "$DATABASE_URL" -c "SELECT 1"` succeeds. - [ ] From a debug pod: `psql "$DATABASE_URL" -c "INSERT INTO <some_test_table> ..."` succeeds. - [ ] From a debug pod: `psql "$DATABASE_URL" -c "CREATE TABLE foo (id int)"` fails with `permission denied`. - [ ] As basketball superuser, `CREATE TABLE bar (id int)`; verify admin_app immediately has SELECT/INSERT/UPDATE/DELETE on `bar` (proves ALTER DEFAULT PRIVILEGES works). - [ ] Rotate password in pillar, re-apply; verify Job is replaced and connection still works with the new password. ## Discovered Scope - **Salt pillar entry** — `admin_app_db_password` must be added to `salt/pillar/secrets/platform.sls` (encrypted via existing GPG flow). Out of scope for this PR; operator step before `make tofu-apply`. - **Cross-namespace secret surfacing** — Track B (`pal-e-deployments#133`) needs to either re-emit the URL via a kustomize secretGenerator in the westside-admin overlay or stand up a small replicator. Documented in the inline module comment. - **SOP `sop-admin-app-password-rotation`** — pal-e-docs note creation belongs to main session (agents do not write to pal-e-docs). Surfacing for follow-up. ## Review Checklist - [ ] `tofu fmt` clean (verified) - [ ] `tofu validate` passes (verified) - [ ] No new terraform providers added - [ ] Secret values not committed (only `CHANGEME` placeholders in example file) - [ ] `TF_SECRET_VARS` updated in Makefile - [ ] Job is idempotent (re-runnable without error) - [ ] Least-privilege grants only (no CREATE/ALTER/DROP/superuser) - [ ] Inline module documentation explains rationale + scope-revision lineage - [ ] Story + arch labels noted in PR body ## Related Notes - `review-1097-2026-04-25` — scope investigation that drove path A (k8s Job over cyrilgdn provider) - `feedback_never_write_prod_db` — admin_app is the legitimized DML write user - `project-westside-admin` — consuming project - `arch-deployment-westside-admin` — architecture anchor ## Related - Forgejo issue: #302 - Blocks: `forgejo_admin/westside-admin#1` (Drizzle), `forgejo_admin/pal-e-deployments#133` (overlay secrets) - Companion: #301 (westside-admin Keycloak client)
feat(database): provision admin_app Postgres role on basketball-api (#302)
Some checks failed
ci/woodpecker/push/woodpecker Pipeline was successful
ci/woodpecker/pr/woodpecker Pipeline failed
ci/woodpecker/pull_request_closed/woodpecker Pipeline was successful
48a51953fd
Adds a one-shot, idempotent k8s Job (`admin-app-user-provision`) and
companion DATABASE_URL Secret (`admin-app-db-url`) to the database
terraform module. Provides westside-admin a least-privilege DML user
on the basketball-api Postgres database.

The Job:
- Connects to postgres.basketball-api.svc.cluster.local as the
  basketball superuser (basketball-api-secrets/postgres-password).
- CREATE ROLE admin_app (or ALTER ROLE on rotation) with login + password.
- GRANTs USAGE on schema public, SELECT/INSERT/UPDATE/DELETE on all tables,
  USAGE on all sequences.
- ALTER DEFAULT PRIVILEGES so future Drizzle-migrated tables/sequences are
  immediately accessible to admin_app.
- Job name embeds a short hash of the password so password rotations
  trigger a replacement Job (k8s Job pod template is immutable).

The Secret encodes the canonical
`postgresql://admin_app:<pw>@postgres.basketball-api.svc.cluster.local:5432/basketball`
URL in the basketball-api namespace. Track B's westside-admin overlay
(pal-e-deployments#133) consumes it.

Scope-revision lineage: review-1097-2026-04-25 rejected the
cyrilgdn/postgresql terraform provider (not registered, would require
provider expansion mid-flight). k8s Job approach selected as the lowest-
blast-radius path.

Story: admin-row-crud   Arch: postgres
Closes #302

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Author
Contributor

PR #304 Review

DOMAIN REVIEW

Stack: OpenTofu + hashicorp/kubernetes provider, in-cluster postgres provisioning via a one-shot k8s Job (postgres:16-alpine running psql). Plus root variable plumbing and Makefile secret-vars wiring.

Pivot from ticket spec (kustomize Job in pal-e-deployments → terraform-managed kubernetes_job_v1 + kubernetes_secret_v1 in pal-e-platform/modules/database): Verdict — sound and aligns with existing convention.

Sibling pattern in the same module (paledocs_db_url, lines 67-78 of terraform/modules/database/main.tf):

  • Uses data "kubernetes_namespace_v1" to land the secret in a foreign namespace (pal-e-app)
  • Password is a sensitive var.paledocs_db_password
  • Secret name pattern <service>-db-url with DATABASE_URL key

PR #304 mirrors this shape exactly (admin-app-db-url Secret, DATABASE_URL key, data.kubernetes_namespace_v1.basketball_api, var.admin_app_db_password sensitive). The added Job is only required because basketball-api postgres is a plain Deployment (not CNPG operator-managed), so role provisioning can't piggyback on a CNPG Cluster.spec.bootstrap. The lineage in review-1097-2026-04-25 (avoid adding cyrilgdn/postgresql provider mid-flight) is the right call — provider expansion has higher blast radius than a one-shot Job using the already-installed kubernetes provider, and there is in-module precedent (kubernetes_cron_job_v1.cnpg_backup_verify).

Job/SQL quality:

  • Idempotent: DO $$ ... IF NOT EXISTS $$ for CREATE ROLE, ALTER ROLE on rotation, GRANT and ALTER DEFAULT PRIVILEGES are no-op on re-apply. Correct.
  • Password rotation handled cleanly via SHA256 prefix in Job name (k8s pod template is immutable; rename forces replacement). Nice touch.
  • Least privilege honored: SELECT/INSERT/UPDATE/DELETE + USAGE only. No CREATE/ALTER/DROP/superuser.
  • ALTER DEFAULT PRIVILEGES FOR ROLE basketball correctly handles forward grants for Drizzle migrations run as superuser.
  • psql -v admin_pw=... with format(... %L ...) is the safe quoting pattern (no SQL injection vector via password contents).
  • wait_for_completion = true + backoff_limit = 4 + ttl_seconds_after_finished = 3600 + resource requests/limits — all good k8s hygiene.
  • Reuses existing basketball-api-secrets/postgres-password for PGPASSWORD via secret_key_ref instead of duplicating the superuser password into terraform state. Correct.

Secret namespace (basketball-api vs westside-admin): Intentional and correct. Secret lives where the database lives. Cross-namespace surfacing is correctly out of scope and tracked at pal-e-deployments#133 (note: PR body mentions #133, dispatch context says #135 — minor doc reconciliation but not blocking).

BLOCKERS

None.

NITS

  1. ADMIN_APP_PASSWORD is passed as a literal env var (value = var.admin_app_db_password) rather than via a Secret + value_from. The PGPASSWORD pattern in the same Job uses secret_key_ref — would be more consistent (and avoid the value showing in kubectl describe pod for anyone with namespace read). Not a blocker since it's already in the admin-app-db-url Secret which exists right next door, but the symmetry would be nice.
  2. PR body says tofu plan deferred until the Salt pillar is populated — fine, but the eventual apply must include -lock=false (per feedback_tofu_lock_false). Worth a one-line callout in the apply runbook.
  3. PR body references pal-e-deployments#133; dispatch context says #135. Reconcile in the cross-repo issue link.
  4. arch:postgres label on Job/Secret is good — but the PR body lists story:admin-row-crud, arch:postgres while the actual Forgejo PR labels weren't reported in the diff. Verify labels are set on the PR itself.

SOP COMPLIANCE

  • Branch named after issue (302-admin-postgres-job)
  • PR body has Summary / Changes / Design Decisions / Test Plan / Discovered Scope / Related
  • Closes #302
  • Story + arch trace present (story:admin-row-crud, arch:postgres)
  • No new terraform providers added (avoided cyrilgdn/postgresql)
  • No secrets committed (only CHANGEME placeholders)
  • TF_SECRET_VARS updated in Makefile
  • Discovered scope tracked: Salt pillar (#306), cross-ns secret surfacing (deployments#133), rotation SOP (deferred to main session)
  • tofu validate clean (per PR body); tofu fmt claimed clean — re-verify on rebase if any
  • [~] tofu plan output not in PR body — acceptable here because pillar must land first; documented

PROCESS OBSERVATIONS

  • Pivot from ticket → implementation was justified in writing (review-1097-2026-04-25) before the agent committed. Good DORA hygiene: scope-revision lineage is in the module comment so a future reader won't re-litigate the choice.
  • This change has hard sequencing: #306 (Salt pillar) MUST land before make tofu-apply or the run aborts on the missing required variable. Track as an apply gate, not a merge gate.
  • Track B (pal-e-deployments#133) is downstream consumer; ensure its overlay references admin-app-db-url / DATABASE_URL exactly (matches what this PR creates).
  • DORA: deployment frequency neutral (additive resources), change failure risk low (idempotent + least-privilege + no schema mutation), MTTR positive (rotation procedure is documented in module comment).

APPLY GATES (do NOT apply until all green)

  • #306admin_app_db_password added to salt/pillar/secrets/platform.sls
  • make tofu-plan (with -lock=false) reviewed by operator
  • Track B (pal-e-deployments#133) ready or sequenced so westside-admin can consume the Secret post-apply

VERDICT: APPROVED

Merge is fine. Apply is gated on #306. Recommend Lucas merge, then drive #306 to completion before any make tofu-apply against this module.

## PR #304 Review ### DOMAIN REVIEW Stack: OpenTofu + `hashicorp/kubernetes` provider, in-cluster postgres provisioning via a one-shot k8s Job (`postgres:16-alpine` running `psql`). Plus root variable plumbing and Makefile secret-vars wiring. **Pivot from ticket spec (kustomize Job in pal-e-deployments → terraform-managed `kubernetes_job_v1` + `kubernetes_secret_v1` in pal-e-platform/modules/database):** Verdict — **sound and aligns with existing convention.** Sibling pattern in the same module (`paledocs_db_url`, lines 67-78 of `terraform/modules/database/main.tf`): - Uses `data "kubernetes_namespace_v1"` to land the secret in a foreign namespace (`pal-e-app`) - Password is a sensitive `var.paledocs_db_password` - Secret name pattern `<service>-db-url` with `DATABASE_URL` key PR #304 mirrors this shape exactly (`admin-app-db-url` Secret, `DATABASE_URL` key, `data.kubernetes_namespace_v1.basketball_api`, `var.admin_app_db_password` sensitive). The added Job is only required because basketball-api postgres is a plain Deployment (not CNPG operator-managed), so role provisioning can't piggyback on a CNPG `Cluster.spec.bootstrap`. The lineage in `review-1097-2026-04-25` (avoid adding cyrilgdn/postgresql provider mid-flight) is the right call — provider expansion has higher blast radius than a one-shot Job using the already-installed kubernetes provider, and there is in-module precedent (`kubernetes_cron_job_v1.cnpg_backup_verify`). **Job/SQL quality:** - Idempotent: `DO $$ ... IF NOT EXISTS $$` for CREATE ROLE, `ALTER ROLE` on rotation, GRANT and ALTER DEFAULT PRIVILEGES are no-op on re-apply. Correct. - Password rotation handled cleanly via SHA256 prefix in Job name (k8s pod template is immutable; rename forces replacement). Nice touch. - Least privilege honored: SELECT/INSERT/UPDATE/DELETE + USAGE only. No CREATE/ALTER/DROP/superuser. - `ALTER DEFAULT PRIVILEGES FOR ROLE basketball` correctly handles forward grants for Drizzle migrations run as superuser. - `psql -v admin_pw=...` with `format(... %L ...)` is the safe quoting pattern (no SQL injection vector via password contents). - `wait_for_completion = true` + `backoff_limit = 4` + `ttl_seconds_after_finished = 3600` + resource requests/limits — all good k8s hygiene. - Reuses existing `basketball-api-secrets/postgres-password` for PGPASSWORD via `secret_key_ref` instead of duplicating the superuser password into terraform state. Correct. **Secret namespace (basketball-api vs westside-admin):** Intentional and correct. Secret lives where the database lives. Cross-namespace surfacing is correctly out of scope and tracked at `pal-e-deployments#133` (note: PR body mentions `#133`, dispatch context says `#135` — minor doc reconciliation but not blocking). ### BLOCKERS None. ### NITS 1. `ADMIN_APP_PASSWORD` is passed as a literal env var (`value = var.admin_app_db_password`) rather than via a Secret + `value_from`. The PGPASSWORD pattern in the same Job uses `secret_key_ref` — would be more consistent (and avoid the value showing in `kubectl describe pod` for anyone with namespace read). Not a blocker since it's already in the `admin-app-db-url` Secret which exists right next door, but the symmetry would be nice. 2. PR body says `tofu plan` deferred until the Salt pillar is populated — fine, but the eventual apply must include `-lock=false` (per `feedback_tofu_lock_false`). Worth a one-line callout in the apply runbook. 3. PR body references `pal-e-deployments#133`; dispatch context says `#135`. Reconcile in the cross-repo issue link. 4. `arch:postgres` label on Job/Secret is good — but the PR body lists `story:admin-row-crud, arch:postgres` while the actual Forgejo PR labels weren't reported in the diff. Verify labels are set on the PR itself. ### SOP COMPLIANCE - [x] Branch named after issue (`302-admin-postgres-job`) - [x] PR body has Summary / Changes / Design Decisions / Test Plan / Discovered Scope / Related - [x] `Closes #302` - [x] Story + arch trace present (`story:admin-row-crud`, `arch:postgres`) - [x] No new terraform providers added (avoided cyrilgdn/postgresql) - [x] No secrets committed (only `CHANGEME` placeholders) - [x] `TF_SECRET_VARS` updated in Makefile - [x] Discovered scope tracked: Salt pillar (#306), cross-ns secret surfacing (deployments#133), rotation SOP (deferred to main session) - [x] `tofu validate` clean (per PR body); `tofu fmt` claimed clean — re-verify on rebase if any - [~] `tofu plan` output not in PR body — acceptable here because pillar must land first; documented ### PROCESS OBSERVATIONS - Pivot from ticket → implementation was justified in writing (`review-1097-2026-04-25`) before the agent committed. Good DORA hygiene: scope-revision lineage is in the module comment so a future reader won't re-litigate the choice. - This change has hard sequencing: **#306 (Salt pillar) MUST land before `make tofu-apply`** or the run aborts on the missing required variable. Track as an apply gate, not a merge gate. - Track B (`pal-e-deployments#133`) is downstream consumer; ensure its overlay references `admin-app-db-url` / `DATABASE_URL` exactly (matches what this PR creates). - DORA: deployment frequency neutral (additive resources), change failure risk low (idempotent + least-privilege + no schema mutation), MTTR positive (rotation procedure is documented in module comment). ### APPLY GATES (do NOT apply until all green) - [ ] #306 — `admin_app_db_password` added to `salt/pillar/secrets/platform.sls` - [ ] `make tofu-plan` (with `-lock=false`) reviewed by operator - [ ] Track B (`pal-e-deployments#133`) ready or sequenced so westside-admin can consume the Secret post-apply ### VERDICT: APPROVED Merge is fine. **Apply is gated on #306.** Recommend Lucas merge, then drive #306 to completion before any `make tofu-apply` against this module.
forgejo_admin deleted branch 302-admin-postgres-job 2026-04-26 01:13:01 +00:00
Sign in to join this conversation.
No description provided.