ldraney/pal-e-platform

Fork 0

Remove stale basketball-api and pal-e-app namespace references blocking CI apply #449

New issue

Closed

opened 2026-06-16 18:31:54 +00:00 by ldraney · 2 comments

ldraney commented

2026-06-16 18:31:54 +00:00

Owner

Type

Bug

Lineage

Discovered during DNS record deployment (godaddy-tofu integration). No parent plan — standalone fix.

Repo

ldraney/pal-e-platform

What Broke

Every push-to-main CI apply fails because Terraform references two namespaces that no longer exist on the cluster: basketball-api and pal-e-app. These are data sources in module.database — Terraform plans using stale state data, then the apply fails when the k8s API rejects creates in non-existent namespaces. This blocks ALL infrastructure changes, including DNS record creation for palinks.app and landscaping-assistant.app.

Repro Steps

Push any change to main on pal-e-platform
CI pipeline runs full apply
Apply fails with namespaces "basketball-api" not found and namespaces "pal-e-app" not found

Expected Behavior

CI apply should succeed. Resources targeting non-existent namespaces should have been removed when the namespaces were deleted.

Environment

Woodpecker CI pipeline #816 (first pipeline with visible apply output, thanks to PR #448)
kubectl get namespaces confirms neither namespace exists
Related namespaces that DO exist: palinks, paldocs, pal-e-admin, pal-e-docs, westside-ror

Scope

Data sources in state (stale — namespace gone):

data.kubernetes_namespace_v1.basketball_api (module.database)
data.kubernetes_namespace_v1.pal_e_production (module.database)

Managed resources in state (orphaned — target namespace gone):

kubernetes_config_map_v1.basketball_api_dashboard (module.monitoring)
kubernetes_config_map_v1.pal_e_production_dashboard (module.monitoring)
kubernetes_manifest.embedding_alerts (module.monitoring)
kubernetes_manifest.embedding_worker_service_monitor (module.monitoring)
kubernetes_manifest.gmail_oauth_expiry_alert (module.monitoring)
kubernetes_manifest.payment_pipeline_alerts (module.monitoring)

New resources in code but NOT in state (fail on create):

kubernetes_secret_v1.paledocs_db_url (module.database) — targets pal-e-app
kubernetes_job_v1.admin_app_user_provision (module.database) — targets basketball-api
kubernetes_service_v1.embedding_worker_metrics (module.ops) — targets pal-e-app
kubernetes_manifest.netpol_basketball_api (root) — targets basketball-api

Files to modify:

terraform/modules/database/main.tf — remove data sources (lines 61-64, 110-114), paledocs_db_url secret, admin_app_user_provision job
terraform/modules/database/outputs.tf — remove namespace outputs
terraform/modules/database/variables.tf — remove admin_app_db_password references
terraform/modules/ops/main.tf — remove embedding worker metrics service (lines 28-55)
terraform/modules/ops/variables.tf — remove pal-e-app namespace variable
terraform/network-policies.tf — remove netpol_basketball_api resource (lines 192-215), remove basketball-api/pal-e-app from from rules
terraform/modules/monitoring/main.tf — remove basketball-api and pal-e-app dashboards, alerts, service monitors
terraform/variables.tf — remove admin_app_db_password variable if fully orphaned

State operations (must run with cluster access):

tofu state rm 'module.database.data.kubernetes_namespace_v1.basketball_api'
tofu state rm 'module.database.data.kubernetes_namespace_v1.pal_e_production'
tofu state rm 'module.monitoring.kubernetes_config_map_v1.basketball_api_dashboard'
tofu state rm 'module.monitoring.kubernetes_config_map_v1.pal_e_production_dashboard'
tofu state rm 'module.monitoring.kubernetes_manifest.embedding_alerts'
tofu state rm 'module.monitoring.kubernetes_manifest.embedding_worker_service_monitor'
tofu state rm 'module.monitoring.kubernetes_manifest.gmail_oauth_expiry_alert'
tofu state rm 'module.monitoring.kubernetes_manifest.payment_pipeline_alerts'

Approach

Remove all code references to basketball-api and pal-e-app namespaces
Remove the admin_app_db_password variable and secret if fully orphaned
Run tofu state rm for orphaned resources before merge (state ops must happen before the apply — one-time manual operation with kubectl/kubeconfig access)
Verify tofu plan shows clean diff
Merge and confirm push-to-main apply succeeds — including DNS record creation

Acceptance Criteria

tofu plan runs clean with no namespace errors
Push-to-main CI apply succeeds
DNS records for palinks.app and landscaping-assistant.app are created
No orphaned resources remain in state for deleted namespaces

Risk

State manipulation — tofu state rm is irreversible but safe here: the orphaned resources don't exist on-cluster, so removing from state won't destroy anything.
Monitoring gaps — removing dashboards/alerts for basketball-api and pal-e-app is correct since the services don't exist.
Variable removal — admin_app_db_password is in Salt pillars and Woodpecker secrets. Removing the TF variable is safe; cleaning up Salt/Woodpecker is optional follow-up.

PR #446 — wired missing TF_VAR secrets
PR #447 — added tail-100 for apply output
PR #448 — extracted apply to script file (made this error visible for the first time)
DNS deployment depends on this fix (godaddy-tofu provider integration)

### Type Bug ### Lineage Discovered during DNS record deployment (godaddy-tofu integration). No parent plan — standalone fix. ### Repo ldraney/pal-e-platform ### What Broke Every push-to-main CI apply fails because Terraform references two namespaces that no longer exist on the cluster: `basketball-api` and `pal-e-app`. These are `data` sources in `module.database` — Terraform plans using stale state data, then the apply fails when the k8s API rejects creates in non-existent namespaces. This blocks ALL infrastructure changes, including DNS record creation for `palinks.app` and `landscaping-assistant.app`. ### Repro Steps 1. Push any change to main on pal-e-platform 2. CI pipeline runs full apply 3. Apply fails with `namespaces "basketball-api" not found` and `namespaces "pal-e-app" not found` ### Expected Behavior CI apply should succeed. Resources targeting non-existent namespaces should have been removed when the namespaces were deleted. ### Environment - Woodpecker CI pipeline #816 (first pipeline with visible apply output, thanks to PR #448) - `kubectl get namespaces` confirms neither namespace exists - Related namespaces that DO exist: `palinks`, `paldocs`, `pal-e-admin`, `pal-e-docs`, `westside-ror` ### Scope **Data sources in state (stale — namespace gone):** - `data.kubernetes_namespace_v1.basketball_api` (module.database) - `data.kubernetes_namespace_v1.pal_e_production` (module.database) **Managed resources in state (orphaned — target namespace gone):** - `kubernetes_config_map_v1.basketball_api_dashboard` (module.monitoring) - `kubernetes_config_map_v1.pal_e_production_dashboard` (module.monitoring) - `kubernetes_manifest.embedding_alerts` (module.monitoring) - `kubernetes_manifest.embedding_worker_service_monitor` (module.monitoring) - `kubernetes_manifest.gmail_oauth_expiry_alert` (module.monitoring) - `kubernetes_manifest.payment_pipeline_alerts` (module.monitoring) **New resources in code but NOT in state (fail on create):** - `kubernetes_secret_v1.paledocs_db_url` (module.database) — targets pal-e-app - `kubernetes_job_v1.admin_app_user_provision` (module.database) — targets basketball-api - `kubernetes_service_v1.embedding_worker_metrics` (module.ops) — targets pal-e-app - `kubernetes_manifest.netpol_basketball_api` (root) — targets basketball-api **Files to modify:** - `terraform/modules/database/main.tf` — remove data sources (lines 61-64, 110-114), paledocs_db_url secret, admin_app_user_provision job - `terraform/modules/database/outputs.tf` — remove namespace outputs - `terraform/modules/database/variables.tf` — remove admin_app_db_password references - `terraform/modules/ops/main.tf` — remove embedding worker metrics service (lines 28-55) - `terraform/modules/ops/variables.tf` — remove pal-e-app namespace variable - `terraform/network-policies.tf` — remove netpol_basketball_api resource (lines 192-215), remove basketball-api/pal-e-app from `from` rules - `terraform/modules/monitoring/main.tf` — remove basketball-api and pal-e-app dashboards, alerts, service monitors - `terraform/variables.tf` — remove admin_app_db_password variable if fully orphaned **State operations (must run with cluster access):** ``` tofu state rm 'module.database.data.kubernetes_namespace_v1.basketball_api' tofu state rm 'module.database.data.kubernetes_namespace_v1.pal_e_production' tofu state rm 'module.monitoring.kubernetes_config_map_v1.basketball_api_dashboard' tofu state rm 'module.monitoring.kubernetes_config_map_v1.pal_e_production_dashboard' tofu state rm 'module.monitoring.kubernetes_manifest.embedding_alerts' tofu state rm 'module.monitoring.kubernetes_manifest.embedding_worker_service_monitor' tofu state rm 'module.monitoring.kubernetes_manifest.gmail_oauth_expiry_alert' tofu state rm 'module.monitoring.kubernetes_manifest.payment_pipeline_alerts' ``` ### Approach 1. Remove all code references to `basketball-api` and `pal-e-app` namespaces 2. Remove the `admin_app_db_password` variable and secret if fully orphaned 3. Run `tofu state rm` for orphaned resources before merge (state ops must happen before the apply — one-time manual operation with kubectl/kubeconfig access) 4. Verify `tofu plan` shows clean diff 5. Merge and confirm push-to-main apply succeeds — including DNS record creation ### Acceptance Criteria - [ ] `tofu plan` runs clean with no namespace errors - [ ] Push-to-main CI apply succeeds - [ ] DNS records for `palinks.app` and `landscaping-assistant.app` are created - [ ] No orphaned resources remain in state for deleted namespaces ### Risk - **State manipulation** — `tofu state rm` is irreversible but safe here: the orphaned resources don't exist on-cluster, so removing from state won't destroy anything. - **Monitoring gaps** — removing dashboards/alerts for basketball-api and pal-e-app is correct since the services don't exist. - **Variable removal** — `admin_app_db_password` is in Salt pillars and Woodpecker secrets. Removing the TF variable is safe; cleaning up Salt/Woodpecker is optional follow-up. ### Related - PR #446 — wired missing TF_VAR secrets - PR #447 — added tail-100 for apply output - PR #448 — extracted apply to script file (made this error visible for the first time) - DNS deployment depends on this fix (godaddy-tofu provider integration)

ldraney commented

2026-06-16 18:35:44 +00:00

Author

Owner

Issue #449 Template Review

TEMPLATE CONFORMANCE

### Type present and valid (Bug)
### Lineage present and descriptive
### Repo present
### What Broke present and specific (exact error messages, blocking impact)
### Repro Steps present and reproducible
### Expected Behavior present
### Environment present with CI pipeline reference
### Acceptance Criteria present with checkboxes
### Related present with PR cross-references
Additional sections (### Scope, ### Approach, ### Risk) add value beyond template minimum

Template conformance: PASS. All required bug template sections present and non-empty.

CONTENT QUALITY

Strengths:

The resource inventory is thorough and well-categorized (data sources vs managed resources vs new-but-unappliable resources)
State rm commands use correct module path syntax (module.database.data.kubernetes_namespace_v1.basketball_api)
Risk section correctly identifies that tofu state rm is safe because the underlying k8s resources do not exist
The distinction between "in state but orphaned" vs "in code but never applied" is operationally important and well-articulated

Findings requiring attention:

1. MISSED FILE: terraform/main.tf (root orchestrator)

The issue lists 8 files under "Files to modify" but misses terraform/main.tf, which contains:

Line 126: pal_e_production_namespace = module.database.pal_e_production_namespace -- passes the stale namespace output from database module into the ops module. Once outputs.tf removes pal_e_production_namespace, this line will cause a validation error unless also removed.
Line 289: moved block: from = kubernetes_config_map_v1.pal_e_docs_dashboard / to = module.monitoring.kubernetes_config_map_v1.pal_e_production_dashboard -- references the dashboard resource being deleted. This moved block must be removed or tofu plan will error referencing a nonexistent target.
Lines 297-300: moved blocks for embedding_worker_service_monitor and embedding_alerts -- both point to resources listed for state rm. These must also be cleaned up.

This is a scope gap. Without cleaning terraform/main.tf, the code removal in modules will break validation before apply even runs.

2. Network policy from rules at lines 176-177 of network-policies.tf

The issue mentions removing basketball-api/pal-e-app from 'from' rules which is correct. Verified at lines 176-177 of terraform/network-policies.tf -- these are namespaceSelector entries in the postgres namespace ingress policy. Removing them is safe since the source namespaces no longer exist, but note: these are not blocking (a from rule referencing a nonexistent namespace simply never matches -- it does not cause apply failure). The issue correctly identifies them for cleanup but does not distinguish their severity from the blocking items.

3. Dashboard JSON files will become orphaned

terraform/modules/monitoring/main.tf line 756 references dashboards/pal-e-app-golden-signals.json and line 920 references dashboards/basketball-api-golden-signals.json. When the ConfigMap resources are removed, these JSON files become dead code. The issue does not mention cleaning them up. Non-blocking but should be noted as follow-up to avoid dashboard file rot.

4. Approach ordering: state-rm-first is correct but needs emphasis

The issue says step 3 is "Run tofu state rm for orphaned resources before merge." This is the correct ordering -- state rm MUST happen before the code-removal PR is applied. If code is removed first and CI runs apply, Terraform will attempt to destroy the orphaned resources (which don't exist on-cluster), potentially causing confusing errors. The issue gets this right but buries it in step 3. For a production state manipulation, the approach section should lead with the state operations and explicitly call out: "state rm is a prerequisite gate -- do NOT merge the code PR until state ops are confirmed."

5. Overlap with issues #411 and #412

Issue #411 ("Remove deprecated pal-e-app references from Terraform") and #412 ("Remove deprecated westside-admin references from Terraform") are both open and overlap significantly with #449's scope:

#411 targets the same paledocs_db_url and embedding_worker_metrics resources
#412 targets westside-admin references including admin_app_db_url_westside_admin

The ### Lineage section says "standalone" but this is actually a superset of #411 and partially overlaps #412. When #449 is completed, #411 should be closed as resolved-by and #412 should be checked for remaining scope. The issue should cross-reference these to prevent duplicate work.

6. State rm command for kubernetes_manifest resources

The tofu state rm commands for kubernetes_manifest resources (embedding_alerts, embedding_worker_service_monitor, gmail_oauth_expiry_alert, payment_pipeline_alerts) use the format module.monitoring.kubernetes_manifest.X. This is syntactically correct for the Plugin Framework. However, kubernetes_manifest resources sometimes have composite state keys (e.g., including the GVK). If the state rm fails with "no matching resource," the operator should run tofu state list | grep embedding_alerts first to verify the exact state address. The issue should note this as a precaution.

7. outputs.tf has additional stale references beyond namespace output

terraform/modules/database/outputs.tf lines 15-26 contain admin_app_db_url_secret_name and admin_app_db_url_namespaces outputs that reference kubernetes_secret_v1.admin_app_db_url -- a basketball-api-targeted resource. These outputs will break validation if the secret resource is removed but the outputs are not. The issue mentions "remove namespace outputs" but does not enumerate all affected outputs. The admin_app_db_url_* outputs must also be removed (or will cause a downstream error in any module consuming them).

BLOCKERS

1. Incomplete file target list -- terraform/main.tf is missing from "Files to modify." The moved blocks and module parameter wiring in this file will cause validation failure after the module-level code is removed. This must be added to scope.

NITS

The ### Environment section lists Woodpecker CI pipeline #816 but does not include the cluster/namespace format from the template (Cluster: archbox k3s). Minor template conformance gap.
Acceptance criteria bullet 3 ("DNS records for palinks.app and landscaping-assistant.app are created") is a downstream effect, not directly related to this bug fix. It would be cleaner as a separate verification step or linked issue rather than an AC of this ticket.
The ### Scope section is excellent but not part of the standard bug template. Consider whether this level of detail belongs in the issue body or in a linked implementation plan. For a state-manipulation ticket of this severity, having it inline is the right call.

PROCESS OBSERVATIONS

This is a high-severity infrastructure blocker that has been accumulating since the namespace deletions. The fact that it was only discovered when CI apply output became visible (PR #448) suggests that CI was previously running blind -- apply failures were silent. This is a systemic observability gap worth a follow-up issue.
Issues #411, #412, and #449 represent the same class of problem (stale namespace references after deletion). A "Rename/Deletion Audit" SOP (issue #294 is already open for this) would prevent recurrence.
The state rm operations are irreversible and target production state. The issue correctly identifies the risk as low (orphaned resources), but the execution should be documented with before/after tofu state list output for auditability.

VERDICT: NOT APPROVED

Single blocker: terraform/main.tf is missing from the file target list. The moved blocks (lines 287-300) and module parameter wiring (line 126) in this file will cause tofu validate failure after module-level code is removed. Add this file to scope, cross-reference overlapping issues #411/#412, and note the outputs.tf additional stale outputs -- then this is ready to move forward.

## Issue #449 Template Review ### TEMPLATE CONFORMANCE - [x] `### Type` present and valid (Bug) - [x] `### Lineage` present and descriptive - [x] `### Repo` present - [x] `### What Broke` present and specific (exact error messages, blocking impact) - [x] `### Repro Steps` present and reproducible - [x] `### Expected Behavior` present - [x] `### Environment` present with CI pipeline reference - [x] `### Acceptance Criteria` present with checkboxes - [x] `### Related` present with PR cross-references - [x] Additional sections (`### Scope`, `### Approach`, `### Risk`) add value beyond template minimum Template conformance: PASS. All required bug template sections present and non-empty. ### CONTENT QUALITY **Strengths:** - The resource inventory is thorough and well-categorized (data sources vs managed resources vs new-but-unappliable resources) - State rm commands use correct module path syntax (`module.database.data.kubernetes_namespace_v1.basketball_api`) - Risk section correctly identifies that `tofu state rm` is safe because the underlying k8s resources do not exist - The distinction between "in state but orphaned" vs "in code but never applied" is operationally important and well-articulated **Findings requiring attention:** **1. MISSED FILE: `terraform/main.tf` (root orchestrator)** The issue lists 8 files under "Files to modify" but misses `terraform/main.tf`, which contains: - **Line 126:** `pal_e_production_namespace = module.database.pal_e_production_namespace` -- passes the stale namespace output from database module into the ops module. Once `outputs.tf` removes `pal_e_production_namespace`, this line will cause a validation error unless also removed. - **Line 289:** `moved` block: `from = kubernetes_config_map_v1.pal_e_docs_dashboard` / `to = module.monitoring.kubernetes_config_map_v1.pal_e_production_dashboard` -- references the dashboard resource being deleted. This `moved` block must be removed or `tofu plan` will error referencing a nonexistent target. - **Lines 297-300:** `moved` blocks for `embedding_worker_service_monitor` and `embedding_alerts` -- both point to resources listed for state rm. These must also be cleaned up. This is a scope gap. Without cleaning `terraform/main.tf`, the code removal in modules will break validation before apply even runs. **2. Network policy `from` rules at lines 176-177 of `network-policies.tf`** The issue mentions removing `basketball-api/pal-e-app from 'from' rules` which is correct. Verified at lines 176-177 of `terraform/network-policies.tf` -- these are `namespaceSelector` entries in the postgres namespace ingress policy. Removing them is safe since the source namespaces no longer exist, but note: these are not blocking (a `from` rule referencing a nonexistent namespace simply never matches -- it does not cause apply failure). The issue correctly identifies them for cleanup but does not distinguish their severity from the blocking items. **3. Dashboard JSON files will become orphaned** `terraform/modules/monitoring/main.tf` line 756 references `dashboards/pal-e-app-golden-signals.json` and line 920 references `dashboards/basketball-api-golden-signals.json`. When the ConfigMap resources are removed, these JSON files become dead code. The issue does not mention cleaning them up. Non-blocking but should be noted as follow-up to avoid dashboard file rot. **4. Approach ordering: state-rm-first is correct but needs emphasis** The issue says step 3 is "Run `tofu state rm` for orphaned resources before merge." This is the correct ordering -- state rm MUST happen before the code-removal PR is applied. If code is removed first and CI runs apply, Terraform will attempt to destroy the orphaned resources (which don't exist on-cluster), potentially causing confusing errors. The issue gets this right but buries it in step 3. For a production state manipulation, the approach section should lead with the state operations and explicitly call out: "state rm is a prerequisite gate -- do NOT merge the code PR until state ops are confirmed." **5. Overlap with issues #411 and #412** Issue #411 ("Remove deprecated pal-e-app references from Terraform") and #412 ("Remove deprecated westside-admin references from Terraform") are both open and overlap significantly with #449's scope: - #411 targets the same `paledocs_db_url` and `embedding_worker_metrics` resources - #412 targets `westside-admin` references including `admin_app_db_url_westside_admin` The `### Lineage` section says "standalone" but this is actually a superset of #411 and partially overlaps #412. When #449 is completed, #411 should be closed as resolved-by and #412 should be checked for remaining scope. The issue should cross-reference these to prevent duplicate work. **6. State rm command for `kubernetes_manifest` resources** The `tofu state rm` commands for `kubernetes_manifest` resources (embedding_alerts, embedding_worker_service_monitor, gmail_oauth_expiry_alert, payment_pipeline_alerts) use the format `module.monitoring.kubernetes_manifest.X`. This is syntactically correct for the Plugin Framework. However, `kubernetes_manifest` resources sometimes have composite state keys (e.g., including the GVK). If the state rm fails with "no matching resource," the operator should run `tofu state list | grep embedding_alerts` first to verify the exact state address. The issue should note this as a precaution. **7. `outputs.tf` has additional stale references beyond namespace output** `terraform/modules/database/outputs.tf` lines 15-26 contain `admin_app_db_url_secret_name` and `admin_app_db_url_namespaces` outputs that reference `kubernetes_secret_v1.admin_app_db_url` -- a basketball-api-targeted resource. These outputs will break validation if the secret resource is removed but the outputs are not. The issue mentions "remove namespace outputs" but does not enumerate all affected outputs. The `admin_app_db_url_*` outputs must also be removed (or will cause a downstream error in any module consuming them). ### BLOCKERS **1. Incomplete file target list** -- `terraform/main.tf` is missing from "Files to modify." The `moved` blocks and module parameter wiring in this file will cause validation failure after the module-level code is removed. This must be added to scope. ### NITS - The `### Environment` section lists Woodpecker CI pipeline #816 but does not include the cluster/namespace format from the template (`Cluster: archbox k3s`). Minor template conformance gap. - Acceptance criteria bullet 3 ("DNS records for palinks.app and landscaping-assistant.app are created") is a downstream effect, not directly related to this bug fix. It would be cleaner as a separate verification step or linked issue rather than an AC of this ticket. - The `### Scope` section is excellent but not part of the standard bug template. Consider whether this level of detail belongs in the issue body or in a linked implementation plan. For a state-manipulation ticket of this severity, having it inline is the right call. ### PROCESS OBSERVATIONS - This is a high-severity infrastructure blocker that has been accumulating since the namespace deletions. The fact that it was only discovered when CI apply output became visible (PR #448) suggests that CI was previously running blind -- apply failures were silent. This is a systemic observability gap worth a follow-up issue. - Issues #411, #412, and #449 represent the same class of problem (stale namespace references after deletion). A "Rename/Deletion Audit" SOP (issue #294 is already open for this) would prevent recurrence. - The state rm operations are irreversible and target production state. The issue correctly identifies the risk as low (orphaned resources), but the execution should be documented with before/after `tofu state list` output for auditability. ### VERDICT: NOT APPROVED Single blocker: `terraform/main.tf` is missing from the file target list. The `moved` blocks (lines 287-300) and module parameter wiring (line 126) in this file will cause `tofu validate` failure after module-level code is removed. Add this file to scope, cross-reference overlapping issues #411/#412, and note the `outputs.tf` additional stale outputs -- then this is ready to move forward.

ldraney commented

2026-06-16 18:36:24 +00:00

Author

Owner

Scope Update (from QA review)

QA flagged missing items. Adding to scope:

Additional files to modify:

terraform/main.tf — remove module wiring:
- Line 118: admin_app_db_password passed to database module
- Line 126: pal_e_production_namespace passed from database to ops
- Lines 289, 298-299, 303-304: moved blocks for pal_e_production_dashboard, embedding_worker_service_monitor, embedding_alerts
- Lines 502-503: moved block for embedding_worker_metrics

Additional outputs to remove from terraform/modules/database/outputs.tf:

admin_app_db_url_secret_name (line 16)
admin_app_db_url_namespaces (line 21)

Dashboard JSON files to delete:

terraform/dashboards/basketball-api-golden-signals.json
terraform/dashboards/pal-e-app-golden-signals.json

State rm precaution: Run tofu state list | grep -E 'basketball|pal_e_production|embedding|admin_app' first to confirm exact resource addresses before running state rm commands.

Ordering note: State rm MUST complete before the code-removal PR is merged. Otherwise the apply will try to destroy resources it can't reach (namespace gone).

## Scope Update (from QA review) QA flagged missing items. Adding to scope: **Additional files to modify:** - `terraform/main.tf` — remove module wiring: - Line 118: `admin_app_db_password` passed to database module - Line 126: `pal_e_production_namespace` passed from database to ops - Lines 289, 298-299, 303-304: `moved` blocks for pal_e_production_dashboard, embedding_worker_service_monitor, embedding_alerts - Lines 502-503: `moved` block for embedding_worker_metrics **Additional outputs to remove from `terraform/modules/database/outputs.tf`:** - `admin_app_db_url_secret_name` (line 16) - `admin_app_db_url_namespaces` (line 21) **Dashboard JSON files to delete:** - `terraform/dashboards/basketball-api-golden-signals.json` - `terraform/dashboards/pal-e-app-golden-signals.json` **State rm precaution:** Run `tofu state list | grep -E 'basketball|pal_e_production|embedding|admin_app'` first to confirm exact resource addresses before running state rm commands. **Ordering note:** State rm MUST complete before the code-removal PR is merged. Otherwise the apply will try to destroy resources it can't reach (namespace gone).