Fix: restore pal-e-docs and pal-e-ror in postgres NetworkPolicy #378

Merged

ldraney merged 1 commit from fix-postgres-netpol-restore-pal-e-docs into main 2026-05-24 15:17:24 +00:00

Conversation 2 Commits 1 Files changed 1 +2

ldraney commented 2026-05-24 14:08:19 +00:00

Owner

Copy link

Summary

• PR #376 (landscaping-assistant) rewrote the postgres NetworkPolicy ingress list and silently dropped pal-e-docs and pal-e-ror
• pal-e-docs was down 6 days — CrashLoopBackOff on postgres connection refused
• Restores both namespaces to the allow list

Changes

• terraform/network-policies.tf: added pal-e-docs and pal-e-ror to netpol_postgres ingress list (2 lines)

Test Plan

• tofu fmt — no changes
• tofu validate — Success
• tofu plan -target=kubernetes_manifest.netpol_postgres — 0 add, 1 change, 0 destroy
• tofu apply executed — NetworkPolicy updated in cluster
• pal-e-docs pods recovered to 1/1 Running
• pal-e-docs-embedding-worker recovered to 1/1 Running
• No regression — all other namespaces still allowed (landscaping-assistant, basketball-api, etc.)

Review Checklist

• No secrets committed
• No unnecessary file changes
• Commit message is descriptive

Related Notes

• Closes #377
• pal-e-platform — project this affects
• ldraney/pal-e-platform #376 — PR that caused the regression

## Summary - PR #376 (landscaping-assistant) rewrote the postgres NetworkPolicy ingress list and silently dropped `pal-e-docs` and `pal-e-ror` - pal-e-docs was down 6 days — CrashLoopBackOff on postgres connection refused - Restores both namespaces to the allow list ## Changes - `terraform/network-policies.tf`: added `pal-e-docs` and `pal-e-ror` to `netpol_postgres` ingress list (2 lines) ## Test Plan - [x] `tofu fmt` — no changes - [x] `tofu validate` — Success - [x] `tofu plan -target=kubernetes_manifest.netpol_postgres` — 0 add, 1 change, 0 destroy - [x] `tofu apply` executed — NetworkPolicy updated in cluster - [x] pal-e-docs pods recovered to 1/1 Running - [x] pal-e-docs-embedding-worker recovered to 1/1 Running - [ ] No regression — all other namespaces still allowed (landscaping-assistant, basketball-api, etc.) ## Review Checklist - [x] No secrets committed - [x] No unnecessary file changes - [x] Commit message is descriptive ## Related Notes - Closes #377 - `pal-e-platform` — project this affects - `ldraney/pal-e-platform #376` — PR that caused the regression

ldraney added 1 commit 2026-05-24 14:08:19 +00:00

Restore pal-e-docs and pal-e-ror in postgres NetworkPolicy ... 826cf2a1b3

PR #376 (landscaping-assistant) rewrote the ingress list and silently
dropped both namespaces. pal-e-docs was down 6 days as a result.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

forgejo_admin commented 2026-05-24 14:09:03 +00:00

Contributor

Copy link

Tofu Plan Output (full)

module.networking.tailscale_acl.this: Refreshing state... [id=acl]
module.forgejo.kubernetes_namespace_v1.forgejo: Refreshing state... [id=forgejo]
module.monitoring.kubernetes_namespace_v1.monitoring: Refreshing state... [id=monitoring]
module.staging.kubernetes_namespace_v1.staging: Refreshing state... [id=staging]
module.database.data.kubernetes_namespace_v1.pal_e_production: Reading...
module.database.kubernetes_namespace_v1.postgres: Refreshing state... [id=postgres]
module.keycloak.kubernetes_namespace_v1.keycloak: Refreshing state... [id=keycloak]
module.database.data.kubernetes_namespace_v1.westside_admin: Reading...
module.networking.kubernetes_namespace_v1.tailscale: Refreshing state... [id=tailscale]
module.database.data.kubernetes_namespace_v1.westside_admin: Read complete after 0s [id=westside-admin]
module.database.data.kubernetes_namespace_v1.pal_e_production: Read complete after 0s [id=pal-e-app]
module.database.data.kubernetes_namespace_v1.basketball_api: Reading...
module.database.kubernetes_namespace_v1.cnpg_system: Refreshing state... [id=cnpg-system]
module.database.data.kubernetes_namespace_v1.basketball_api: Read complete after 0s [id=basketball-api]
module.keycloak.kubernetes_secret_v1.keycloak_admin: Refreshing state... [id=keycloak/keycloak-admin]
module.keycloak.kubernetes_persistent_volume_claim_v1.keycloak_data: Refreshing state... [id=keycloak/keycloak-data]
module.keycloak.kubernetes_config_map_v1.keycloak_westside_theme: Refreshing state... [id=keycloak/keycloak-westside-theme]
module.forgejo.kubernetes_secret_v1.forgejo_oidc: Refreshing state... [id=forgejo/forgejo-oidc]
module.forgejo.kubernetes_config_map_v1.forgejo_custom_css: Refreshing state... [id=forgejo/forgejo-custom-css]
module.keycloak.kubernetes_service_v1.keycloak: Refreshing state... [id=keycloak/keycloak]
module.networking.helm_release.tailscale_operator: Refreshing state... [id=tailscale-operator]
module.monitoring.kubernetes_secret_v1.grafana_oidc: Refreshing state... [id=monitoring/grafana-oidc]
module.monitoring.kubernetes_secret_v1.dora_exporter: Refreshing state... [id=monitoring/dora-exporter]
module.monitoring.helm_release.kube_prometheus_stack: Refreshing state... [id=kube-prometheus-stack]
module.monitoring.helm_release.loki_stack: Refreshing state... [id=loki-stack]
module.monitoring.kubernetes_config_map_v1.uptime_dashboard: Refreshing state... [id=monitoring/uptime-dashboard]
module.monitoring.kubernetes_service_v1.dora_exporter: Refreshing state... [id=monitoring/dora-exporter]
module.database.kubernetes_job_v1.admin_app_user_provision: Refreshing state... [id=basketball-api/admin-app-user-provision-c5662180]
module.database.helm_release.cnpg: Refreshing state... [id=cnpg]
kubernetes_manifest.netpol_basketball_api: Refreshing state...
kubernetes_manifest.netpol_postgres: Refreshing state...
kubernetes_manifest.netpol_forgejo: Refreshing state...
kubernetes_manifest.netpol_keycloak: Refreshing state...
kubernetes_manifest.netpol_staging: Refreshing state...
kubernetes_manifest.netpol_monitoring: Refreshing state...
module.forgejo.helm_release.forgejo: Refreshing state... [id=forgejo]
module.database.kubernetes_secret_v1.admin_app_db_url: Refreshing state... [id=basketball-api/admin-app-db-url]
module.keycloak.kubernetes_deployment_v1.keycloak: Refreshing state... [id=keycloak/keycloak]
kubernetes_manifest.netpol_cnpg_system: Refreshing state...
module.admin.kubernetes_namespace_v1.admin: Refreshing state... [id=pal-e-admin]
module.admin.kubernetes_secret_v1.admin_auth: Refreshing state... [id=pal-e-admin/admin-auth]
module.admin.kubernetes_secret_v1.harbor_creds: Refreshing state... [id=pal-e-admin/harbor-creds]
module.admin.kubernetes_service_v1.admin: Refreshing state... [id=pal-e-admin/pal-e-admin]
module.admin.kubernetes_deployment_v1.admin: Refreshing state... [id=pal-e-admin/pal-e-admin]
module.networking.kubernetes_ingress_v1.keycloak_funnel: Refreshing state... [id=keycloak/keycloak-funnel]
module.networking.kubernetes_ingress_v1.forgejo_funnel: Refreshing state... [id=forgejo/forgejo-funnel]
module.networking.kubernetes_ingress_v1.grafana_funnel: Refreshing state... [id=monitoring/grafana-funnel]
module.networking.kubernetes_ingress_v1.admin_funnel: Refreshing state... [id=pal-e-admin/admin-funnel]
module.networking.kubernetes_ingress_v1.alertmanager_funnel: Refreshing state... [id=monitoring/alertmanager-funnel]
module.networking.kubernetes_manifest.tailscale_subnet_router: Refreshing state...
module.monitoring.kubernetes_config_map_v1.playme2k_dashboard: Refreshing state... [id=monitoring/playme2k-dashboard]
module.monitoring.kubernetes_config_map_v1.basketball_api_dashboard: Refreshing state... [id=monitoring/basketball-api-dashboard]
module.monitoring.kubernetes_config_map_v1.mac_agent_dashboard: Refreshing state... [id=monitoring/mac-agent-dashboard]
module.monitoring.kubernetes_deployment_v1.dora_exporter: Refreshing state... [id=monitoring/dora-exporter]
module.monitoring.kubernetes_manifest.payment_pipeline_alerts: Refreshing state...
module.monitoring.kubernetes_config_map_v1.pal_e_production_dashboard: Refreshing state... [id=monitoring/pal-e-app-dashboard]
module.monitoring.helm_release.blackbox_exporter: Refreshing state... [id=blackbox-exporter]
module.monitoring.kubernetes_manifest.embedding_alerts: Refreshing state...
module.monitoring.kubernetes_manifest.blackbox_alerts: Refreshing state...
module.monitoring.kubernetes_manifest.gmail_oauth_expiry_alert: Refreshing state...
module.monitoring.kubernetes_config_map_v1.dora_dashboard: Refreshing state... [id=monitoring/dora-dashboard]
module.monitoring.kubernetes_manifest.embedding_worker_service_monitor: Refreshing state...
module.monitoring.kubernetes_manifest.dora_exporter_service_monitor: Refreshing state...
module.monitoring.kubernetes_config_map_v1.grafana_loki_datasource: Refreshing state... [id=monitoring/grafana-loki-datasource]
module.harbor.kubernetes_namespace_v1.harbor: Refreshing state... [id=harbor]
module.storage.kubernetes_namespace_v1.minio: Refreshing state... [id=minio]
module.storage.kubernetes_config_map_v1.minio_console_css: Refreshing state... [id=minio/minio-console-css]
module.storage.kubernetes_config_map_v1.minio_console_nginx: Refreshing state... [id=minio/minio-console-nginx]
module.networking.kubernetes_ingress_v1.harbor_funnel: Refreshing state... [id=harbor/harbor-funnel]
module.storage.kubernetes_service_v1.minio_console_proxy: Refreshing state... [id=minio/minio-console-proxy]
module.harbor.kubernetes_config_map_v1.harbor_portal_nginx: Refreshing state... [id=harbor/harbor-portal-nginx]
module.harbor.kubernetes_service_v1.harbor_portal_proxy: Refreshing state... [id=harbor/harbor-portal-proxy]
module.storage.helm_release.minio: Refreshing state... [id=minio]
module.harbor.helm_release.harbor: Refreshing state... [id=harbor]
module.harbor.kubernetes_config_map_v1.harbor_portal_css: Refreshing state... [id=harbor/harbor-portal-css]
module.networking.kubernetes_ingress_v1.minio_api_funnel: Refreshing state... [id=minio/minio-api-funnel]
kubernetes_manifest.netpol_minio: Refreshing state...
module.networking.kubernetes_ingress_v1.minio_funnel: Refreshing state... [id=minio/minio-funnel]
kubernetes_manifest.netpol_harbor: Refreshing state...
module.harbor.null_resource.harbor_oidc: Refreshing state... [id=661278317834906207]
module.harbor.kubernetes_deployment_v1.harbor_portal_proxy: Refreshing state... [id=harbor/harbor-portal-proxy]
module.storage.minio_iam_policy.tf_backup: Refreshing state... [id=tf-backup]
module.storage.minio_s3_bucket.assets: Refreshing state... [id=assets]
module.storage.minio_iam_user.tf_backup: Refreshing state... [id=tf-backup]
module.storage.minio_iam_policy.cnpg_wal: Refreshing state... [id=cnpg-wal]
module.storage.minio_iam_user.cnpg: Refreshing state... [id=cnpg]
module.storage.minio_s3_bucket.tf_state_backups: Refreshing state... [id=tf-state-backups]
module.storage.minio_s3_bucket.postgres_wal: Refreshing state... [id=postgres-wal]
module.storage.minio_iam_user_policy_attachment.tf_backup: Refreshing state... [id=tf-backup-20260314163610110100000001]
module.storage.minio_iam_user_policy_attachment.cnpg: Refreshing state... [id=cnpg-20260302210642491000000001]
module.database.kubernetes_secret_v1.cnpg_s3_creds: Refreshing state... [id=postgres/cnpg-s3-creds]
module.storage.minio_s3_bucket_policy.assets_public_read: Refreshing state... [id=assets]
module.storage.kubernetes_deployment_v1.minio_console_proxy: Refreshing state... [id=minio/minio-console-proxy]
module.database.kubernetes_cron_job_v1.cnpg_backup_verify: Refreshing state... [id=postgres/cnpg-backup-verify]
module.ops.kubernetes_namespace_v1.ollama: Refreshing state... [id=ollama]
module.ops.kubernetes_service_account_v1.tf_backup: Refreshing state... [id=tofu-state/tf-state-backup]
module.ops.kubernetes_secret_v1.tf_backup_s3_creds: Refreshing state... [id=tofu-state/tf-backup-s3-creds]
module.ops.kubernetes_role_v1.tf_backup: Refreshing state... [id=tofu-state/tf-state-backup]
module.ops.helm_release.nvidia_device_plugin: Refreshing state... [id=nvidia-device-plugin]
module.ops.kubernetes_role_binding_v1.tf_backup: Refreshing state... [id=tofu-state/tf-state-backup]
module.ci.kubernetes_namespace_v1.woodpecker: Refreshing state... [id=woodpecker]
kubernetes_manifest.netpol_ollama: Refreshing state...
module.ci.kubernetes_secret_v1.woodpecker_db_credentials: Refreshing state... [id=woodpecker/woodpecker-db-credentials]
module.ci.kubernetes_secret_v1.woodpecker_cnpg_s3_creds: Refreshing state... [id=woodpecker/cnpg-s3-creds]
module.ops.kubernetes_cron_job_v1.tf_state_backup: Refreshing state... [id=tofu-state/tf-state-backup]
module.networking.kubernetes_ingress_v1.woodpecker_funnel: Refreshing state... [id=woodpecker/woodpecker-funnel]
kubernetes_manifest.netpol_woodpecker: Refreshing state...
module.ci.kubernetes_manifest.woodpecker_postgres: Refreshing state...
module.ops.helm_release.ollama: Refreshing state... [id=ollama]
module.ci.helm_release.woodpecker: Refreshing state... [id=woodpecker]
module.ci.kubernetes_manifest.woodpecker_postgres_scheduled_backup: Refreshing state...
module.ci.kubernetes_manifest.woodpecker_postgres_podmonitor: Refreshing state...

Note: Objects have changed outside of OpenTofu

OpenTofu detected the following changes made outside of OpenTofu since the
last "tofu apply" which may have affected this plan:

  # module.harbor.kubernetes_config_map_v1.harbor_portal_nginx has changed
  ~ resource "kubernetes_config_map_v1" "harbor_portal_nginx" {
      ~ data        = {
          ~ "nginx.conf" = <<-EOT
                worker_processes 1;
                events { worker_connections 128; }
                
                http {
                  include       /etc/nginx/mime.types;
                  default_type  application/octet-stream;
                
                  server {
                    listen 8080;
                
                    # Serve custom CSS from ConfigMap mount
                    location /custom/ {
                      alias /usr/share/nginx/custom/;
                      expires 1h;
                      add_header Cache-Control "public, immutable";
                    }
                
                    # Proxy everything to Harbor's internal nginx (which routes to portal, core, registry, etc.)
                    location / {
              -       proxy_pass http://harbor:80;
              +       client_max_body_size 0;
              +           proxy_pass http://harbor:80;
                      proxy_http_version 1.1;
                
                      # Pass through headers for auth and websocket
                      proxy_set_header Host $host;
                      proxy_set_header X-Real-IP $remote_addr;
                      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                      proxy_set_header X-Forwarded-Proto https;
                      proxy_set_header Upgrade $http_upgrade;
                      proxy_set_header Connection "upgrade";
                
                      # Disable compression so sub_filter can operate on the response body
                      proxy_set_header Accept-Encoding "";
                
                      # Inject CSS link before </head>
                      sub_filter '</head>' '<link rel="stylesheet" href="/custom/mobile.css"></head>';
                      sub_filter_once on;
                      sub_filter_types text/html;
                    }
                  }
                }
            EOT
        }
        id          = "harbor/harbor-portal-nginx"
        # (2 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }


Unless you have made equivalent changes to your configuration, or ignored the
relevant attributes using ignore_changes, the following plan may include
actions to undo or respond to these changes.

─────────────────────────────────────────────────────────────────────────────

OpenTofu used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create
  ~ update in-place
 <= read (data resources)

OpenTofu will perform the following actions:

  # module.database.kubernetes_job_v1.admin_app_user_provision will be created
  + resource "kubernetes_job_v1" "admin_app_user_provision" {
      + id                  = (known after apply)
      + wait_for_completion = true

      + metadata {
          + generation       = (known after apply)
          + labels           = {
              + "app.kubernetes.io/managed-by" = "pal-e-platform-terraform"
              + "app.kubernetes.io/name"       = "admin-app-user-provision"
              + "arch"                         = "postgres"
              + "story"                        = "admin-row-crud"
            }
          + name             = (sensitive value)
          + namespace        = "basketball-api"
          + resource_version = (known after apply)
          + uid              = (known after apply)
        }

      + spec {
          + backoff_limit              = 4
          + completion_mode            = (known after apply)
          + completions                = 1
          + parallelism                = 1
          + ttl_seconds_after_finished = "3600"

          + selector (known after apply)

          + template {
              + metadata {
                  + generation       = (known after apply)
                  + labels           = {
                      + "app.kubernetes.io/name" = "admin-app-user-provision"
                    }
                  + name             = (known after apply)
                  + resource_version = (known after apply)
                  + uid              = (known after apply)
                }
              + spec {
                  + automount_service_account_token  = true
                  + dns_policy                       = "ClusterFirst"
                  + enable_service_links             = true
                  + host_ipc                         = false
                  + host_network                     = false
                  + host_pid                         = false
                  + hostname                         = (known after apply)
                  + node_name                        = (known after apply)
                  + restart_policy                   = "OnFailure"
                  + scheduler_name                   = (known after apply)
                  + service_account_name             = (known after apply)
                  + share_process_namespace          = false
                  + termination_grace_period_seconds = 30

                  + container {
                      + args                       = [
                          + <<-EOT
                                set -euo pipefail
                                
                                echo "==> Provisioning admin_app role on ${PGHOST}/${PGDATABASE}"
                                
                                # Idempotent: CREATE ROLE if missing, otherwise rotate password.
                                psql -v ON_ERROR_STOP=1 -v admin_pw="${ADMIN_APP_PASSWORD}" <<'SQL'
                                  -- Idempotent role creation/rotation. Conditional uses psql's \gset + \if
                                  -- (client-side) so we can use :'admin_pw' substitution. psql variable
                                  -- substitution does NOT work inside DO $$...$$ dollar-quoted blocks.
                                  SELECT EXISTS(SELECT 1 FROM pg_roles WHERE rolname = 'admin_app') AS role_exists \gset
                                  \if :role_exists
                                    ALTER ROLE admin_app WITH LOGIN PASSWORD :'admin_pw';
                                  \else
                                    CREATE ROLE admin_app WITH LOGIN PASSWORD :'admin_pw';
                                  \endif
                                
                                  -- DML-only grants on schema public. Idempotent (re-grant is a no-op).
                                  GRANT USAGE ON SCHEMA public TO admin_app;
                                  GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO admin_app;
                                  GRANT USAGE ON ALL SEQUENCES IN SCHEMA public TO admin_app;
                                
                                  -- Forward grants for tables/sequences created later by Drizzle migrations
                                  -- (which run as the basketball superuser).
                                  ALTER DEFAULT PRIVILEGES FOR ROLE basketball IN SCHEMA public
                                    GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO admin_app;
                                  ALTER DEFAULT PRIVILEGES FOR ROLE basketball IN SCHEMA public
                                    GRANT USAGE ON SEQUENCES TO admin_app;
                                SQL
                                
                                echo "==> admin_app role provisioned successfully"
                            EOT,
                        ]
                      + command                    = [
                          + "/bin/sh",
                          + "-c",
                        ]
                      + image                      = "postgres:16-alpine"
                      + image_pull_policy          = (known after apply)
                      + name                       = "psql"
                      + stdin                      = false
                      + stdin_once                 = false
                      + termination_message_path   = "/dev/termination-log"
                      + termination_message_policy = (known after apply)
                      + tty                        = false

                      + env {
                          + name  = "PGHOST"
                          + value = "postgres.basketball-api.svc.cluster.local"
                        }
                      + env {
                          + name  = "PGPORT"
                          + value = "5432"
                        }
                      + env {
                          + name  = "PGUSER"
                          + value = "basketball"
                        }
                      + env {
                          + name  = "PGDATABASE"
                          + value = "basketball"
                        }
                      + env {
                          + name = "PGPASSWORD"

                          + value_from {
                              + secret_key_ref {
                                  + key  = "postgres-password"
                                  + name = "basketball-api-secrets"
                                }
                            }
                        }
                      + env {
                          + name  = "ADMIN_APP_PASSWORD"
                          + value = (sensitive value)
                        }

                      + resources {
                          + limits   = {
                              + "memory" = "128Mi"
                            }
                          + requests = {
                              + "cpu"    = "50m"
                              + "memory" = "64Mi"
                            }
                        }
                    }

                  + image_pull_secrets (known after apply)

                  + readiness_gate (known after apply)
                }
            }
        }

      + timeouts {
          + create = "5m"
          + update = "5m"
        }
    }

  # module.database.kubernetes_secret_v1.admin_app_db_url_westside_admin will be created
  + resource "kubernetes_secret_v1" "admin_app_db_url_westside_admin" {
      + data                           = (sensitive value)
      + id                             = (known after apply)
      + type                           = "Opaque"
      + wait_for_service_account_token = true

      + metadata {
          + generation       = (known after apply)
          + labels           = {
              + "app.kubernetes.io/managed-by" = "pal-e-platform-terraform"
              + "arch"                         = "postgres"
              + "mirror-of"                    = "basketball-api.admin-app-db-url"
              + "story"                        = "admin-row-crud"
            }
          + name             = "admin-app-db-url"
          + namespace        = "westside-admin"
          + resource_version = (known after apply)
          + uid              = (known after apply)
        }
    }

  # module.database.kubernetes_secret_v1.paledocs_db_url will be created
  + resource "kubernetes_secret_v1" "paledocs_db_url" {
      + data                           = (sensitive value)
      + id                             = (known after apply)
      + type                           = "Opaque"
      + wait_for_service_account_token = true

      + metadata {
          + generation       = (known after apply)
          + name             = "paledocs-db-url"
          + namespace        = "pal-e-app"
          + resource_version = (known after apply)
          + uid              = (known after apply)
        }
    }

  # module.harbor.kubernetes_config_map_v1.harbor_portal_nginx will be updated in-place
  ~ resource "kubernetes_config_map_v1" "harbor_portal_nginx" {
      ~ data        = {
          ~ "nginx.conf" = <<-EOT
                worker_processes 1;
                events { worker_connections 128; }
                
                http {
                  include       /etc/nginx/mime.types;
                  default_type  application/octet-stream;
                
                  server {
                    listen 8080;
                
                    # Serve custom CSS from ConfigMap mount
                    location /custom/ {
                      alias /usr/share/nginx/custom/;
                      expires 1h;
                      add_header Cache-Control "public, immutable";
                    }
                
                    # Proxy everything to Harbor's internal nginx (which routes to portal, core, registry, etc.)
                    location / {
              -       client_max_body_size 0;
              -           proxy_pass http://harbor:80;
              +       proxy_pass http://harbor:80;
                      proxy_http_version 1.1;
                
                      # Pass through headers for auth and websocket
                      proxy_set_header Host $host;
                      proxy_set_header X-Real-IP $remote_addr;
                      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                      proxy_set_header X-Forwarded-Proto https;
                      proxy_set_header Upgrade $http_upgrade;
                      proxy_set_header Connection "upgrade";
                
                      # Disable compression so sub_filter can operate on the response body
                      proxy_set_header Accept-Encoding "";
                
                      # Inject CSS link before </head>
                      sub_filter '</head>' '<link rel="stylesheet" href="/custom/mobile.css"></head>';
                      sub_filter_once on;
                      sub_filter_types text/html;
                    }
                  }
                }
            EOT
        }
        id          = "harbor/harbor-portal-nginx"
        # (2 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

  # module.harbor.kubernetes_deployment_v1.harbor_portal_proxy will be updated in-place
  ~ resource "kubernetes_deployment_v1" "harbor_portal_proxy" {
        id               = "harbor/harbor-portal-proxy"
        # (1 unchanged attribute hidden)

      ~ spec {
            # (5 unchanged attributes hidden)

          ~ template {
              ~ metadata {
                  ~ annotations = {
                      - "kubectl.kubernetes.io/restartedAt" = "2026-05-23T19:04:41-06:00" -> null
                        # (2 unchanged elements hidden)
                    }
                    # (2 unchanged attributes hidden)
                }

                # (1 unchanged block hidden)
            }

            # (2 unchanged blocks hidden)
        }

        # (1 unchanged block hidden)
    }

  # module.ops.data.kubernetes_namespace_v1.tofu_state will be read during apply
  # (depends on a resource or a module with changes pending)
 <= data "kubernetes_namespace_v1" "tofu_state" {
      + id   = (known after apply)
      + spec = (known after apply)

      + metadata {
          + generation       = (known after apply)
          + name             = "tofu-state"
          + resource_version = (known after apply)
          + uid              = (known after apply)
        }
    }

  # module.ops.kubernetes_service_v1.embedding_worker_metrics will be created
  + resource "kubernetes_service_v1" "embedding_worker_metrics" {
      + id                     = (known after apply)
      + status                 = (known after apply)
      + wait_for_load_balancer = true

      + metadata {
          + generation       = (known after apply)
          + labels           = {
              + "app" = "pal-e-app-embedding-worker"
            }
          + name             = "embedding-worker-metrics"
          + namespace        = "pal-e-app"
          + resource_version = (known after apply)
          + uid              = (known after apply)
        }

      + spec {
          + allocate_load_balancer_node_ports = true
          + cluster_ip                        = (known after apply)
          + cluster_ips                       = (known after apply)
          + external_traffic_policy           = (known after apply)
          + health_check_node_port            = (known after apply)
          + internal_traffic_policy           = (known after apply)
          + ip_families                       = (known after apply)
          + ip_family_policy                  = (known after apply)
          + publish_not_ready_addresses       = false
          + selector                          = {
              + "app" = "pal-e-app-embedding-worker"
            }
          + session_affinity                  = "None"
          + type                              = "ClusterIP"

          + port {
              + name        = "metrics"
              + node_port   = (known after apply)
              + port        = 8001
              + protocol    = "TCP"
              + target_port = "8001"
            }

          + session_affinity_config (known after apply)
        }
    }

  # module.storage.helm_release.minio will be updated in-place
  ~ resource "helm_release" "minio" {
        id                         = "minio"
      ~ metadata                   = [
          - {
              - app_version    = "RELEASE.2024-12-18T13-15-44Z"
              - chart          = "minio"
              - first_deployed = 1772077300
              - last_deployed  = 1779583393
              - name           = "minio"
              - namespace      = "minio"
              - notes          = <<-EOT
                    MinIO can be accessed via port 9000 on the following DNS name from within your cluster:
                    minio.minio.cluster.local
                    
                    To access MinIO from localhost, run the below commands:
                    
                      1. export POD_NAME=$(kubectl get pods --namespace minio -l "release=minio" -o jsonpath="{.items[0].metadata.name}")
                    
                      2. kubectl port-forward $POD_NAME 9000 --namespace minio
                    
                    Read more about port forwarding here: http://kubernetes.io/docs/user-guide/kubectl/kubectl_port-forward/
                    
                    You can now access MinIO server on http://localhost:9000. Follow the below steps to connect to MinIO server with mc client:
                    
                      1. Download the MinIO mc client - https://min.io/docs/minio/linux/reference/minio-mc.html#quickstart
                    
                      2. export MC_HOST_minio-local=http://$(kubectl get secret --namespace minio minio -o jsonpath="{.data.rootUser}" | base64 --decode):$(kubectl get secret --namespace minio minio -o jsonpath="{.data.rootPassword}" | base64 --decode)@localhost:9000
                    
                      3. mc ls minio-local
                EOT
              - revision       = 13
              - values         = jsonencode(
                    {
                      - buckets        = []
                      - consoleService = {
                          - port = 9001
                          - type = "ClusterIP"
                        }
                      - customCommands = []
                      - metrics        = {
                          - serviceMonitor = {
                              - enabled = true
                            }
                        }
                      - mode           = "standalone"
                      - oidc           = {
                          - claimName    = "policy"
                          - clientId     = "minio"
                          - clientSecret = "Bh0VWHafwomHu4yNHjfLeCXh5oSghOPG"
                          - configUrl    = "https://keycloak.tail5b443a.ts.net/realms/platform/.well-known/openid-configuration"
                          - enabled      = true
                          - redirectUri  = "https://minio.tail5b443a.ts.net/oauth_callback"
                          - scopes       = "openid,profile,email"
                        }
                      - persistence    = {
                          - enabled      = true
                          - size         = "10Gi"
                          - storageClass = "local-path"
                        }
                      - policies       = []
                      - postJob        = {
                          - enabled = false
                        }
                      - resources      = {
                          - limits   = {
                              - memory = "512Mi"
                            }
                          - requests = {
                              - cpu    = "100m"
                              - memory = "256Mi"
                            }
                        }
                      - rootPassword   = "(sensitive value)"
                      - rootUser       = "(sensitive value)"
                      - service        = {
                          - port = 9000
                          - type = "ClusterIP"
                        }
                      - users          = []
                    }
                )
              - version        = "5.4.0"
            },
        ] -> (known after apply)
        name                       = "minio"
      ~ values                     = [
          - (sensitive value),
          + (sensitive value),
        ]
        # (26 unchanged attributes hidden)

        # (2 unchanged blocks hidden)
    }

Plan: 4 to add, 3 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so OpenTofu can't
guarantee to take exactly these actions if you run "tofu apply" now.

## Tofu Plan Output (full) ``` module.networking.tailscale_acl.this: Refreshing state... [id=acl] module.forgejo.kubernetes_namespace_v1.forgejo: Refreshing state... [id=forgejo] module.monitoring.kubernetes_namespace_v1.monitoring: Refreshing state... [id=monitoring] module.staging.kubernetes_namespace_v1.staging: Refreshing state... [id=staging] module.database.data.kubernetes_namespace_v1.pal_e_production: Reading... module.database.kubernetes_namespace_v1.postgres: Refreshing state... [id=postgres] module.keycloak.kubernetes_namespace_v1.keycloak: Refreshing state... [id=keycloak] module.database.data.kubernetes_namespace_v1.westside_admin: Reading... module.networking.kubernetes_namespace_v1.tailscale: Refreshing state... [id=tailscale] module.database.data.kubernetes_namespace_v1.westside_admin: Read complete after 0s [id=westside-admin] module.database.data.kubernetes_namespace_v1.pal_e_production: Read complete after 0s [id=pal-e-app] module.database.data.kubernetes_namespace_v1.basketball_api: Reading... module.database.kubernetes_namespace_v1.cnpg_system: Refreshing state... [id=cnpg-system] module.database.data.kubernetes_namespace_v1.basketball_api: Read complete after 0s [id=basketball-api] module.keycloak.kubernetes_secret_v1.keycloak_admin: Refreshing state... [id=keycloak/keycloak-admin] module.keycloak.kubernetes_persistent_volume_claim_v1.keycloak_data: Refreshing state... [id=keycloak/keycloak-data] module.keycloak.kubernetes_config_map_v1.keycloak_westside_theme: Refreshing state... [id=keycloak/keycloak-westside-theme] module.forgejo.kubernetes_secret_v1.forgejo_oidc: Refreshing state... [id=forgejo/forgejo-oidc] module.forgejo.kubernetes_config_map_v1.forgejo_custom_css: Refreshing state... [id=forgejo/forgejo-custom-css] module.keycloak.kubernetes_service_v1.keycloak: Refreshing state... [id=keycloak/keycloak] module.networking.helm_release.tailscale_operator: Refreshing state... [id=tailscale-operator] module.monitoring.kubernetes_secret_v1.grafana_oidc: Refreshing state... [id=monitoring/grafana-oidc] module.monitoring.kubernetes_secret_v1.dora_exporter: Refreshing state... [id=monitoring/dora-exporter] module.monitoring.helm_release.kube_prometheus_stack: Refreshing state... [id=kube-prometheus-stack] module.monitoring.helm_release.loki_stack: Refreshing state... [id=loki-stack] module.monitoring.kubernetes_config_map_v1.uptime_dashboard: Refreshing state... [id=monitoring/uptime-dashboard] module.monitoring.kubernetes_service_v1.dora_exporter: Refreshing state... [id=monitoring/dora-exporter] module.database.kubernetes_job_v1.admin_app_user_provision: Refreshing state... [id=basketball-api/admin-app-user-provision-c5662180] module.database.helm_release.cnpg: Refreshing state... [id=cnpg] kubernetes_manifest.netpol_basketball_api: Refreshing state... kubernetes_manifest.netpol_postgres: Refreshing state... kubernetes_manifest.netpol_forgejo: Refreshing state... kubernetes_manifest.netpol_keycloak: Refreshing state... kubernetes_manifest.netpol_staging: Refreshing state... kubernetes_manifest.netpol_monitoring: Refreshing state... module.forgejo.helm_release.forgejo: Refreshing state... [id=forgejo] module.database.kubernetes_secret_v1.admin_app_db_url: Refreshing state... [id=basketball-api/admin-app-db-url] module.keycloak.kubernetes_deployment_v1.keycloak: Refreshing state... [id=keycloak/keycloak] kubernetes_manifest.netpol_cnpg_system: Refreshing state... module.admin.kubernetes_namespace_v1.admin: Refreshing state... [id=pal-e-admin] module.admin.kubernetes_secret_v1.admin_auth: Refreshing state... [id=pal-e-admin/admin-auth] module.admin.kubernetes_secret_v1.harbor_creds: Refreshing state... [id=pal-e-admin/harbor-creds] module.admin.kubernetes_service_v1.admin: Refreshing state... [id=pal-e-admin/pal-e-admin] module.admin.kubernetes_deployment_v1.admin: Refreshing state... [id=pal-e-admin/pal-e-admin] module.networking.kubernetes_ingress_v1.keycloak_funnel: Refreshing state... [id=keycloak/keycloak-funnel] module.networking.kubernetes_ingress_v1.forgejo_funnel: Refreshing state... [id=forgejo/forgejo-funnel] module.networking.kubernetes_ingress_v1.grafana_funnel: Refreshing state... [id=monitoring/grafana-funnel] module.networking.kubernetes_ingress_v1.admin_funnel: Refreshing state... [id=pal-e-admin/admin-funnel] module.networking.kubernetes_ingress_v1.alertmanager_funnel: Refreshing state... [id=monitoring/alertmanager-funnel] module.networking.kubernetes_manifest.tailscale_subnet_router: Refreshing state... module.monitoring.kubernetes_config_map_v1.playme2k_dashboard: Refreshing state... [id=monitoring/playme2k-dashboard] module.monitoring.kubernetes_config_map_v1.basketball_api_dashboard: Refreshing state... [id=monitoring/basketball-api-dashboard] module.monitoring.kubernetes_config_map_v1.mac_agent_dashboard: Refreshing state... [id=monitoring/mac-agent-dashboard] module.monitoring.kubernetes_deployment_v1.dora_exporter: Refreshing state... [id=monitoring/dora-exporter] module.monitoring.kubernetes_manifest.payment_pipeline_alerts: Refreshing state... module.monitoring.kubernetes_config_map_v1.pal_e_production_dashboard: Refreshing state... [id=monitoring/pal-e-app-dashboard] module.monitoring.helm_release.blackbox_exporter: Refreshing state... [id=blackbox-exporter] module.monitoring.kubernetes_manifest.embedding_alerts: Refreshing state... module.monitoring.kubernetes_manifest.blackbox_alerts: Refreshing state... module.monitoring.kubernetes_manifest.gmail_oauth_expiry_alert: Refreshing state... module.monitoring.kubernetes_config_map_v1.dora_dashboard: Refreshing state... [id=monitoring/dora-dashboard] module.monitoring.kubernetes_manifest.embedding_worker_service_monitor: Refreshing state... module.monitoring.kubernetes_manifest.dora_exporter_service_monitor: Refreshing state... module.monitoring.kubernetes_config_map_v1.grafana_loki_datasource: Refreshing state... [id=monitoring/grafana-loki-datasource] module.harbor.kubernetes_namespace_v1.harbor: Refreshing state... [id=harbor] module.storage.kubernetes_namespace_v1.minio: Refreshing state... [id=minio] module.storage.kubernetes_config_map_v1.minio_console_css: Refreshing state... [id=minio/minio-console-css] module.storage.kubernetes_config_map_v1.minio_console_nginx: Refreshing state... [id=minio/minio-console-nginx] module.networking.kubernetes_ingress_v1.harbor_funnel: Refreshing state... [id=harbor/harbor-funnel] module.storage.kubernetes_service_v1.minio_console_proxy: Refreshing state... [id=minio/minio-console-proxy] module.harbor.kubernetes_config_map_v1.harbor_portal_nginx: Refreshing state... [id=harbor/harbor-portal-nginx] module.harbor.kubernetes_service_v1.harbor_portal_proxy: Refreshing state... [id=harbor/harbor-portal-proxy] module.storage.helm_release.minio: Refreshing state... [id=minio] module.harbor.helm_release.harbor: Refreshing state... [id=harbor] module.harbor.kubernetes_config_map_v1.harbor_portal_css: Refreshing state... [id=harbor/harbor-portal-css] module.networking.kubernetes_ingress_v1.minio_api_funnel: Refreshing state... [id=minio/minio-api-funnel] kubernetes_manifest.netpol_minio: Refreshing state... module.networking.kubernetes_ingress_v1.minio_funnel: Refreshing state... [id=minio/minio-funnel] kubernetes_manifest.netpol_harbor: Refreshing state... module.harbor.null_resource.harbor_oidc: Refreshing state... [id=661278317834906207] module.harbor.kubernetes_deployment_v1.harbor_portal_proxy: Refreshing state... [id=harbor/harbor-portal-proxy] module.storage.minio_iam_policy.tf_backup: Refreshing state... [id=tf-backup] module.storage.minio_s3_bucket.assets: Refreshing state... [id=assets] module.storage.minio_iam_user.tf_backup: Refreshing state... [id=tf-backup] module.storage.minio_iam_policy.cnpg_wal: Refreshing state... [id=cnpg-wal] module.storage.minio_iam_user.cnpg: Refreshing state... [id=cnpg] module.storage.minio_s3_bucket.tf_state_backups: Refreshing state... [id=tf-state-backups] module.storage.minio_s3_bucket.postgres_wal: Refreshing state... [id=postgres-wal] module.storage.minio_iam_user_policy_attachment.tf_backup: Refreshing state... [id=tf-backup-20260314163610110100000001] module.storage.minio_iam_user_policy_attachment.cnpg: Refreshing state... [id=cnpg-20260302210642491000000001] module.database.kubernetes_secret_v1.cnpg_s3_creds: Refreshing state... [id=postgres/cnpg-s3-creds] module.storage.minio_s3_bucket_policy.assets_public_read: Refreshing state... [id=assets] module.storage.kubernetes_deployment_v1.minio_console_proxy: Refreshing state... [id=minio/minio-console-proxy] module.database.kubernetes_cron_job_v1.cnpg_backup_verify: Refreshing state... [id=postgres/cnpg-backup-verify] module.ops.kubernetes_namespace_v1.ollama: Refreshing state... [id=ollama] module.ops.kubernetes_service_account_v1.tf_backup: Refreshing state... [id=tofu-state/tf-state-backup] module.ops.kubernetes_secret_v1.tf_backup_s3_creds: Refreshing state... [id=tofu-state/tf-backup-s3-creds] module.ops.kubernetes_role_v1.tf_backup: Refreshing state... [id=tofu-state/tf-state-backup] module.ops.helm_release.nvidia_device_plugin: Refreshing state... [id=nvidia-device-plugin] module.ops.kubernetes_role_binding_v1.tf_backup: Refreshing state... [id=tofu-state/tf-state-backup] module.ci.kubernetes_namespace_v1.woodpecker: Refreshing state... [id=woodpecker] kubernetes_manifest.netpol_ollama: Refreshing state... module.ci.kubernetes_secret_v1.woodpecker_db_credentials: Refreshing state... [id=woodpecker/woodpecker-db-credentials] module.ci.kubernetes_secret_v1.woodpecker_cnpg_s3_creds: Refreshing state... [id=woodpecker/cnpg-s3-creds] module.ops.kubernetes_cron_job_v1.tf_state_backup: Refreshing state... [id=tofu-state/tf-state-backup] module.networking.kubernetes_ingress_v1.woodpecker_funnel: Refreshing state... [id=woodpecker/woodpecker-funnel] kubernetes_manifest.netpol_woodpecker: Refreshing state... module.ci.kubernetes_manifest.woodpecker_postgres: Refreshing state... module.ops.helm_release.ollama: Refreshing state... [id=ollama] module.ci.helm_release.woodpecker: Refreshing state... [id=woodpecker] module.ci.kubernetes_manifest.woodpecker_postgres_scheduled_backup: Refreshing state... module.ci.kubernetes_manifest.woodpecker_postgres_podmonitor: Refreshing state... Note: Objects have changed outside of OpenTofu OpenTofu detected the following changes made outside of OpenTofu since the last "tofu apply" which may have affected this plan: # module.harbor.kubernetes_config_map_v1.harbor_portal_nginx has changed ~ resource "kubernetes_config_map_v1" "harbor_portal_nginx" { ~ data = { ~ "nginx.conf" = <<-EOT worker_processes 1; events { worker_connections 128; } http { include /etc/nginx/mime.types; default_type application/octet-stream; server { listen 8080; # Serve custom CSS from ConfigMap mount location /custom/ { alias /usr/share/nginx/custom/; expires 1h; add_header Cache-Control "public, immutable"; } # Proxy everything to Harbor's internal nginx (which routes to portal, core, registry, etc.) location / { - proxy_pass http://harbor:80; + client_max_body_size 0; + proxy_pass http://harbor:80; proxy_http_version 1.1; # Pass through headers for auth and websocket proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto https; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; # Disable compression so sub_filter can operate on the response body proxy_set_header Accept-Encoding ""; # Inject CSS link before </head> sub_filter '</head>' '<link rel="stylesheet" href="/custom/mobile.css"></head>'; sub_filter_once on; sub_filter_types text/html; } } } EOT } id = "harbor/harbor-portal-nginx" # (2 unchanged attributes hidden) # (1 unchanged block hidden) } Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan may include actions to undo or respond to these changes. ───────────────────────────────────────────────────────────────────────────── OpenTofu used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: + create ~ update in-place <= read (data resources) OpenTofu will perform the following actions: # module.database.kubernetes_job_v1.admin_app_user_provision will be created + resource "kubernetes_job_v1" "admin_app_user_provision" { + id = (known after apply) + wait_for_completion = true + metadata { + generation = (known after apply) + labels = { + "app.kubernetes.io/managed-by" = "pal-e-platform-terraform" + "app.kubernetes.io/name" = "admin-app-user-provision" + "arch" = "postgres" + "story" = "admin-row-crud" } + name = (sensitive value) + namespace = "basketball-api" + resource_version = (known after apply) + uid = (known after apply) } + spec { + backoff_limit = 4 + completion_mode = (known after apply) + completions = 1 + parallelism = 1 + ttl_seconds_after_finished = "3600" + selector (known after apply) + template { + metadata { + generation = (known after apply) + labels = { + "app.kubernetes.io/name" = "admin-app-user-provision" } + name = (known after apply) + resource_version = (known after apply) + uid = (known after apply) } + spec { + automount_service_account_token = true + dns_policy = "ClusterFirst" + enable_service_links = true + host_ipc = false + host_network = false + host_pid = false + hostname = (known after apply) + node_name = (known after apply) + restart_policy = "OnFailure" + scheduler_name = (known after apply) + service_account_name = (known after apply) + share_process_namespace = false + termination_grace_period_seconds = 30 + container { + args = [ + <<-EOT set -euo pipefail echo "==> Provisioning admin_app role on ${PGHOST}/${PGDATABASE}" # Idempotent: CREATE ROLE if missing, otherwise rotate password. psql -v ON_ERROR_STOP=1 -v admin_pw="${ADMIN_APP_PASSWORD}" <<'SQL' -- Idempotent role creation/rotation. Conditional uses psql's \gset + \if -- (client-side) so we can use :'admin_pw' substitution. psql variable -- substitution does NOT work inside DO $$...$$ dollar-quoted blocks. SELECT EXISTS(SELECT 1 FROM pg_roles WHERE rolname = 'admin_app') AS role_exists \gset \if :role_exists ALTER ROLE admin_app WITH LOGIN PASSWORD :'admin_pw'; \else CREATE ROLE admin_app WITH LOGIN PASSWORD :'admin_pw'; \endif -- DML-only grants on schema public. Idempotent (re-grant is a no-op). GRANT USAGE ON SCHEMA public TO admin_app; GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO admin_app; GRANT USAGE ON ALL SEQUENCES IN SCHEMA public TO admin_app; -- Forward grants for tables/sequences created later by Drizzle migrations -- (which run as the basketball superuser). ALTER DEFAULT PRIVILEGES FOR ROLE basketball IN SCHEMA public GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO admin_app; ALTER DEFAULT PRIVILEGES FOR ROLE basketball IN SCHEMA public GRANT USAGE ON SEQUENCES TO admin_app; SQL echo "==> admin_app role provisioned successfully" EOT, ] + command = [ + "/bin/sh", + "-c", ] + image = "postgres:16-alpine" + image_pull_policy = (known after apply) + name = "psql" + stdin = false + stdin_once = false + termination_message_path = "/dev/termination-log" + termination_message_policy = (known after apply) + tty = false + env { + name = "PGHOST" + value = "postgres.basketball-api.svc.cluster.local" } + env { + name = "PGPORT" + value = "5432" } + env { + name = "PGUSER" + value = "basketball" } + env { + name = "PGDATABASE" + value = "basketball" } + env { + name = "PGPASSWORD" + value_from { + secret_key_ref { + key = "postgres-password" + name = "basketball-api-secrets" } } } + env { + name = "ADMIN_APP_PASSWORD" + value = (sensitive value) } + resources { + limits = { + "memory" = "128Mi" } + requests = { + "cpu" = "50m" + "memory" = "64Mi" } } } + image_pull_secrets (known after apply) + readiness_gate (known after apply) } } } + timeouts { + create = "5m" + update = "5m" } } # module.database.kubernetes_secret_v1.admin_app_db_url_westside_admin will be created + resource "kubernetes_secret_v1" "admin_app_db_url_westside_admin" { + data = (sensitive value) + id = (known after apply) + type = "Opaque" + wait_for_service_account_token = true + metadata { + generation = (known after apply) + labels = { + "app.kubernetes.io/managed-by" = "pal-e-platform-terraform" + "arch" = "postgres" + "mirror-of" = "basketball-api.admin-app-db-url" + "story" = "admin-row-crud" } + name = "admin-app-db-url" + namespace = "westside-admin" + resource_version = (known after apply) + uid = (known after apply) } } # module.database.kubernetes_secret_v1.paledocs_db_url will be created + resource "kubernetes_secret_v1" "paledocs_db_url" { + data = (sensitive value) + id = (known after apply) + type = "Opaque" + wait_for_service_account_token = true + metadata { + generation = (known after apply) + name = "paledocs-db-url" + namespace = "pal-e-app" + resource_version = (known after apply) + uid = (known after apply) } } # module.harbor.kubernetes_config_map_v1.harbor_portal_nginx will be updated in-place ~ resource "kubernetes_config_map_v1" "harbor_portal_nginx" { ~ data = { ~ "nginx.conf" = <<-EOT worker_processes 1; events { worker_connections 128; } http { include /etc/nginx/mime.types; default_type application/octet-stream; server { listen 8080; # Serve custom CSS from ConfigMap mount location /custom/ { alias /usr/share/nginx/custom/; expires 1h; add_header Cache-Control "public, immutable"; } # Proxy everything to Harbor's internal nginx (which routes to portal, core, registry, etc.) location / { - client_max_body_size 0; - proxy_pass http://harbor:80; + proxy_pass http://harbor:80; proxy_http_version 1.1; # Pass through headers for auth and websocket proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto https; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; # Disable compression so sub_filter can operate on the response body proxy_set_header Accept-Encoding ""; # Inject CSS link before </head> sub_filter '</head>' '<link rel="stylesheet" href="/custom/mobile.css"></head>'; sub_filter_once on; sub_filter_types text/html; } } } EOT } id = "harbor/harbor-portal-nginx" # (2 unchanged attributes hidden) # (1 unchanged block hidden) } # module.harbor.kubernetes_deployment_v1.harbor_portal_proxy will be updated in-place ~ resource "kubernetes_deployment_v1" "harbor_portal_proxy" { id = "harbor/harbor-portal-proxy" # (1 unchanged attribute hidden) ~ spec { # (5 unchanged attributes hidden) ~ template { ~ metadata { ~ annotations = { - "kubectl.kubernetes.io/restartedAt" = "2026-05-23T19:04:41-06:00" -> null # (2 unchanged elements hidden) } # (2 unchanged attributes hidden) } # (1 unchanged block hidden) } # (2 unchanged blocks hidden) } # (1 unchanged block hidden) } # module.ops.data.kubernetes_namespace_v1.tofu_state will be read during apply # (depends on a resource or a module with changes pending) <= data "kubernetes_namespace_v1" "tofu_state" { + id = (known after apply) + spec = (known after apply) + metadata { + generation = (known after apply) + name = "tofu-state" + resource_version = (known after apply) + uid = (known after apply) } } # module.ops.kubernetes_service_v1.embedding_worker_metrics will be created + resource "kubernetes_service_v1" "embedding_worker_metrics" { + id = (known after apply) + status = (known after apply) + wait_for_load_balancer = true + metadata { + generation = (known after apply) + labels = { + "app" = "pal-e-app-embedding-worker" } + name = "embedding-worker-metrics" + namespace = "pal-e-app" + resource_version = (known after apply) + uid = (known after apply) } + spec { + allocate_load_balancer_node_ports = true + cluster_ip = (known after apply) + cluster_ips = (known after apply) + external_traffic_policy = (known after apply) + health_check_node_port = (known after apply) + internal_traffic_policy = (known after apply) + ip_families = (known after apply) + ip_family_policy = (known after apply) + publish_not_ready_addresses = false + selector = { + "app" = "pal-e-app-embedding-worker" } + session_affinity = "None" + type = "ClusterIP" + port { + name = "metrics" + node_port = (known after apply) + port = 8001 + protocol = "TCP" + target_port = "8001" } + session_affinity_config (known after apply) } } # module.storage.helm_release.minio will be updated in-place ~ resource "helm_release" "minio" { id = "minio" ~ metadata = [ - { - app_version = "RELEASE.2024-12-18T13-15-44Z" - chart = "minio" - first_deployed = 1772077300 - last_deployed = 1779583393 - name = "minio" - namespace = "minio" - notes = <<-EOT MinIO can be accessed via port 9000 on the following DNS name from within your cluster: minio.minio.cluster.local To access MinIO from localhost, run the below commands: 1. export POD_NAME=$(kubectl get pods --namespace minio -l "release=minio" -o jsonpath="{.items[0].metadata.name}") 2. kubectl port-forward $POD_NAME 9000 --namespace minio Read more about port forwarding here: http://kubernetes.io/docs/user-guide/kubectl/kubectl_port-forward/ You can now access MinIO server on http://localhost:9000. Follow the below steps to connect to MinIO server with mc client: 1. Download the MinIO mc client - https://min.io/docs/minio/linux/reference/minio-mc.html#quickstart 2. export MC_HOST_minio-local=http://$(kubectl get secret --namespace minio minio -o jsonpath="{.data.rootUser}" | base64 --decode):$(kubectl get secret --namespace minio minio -o jsonpath="{.data.rootPassword}" | base64 --decode)@localhost:9000 3. mc ls minio-local EOT - revision = 13 - values = jsonencode( { - buckets = [] - consoleService = { - port = 9001 - type = "ClusterIP" } - customCommands = [] - metrics = { - serviceMonitor = { - enabled = true } } - mode = "standalone" - oidc = { - claimName = "policy" - clientId = "minio" - clientSecret = "Bh0VWHafwomHu4yNHjfLeCXh5oSghOPG" - configUrl = "https://keycloak.tail5b443a.ts.net/realms/platform/.well-known/openid-configuration" - enabled = true - redirectUri = "https://minio.tail5b443a.ts.net/oauth_callback" - scopes = "openid,profile,email" } - persistence = { - enabled = true - size = "10Gi" - storageClass = "local-path" } - policies = [] - postJob = { - enabled = false } - resources = { - limits = { - memory = "512Mi" } - requests = { - cpu = "100m" - memory = "256Mi" } } - rootPassword = "(sensitive value)" - rootUser = "(sensitive value)" - service = { - port = 9000 - type = "ClusterIP" } - users = [] } ) - version = "5.4.0" }, ] -> (known after apply) name = "minio" ~ values = [ - (sensitive value), + (sensitive value), ] # (26 unchanged attributes hidden) # (2 unchanged blocks hidden) } Plan: 4 to add, 3 to change, 0 to destroy. ───────────────────────────────────────────────────────────────────────────── Note: You didn't use the -out option to save this plan, so OpenTofu can't guarantee to take exactly these actions if you run "tofu apply" now. ```

ldraney commented 2026-05-24 14:10:12 +00:00

Author

Owner

Copy link

PR #378 Review

DOMAIN REVIEW

Tech stack: OpenTofu / Kubernetes NetworkPolicy (HCL)

The diff adds two lines to terraform/network-policies.tf, restoring pal-e-docs and pal-e-ror namespaces to the netpol_postgres ingress allowlist. Both were silently dropped by PR #376 (landscaping-assistant rewrite).

Terraform correctness:

• The two new entries follow the identical structure as every other entry in the ingress list: { from = [{ namespaceSelector = { matchLabels = { "kubernetes.io/metadata.name" = "<namespace>" } } }] }. Consistent and correct.
• Placement is logical -- inserted after westside-ror and before landscaping-assistant, maintaining readable ordering.
• tofu fmt and tofu validate both pass per the PR body.
• tofu plan shows 0 add, 1 change, 0 destroy -- expected for an in-place NetworkPolicy update.

k8s security:

• No RBAC changes. No privilege escalation. No new namespaces created. This is strictly restoring previously-existing ingress rules.
• The postgres namespace retains its default-deny posture -- only explicitly listed namespaces can reach it.

State safety:

• In-place update to an existing kubernetes_manifest resource. No state migration needed. No risk of resource recreation.

Incident context:

• This is the second time namespace entries have been dropped from this list (see issue #334 for a prior incident during the pal-e-production rename). The ingress list is a flat array with no programmatic guard against accidental deletions. This is a structural risk worth tracking but not a blocker for this fix PR.

BLOCKERS

None. This is a 2-line restoration of accidentally dropped config. No new functionality requiring tests. No user input. No secrets. No auth changes.

NITS

1. Structural fragility: The ingress list is a hand-maintained flat array. PR #376 proved that additive PRs can silently drop entries when the author rewrites the block. Consider extracting the namespace list into a local variable (e.g., locals { postgres_allowed_namespaces = [...] }) to make additions append-only and deletions visible in diffs. Not blocking -- track as a separate improvement issue. Note: issue #334 documents a prior identical incident.

SOP COMPLIANCE

• Branch named after issue: fix-postgres-netpol-restore-pal-e-docs references the fix. Branch does not follow strict {issue-number}-{purpose} convention (would be 377-restore-postgres-netpol), but intent is clear.
• PR body follows template: Summary, Changes, Test Plan, Review Checklist, Related all present.
• Related references parent issue: Closes #377 present.
• Related references plan slug: No plan slug -- acceptable for a hotfix/bug.
• No secrets committed.
• No unnecessary file changes: 1 file, 2 lines, exactly scoped.
• Commit message is descriptive.
• tofu fmt and tofu validate pass.
• tofu plan output documented.
• Applied and verified: pods recovered to Running state.

PROCESS OBSERVATIONS

• MTTR: 6-day outage for pal-e-docs caused by a regression in PR #376. The root cause (flat array rewrite without diffing against main) is a recurring pattern (also seen in #334). The fix itself is correct and minimal.
• Change failure risk: Low for this PR. The structural nit (flat array fragility) is a medium-term risk for future PRs touching this file.
• Recommendation: A post-incident item to refactor the ingress list into a locals variable would reduce recurrence risk. This is the second time this failure mode has occurred.

VERDICT: APPROVED

## PR #378 Review ### DOMAIN REVIEW **Tech stack**: OpenTofu / Kubernetes NetworkPolicy (HCL) The diff adds two lines to `terraform/network-policies.tf`, restoring `pal-e-docs` and `pal-e-ror` namespaces to the `netpol_postgres` ingress allowlist. Both were silently dropped by PR #376 (landscaping-assistant rewrite). **Terraform correctness**: - The two new entries follow the identical structure as every other entry in the ingress list: `{ from = [{ namespaceSelector = { matchLabels = { "kubernetes.io/metadata.name" = "<namespace>" } } }] }`. Consistent and correct. - Placement is logical -- inserted after `westside-ror` and before `landscaping-assistant`, maintaining readable ordering. - `tofu fmt` and `tofu validate` both pass per the PR body. - `tofu plan` shows 0 add, 1 change, 0 destroy -- expected for an in-place NetworkPolicy update. **k8s security**: - No RBAC changes. No privilege escalation. No new namespaces created. This is strictly restoring previously-existing ingress rules. - The postgres namespace retains its default-deny posture -- only explicitly listed namespaces can reach it. **State safety**: - In-place update to an existing `kubernetes_manifest` resource. No state migration needed. No risk of resource recreation. **Incident context**: - This is the second time namespace entries have been dropped from this list (see issue #334 for a prior incident during the pal-e-production rename). The ingress list is a flat array with no programmatic guard against accidental deletions. This is a structural risk worth tracking but not a blocker for this fix PR. ### BLOCKERS None. This is a 2-line restoration of accidentally dropped config. No new functionality requiring tests. No user input. No secrets. No auth changes. ### NITS 1. **Structural fragility**: The ingress list is a hand-maintained flat array. PR #376 proved that additive PRs can silently drop entries when the author rewrites the block. Consider extracting the namespace list into a local variable (e.g., `locals { postgres_allowed_namespaces = [...] }`) to make additions append-only and deletions visible in diffs. Not blocking -- track as a separate improvement issue. Note: issue #334 documents a prior identical incident. ### SOP COMPLIANCE - [x] Branch named after issue: `fix-postgres-netpol-restore-pal-e-docs` references the fix. Branch does not follow strict `{issue-number}-{purpose}` convention (would be `377-restore-postgres-netpol`), but intent is clear. - [x] PR body follows template: Summary, Changes, Test Plan, Review Checklist, Related all present. - [x] Related references parent issue: `Closes #377` present. - [ ] Related references plan slug: No plan slug -- acceptable for a hotfix/bug. - [x] No secrets committed. - [x] No unnecessary file changes: 1 file, 2 lines, exactly scoped. - [x] Commit message is descriptive. - [x] `tofu fmt` and `tofu validate` pass. - [x] `tofu plan` output documented. - [x] Applied and verified: pods recovered to Running state. ### PROCESS OBSERVATIONS - **MTTR**: 6-day outage for pal-e-docs caused by a regression in PR #376. The root cause (flat array rewrite without diffing against main) is a recurring pattern (also seen in #334). The fix itself is correct and minimal. - **Change failure risk**: Low for this PR. The structural nit (flat array fragility) is a medium-term risk for future PRs touching this file. - **Recommendation**: A post-incident item to refactor the ingress list into a `locals` variable would reduce recurrence risk. This is the second time this failure mode has occurred. ### VERDICT: APPROVED

ldraney merged commit bb0861fccb into main 2026-05-24 15:17:24 +00:00

ldraney deleted branch fix-postgres-netpol-restore-pal-e-docs 2026-05-24 15:17:24 +00:00

ldraney referenced this pull request from a commit 2026-05-24 15:17:25 +00:00

Fix: restore pal-e-docs and pal-e-ror in postgres NetworkPolicy (#378)

ldraney referenced this pull request 2026-05-24 17:40:11 +00:00

Fix 413 on Docker push, add palinks to postgres NetworkPolicy #382

ldraney referenced this pull request 2026-06-05 03:55:46 +00:00

Add paldocs to Postgres NetworkPolicy allowlist (#397) #398

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

domain:backend

  

domain:devops

  

domain:frontend

  

status:approved

QA passed, awaiting merge approval

  

status:in-progress

Dev agent is actively working

  

status:needs-fix

QA found issues, back to dev

  

status:qa

PR submitted, awaiting QA review

  

type:bug

Bug fix

  

type:devops

Infrastructure/CI/config work

  

type:feature

New functionality

No labels

domain:backend

domain:devops

domain:frontend

status:approved

status:in-progress

status:needs-fix

status:qa

type:bug

type:devops

type:feature

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

ldraney/pal-e-platform!378

No description provided.