forgejo_admin/pal-e-platform

Fork 1

fix: add default-deny-ingress NetworkPolicy for basketball-api #269

Merged

forgejo_admin merged 1 commit from 268-add-default-deny-ingress-networkpolicy-f into main

2026-04-05 20:07:21 +00:00

forgejo_admin commented

2026-04-05 19:09:32 +00:00

Owner

Summary

Adds a default-deny-ingress NetworkPolicy for the basketball-api namespace, closing the only security gap among service namespaces. Investigation of forgejo_admin/basketball-api#343 confirmed the Tailscale Funnel is correctly configured and routing public traffic — the webhook delivery failures correlate with pod downtime during Recreate strategy deployments, not a funnel misconfiguration.

Changes

terraform/network-policies.tf — added netpol_basketball_api resource allowing ingress from tailscale and monitoring namespaces only, matching the pattern used by all other service namespaces

Investigation Findings

Funnel is working: DNS resolves to public Tailscale relay IPs (208.111.35.209) from external DNS. POST to /webhooks/stripe via public IP returns 400 "Missing stripe-signature header" — proving full reachability.
Proxy logs show intermittent "connection refused": These correlate with basketball-api pod restarts during Recreate deployments (7+ minutes of errors on 2026-04-04 22:40-22:47 UTC).
Security gap: basketball-api was the only service namespace without a default-deny-ingress NetworkPolicy. All 10 other namespaces have one.
ACLs correct: nodeAttrs grants funnel to all tag:k8s nodes. The proxy pod has AllowFunnel: true and the serve config matches the ClusterIP.

tofu plan Output

Plan: 1 to add, 0 to change, 0 to destroy.

  # kubernetes_manifest.netpol_basketball_api will be created
  + resource "kubernetes_manifest" "netpol_basketball_api" {
      + manifest = {
          + apiVersion = "networking.k8s.io/v1"
          + kind       = "NetworkPolicy"
          + metadata   = {
              + name      = "default-deny-ingress"
              + namespace = "basketball-api"
            }
          + spec       = {
              + ingress     = [
                  + { from = [{ namespaceSelector = { matchLabels = { "kubernetes.io/metadata.name" = "tailscale" } } }] },
                  + { from = [{ namespaceSelector = { matchLabels = { "kubernetes.io/metadata.name" = "monitoring" } } }] },
                ]
              + podSelector = {}
              + policyTypes = ["Ingress"]
            }
        }
    }

Discovered Scope

The root cause of Stripe webhook failures is pod downtime during Recreate deployments, not a funnel issue. Changing to RollingUpdate strategy would eliminate webhook delivery gaps during deploys — this is a pal-e-deployments change (separate ticket).

Test Plan

tofu fmt — passed
tofu validate — passed
tofu plan -lock=false — 1 to add, 0 to change, 0 to destroy
After apply: kubectl get networkpolicy -n basketball-api returns default-deny-ingress
After apply: funnel traffic still reaches basketball-api via public IP

Review Checklist

tofu fmt run
tofu validate passes
tofu plan -lock=false output included
Follows existing NetworkPolicy pattern in network-policies.tf
No README roadmap update needed

forgejo_admin/basketball-api#343 — parent issue (Stripe webhooks unreachable)

Closes forgejo_admin/basketball-api#343
Forgejo issue: #268

## Summary Adds a default-deny-ingress NetworkPolicy for the basketball-api namespace, closing the only security gap among service namespaces. Investigation of forgejo_admin/basketball-api#343 confirmed the Tailscale Funnel is correctly configured and routing public traffic — the webhook delivery failures correlate with pod downtime during `Recreate` strategy deployments, not a funnel misconfiguration. ## Changes - `terraform/network-policies.tf` — added `netpol_basketball_api` resource allowing ingress from `tailscale` and `monitoring` namespaces only, matching the pattern used by all other service namespaces ## Investigation Findings 1. **Funnel is working**: DNS resolves to public Tailscale relay IPs (208.111.35.209) from external DNS. POST to `/webhooks/stripe` via public IP returns 400 "Missing stripe-signature header" — proving full reachability. 2. **Proxy logs show intermittent "connection refused"**: These correlate with basketball-api pod restarts during `Recreate` deployments (7+ minutes of errors on 2026-04-04 22:40-22:47 UTC). 3. **Security gap**: basketball-api was the only service namespace without a default-deny-ingress NetworkPolicy. All 10 other namespaces have one. 4. **ACLs correct**: `nodeAttrs` grants funnel to all `tag:k8s` nodes. The proxy pod has `AllowFunnel: true` and the serve config matches the ClusterIP. ## tofu plan Output ``` Plan: 1 to add, 0 to change, 0 to destroy. # kubernetes_manifest.netpol_basketball_api will be created + resource "kubernetes_manifest" "netpol_basketball_api" { + manifest = { + apiVersion = "networking.k8s.io/v1" + kind = "NetworkPolicy" + metadata = { + name = "default-deny-ingress" + namespace = "basketball-api" } + spec = { + ingress = [ + { from = [{ namespaceSelector = { matchLabels = { "kubernetes.io/metadata.name" = "tailscale" } } }] }, + { from = [{ namespaceSelector = { matchLabels = { "kubernetes.io/metadata.name" = "monitoring" } } }] }, ] + podSelector = {} + policyTypes = ["Ingress"] } } } ``` ## Discovered Scope The root cause of Stripe webhook failures is pod downtime during `Recreate` deployments, not a funnel issue. Changing to `RollingUpdate` strategy would eliminate webhook delivery gaps during deploys — this is a pal-e-deployments change (separate ticket). ## Test Plan - `tofu fmt` — passed - `tofu validate` — passed - `tofu plan -lock=false` — 1 to add, 0 to change, 0 to destroy - After apply: `kubectl get networkpolicy -n basketball-api` returns `default-deny-ingress` - After apply: funnel traffic still reaches basketball-api via public IP ## Review Checklist - [x] `tofu fmt` run - [x] `tofu validate` passes - [x] `tofu plan -lock=false` output included - [x] Follows existing NetworkPolicy pattern in network-policies.tf - [x] No README roadmap update needed ## Related Notes - forgejo_admin/basketball-api#343 — parent issue (Stripe webhooks unreachable) ## Related - Closes forgejo_admin/basketball-api#343 - Forgejo issue: #268

forgejo_admin added 1 commit

2026-04-05 19:09:32 +00:00

fix: add default-deny-ingress NetworkPolicy for basketball-api namespace

ci/woodpecker/push/woodpecker Pipeline was successful

Details

ci/woodpecker/pr/woodpecker Pipeline was successful

Details

ci/woodpecker/pull_request_closed/woodpecker Pipeline was successful

Details

dd1d767fc1

The basketball-api namespace was the only service namespace without a
default-deny-ingress NetworkPolicy. All other namespaces (monitoring,
forgejo, woodpecker, harbor, minio, keycloak, postgres, ollama, staging)
have one. This closes the security gap by restricting ingress to only
the tailscale namespace (funnel proxy traffic) and monitoring namespace
(Prometheus scraping).

Closes forgejo_admin/basketball-api#343

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

forgejo_admin commented

2026-04-05 19:09:49 +00:00

Author

Owner

QA Review

Verdict: APPROVED

Diff Review

Single resource added: kubernetes_manifest.netpol_basketball_api
Follows exact pattern of all 10 sibling network policies in the file
field_manager { force_conflicts = true } -- consistent
Allows ingress from tailscale (funnel proxy) and monitoring (Prometheus) -- minimum required
Placement between netpol_ollama and netpol_staging is logical

Validation

tofu fmt -- clean (no formatting changes needed)
tofu validate -- passed
tofu plan -lock=false -- 1 to add, 0 to change, 0 to destroy

Investigation Quality

The PR body documents thorough investigation of basketball-api#343:

Confirmed funnel works via public IP (208.111.35.209)
Identified proxy "connection refused" errors correlate with pod restarts
Correctly identified discovered scope (Recreate -> RollingUpdate) as separate ticket

No issues found.

## QA Review **Verdict: APPROVED** ### Diff Review - Single resource added: `kubernetes_manifest.netpol_basketball_api` - Follows exact pattern of all 10 sibling network policies in the file - `field_manager { force_conflicts = true }` -- consistent - Allows ingress from `tailscale` (funnel proxy) and `monitoring` (Prometheus) -- minimum required - Placement between `netpol_ollama` and `netpol_staging` is logical ### Validation - `tofu fmt` -- clean (no formatting changes needed) - `tofu validate` -- passed - `tofu plan -lock=false` -- 1 to add, 0 to change, 0 to destroy ### Investigation Quality The PR body documents thorough investigation of basketball-api#343: - Confirmed funnel works via public IP (208.111.35.209) - Identified proxy "connection refused" errors correlate with pod restarts - Correctly identified discovered scope (Recreate -> RollingUpdate) as separate ticket No issues found.

forgejo_admin commented

2026-04-05 19:26:58 +00:00

Author

Owner

PR #269 Review

DOMAIN REVIEW

Tech stack: Terraform (OpenTofu) / Kubernetes NetworkPolicy

Pattern compliance: The new netpol_basketball_api resource exactly matches the established pattern used by all 10 existing netpol resources in terraform/network-policies.tf:

field_manager { force_conflicts = true } -- present
kubernetes_manifest resource type -- correct
podSelector = {} (all pods in namespace) -- correct
policyTypes = ["Ingress"] -- correct (egress unrestricted per file header comment)
Ingress rules allow tailscale + monitoring namespaces only

Namespace reference: Uses hardcoded string "basketball-api" rather than a module output. This is correct -- basketball-api is deployed via pal-e-deployments/ArgoCD, not by this bootstrap repo, so no module output exists for it. Same pattern would apply to any externally-managed namespace.

Ingress rule correctness: The two allowed namespaces are correct and sufficient:

tailscale -- required for funnel-routed public traffic (Stripe webhooks, client requests)
monitoring -- required for Prometheus scraping

ArgoCD does not need direct pod ingress (it works via the k8s API server). Woodpecker CI does not need to reach running basketball-api pods. No other namespace-to-namespace traffic is needed.

Terraform quality:

tofu plan output in PR body shows clean 1 to add, 0 to change, 0 to destroy
tofu fmt and tofu validate both passed per PR body
-lock=false used in plan (per convention)

BLOCKERS

None.

NITS

Alphabetical/logical ordering: The new resource is inserted between netpol_ollama (line 200) and netpol_staging (line 203). The file does not follow strict alphabetical order (monitoring, forgejo, woodpecker, harbor, minio, keycloak, postgres, ollama, basketball_api, staging, cnpg_system), so this placement is fine, but a future cleanup could alphabetize. Non-blocking.

SOP COMPLIANCE

Branch named after issue (268-add-default-deny-ingress-networkpolicy-f references issue #268)
PR body follows template (Summary, Changes, Test Plan, Related all present)
Related section references issue #268 and cross-repo basketball-api#343
No secrets committed
tofu fmt run
tofu validate passes
tofu plan -lock=false output included
Discovered scope documented (RollingUpdate strategy -- separate ticket)
No scope creep (single resource addition, nothing unrelated)

PROCESS OBSERVATIONS

Good investigative work documented in the PR body. The root cause analysis (Recreate strategy causing pod downtime, not a funnel issue) and the decision to split the actual fix into a separate ticket demonstrates proper scope discipline.
This is a security hardening change that closes the last gap in namespace-level network isolation. Low change failure risk -- additive only, no modifications to existing resources.
Deployment frequency: straightforward tofu apply with no dependencies.

VERDICT: APPROVED

## PR #269 Review ### DOMAIN REVIEW **Tech stack:** Terraform (OpenTofu) / Kubernetes NetworkPolicy **Pattern compliance:** The new `netpol_basketball_api` resource exactly matches the established pattern used by all 10 existing netpol resources in `terraform/network-policies.tf`: - `field_manager { force_conflicts = true }` -- present - `kubernetes_manifest` resource type -- correct - `podSelector = {}` (all pods in namespace) -- correct - `policyTypes = ["Ingress"]` -- correct (egress unrestricted per file header comment) - Ingress rules allow `tailscale` + `monitoring` namespaces only **Namespace reference:** Uses hardcoded string `"basketball-api"` rather than a module output. This is correct -- basketball-api is deployed via pal-e-deployments/ArgoCD, not by this bootstrap repo, so no module output exists for it. Same pattern would apply to any externally-managed namespace. **Ingress rule correctness:** The two allowed namespaces are correct and sufficient: - `tailscale` -- required for funnel-routed public traffic (Stripe webhooks, client requests) - `monitoring` -- required for Prometheus scraping ArgoCD does not need direct pod ingress (it works via the k8s API server). Woodpecker CI does not need to reach running basketball-api pods. No other namespace-to-namespace traffic is needed. **Terraform quality:** - `tofu plan` output in PR body shows clean `1 to add, 0 to change, 0 to destroy` - `tofu fmt` and `tofu validate` both passed per PR body - `-lock=false` used in plan (per convention) ### BLOCKERS None. ### NITS 1. **Alphabetical/logical ordering**: The new resource is inserted between `netpol_ollama` (line 200) and `netpol_staging` (line 203). The file does not follow strict alphabetical order (monitoring, forgejo, woodpecker, harbor, minio, keycloak, postgres, ollama, basketball_api, staging, cnpg_system), so this placement is fine, but a future cleanup could alphabetize. Non-blocking. ### SOP COMPLIANCE - [x] Branch named after issue (`268-add-default-deny-ingress-networkpolicy-f` references issue #268) - [x] PR body follows template (Summary, Changes, Test Plan, Related all present) - [x] Related section references issue #268 and cross-repo basketball-api#343 - [x] No secrets committed - [x] `tofu fmt` run - [x] `tofu validate` passes - [x] `tofu plan -lock=false` output included - [x] Discovered scope documented (RollingUpdate strategy -- separate ticket) - [x] No scope creep (single resource addition, nothing unrelated) ### PROCESS OBSERVATIONS - Good investigative work documented in the PR body. The root cause analysis (Recreate strategy causing pod downtime, not a funnel issue) and the decision to split the actual fix into a separate ticket demonstrates proper scope discipline. - This is a security hardening change that closes the last gap in namespace-level network isolation. Low change failure risk -- additive only, no modifications to existing resources. - Deployment frequency: straightforward `tofu apply` with no dependencies. ### VERDICT: APPROVED