ldraney/landscaping-assistant
1
0
Fork 0
Code Issues 32 Pull requests 1 Projects Releases Packages Wiki Activity Actions 4

Fix HostAuthorization blocking k8s health probes and Tailscale hostname #231

New issue
Closed
opened 2026-06-16 01:46:12 +00:00 by ldraney · 0 comments
ldraney commented 2026-06-16 01:46:12 +00:00
Owner
Copy link

Type

Bug

Lineage

Regression from ldraney/landscaping-assistant #224 — PR #230 added config.hosts entry that broke probe authorization and Tailscale access.

Repo

ldraney/landscaping-assistant

What Broke

PR #230 added config.hosts << "landscaping-assistant.app" to production config. This enables Rails HostAuthorization middleware but only whitelists the custom domain. Two things break:

  1. k8s liveness/readiness probes fail — kubelet hits the pod IP directly (10.42.0.x:3000), not in config.hosts. Rails returns 403, probe fails, pod crash-loops (5+ restarts in 10 minutes).
  2. Tailscale production URL blocked — landscaping-assistant.tail5b443a.ts.net not in the allowed hosts list.
[ActionDispatch::HostAuthorization::DefaultResponseApp] Blocked hosts: 10.42.0.165:3000
Liveness probe failed: HTTP probe failed with statuscode: 403
Readiness probe failed: HTTP probe failed with statuscode: 403

Old pod (46h, pre-#230 image) still serving. New pod cannot roll out.

Repro Steps

  1. Pipeline #586 pushes image b654b58 to Harbor
  2. ArgoCD deploys new pod with the updated image
  3. Kubelet sends liveness probe to pod IP 10.42.0.165:3000/up
  4. Observe: Rails HostAuthorization returns 403, pod enters CrashLoopBackOff

Expected Behavior

  • /up health endpoint responds 200 regardless of Host header
  • Production accessible via both landscaping-assistant.tail5b443a.ts.net and landscaping-assistant.app
  • Dev accessible via landscaping-dev.tail5b443a.ts.net

Environment

  • Cluster/namespace: prod / landscaping-assistant
  • Service version/commit: b654b58
  • Pipeline: #586 (all steps green, pod crash-loops post-deploy)

Acceptance Criteria

  • New pod starts and passes liveness/readiness probes
  • Production accessible via landscaping-assistant.tail5b443a.ts.net and landscaping-assistant.app
  • Dev accessible via landscaping-dev.tail5b443a.ts.net
  • /up health endpoint responds 200 regardless of Host header
  • docs/networking.md updated with allowed hosts section
  • No regression in existing functionality

Related

  • Regression from: #224 / PR #230
  • Prior HostAuthorization fix: #103
  • project-landscaping-assistant
### Type Bug ### Lineage Regression from `ldraney/landscaping-assistant #224` — PR #230 added `config.hosts` entry that broke probe authorization and Tailscale access. ### Repo `ldraney/landscaping-assistant` ### What Broke PR #230 added `config.hosts << "landscaping-assistant.app"` to production config. This enables Rails `HostAuthorization` middleware but only whitelists the custom domain. Two things break: 1. **k8s liveness/readiness probes fail** — kubelet hits the pod IP directly (`10.42.0.x:3000`), not in `config.hosts`. Rails returns 403, probe fails, pod crash-loops (5+ restarts in 10 minutes). 2. **Tailscale production URL blocked** — `landscaping-assistant.tail5b443a.ts.net` not in the allowed hosts list. ``` [ActionDispatch::HostAuthorization::DefaultResponseApp] Blocked hosts: 10.42.0.165:3000 Liveness probe failed: HTTP probe failed with statuscode: 403 Readiness probe failed: HTTP probe failed with statuscode: 403 ``` Old pod (46h, pre-#230 image) still serving. New pod cannot roll out. ### Repro Steps 1. Pipeline #586 pushes image `b654b58` to Harbor 2. ArgoCD deploys new pod with the updated image 3. Kubelet sends liveness probe to pod IP `10.42.0.165:3000/up` 4. Observe: Rails HostAuthorization returns 403, pod enters CrashLoopBackOff ### Expected Behavior - `/up` health endpoint responds 200 regardless of Host header - Production accessible via both `landscaping-assistant.tail5b443a.ts.net` and `landscaping-assistant.app` - Dev accessible via `landscaping-dev.tail5b443a.ts.net` ### Environment - Cluster/namespace: prod / landscaping-assistant - Service version/commit: `b654b58` - Pipeline: #586 (all steps green, pod crash-loops post-deploy) ### Acceptance Criteria - [ ] New pod starts and passes liveness/readiness probes - [ ] Production accessible via `landscaping-assistant.tail5b443a.ts.net` and `landscaping-assistant.app` - [ ] Dev accessible via `landscaping-dev.tail5b443a.ts.net` - [ ] `/up` health endpoint responds 200 regardless of Host header - [ ] `docs/networking.md` updated with allowed hosts section - [ ] No regression in existing functionality ### Related - Regression from: #224 / PR #230 - Prior HostAuthorization fix: #103 - `project-landscaping-assistant`
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
docs/update-app-architecture
revert-213
move-other-crew-bottom-remove-schedule
feature/today-view-improvements
154-enable-assume-ssl
revert-145-person-icon
docs/user-stories-auth
docs/update-infra-pipeline-post-incident
fix-cache-step-blocking-deploy
fix-kaniko-registry-pull
deactivate-from-today
feature/quick-address-entry
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
ldraney/landscaping-assistant#231
No description provided.