ldraney/pal-e-platform
1
0
Fork 0
Code Issues 91 Pull requests 5 Projects Releases Packages Wiki Activity Actions

Post-move network recovery -- Archbox at new location #153

New issue
Closed
opened 2026-03-24 20:06:54 +00:00 by forgejo_admin · 0 comments
forgejo_admin commented 2026-03-24 20:06:54 +00:00
Contributor
Copy link

Type

Feature

Lineage

Standalone -- discovered during physical server relocation

Repo

forgejo_admin/pal-e-platform

User Story

As a platform operator
I want all services restored after the physical server move
So that the production environment is fully operational at the new location

Context

Archbox moved to a new physical location (Xfinity network). LAN IP changed from 10.0.0.217 to 10.0.0.149 (DHCP). Tailscale overlay is up and cluster is running, but several issues need attention to restore full production.

What is working: Tailscale UP, K3s healthy, funnel proxies running, public URLs responding (forgejo 200, grafana 302, woodpecker 200, argocd 200).

Action items by priority:

  • P0: Fix systemd-resolved for Tailscale DNS (MagicDNS broken)
  • P1: Investigate pal-e-docs 404 on funnel
  • P2: Fix unhealthy pods (basketball-api ErrImagePull, mcd-tracker ErrImagePull, mcd-tracker-app CrashLoopBackOff, westside-app ImagePullBackOff, ollama UnexpectedAdmissionError)
  • P3: Update archbox IP in memory/docs
  • P4: Salt master/minion decision

File Targets

Files to check/modify:

  • Salt pillar files if IP references exist
  • MEMORY.md for IP updates

Files NOT to touch:

  • Terraform files -- no hardcoded IPs, routes through Tailscale

Acceptance Criteria

  • systemd-resolved enabled and Tailscale MagicDNS working
  • pal-e-docs accessible via funnel URL
  • All pods in Running state
  • IP references updated
  • Salt decision documented

Test Expectations

  • tailscale status shows all nodes reachable
  • kubectl get pods -A shows no CrashLoopBackOff or ImagePullBackOff
  • All funnel URLs return expected status codes
  • Run command: kubectl get pods -A --field-selector=status.phase!=Running

Constraints

  • Do not break existing Tailscale funnel configuration
  • DHCP reservation should be set on router for stability
  • Salt changes require careful consideration of whether to re-enable

Checklist

  • PR opened
  • Tests pass
  • No unrelated changes

Related

  • pal-e-platform -- parent project
### Type Feature ### Lineage Standalone -- discovered during physical server relocation ### Repo `forgejo_admin/pal-e-platform` ### User Story As a platform operator I want all services restored after the physical server move So that the production environment is fully operational at the new location ### Context Archbox moved to a new physical location (Xfinity network). LAN IP changed from `10.0.0.217` to `10.0.0.149` (DHCP). Tailscale overlay is up and cluster is running, but several issues need attention to restore full production. What is working: Tailscale UP, K3s healthy, funnel proxies running, public URLs responding (forgejo 200, grafana 302, woodpecker 200, argocd 200). Action items by priority: - P0: Fix systemd-resolved for Tailscale DNS (MagicDNS broken) - P1: Investigate pal-e-docs 404 on funnel - P2: Fix unhealthy pods (basketball-api ErrImagePull, mcd-tracker ErrImagePull, mcd-tracker-app CrashLoopBackOff, westside-app ImagePullBackOff, ollama UnexpectedAdmissionError) - P3: Update archbox IP in memory/docs - P4: Salt master/minion decision ### File Targets Files to check/modify: - Salt pillar files if IP references exist - MEMORY.md for IP updates Files NOT to touch: - Terraform files -- no hardcoded IPs, routes through Tailscale ### Acceptance Criteria - [ ] systemd-resolved enabled and Tailscale MagicDNS working - [ ] pal-e-docs accessible via funnel URL - [ ] All pods in Running state - [ ] IP references updated - [ ] Salt decision documented ### Test Expectations - [ ] `tailscale status` shows all nodes reachable - [ ] `kubectl get pods -A` shows no CrashLoopBackOff or ImagePullBackOff - [ ] All funnel URLs return expected status codes - Run command: `kubectl get pods -A --field-selector=status.phase!=Running` ### Constraints - Do not break existing Tailscale funnel configuration - DHCP reservation should be set on router for stability - Salt changes require careful consideration of whether to re-enable ### Checklist - [ ] PR opened - [ ] Tests pass - [ ] No unrelated changes ### Related - `pal-e-platform` -- parent project
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
fix-gem-bin-path
netpol-westside-ror
fix-woodpecker-multi-pipeline
345-harbor-mobile-css
290-payment-pipeline-observability
spec-board-centric-renderer
284-fix-add-pal-e-docs-namespace-to-postgres
284-add-pal-e-docs-postgres-netpol
188-cross-repo-worktree-isolation-for-parall
111-fix-keycloak-probe
86-fix-rotate-woodpecker-api-token-in-salt
71-feat-deploy-pyrra-slo-manager-with-servi
64-hotfix-woodpecker-oauth-login-broken-for
18-fix-grafana-duplicate-default-datasource-v2
18-fix-grafana-duplicate-default-datasource
13-fix-cnpg-all-parameters
No results found.
Labels
Clear labels   
domain:backend

   
domain:devops

   
domain:frontend

   
status:approved

QA passed, awaiting merge approval

  
status:in-progress

Dev agent is actively working

  
status:needs-fix

QA found issues, back to dev

  
status:qa

PR submitted, awaiting QA review

  
type:bug

Bug fix

  
type:devops

Infrastructure/CI/config work

  
type:feature

New functionality

No labels
domain:backend
domain:devops
domain:frontend
status:approved
status:in-progress
status:needs-fix
status:qa
type:bug
type:devops
type:feature
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
ldraney/pal-e-platform#153
No description provided.