Cluster at pod-scheduling ceiling (125/110 maxPods on single-node k3s) #331

New issue

Open

opened 2026-05-04 12:27:50 +00:00 by forgejo_admin · 0 comments

forgejo_admin commented

2026-05-04 12:27:50 +00:00

Contributor

Type

Bug

Lineage

Discovered 2026-05-04 during the notion-mcp-remote cascade. tofu apply succeeded for notion-mcp-remote infra (5 resources created), but the notion-mcp-remote deployment pod cannot schedule. This is platform-level capacity, not a notion-mcp-remote issue. Will block onboarding any future funnel-enabled service until resolved.

Repo

forgejo_admin/pal-e-platform

What Broke

The single-node k3s cluster (archbox) is at or above the kubelet maxPods=110 default. New pods cannot schedule.

Pod event on the stuck pod:

Warning  FailedScheduling  default-scheduler
0/1 nodes are available: 1 Too many pods. no new claims to deallocate,
preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.

Total pod count at time of discovery: 125 (over the 110 default).

Distribution:

Namespace	Pods
tailscale	38
monitoring	10
harbor	9
argocd	8
palworld	5
woodpecker	4
postgres	4
kube-system	4
default	4
tofu-state	3
(other)	36

The tailscale namespace dominates because every funnel-enabled service spawns its own operator-managed pod. Onboarding scales pod count linearly.

Repro Steps

kubectl get pods -A --no-headers | wc -l → 125
kubectl describe node archbox | grep -i "Too many pods" → confirms scheduling rejection
Any tofu apply that creates a new Deployment/StatefulSet → pod stays Pending forever

Expected Behavior

Per feedback_enterprise_no_workarounds.md (be ready to scale): the platform should accommodate ongoing service onboarding. New services that pass service-onboarding-sop should not be blocked by static cluster capacity limits.

Environment

Cluster: archbox (single-node k3s)
Affected services: any new pod scheduling, including notion-mcp-remote deployment
Related: ArgoCD Application notion-mcp-remote is Synced/Progressing but pod is Pending

Acceptance Criteria

New pods can schedule on the cluster
notion-mcp-remote pod transitions Pending → Running (assuming Harbor image is also resolved via #1048)
Pod-onboarding documented in service-onboarding-sop Pre-Deploy Validation Checklist (capacity check before adding funnel-enabled service)
No regression: existing services remain healthy after the chosen fix

Proposed fixes (operator decision)

Raise kubelet maxPods on archbox to 250: --kubelet-arg=max-pods=250 in k3s service config. Requires k3s restart (~30s API downtime). Single change, scales platform.
Add a second node: more durable but slower; requires k3s join + node provisioning.
Consolidate Tailscale funnels: investigate if multiple funnels can share an operator pod (38 → ~5 frees 33 slots). Tailscale operator config spike.
Decommission palworld namespace: frees 5 slots, mitigates short-term, doesn't solve scaling.

Recommended path: Option 1 short-term, Option 3 spike for medium-term.

project-pal-e-platform
arch-deployment-notion-mcp-remote — the cascade that surfaced this
feedback_enterprise_no_workarounds.md
service-onboarding-sop

### Type Bug ### Lineage Discovered 2026-05-04 during the notion-mcp-remote cascade. `tofu apply` succeeded for notion-mcp-remote infra (5 resources created), but the `notion-mcp-remote` deployment pod cannot schedule. This is platform-level capacity, not a notion-mcp-remote issue. Will block onboarding any future funnel-enabled service until resolved. ### Repo `forgejo_admin/pal-e-platform` ### What Broke The single-node k3s cluster (`archbox`) is at or above the kubelet `maxPods=110` default. New pods cannot schedule. Pod event on the stuck pod: ``` Warning FailedScheduling default-scheduler 0/1 nodes are available: 1 Too many pods. no new claims to deallocate, preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod. ``` Total pod count at time of discovery: **125** (over the 110 default). Distribution: | Namespace | Pods | |---|---| | tailscale | 38 | | monitoring | 10 | | harbor | 9 | | argocd | 8 | | palworld | 5 | | woodpecker | 4 | | postgres | 4 | | kube-system | 4 | | default | 4 | | tofu-state | 3 | | (other) | 36 | The `tailscale` namespace dominates because every funnel-enabled service spawns its own operator-managed pod. Onboarding scales pod count linearly. ### Repro Steps 1. `kubectl get pods -A --no-headers | wc -l` → 125 2. `kubectl describe node archbox | grep -i "Too many pods"` → confirms scheduling rejection 3. Any `tofu apply` that creates a new Deployment/StatefulSet → pod stays Pending forever ### Expected Behavior Per `feedback_enterprise_no_workarounds.md` (be ready to scale): the platform should accommodate ongoing service onboarding. New services that pass `service-onboarding-sop` should not be blocked by static cluster capacity limits. ### Environment - Cluster: `archbox` (single-node k3s) - Affected services: any new pod scheduling, including `notion-mcp-remote` deployment - Related: ArgoCD Application `notion-mcp-remote` is Synced/Progressing but pod is Pending ### Acceptance Criteria - [ ] New pods can schedule on the cluster - [ ] `notion-mcp-remote` pod transitions Pending → Running (assuming Harbor image is also resolved via #1048) - [ ] Pod-onboarding documented in `service-onboarding-sop` Pre-Deploy Validation Checklist (capacity check before adding funnel-enabled service) - [ ] No regression: existing services remain healthy after the chosen fix ### Proposed fixes (operator decision) 1. **Raise kubelet maxPods on archbox** to 250: `--kubelet-arg=max-pods=250` in k3s service config. Requires k3s restart (~30s API downtime). Single change, scales platform. 2. **Add a second node**: more durable but slower; requires k3s join + node provisioning. 3. **Consolidate Tailscale funnels**: investigate if multiple funnels can share an operator pod (38 → ~5 frees 33 slots). Tailscale operator config spike. 4. **Decommission `palworld` namespace**: frees 5 slots, mitigates short-term, doesn't solve scaling. Recommended path: Option 1 short-term, Option 3 spike for medium-term. ### Related - `project-pal-e-platform` - `arch-deployment-notion-mcp-remote` — the cascade that surfaced this - `feedback_enterprise_no_workarounds.md` - `service-onboarding-sop`