Harbor cluster-internal service unreachable from CI pods #369

New issue

Closed

opened 2026-05-10 20:15:27 +00:00 by ldraney · 0 comments

ldraney commented

2026-05-10 20:15:27 +00:00

Owner

Type

Bug

Description

Kaniko pods in Woodpecker CI cannot reach Harbor via the cluster-internal service URL harbor.harbor.svc.cluster.local. Both HTTPS (port 443, i/o timeout) and HTTP (port 80, connection refused) fail.

Error from pipeline logs:

error building image: unable to complete operation after 0 attempts, last error:
Get "https://harbor.harbor.svc.cluster.local/v2/": dial tcp 10.43.131.178:443: i/o timeout;
Get "http://harbor.harbor.svc.cluster.local/v2/": dial tcp 10.43.131.178:80: connect: connection refused

Harbor is healthy via external URL (harbor.tail5b443a.ts.net returns 200). The cluster-internal service/endpoint is broken.

Impact

All CI build-and-push steps are broken platform-wide (pal-enterprises, basketball-api, etc.)
Images cannot be pushed to Harbor from CI
Deployments stall because no new images are built

Evidence

ldraney/pal-enterprises pipeline #15: test passes, build-and-push fails (exit 1)
ldraney/basketball-api pipeline #512: same build-and-push failure
Harbor external health check: 200 OK

Investigation Steps

Check if Harbor pods are running: kubectl get pods -n harbor
Check the Harbor Service and Endpoints: kubectl get svc,endpoints -n harbor
Verify the ClusterIP 10.43.131.178 matches current service: kubectl get svc harbor-core -n harbor -o wide
Check if Harbor was recently restarted or recreated (stale ClusterIP?)
Check network policies that might block pod-to-service traffic in the harbor namespace

Acceptance Criteria

Kaniko can reach harbor.harbor.svc.cluster.local from CI pods
ldraney/pal-enterprises build-and-push step succeeds
Platform-wide CI image push is restored

### Type Bug ### Description Kaniko pods in Woodpecker CI cannot reach Harbor via the cluster-internal service URL `harbor.harbor.svc.cluster.local`. Both HTTPS (port 443, i/o timeout) and HTTP (port 80, connection refused) fail. Error from pipeline logs: ``` error building image: unable to complete operation after 0 attempts, last error: Get "https://harbor.harbor.svc.cluster.local/v2/": dial tcp 10.43.131.178:443: i/o timeout; Get "http://harbor.harbor.svc.cluster.local/v2/": dial tcp 10.43.131.178:80: connect: connection refused ``` Harbor is healthy via external URL (`harbor.tail5b443a.ts.net` returns 200). The cluster-internal service/endpoint is broken. ### Impact - All CI build-and-push steps are broken platform-wide (pal-enterprises, basketball-api, etc.) - Images cannot be pushed to Harbor from CI - Deployments stall because no new images are built ### Evidence - `ldraney/pal-enterprises` pipeline #15: test passes, build-and-push fails (exit 1) - `ldraney/basketball-api` pipeline #512: same build-and-push failure - Harbor external health check: 200 OK ### Investigation Steps 1. Check if Harbor pods are running: `kubectl get pods -n harbor` 2. Check the Harbor Service and Endpoints: `kubectl get svc,endpoints -n harbor` 3. Verify the ClusterIP 10.43.131.178 matches current service: `kubectl get svc harbor-core -n harbor -o wide` 4. Check if Harbor was recently restarted or recreated (stale ClusterIP?) 5. Check network policies that might block pod-to-service traffic in the harbor namespace ### Acceptance Criteria - [ ] Kaniko can reach `harbor.harbor.svc.cluster.local` from CI pods - [ ] `ldraney/pal-enterprises` build-and-push step succeeds - [ ] Platform-wide CI image push is restored