Woodpecker server log noise: orphaned queue.Done / stream errors #241

New issue

Open

opened 2026-03-28 22:50:32 +00:00 by forgejo_admin · 1 comment

forgejo_admin commented

2026-03-28 22:50:32 +00:00

Contributor

Type

Bug

Lineage

Board: board-pal-e-platform
Story: story:superuser-deploy
Arch: arch:ci-pipeline
Discovered from: #184 (Harbor connectivity timeout resolution)

Repo

forgejo_admin/pal-e-platform (Woodpecker deployed via Helm in this repo)

What Broke

Woodpecker server emits ~5 errors/hour:

queue.Done: cannot ack workflow: sql: no rows in result set
stream: not found

These fire on queue.Done ack and log stream close for completed pipelines. Pipelines succeed but get logged as errored.

Repro Steps

kubectl logs -n woodpecker deploy/woodpecker-server --tail=200 | grep -c "no rows"
Observe ~5+ matches per hour
Cross-reference with pipeline status — pipelines succeed despite errors

Expected Behavior

No errors for successfully completed pipelines.

Environment

Namespace: woodpecker
Pods: woodpecker-server-0, woodpecker-agent-0, woodpecker-db-1
All pods Running, zero recent restarts

Fix Path

Server restart or DB vacuum of orphaned workflow records should clear this. Low priority — cosmetic only.

Acceptance Criteria

Woodpecker server logs clean of orphaned queue/stream errors
Pipelines still succeed after fix

Constraints

Do NOT restart server during active pipeline runs
Verify no stuck workflows exist before restart

#184 — Harbor connectivity timeout (parent incident, now resolved)

### Type Bug ### Lineage - Board: board-pal-e-platform - Story: story:superuser-deploy - Arch: arch:ci-pipeline - Discovered from: #184 (Harbor connectivity timeout resolution) ### Repo `forgejo_admin/pal-e-platform` (Woodpecker deployed via Helm in this repo) ### What Broke Woodpecker server emits ~5 errors/hour: - `queue.Done: cannot ack workflow: sql: no rows in result set` - `stream: not found` These fire on `queue.Done` ack and log stream close for completed pipelines. Pipelines succeed but get logged as errored. ### Repro Steps 1. `kubectl logs -n woodpecker deploy/woodpecker-server --tail=200 | grep -c "no rows"` 2. Observe ~5+ matches per hour 3. Cross-reference with pipeline status — pipelines succeed despite errors ### Expected Behavior No errors for successfully completed pipelines. ### Environment - Namespace: woodpecker - Pods: woodpecker-server-0, woodpecker-agent-0, woodpecker-db-1 - All pods Running, zero recent restarts ### Fix Path Server restart or DB vacuum of orphaned workflow records should clear this. Low priority — cosmetic only. ### Acceptance Criteria - [ ] Woodpecker server logs clean of orphaned queue/stream errors - [ ] Pipelines still succeed after fix ### Constraints - Do NOT restart server during active pipeline runs - Verify no stuck workflows exist before restart ### Related - #184 — Harbor connectivity timeout (parent incident, now resolved)

forgejo_admin commented

2026-03-29 04:18:13 +00:00

Author

Contributor

Scope Review: NEEDS_REFINEMENT

Review note: review-631-2026-03-28

Well-structured bug report with complete traceability (story:superuser-deploy, arch:ci-pipeline, Forgejo #241). Template sections are all present. Low blast radius — cosmetic log noise only.

Issues found:

Fix Path is ambiguous: "server restart or DB vacuum" — which approach? Specify exact commands.
No diagnostic step to confirm root cause (orphaned DB records vs. recurring Woodpecker bug).
Fix is operational (kubectl/SQL), not a code change — may not be suitable for standard agent dispatch without clarification.
[SCOPE] Needs human decision: is this a one-time manual cleanup or should a config change prevent recurrence?

## Scope Review: NEEDS_REFINEMENT Review note: `review-631-2026-03-28` Well-structured bug report with complete traceability (story:superuser-deploy, arch:ci-pipeline, Forgejo #241). Template sections are all present. Low blast radius — cosmetic log noise only. **Issues found:** - Fix Path is ambiguous: "server restart or DB vacuum" — which approach? Specify exact commands. - No diagnostic step to confirm root cause (orphaned DB records vs. recurring Woodpecker bug). - Fix is operational (kubectl/SQL), not a code change — may not be suitable for standard agent dispatch without clarification. - `[SCOPE]` Needs human decision: is this a one-time manual cleanup or should a config change prevent recurrence?