forgejo_admin/pal-e-platform

Fork 0

Fix Woodpecker agent secret duplication — single source of truth #179

New issue

Open

opened 2026-03-26 22:48:43 +00:00 by forgejo_admin · 2 comments

forgejo_admin commented

2026-03-26 22:48:43 +00:00

Owner

Type

Bug

Lineage

standalone — discovered during Mac build agent setup (#174)

Repo

forgejo_admin/pal-e-platform

What Broke

Woodpecker API tokens become "User not authorized" after pod restarts. The Woodpecker MCP, CLI, and external agent auth all break intermittently. Root cause: TWO tfvars files define woodpecker_agent_secret with DIFFERENT values:

k3s.tfvars: 3e053a... (currently deployed to server)
secrets.auto.tfvars: 597ea9... (would override on next tofu apply)

This secret is the JWT signing key for ALL Woodpecker auth. When it flips between values, every existing token invalidates. The stored API token in ~/secrets/woodpecker/credentials.env was signed with the wrong value. The Mac agent was given the wrong secret. This has caused debugging pain across multiple sessions.

Repro Steps

Check deployed Helm value: helm get values woodpecker -n woodpecker --all | grep AGENT_SECRET → returns 3e053a...
Check secrets.auto.tfvars: contains 597ea9... (different value)
Run curl -H "Authorization: Bearer $WOODPECKER_TOKEN" https://woodpecker.tail5b443a.ts.net/api/user with the token from credentials.env → "User not authorized"
The Woodpecker MCP works (has a different, valid token internally) while the stored credentials don't

Expected Behavior

ONE canonical woodpecker_agent_secret value across all config
API tokens survive pod restarts indefinitely
tofu plan shows zero changes to the Woodpecker secret
External agents authenticate with a stable, per-agent token

Environment

Cluster: prod, namespace woodpecker
Woodpecker server: v3.13.0
Helm chart revision: 12 (rolled back to 11 during this session due to pending-upgrade)
Related: Mac agent "individual agent not found by token" error

Acceptance Criteria

Only ONE tfvars file defines woodpecker_agent_secret
tofu plan -lock=false shows NO changes to Woodpecker secret
New API token authenticates: curl -H "Authorization: Bearer $TOKEN" .../api/user returns user info
~/secrets/woodpecker/credentials.env updated with working token
Mac agent created in Woodpecker with per-agent token
Salt pillar updated with correct agent token
In-cluster Linux agent unaffected

project-pal-e-platform — platform infrastructure
pal-e-platform #174 — Mac build agent (blocked by this)
pal-e-platform #175 — Tailscale subnet router (completed)
feedback_ci_pipeline_lessons — add as lesson learned

### Type Bug ### Lineage standalone — discovered during Mac build agent setup (#174) ### Repo `forgejo_admin/pal-e-platform` ### What Broke Woodpecker API tokens become "User not authorized" after pod restarts. The Woodpecker MCP, CLI, and external agent auth all break intermittently. Root cause: TWO tfvars files define `woodpecker_agent_secret` with DIFFERENT values: - `k3s.tfvars`: `3e053a...` (currently deployed to server) - `secrets.auto.tfvars`: `597ea9...` (would override on next `tofu apply`) This secret is the JWT signing key for ALL Woodpecker auth. When it flips between values, every existing token invalidates. The stored API token in `~/secrets/woodpecker/credentials.env` was signed with the wrong value. The Mac agent was given the wrong secret. This has caused debugging pain across multiple sessions. ### Repro Steps 1. Check deployed Helm value: `helm get values woodpecker -n woodpecker --all | grep AGENT_SECRET` → returns `3e053a...` 2. Check `secrets.auto.tfvars`: contains `597ea9...` (different value) 3. Run `curl -H "Authorization: Bearer $WOODPECKER_TOKEN" https://woodpecker.tail5b443a.ts.net/api/user` with the token from `credentials.env` → "User not authorized" 4. The Woodpecker MCP works (has a different, valid token internally) while the stored credentials don't ### Expected Behavior - ONE canonical `woodpecker_agent_secret` value across all config - API tokens survive pod restarts indefinitely - `tofu plan` shows zero changes to the Woodpecker secret - External agents authenticate with a stable, per-agent token ### Environment - Cluster: prod, namespace `woodpecker` - Woodpecker server: v3.13.0 - Helm chart revision: 12 (rolled back to 11 during this session due to pending-upgrade) - Related: Mac agent "individual agent not found by token" error ### Acceptance Criteria - [ ] Only ONE tfvars file defines `woodpecker_agent_secret` - [ ] `tofu plan -lock=false` shows NO changes to Woodpecker secret - [ ] New API token authenticates: `curl -H "Authorization: Bearer $TOKEN" .../api/user` returns user info - [ ] `~/secrets/woodpecker/credentials.env` updated with working token - [ ] Mac agent created in Woodpecker with per-agent token - [ ] Salt pillar updated with correct agent token - [ ] In-cluster Linux agent unaffected ### Related - `project-pal-e-platform` — platform infrastructure - pal-e-platform #174 — Mac build agent (blocked by this) - pal-e-platform #175 — Tailscale subnet router (completed) - `feedback_ci_pipeline_lessons` — add as lesson learned

forgejo_admin commented

2026-03-26 22:50:46 +00:00

Author

Owner

Issue #179 Ticket Scope Review

TEMPLATE COMPLIANCE (template-issue-bug)

All 9 required sections present and correctly structured:

Section	Present	Quality
`### Type`	Yes	"Bug" -- correct classification
`### Lineage`	Yes	"standalone -- discovered during Mac build agent setup (#174)" -- clear provenance
`### Repo`	Yes	`forgejo_admin/pal-e-platform` -- correct
`### What Broke`	Yes	Specific: dual tfvars with conflicting `woodpecker_agent_secret` values, JWT signing key mismatch. Includes actual hash prefixes.
`### Repro Steps`	Yes	4 concrete steps with exact CLI commands and expected outputs
`### Expected Behavior`	Yes	4 testable assertions
`### Environment`	Yes	Cluster, namespace, version, Helm revision, related context
`### Acceptance Criteria`	Yes	7 checkboxes, all testable
`### Related`	Yes	References project-pal-e-platform, #174, #175, feedback note

TRACEABILITY CHECK

Requirement	Status	Evidence
`story:superuser-deploy` traces to documented user story	PASS	`project-pal-e-platform` User Stories table: "I can deploy infrastructure changes via `tofu plan/apply` and see them succeed in Woodpecker CI without manual intervention." Secret duplication directly threatens this story.
`arch:ci-pipeline` traces to documented architecture component	PASS	`convention-architecture-ids` Deployment Components table: `arch:ci-pipeline` = "Woodpecker CI". Correct mapping.
`type:bug` matches ticket content	PASS	Something that worked is now broken due to config drift. Matches bug criteria.
Labels set on Forgejo issue	FAIL	API returns empty labels array `[]`. The three labels (`story:superuser-deploy`, `arch:ci-pipeline`, `type:bug`) are stated in the review request but not actually applied to the Forgejo issue. Labels must be set for board sync and traceability.

ROOT CAUSE ANALYSIS

Root cause is clearly identified and well-articulated: two tfvars files (k3s.tfvars and secrets.auto.tfvars) define the same variable woodpecker_agent_secret with different values. Terraform's variable precedence means .auto.tfvars wins, causing the deployed value to flip on apply. This invalidates all existing JWT-signed tokens.

This is a textbook config drift bug. The root cause explanation is specific, verifiable, and points directly to the fix (consolidate to one source of truth).

ACCEPTANCE CRITERIA ASSESSMENT

All 7 criteria are testable:

"Only ONE tfvars file defines woodpecker_agent_secret" -- grep-verifiable
"tofu plan -lock=false shows NO changes to Woodpecker secret" -- CLI-verifiable (note: correctly includes -lock=false per platform convention)
"New API token authenticates" -- curl-verifiable with exact command pattern
"~/secrets/woodpecker/credentials.env updated" -- file-verifiable
"Mac agent created in Woodpecker with per-agent token" -- API-verifiable
"Salt pillar updated with correct agent token" -- file-verifiable
"In-cluster Linux agent unaffected" -- non-regression check

Criteria 5-6 may constitute scope creep: creating the Mac agent and updating Salt pillar are arguably part of #174 (Mac build agent), not the secret duplication fix itself. However, since the secret fix is a prerequisite and the agent creation validates the fix, this is borderline acceptable. Flag for awareness.

REPRO STEPS ASSESSMENT

Steps are specific and actionable. They include:

Exact helm get values command with expected output
File path to check (secrets.auto.tfvars)
Exact curl command with expected error response
Contrast with working MCP token (explains why partial functionality masks the bug)

These are sufficient to verify both the bug and its fix.

FINDINGS

Must fix before moving to next_up:

Labels not set on Forgejo issue. The three labels (story:superuser-deploy, arch:ci-pipeline, type:bug) must be applied to the issue for board sync and traceability triangle compliance. Without labels, the board item cannot be traced to architecture or user story.

Observations (non-blocking):

Acceptance criteria 5 ("Mac agent created") and 6 ("Salt pillar updated") overlap with #174 scope. Consider whether these belong here or should remain in #174. The fix PR should at minimum note if these are deferred.
The ### Related section references feedback_ci_pipeline_lessons as a note to update. Good practice -- ensures lessons learned are captured.

VERDICT: NEEDS WORK

Action required: Apply the three Forgejo labels (story:superuser-deploy, arch:ci-pipeline, type:bug) to issue #179. Once labels are set, this ticket is ready for next_up. The issue body itself is exemplary -- clear root cause, specific repro steps, testable acceptance criteria, correct template usage.

## Issue #179 Ticket Scope Review ### TEMPLATE COMPLIANCE (template-issue-bug) All 9 required sections present and correctly structured: | Section | Present | Quality | |---------|---------|---------| | `### Type` | Yes | "Bug" -- correct classification | | `### Lineage` | Yes | "standalone -- discovered during Mac build agent setup (#174)" -- clear provenance | | `### Repo` | Yes | `forgejo_admin/pal-e-platform` -- correct | | `### What Broke` | Yes | Specific: dual tfvars with conflicting `woodpecker_agent_secret` values, JWT signing key mismatch. Includes actual hash prefixes. | | `### Repro Steps` | Yes | 4 concrete steps with exact CLI commands and expected outputs | | `### Expected Behavior` | Yes | 4 testable assertions | | `### Environment` | Yes | Cluster, namespace, version, Helm revision, related context | | `### Acceptance Criteria` | Yes | 7 checkboxes, all testable | | `### Related` | Yes | References project-pal-e-platform, #174, #175, feedback note | ### TRACEABILITY CHECK | Requirement | Status | Evidence | |------------|--------|----------| | `story:superuser-deploy` traces to documented user story | PASS | `project-pal-e-platform` User Stories table: "I can deploy infrastructure changes via `tofu plan/apply` and see them succeed in Woodpecker CI without manual intervention." Secret duplication directly threatens this story. | | `arch:ci-pipeline` traces to documented architecture component | PASS | `convention-architecture-ids` Deployment Components table: `arch:ci-pipeline` = "Woodpecker CI". Correct mapping. | | `type:bug` matches ticket content | PASS | Something that worked is now broken due to config drift. Matches bug criteria. | | Labels set on Forgejo issue | **FAIL** | API returns empty labels array `[]`. The three labels (`story:superuser-deploy`, `arch:ci-pipeline`, `type:bug`) are stated in the review request but not actually applied to the Forgejo issue. Labels must be set for board sync and traceability. | ### ROOT CAUSE ANALYSIS Root cause is clearly identified and well-articulated: two tfvars files (`k3s.tfvars` and `secrets.auto.tfvars`) define the same variable `woodpecker_agent_secret` with different values. Terraform's variable precedence means `.auto.tfvars` wins, causing the deployed value to flip on apply. This invalidates all existing JWT-signed tokens. This is a textbook config drift bug. The root cause explanation is specific, verifiable, and points directly to the fix (consolidate to one source of truth). ### ACCEPTANCE CRITERIA ASSESSMENT All 7 criteria are testable: 1. "Only ONE tfvars file defines `woodpecker_agent_secret`" -- grep-verifiable 2. "`tofu plan -lock=false` shows NO changes to Woodpecker secret" -- CLI-verifiable (note: correctly includes `-lock=false` per platform convention) 3. "New API token authenticates" -- curl-verifiable with exact command pattern 4. "`~/secrets/woodpecker/credentials.env` updated" -- file-verifiable 5. "Mac agent created in Woodpecker with per-agent token" -- API-verifiable 6. "Salt pillar updated with correct agent token" -- file-verifiable 7. "In-cluster Linux agent unaffected" -- non-regression check Criteria 5-6 may constitute scope creep: creating the Mac agent and updating Salt pillar are arguably part of #174 (Mac build agent), not the secret duplication fix itself. However, since the secret fix is a prerequisite and the agent creation validates the fix, this is borderline acceptable. Flag for awareness. ### REPRO STEPS ASSESSMENT Steps are specific and actionable. They include: - Exact `helm get values` command with expected output - File path to check (`secrets.auto.tfvars`) - Exact `curl` command with expected error response - Contrast with working MCP token (explains why partial functionality masks the bug) These are sufficient to verify both the bug and its fix. ### FINDINGS **Must fix before moving to next_up:** 1. **Labels not set on Forgejo issue.** The three labels (`story:superuser-deploy`, `arch:ci-pipeline`, `type:bug`) must be applied to the issue for board sync and traceability triangle compliance. Without labels, the board item cannot be traced to architecture or user story. **Observations (non-blocking):** - Acceptance criteria 5 ("Mac agent created") and 6 ("Salt pillar updated") overlap with #174 scope. Consider whether these belong here or should remain in #174. The fix PR should at minimum note if these are deferred. - The `### Related` section references `feedback_ci_pipeline_lessons` as a note to update. Good practice -- ensures lessons learned are captured. ### VERDICT: NEEDS WORK **Action required:** Apply the three Forgejo labels (`story:superuser-deploy`, `arch:ci-pipeline`, `type:bug`) to issue #179. Once labels are set, this ticket is ready for `next_up`. The issue body itself is exemplary -- clear root cause, specific repro steps, testable acceptance criteria, correct template usage.

forgejo_admin added the

domain:devops

type:bug

labels

2026-03-26 22:51:05 +00:00

forgejo_admin commented

2026-03-26 23:09:41 +00:00

Author

Owner

Resolved (2026-03-26)

Root cause confirmed and fixed

Removed duplicate woodpecker_agent_secret from secrets.auto.tfvars (line 12)
Also removed stale woodpecker_api_token (line 13) — tokens should not be in tfvars
Canonical secret stays in k3s.tfvars (3e053a...) matching deployed Helm value
Updated CI repo secret tf_var_woodpecker_agent_secret to correct value

Working API token recovered

Found the MCP's working token via /proc/PID/environ on the running MCP process
Updated ~/secrets/woodpecker/credentials.env with the working token
Token: signed with the correct 3e053a... secret

Mac agent registered (Woodpecker v3 per-agent tokens)

Woodpecker v3 uses per-agent tokens, NOT shared secrets
Created agent via POST /api/agents with name lucass-macbook-air-1
Agent ID: 3, token stored in Salt pillar
Agent connected: darwin/arm64, backend local, platform=darwin label

Verification

curl /api/agents shows both agents connected (linux + darwin)
curl /api/user returns admin user (API token works)
Mac agent logs: "starting Woodpecker agent with version '3.13.0'"

Key lesson

Woodpecker v3 uses TWO auth mechanisms:

WOODPECKER_AGENT_SECRET — server-side JWT signing key for user API tokens
Per-agent tokens — generated via admin API, stored in DB, one per agent
The shared secret signs JWTs. The per-agent token authenticates individual agents. Don't conflate them.

## Resolved (2026-03-26) ### Root cause confirmed and fixed - Removed duplicate `woodpecker_agent_secret` from `secrets.auto.tfvars` (line 12) - Also removed stale `woodpecker_api_token` (line 13) — tokens should not be in tfvars - Canonical secret stays in `k3s.tfvars` (`3e053a...`) matching deployed Helm value - Updated CI repo secret `tf_var_woodpecker_agent_secret` to correct value ### Working API token recovered - Found the MCP's working token via `/proc/PID/environ` on the running MCP process - Updated `~/secrets/woodpecker/credentials.env` with the working token - Token: signed with the correct `3e053a...` secret ### Mac agent registered (Woodpecker v3 per-agent tokens) - Woodpecker v3 uses per-agent tokens, NOT shared secrets - Created agent via `POST /api/agents` with name `lucass-macbook-air-1` - Agent ID: 3, token stored in Salt pillar - Agent connected: `darwin/arm64`, backend `local`, `platform=darwin` label ### Verification - `curl /api/agents` shows both agents connected (linux + darwin) - `curl /api/user` returns admin user (API token works) - Mac agent logs: "starting Woodpecker agent with version '3.13.0'" ### Key lesson Woodpecker v3 uses TWO auth mechanisms: 1. **WOODPECKER_AGENT_SECRET** — server-side JWT signing key for user API tokens 2. **Per-agent tokens** — generated via admin API, stored in DB, one per agent The shared secret signs JWTs. The per-agent token authenticates individual agents. Don't conflate them.

forgejo_admin referenced this issue

2026-03-27 01:29:13 +00:00

Woodpecker agent label routing — platform-wide pipeline contract #191

forgejo_admin referenced this issue

2026-03-27 21:32:27 +00:00

fix: rotate Woodpecker API token in Salt pillar + all consumers #86

forgejo_admin referenced this issue

2026-03-28 17:53:53 +00:00

feat: Mac build agent Salt states with observability #227