Switch Playwright tests to preview webServer — prod-URL deadlocks CI #112

New issue

Open

opened 2026-04-17 06:02:42 +00:00 by forgejo_admin · 0 comments

forgejo_admin commented

2026-04-17 06:02:42 +00:00

Contributor

Type

Bug

Lineage

Discovered during CI run of PR #111 (pipeline #125). DNS resolution was restored by reverting the rename, but 3 tests failed against the live prod URL because prod serves stale code (frontend is stuck on PR #95). Any PR that adds UI features can never pass e2e tests against prod — classic deadlock: tests gate deploys, deploys update the prod tests test against.

Repo

forgejo_admin/pal-e-app

What Broke

playwright.config.ts runs tests against process.env.PLAYWRIGHT_BASE_URL ?? 'https://pal-e-app.tail5b443a.ts.net'. That's the LIVE PROD URL. Which means:

Tests for features in a PR hit prod that doesn't have those features yet
update-kustomize-tag is skipped because test fails
deploy never happens
next PR has the same problem, forever

This was masked for weeks by the pal-e-docs-app rename breaking DNS before tests even ran. With DNS fixed, we now see the underlying design flaw.

Repro Steps

Open pipeline #125 logs for PR #111.
Observe tests 28, 34, 37 failing with content-assertion errors (not DNS).
Those tests check for UI from #105/#109 which ARE in main but NOT in prod.

Expected Behavior

Playwright builds the PR's own frontend via npm run preview and runs tests against http://localhost:4173. UI matches the branch's code. API calls still go to pal-e-docs.tail5b443a.ts.net (depends on sibling pal-e-deployments CORS expansion issue).

Environment

Cluster/namespace: CI only (test step runs inside Woodpecker container)
Repo: forgejo_admin/pal-e-app
Related: pal-e-deployments CORS allow_origins must include the preview origin (separate ticket)
Current config: playwright.config.ts lines 14-35 (webServer block exists but commented)

Acceptance Criteria

Uncomment webServer block in playwright.config.ts (builds + serves preview on localhost:4173)
Default baseURL changed to http://localhost:4173 (remove prod URL default)
.woodpecker.yaml test step no longer sets PLAYWRIGHT_BASE_URL env var (or sets it explicitly to localhost:4173)
Pipeline on this branch runs the test step end-to-end with preview
update-kustomize-tag step runs on green pipeline (sibling pal-e-deployments CORS ticket resolved by then)
Verified locally: npm run build && npm run preview then npx playwright test passes against localhost:4173

Out of Scope (follow-ups)

Mocking the backend API (tests still hit live pal-e-docs)
Full stubbed-API test layer
Reviewing each individual test's dependence on prod DB content

forgejo_admin/pal-e-app#110 — rename revert (prereq for this, will bundle into same PR #111)
forgejo_admin/pal-e-deployments#{TBD} — CORS expansion for preview origin
feedback_validate_before_done.md — tests must actually validate, not just run

### Type Bug ### Lineage Discovered during CI run of PR #111 (pipeline #125). DNS resolution was restored by reverting the rename, but 3 tests failed against the live prod URL because prod serves stale code (frontend is stuck on PR #95). Any PR that adds UI features can never pass e2e tests against prod — classic deadlock: tests gate deploys, deploys update the prod tests test against. ### Repo `forgejo_admin/pal-e-app` ### What Broke `playwright.config.ts` runs tests against `process.env.PLAYWRIGHT_BASE_URL ?? 'https://pal-e-app.tail5b443a.ts.net'`. That's the LIVE PROD URL. Which means: - Tests for features in a PR hit prod that doesn't have those features yet - update-kustomize-tag is skipped because test fails - deploy never happens - next PR has the same problem, forever This was masked for weeks by the `pal-e-docs-app` rename breaking DNS before tests even ran. With DNS fixed, we now see the underlying design flaw. ### Repro Steps 1. Open pipeline #125 logs for PR #111. 2. Observe tests 28, 34, 37 failing with content-assertion errors (not DNS). 3. Those tests check for UI from #105/#109 which ARE in `main` but NOT in prod. ### Expected Behavior Playwright builds the PR's own frontend via `npm run preview` and runs tests against `http://localhost:4173`. UI matches the branch's code. API calls still go to `pal-e-docs.tail5b443a.ts.net` (depends on sibling `pal-e-deployments` CORS expansion issue). ### Environment - Cluster/namespace: CI only (test step runs inside Woodpecker container) - Repo: `forgejo_admin/pal-e-app` - Related: `pal-e-deployments` CORS allow_origins must include the preview origin (separate ticket) - Current config: `playwright.config.ts` lines 14-35 (webServer block exists but commented) ### Acceptance Criteria - [ ] Uncomment `webServer` block in `playwright.config.ts` (builds + serves preview on `localhost:4173`) - [ ] Default `baseURL` changed to `http://localhost:4173` (remove prod URL default) - [ ] `.woodpecker.yaml` `test` step no longer sets `PLAYWRIGHT_BASE_URL` env var (or sets it explicitly to localhost:4173) - [ ] Pipeline on this branch runs the test step end-to-end with preview - [ ] `update-kustomize-tag` step runs on green pipeline (sibling `pal-e-deployments` CORS ticket resolved by then) - [ ] Verified locally: `npm run build && npm run preview` then `npx playwright test` passes against localhost:4173 ### Out of Scope (follow-ups) - Mocking the backend API (tests still hit live pal-e-docs) - Full stubbed-API test layer - Reviewing each individual test's dependence on prod DB content ### Related - `forgejo_admin/pal-e-app#110` — rename revert (prereq for this, will bundle into same PR #111) - `forgejo_admin/pal-e-deployments#{TBD}` — CORS expansion for preview origin - `feedback_validate_before_done.md` — tests must actually validate, not just run