ldraney/claude-custom

Fork 0

Validation agent: Playwright-based role validation for westside-basketball #235

New issue

Closed

opened 2026-04-04 22:48:48 +00:00 by forgejo_admin · 3 comments

forgejo_admin commented

2026-04-04 22:48:48 +00:00

Contributor

Type

Feature

User Story

story:validation-execute As the superadmin, I want an automated agent that validates deployed features by logging in as each role (admin, coach, parent) and navigating the actual UI with Playwright, so that I can verify merges work in production without manual clicking.

Lineage

Story: story:validation-execute (added to project-pal-e-agency user stories)
Arch: arch:claude-custom (note creation tracked as discovered scope)
Blocked by: Nothing (Playwright MCP already wired)
Blocks: Schedule ticket validations (#804, #808, #809, #810, #815)

Repo

forgejo_admin/claude-custom

Context

Today's schedule feature session produced 4 merged PRs that need production validation. The Playwright MCP server is already configured in ~/.mcp.json. The concept: a Claude Code skill that dispatches Playwright-based validation against the live app.

What exists today:

Playwright MCP server (mcp__playwright__* tools) — browser automation
Keycloak test accounts with known password Westside2026! for all roles
/validate-ticket skill — runs post-merge checks but doesn't use browser automation

Decision: Skill, not agent. A skill is simpler, follows the existing skills/{name}/SKILL.md convention, and can be invoked inline via /validate-ui. An agent would need its own frontmatter, tool restrictions, and spawn conventions — overkill for this.

Test accounts (all use password Westside2026!):

Admin+Coach: marcusdraney23@gmail.com (roles: admin, coach, player)
Coach only: ken10seka@gmail.com (role: coach)
Parent: apaisasandra@gmail.com (role: player — 3 kids, 2 teams)

Validation mechanism:

mcp__playwright__browser_snapshot — captures page accessibility tree for text content validation
mcp__playwright__browser_take_screenshot — captures visual evidence for the verdict report
mcp__playwright__browser_fill + browser_click — fills Keycloak login form
Validation compares snapshot text against acceptance criteria from the Forgejo issue

Keycloak login flow:

Navigate to app URL (e.g., /coach)
App redirects to Keycloak login page
Fill email field, fill password field, click Sign In
Keycloak redirects back to app with auth token
Page renders authenticated content

File Targets

~/.claude/skills/validate-ui/SKILL.md — skill definition with usage instructions, Playwright flow, and verdict template

Acceptance Criteria

Skill accepts parameters: role (admin/coach/parent), URL, acceptance criteria text
Skill logs into Keycloak via Playwright (fill email/password form, handle redirect)
Skill captures browser_snapshot and validates text content against acceptance criteria
Skill captures browser_take_screenshot as visual evidence
Skill reports PASS/PARTIAL/FAIL verdict with screenshot path and matched/unmatched criteria
Works with existing Playwright MCP server (no new infrastructure)

Test Expectations

Skill logs in as coach (Ken Seka) and validates coach dashboard has schedule content
Skill logs in as parent (Sandra Apaisa) and validates my-players has schedule content
Skill validates public schedule page shows 2 local team cards (not 4)
Screenshots saved to working directory for review

Constraints

Uses existing Playwright MCP tools — no new MCP servers
Skill lives in ~/.claude/skills/validate-ui/SKILL.md (standard skill convention)
Password hardcoded as Westside2026! for all test accounts
Must handle Keycloak redirect flow (app → Keycloak → app callback)
No test data creation — uses existing seeded data and known test accounts

Checklist

Create ~/.claude/skills/validate-ui/SKILL.md
Implement Keycloak login flow (navigate → fill → click → wait for redirect)
Implement snapshot capture + text validation against AC
Implement screenshot capture for evidence
Implement verdict template (PASS/PARTIAL/FAIL + evidence)
Test: public schedule page (no auth)
Test: coach login + dashboard schedule
Test: parent login + my-players schedule
Test: admin login + admin schedule

Playwright MCP in ~/.mcp.json
westside-landing#216, #217, #218 (schedule PRs needing validation)
/validate-ticket skill (existing — /validate-ui is complementary, not a replacement)
project-pal-e-agency user stories — story:validation-execute added

### Type Feature ### User Story `story:validation-execute` As the superadmin, I want an automated agent that validates deployed features by logging in as each role (admin, coach, parent) and navigating the actual UI with Playwright, so that I can verify merges work in production without manual clicking. ### Lineage - **Story:** `story:validation-execute` (added to project-pal-e-agency user stories) - **Arch:** `arch:claude-custom` (note creation tracked as discovered scope) - **Blocked by:** Nothing (Playwright MCP already wired) - **Blocks:** Schedule ticket validations (#804, #808, #809, #810, #815) ### Repo `forgejo_admin/claude-custom` ### Context Today's schedule feature session produced 4 merged PRs that need production validation. The Playwright MCP server is already configured in `~/.mcp.json`. The concept: a Claude Code **skill** that dispatches Playwright-based validation against the live app. **What exists today:** - Playwright MCP server (`mcp__playwright__*` tools) — browser automation - Keycloak test accounts with known password `Westside2026!` for all roles - `/validate-ticket` skill — runs post-merge checks but doesn't use browser automation **Decision: Skill, not agent.** A skill is simpler, follows the existing `skills/{name}/SKILL.md` convention, and can be invoked inline via `/validate-ui`. An agent would need its own frontmatter, tool restrictions, and spawn conventions — overkill for this. **Test accounts (all use password `Westside2026!`):** - Admin+Coach: `marcusdraney23@gmail.com` (roles: admin, coach, player) - Coach only: `ken10seka@gmail.com` (role: coach) - Parent: `apaisasandra@gmail.com` (role: player — 3 kids, 2 teams) **Validation mechanism:** - `mcp__playwright__browser_snapshot` — captures page accessibility tree for text content validation - `mcp__playwright__browser_take_screenshot` — captures visual evidence for the verdict report - `mcp__playwright__browser_fill` + `browser_click` — fills Keycloak login form - Validation compares snapshot text against acceptance criteria from the Forgejo issue **Keycloak login flow:** 1. Navigate to app URL (e.g., `/coach`) 2. App redirects to Keycloak login page 3. Fill email field, fill password field, click Sign In 4. Keycloak redirects back to app with auth token 5. Page renders authenticated content ### File Targets - `~/.claude/skills/validate-ui/SKILL.md` — skill definition with usage instructions, Playwright flow, and verdict template ### Acceptance Criteria 1. Skill accepts parameters: role (admin/coach/parent), URL, acceptance criteria text 2. Skill logs into Keycloak via Playwright (fill email/password form, handle redirect) 3. Skill captures `browser_snapshot` and validates text content against acceptance criteria 4. Skill captures `browser_take_screenshot` as visual evidence 5. Skill reports PASS/PARTIAL/FAIL verdict with screenshot path and matched/unmatched criteria 6. Works with existing Playwright MCP server (no new infrastructure) ### Test Expectations - Skill logs in as coach (Ken Seka) and validates coach dashboard has schedule content - Skill logs in as parent (Sandra Apaisa) and validates my-players has schedule content - Skill validates public schedule page shows 2 local team cards (not 4) - Screenshots saved to working directory for review ### Constraints - Uses existing Playwright MCP tools — no new MCP servers - Skill lives in `~/.claude/skills/validate-ui/SKILL.md` (standard skill convention) - Password hardcoded as `Westside2026!` for all test accounts - Must handle Keycloak redirect flow (app → Keycloak → app callback) - No test data creation — uses existing seeded data and known test accounts ### Checklist - [ ] Create `~/.claude/skills/validate-ui/SKILL.md` - [ ] Implement Keycloak login flow (navigate → fill → click → wait for redirect) - [ ] Implement snapshot capture + text validation against AC - [ ] Implement screenshot capture for evidence - [ ] Implement verdict template (PASS/PARTIAL/FAIL + evidence) - [ ] Test: public schedule page (no auth) - [ ] Test: coach login + dashboard schedule - [ ] Test: parent login + my-players schedule - [ ] Test: admin login + admin schedule ### Related - Playwright MCP in `~/.mcp.json` - westside-landing#216, #217, #218 (schedule PRs needing validation) - `/validate-ticket` skill (existing — `/validate-ui` is complementary, not a replacement) - `project-pal-e-agency` user stories — `story:validation-execute` added

forgejo_admin commented

2026-04-04 22:53:00 +00:00

Author

Contributor

Scope Review: NEEDS_REFINEMENT

Review note: review-820-2026-04-04
Template is complete and traceability is strong (story note verified), but 4 issues need resolution before dispatch:

[SCOPE] Missing architecture note arch-claude-custom — no backing note in pal-e-docs for the arch:claude-custom label
[BODY] Agent vs skill decision is TBD — must be resolved before dispatch, not during implementation. File target paths also don't match skill directory convention (skills/X/SKILL.md, not skills/X.md)
[BODY] Keycloak test account credential source unspecified — agent needs to know where passwords live
[BODY] AC3 validation mechanism unclear — specify browser_snapshot, browser_evaluate DOM queries, or screenshot inspection

## Scope Review: NEEDS_REFINEMENT Review note: `review-820-2026-04-04` Template is complete and traceability is strong (story note verified), but 4 issues need resolution before dispatch: - **[SCOPE]** Missing architecture note `arch-claude-custom` — no backing note in pal-e-docs for the `arch:claude-custom` label - **[BODY]** Agent vs skill decision is TBD — must be resolved before dispatch, not during implementation. File target paths also don't match skill directory convention (`skills/X/SKILL.md`, not `skills/X.md`) - **[BODY]** Keycloak test account credential source unspecified — agent needs to know where passwords live - **[BODY]** AC3 validation mechanism unclear — specify browser_snapshot, browser_evaluate DOM queries, or screenshot inspection

forgejo_admin commented

2026-04-04 22:53:54 +00:00

Author

Contributor

Scope refinement (2026-04-04)

Addressed review-820 findings:

arch-claude-custom note: Flagged as discovered scope, not blocking. Same pattern as arch-landing-site.
Agent vs skill decided: Skill. File target: ~/.claude/skills/validate-ui/SKILL.md. Follows existing convention.
Keycloak passwords: All 3 test accounts use Westside2026!. Documented in issue body with email/role mapping.
AC3 mechanism specified: browser_snapshot for text validation, browser_take_screenshot for visual evidence. Explicit in body.

### Scope refinement (2026-04-04) Addressed review-820 findings: 1. **arch-claude-custom note:** Flagged as discovered scope, not blocking. Same pattern as arch-landing-site. 2. **Agent vs skill decided:** Skill. File target: `~/.claude/skills/validate-ui/SKILL.md`. Follows existing convention. 3. **Keycloak passwords:** All 3 test accounts use `Westside2026!`. Documented in issue body with email/role mapping. 4. **AC3 mechanism specified:** `browser_snapshot` for text validation, `browser_take_screenshot` for visual evidence. Explicit in body.

forgejo_admin commented

2026-04-04 22:56:41 +00:00

Author

Contributor

Scope Review: READY

Review note: review-820-2026-04-04 (re-review, updated from NEEDS_REFINEMENT)

All 4 issues from first review are resolved:

Agent vs skill — decided: skill at skills/validate-ui/SKILL.md
Keycloak passwords — documented: Westside2026! for all 3 test accounts with email/role mapping
AC3 mechanism — specified: browser_snapshot for text validation, browser_take_screenshot for evidence
arch-claude-custom note — still missing, tracked as [SCOPE] discovered scope, not blocking

Template complete. Traceability verified (story note confirmed on project-pal-e-agency). File target follows convention. 6 AC all testable. Single file, single repo, under 5-minute rule. Ready for dispatch.

## Scope Review: READY Review note: `review-820-2026-04-04` (re-review, updated from NEEDS_REFINEMENT) All 4 issues from first review are resolved: - **Agent vs skill** — decided: skill at `skills/validate-ui/SKILL.md` - **Keycloak passwords** — documented: `Westside2026!` for all 3 test accounts with email/role mapping - **AC3 mechanism** — specified: `browser_snapshot` for text validation, `browser_take_screenshot` for evidence - **arch-claude-custom note** — still missing, tracked as `[SCOPE]` discovered scope, not blocking Template complete. Traceability verified (story note confirmed on project-pal-e-agency). File target follows convention. 6 AC all testable. Single file, single repo, under 5-minute rule. Ready for dispatch.