forgejo_admin/basketball-api
1
0
Fork 1
Code Issues 88 Pull requests 9 Projects Releases Packages Wiki Activity Actions

Observability: Grafana dashboard + structured logging + SPA health probes #99

New issue
Closed
opened 2026-03-18 15:58:27 +00:00 by forgejo_admin · 1 comment
forgejo_admin commented 2026-03-18 15:58:27 +00:00
Owner
Copy link

Lineage

plan-wkq → Phase 15 (Production Port) — discovered scope from SPA validation

Repo

forgejo_admin/basketball-api + forgejo_admin/pal-e-platform (for Grafana/blackbox config)

User Story

As an admin
I want to see API health, auth failures, and error rates in Grafana
So that I can debug issues like "Could not load dashboard data" without SSH-ing into pods

Context

During Phase 15 SPA validation, Lucas hit "Could not load dashboard data" on phone but Playwright showed the API working fine. Debugging required manual kubectl logs, curl with tokens, and guesswork. The observability stack exists (Prometheus, Loki, Grafana, blackbox) but there's no app-level dashboard for basketball-api or the westside SPA.

Basketball-api has a ServiceMonitor and /metrics endpoint, but no Grafana dashboard consuming the data. Loki collects pod logs but they're unstructured (plain uvicorn access logs). No SPA-side error tracking exists.

File Targets

basketball-api:

  • src/basketball_api/main.py — add structured JSON logging middleware (request method, path, status, latency, user_id from token)
  • src/basketball_api/routes/health.py — add /health/ready endpoint that checks DB connectivity

pal-e-platform or pal-e-services:

  • Grafana dashboard JSON (ConfigMap or provisioned) — basketball-api request metrics
  • Blackbox exporter target for westside-dev.tail5b443a.ts.net and westsidekingsandqueens.tail5b443a.ts.net
  • PrometheusRule for auth failure rate alert

Acceptance Criteria

  • Grafana dashboard shows: request rate, latency p50/p95/p99, error rate (4xx/5xx), all by endpoint
  • Grafana dashboard shows: auth failure rate (401s) as a separate panel
  • Basketball-api logs are structured JSON (parseable by Loki)
  • Blackbox exporter probes both SPA hostnames (dev + prod)
  • At least one alert rule: basketball-api 5xx rate > 5% for 5 minutes

Test Expectations

  • Grafana dashboard loads with real data
  • Loki query {namespace="basketball-api"} | json | status >= 400 returns structured results
  • Blackbox targets show UP in Prometheus

Constraints

  • Use existing kube-prometheus-stack — no new monitoring tools
  • Dashboard should be provisioned via ConfigMap (not manually created in Grafana UI)
  • Structured logging must not break existing /metrics endpoint

Checklist

  • Structured logging middleware
  • Grafana dashboard provisioned
  • Blackbox probes for SPA
  • Alert rule for 5xx
  • PR opened
  • Tests pass

Related

  • plan-pal-e-platform — platform hardening (observability is a core pillar)
  • sop-incident-response — observability feeds incident detection
  • ServiceMonitor already exists in pal-e-deployments/overlays/basketball-api/
### Lineage `plan-wkq` → Phase 15 (Production Port) — discovered scope from SPA validation ### Repo `forgejo_admin/basketball-api` + `forgejo_admin/pal-e-platform` (for Grafana/blackbox config) ### User Story As an admin I want to see API health, auth failures, and error rates in Grafana So that I can debug issues like "Could not load dashboard data" without SSH-ing into pods ### Context During Phase 15 SPA validation, Lucas hit "Could not load dashboard data" on phone but Playwright showed the API working fine. Debugging required manual kubectl logs, curl with tokens, and guesswork. The observability stack exists (Prometheus, Loki, Grafana, blackbox) but there's no app-level dashboard for basketball-api or the westside SPA. Basketball-api has a ServiceMonitor and `/metrics` endpoint, but no Grafana dashboard consuming the data. Loki collects pod logs but they're unstructured (plain uvicorn access logs). No SPA-side error tracking exists. ### File Targets `basketball-api`: - `src/basketball_api/main.py` — add structured JSON logging middleware (request method, path, status, latency, user_id from token) - `src/basketball_api/routes/health.py` — add `/health/ready` endpoint that checks DB connectivity `pal-e-platform` or `pal-e-services`: - Grafana dashboard JSON (ConfigMap or provisioned) — basketball-api request metrics - Blackbox exporter target for `westside-dev.tail5b443a.ts.net` and `westsidekingsandqueens.tail5b443a.ts.net` - PrometheusRule for auth failure rate alert ### Acceptance Criteria - [ ] Grafana dashboard shows: request rate, latency p50/p95/p99, error rate (4xx/5xx), all by endpoint - [ ] Grafana dashboard shows: auth failure rate (401s) as a separate panel - [ ] Basketball-api logs are structured JSON (parseable by Loki) - [ ] Blackbox exporter probes both SPA hostnames (dev + prod) - [ ] At least one alert rule: basketball-api 5xx rate > 5% for 5 minutes ### Test Expectations - [ ] Grafana dashboard loads with real data - [ ] Loki query `{namespace="basketball-api"} | json | status >= 400` returns structured results - [ ] Blackbox targets show UP in Prometheus ### Constraints - Use existing kube-prometheus-stack — no new monitoring tools - Dashboard should be provisioned via ConfigMap (not manually created in Grafana UI) - Structured logging must not break existing `/metrics` endpoint ### Checklist - [ ] Structured logging middleware - [ ] Grafana dashboard provisioned - [ ] Blackbox probes for SPA - [ ] Alert rule for 5xx - [ ] PR opened - [ ] Tests pass ### Related - `plan-pal-e-platform` — platform hardening (observability is a core pillar) - `sop-incident-response` — observability feeds incident detection - ServiceMonitor already exists in `pal-e-deployments/overlays/basketball-api/`
forgejo_admin 2026-03-18 16:13:29 +00:00
forgejo_admin commented 2026-03-18 16:35:53 +00:00
Author
Owner
Copy link

Progress: Structured Logging (PR #100 merged)

Completed

  • logging_config.py — JSON structured logging middleware (method, path, status_code, latency_ms, client_ip, user_agent, user_id from JWT)
  • /health/ready endpoint — DB connectivity check, 200/503
  • 14 new tests (12 logging, 2 health), 347 total pass
  • QA: approved on second review (blocker: test coverage, fixed)

Remaining

  • Prometheus HTTP instrumentation (prometheus-fastapi-instrumentator) — /metrics only exports basketball_api_up currently
  • Grafana dashboard (needs instrumentation first)
  • Alert rules for 5xx rate
  • Blackbox probe for westside-dev (PR #106 on pal-e-platform, pending CI)
## Progress: Structured Logging (PR #100 merged) ### Completed - `logging_config.py` — JSON structured logging middleware (method, path, status_code, latency_ms, client_ip, user_agent, user_id from JWT) - `/health/ready` endpoint — DB connectivity check, 200/503 - 14 new tests (12 logging, 2 health), 347 total pass - QA: approved on second review (blocker: test coverage, fixed) ### Remaining - Prometheus HTTP instrumentation (`prometheus-fastapi-instrumentator`) — `/metrics` only exports `basketball_api_up` currently - Grafana dashboard (needs instrumentation first) - Alert rules for 5xx rate - Blackbox probe for westside-dev (PR #106 on pal-e-platform, pending CI)
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
510-grant-tournament-tables-to-ro-role
458-skip-proration-v2
458-skip-proration
466-cherry-pick-first-payment
458-merge-first-payment-blast
369-first-payment-email-blast
408-player-nickname
313-local-first-week-email
320-mjml-sponsor-outreach-template
270-contract-reminder-email
205-clean-test-data-from-production-db-playe
114-email-verification
147-feat-add-get-jersey-player-info-endpoint
124-feat-many-to-many-player-team-assignment
125-feat-add-new-players-manually-seydou-gou
126-feat-team-placement-congratulations-emai
121-admin-teams-spa-endpoints
70-add-graduating-class-to-roster-api-respo
59-fix-auth-test
62-phase-4l-sveltekit-dashboard-wkq-tailsca
11-phase-1a-fix-ci-pipeline-venv-in-test-st
No results found.
Labels
Clear labels   
domain:backend

   
domain:devops

   
domain:frontend

   
status:approved

QA passed, awaiting merge approval

  
status:in-progress

Dev agent is actively working

  
status:needs-fix

QA found issues, back to dev

  
status:qa

PR submitted, awaiting QA review

  
type:bug

Bug fix

  
type:devops

Infrastructure/CI/config work

  
type:feature

New functionality

No labels
domain:backend
domain:devops
domain:frontend
status:approved
status:in-progress
status:needs-fix
status:qa
type:bug
type:devops
type:feature
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
forgejo_admin/basketball-api#99
No description provided.