Spike: Token metrics -- correlate agent token usage with DORA and kanban throughput #227

New issue

Open

opened 2026-03-28 02:07:07 +00:00 by forgejo_admin · 0 comments

forgejo_admin commented

2026-03-28 02:07:07 +00:00

Owner

Type

Spike

Lineage

Standalone -- migrated from todo-token-metrics-dora-correlation in pal-e-docs. Original framing referenced sprints and story points, which are now deprecated (continuous kanban, no points). Core idea remains valid.

Repo

forgejo_admin/pal-e-api (multiple repos if implementation proceeds)

Question

What is the minimum viable instrumentation to capture per-issue token usage and correlate it with DORA metrics and kanban throughput, and where should this data live?

What to Explore

How to capture token usage per agent spawn with issue number linkage:
- Claude Code session metadata (total_tokens, tool_uses, duration_ms already in output)
- Post-session hook parsing agent output files
- Push data to pal-e-docs as board item metadata
Where to store the data: pal-e-docs note/block, Prometheus metric, or both
What correlations are meaningful now that we use kanban (not sprints):
- Token cost per successful deployment (DF correlation)
- Token efficiency curve per repo as it matures
- Which SOPs/hooks reduce token waste
Empirical baseline data from 2026-03-10 basketball-api session:
- Full issue cycle: ~150-250K tokens, ~15-20 min
- Dev single endpoint: 71K tokens
- QA full PR review: 38-73K tokens
Review Grafana dashboard feasibility for token metrics alongside DORA

Success Criteria

Instrumentation approach decided (hook-based vs API-based vs manual)
Storage location decided (pal-e-docs metadata, Prometheus, or both)
Follow-up Feature issue(s) created for implementation
Or: "not yet" conclusion with clear prerequisites documented

Time-box

1 session (2 hours max). If time-box expires, document findings and escalate.

pal-e-docs -- project (data storage)
pal-e-platform -- project (Grafana dashboard)
dora-framework -- the metrics this would extend

### Type Spike ### Lineage Standalone -- migrated from `todo-token-metrics-dora-correlation` in pal-e-docs. Original framing referenced sprints and story points, which are now deprecated (continuous kanban, no points). Core idea remains valid. ### Repo `forgejo_admin/pal-e-api` (multiple repos if implementation proceeds) ### Question What is the minimum viable instrumentation to capture per-issue token usage and correlate it with DORA metrics and kanban throughput, and where should this data live? ### What to Explore - How to capture token usage per agent spawn with issue number linkage: - Claude Code session metadata (total_tokens, tool_uses, duration_ms already in output) - Post-session hook parsing agent output files - Push data to pal-e-docs as board item metadata - Where to store the data: pal-e-docs note/block, Prometheus metric, or both - What correlations are meaningful now that we use kanban (not sprints): - Token cost per successful deployment (DF correlation) - Token efficiency curve per repo as it matures - Which SOPs/hooks reduce token waste - Empirical baseline data from 2026-03-10 basketball-api session: - Full issue cycle: ~150-250K tokens, ~15-20 min - Dev single endpoint: 71K tokens - QA full PR review: 38-73K tokens - Review Grafana dashboard feasibility for token metrics alongside DORA ### Success Criteria - [ ] Instrumentation approach decided (hook-based vs API-based vs manual) - [ ] Storage location decided (pal-e-docs metadata, Prometheus, or both) - [ ] Follow-up Feature issue(s) created for implementation - [ ] Or: "not yet" conclusion with clear prerequisites documented ### Time-box 1 session (2 hours max). If time-box expires, document findings and escalate. ### Related - `pal-e-docs` -- project (data storage) - `pal-e-platform` -- project (Grafana dashboard) - `dora-framework` -- the metrics this would extend