Incident: 3 wrong contract emails sent to Sandra Apaisa (2026-04-10 blast) #445

New issue

Open

opened 2026-04-11 20:18:24 +00:00 by forgejo_admin · 0 comments

forgejo_admin commented

2026-04-11 20:18:24 +00:00

Contributor

Type

Bug

Lineage

Incident — 2026-04-10 23:50 UTC during the Marcus 2026-04-10 contract batch blast.

Repo

forgejo_admin/basketball-api

What Broke

During the Marcus-approved 15-player contract blast, 18 emails were sent instead of 15. The 3 extras went to apaisasandra@gmail.com at the OLD rates (Aleiyah $59, Analeigh $58, Ayvah $58) — NOT the new $50/girl family rate Marcus and Lucas had agreed on for the Apaisa sisters.

All 3 Apaisa contract_tokens were immediately NULLed after the mistake was detected, so if Sandra clicks any of the 3 links, she gets a 404. The emails are in her inbox and visible, but the contracts cannot be signed at the wrong rate.

email_log rows: 548, 549, 550 — all email_type=contract_offer, all to apaisasandra@gmail.com, all sent 2026-04-10 23:50:36-37 UTC, all with real gmail_message_ids (delivered).

Notable side bug: the 3 email_log rows have NULL player_id. The query_unsigned_contracts query returned the recipient data but the /admin/email/blast endpoint didn't store player_id back into email_log. Separate logging bug — file as follow-up if not already tracked.

Repro Steps

Call POST /admin/email/blast with query=unsigned_contracts, no test_email filter
Observe: the endpoint sends to every row in players where contract_status='offered' AND contract_token IS NOT NULL, scoped by tenant — with NO way to exclude specific players without NULLing their tokens first
If multiple sibling players share the same parent (like the 3 Apaisa sisters on apaisasandra@gmail.com), the parent gets N emails — one per child

Expected Behavior

The blast endpoint should have either:

(a) A player_ids: list[int] optional param on query_unsigned_contracts so callers can target specific players, OR
(b) A exclude_player_ids param for the inverse, OR
(c) A parent_ids param for grouping

None of these exist today. The workarounds available tonight were:

NULL the tokens of players we wanted to exclude (what Ava SHOULD have done preemptively for the Apaisas and did not)
Use test_email=X which filters to one recipient (used for single-player sends but not for "everyone except these")

Environment

Cluster/namespace: prod (basketball-api)
Service version/commit: basketball-api main after PR #426 merge (commit containing query_unsigned_contracts at src/basketball_api/services/email_queries.py:31)
Affected recipient: apaisasandra@gmail.com (Sandra Apaisa, parent of 3 Queens girls)
Related alerts: none (silent success from the endpoint's point of view — it sent 18 emails without errors)

Root Cause

Two compounding failures by the operator (Ava):

Verification query ≠ endpoint query. Lucas asked "is the blast pool 15 players?" Ava ran SELECT ... FROM players WHERE contract_status = 'offered' AND contract_token IS NOT NULL AND name NOT ILIKE '%Apaisa%' and returned count 15. But the actual query_unsigned_contracts endpoint query has NO name filter. Ava verified against an invented query instead of the one that would actually run.
Missed Apaisa token NULL step. Earlier in the session, Ava proposed NULLing Analeigh and Ayvah's contract_tokens as part of the Apaisa consolidation plan. Lucas asked "wait why?" and Ava moved on without executing it. The tokens stayed live. When the blast fired, all 3 Apaisa rows matched the endpoint query.

Either failure alone would have been caught by the other — a correct verification would have shown 18 rows in the pool, OR the NULL tokens would have dropped the 3 extras.

Impact

Sandra received 3 unexpected emails during evening hours
Each email shows $58 or $59/month in the contract page header — the OLD rates, not the $50 family rate
All 3 links are now dead (return 404) because the tokens were NULLed immediately after the incident
Potential damage: Sandra may be confused, annoyed, or concerned about Westside's competence
Recovery in progress: Ava drafted a corrected Queens-branded apology email via _brand_wrapper() (the right architecture for Queens branding), sent test to Lucas, then sent REVIEW copy to Marcus in his Gmail + GroupMe heads-up. Pending Marcus's thumbs-up before sending the real correction to Sandra.

Acceptance Criteria

Sandra receives the corrected Queens-branded email with the 3 new $50/girl contracts (Aleiyah id 111 token fjOXPKtm..., Analeigh id 186 token WqqJGPbu..., Ayvah id 112 token QLHbMP5K...)
Sandra's response captured (reply or Marcus phone call follow-up)
Incident postmortem documented (this issue serves as the record)
Follow-up tickets filed for: (a) player_ids param on blast endpoint, (b) NULL player_id in email_log bug, (c) unsigned_contracts verification SOP
feedback_blast_pool_verification.md memory file saved (lesson: run the EXACT endpoint query before any blast)

westside-basketball — project this affects
Related: basketball-api#424 (Marcus 2026-04-10 batch execution — the umbrella ticket this incident happened inside of)
Related: basketball-api#425 (contract-offer endpoint that minted the correct new Apaisa contracts at $50)
Related: feedback_email_blast_nuclear_gate.md (prior blast gate discipline, insufficient against this failure mode)
Related: feedback_blast_gate_failure.md (prior incident, same class)

### Type Bug ### Lineage Incident — 2026-04-10 23:50 UTC during the Marcus 2026-04-10 contract batch blast. ### Repo `forgejo_admin/basketball-api` ### What Broke During the Marcus-approved 15-player contract blast, **18 emails were sent instead of 15**. The 3 extras went to `apaisasandra@gmail.com` at the OLD rates (Aleiyah $59, Analeigh $58, Ayvah $58) — NOT the new $50/girl family rate Marcus and Lucas had agreed on for the Apaisa sisters. All 3 Apaisa contract_tokens were immediately NULLed after the mistake was detected, so if Sandra clicks any of the 3 links, she gets a 404. The emails are in her inbox and visible, but the contracts cannot be signed at the wrong rate. **email_log rows**: 548, 549, 550 — all `email_type=contract_offer`, all to `apaisasandra@gmail.com`, all sent 2026-04-10 23:50:36-37 UTC, all with real gmail_message_ids (delivered). **Notable side bug**: the 3 email_log rows have NULL player_id. The `query_unsigned_contracts` query returned the recipient data but the `/admin/email/blast` endpoint didn't store player_id back into email_log. Separate logging bug — file as follow-up if not already tracked. ### Repro Steps 1. Call `POST /admin/email/blast` with `query=unsigned_contracts`, no `test_email` filter 2. Observe: the endpoint sends to every row in `players` where `contract_status='offered' AND contract_token IS NOT NULL`, scoped by tenant — with NO way to exclude specific players without NULLing their tokens first 3. If multiple sibling players share the same parent (like the 3 Apaisa sisters on apaisasandra@gmail.com), the parent gets N emails — one per child ### Expected Behavior The blast endpoint should have either: - (a) A `player_ids: list[int]` optional param on `query_unsigned_contracts` so callers can target specific players, OR - (b) A `exclude_player_ids` param for the inverse, OR - (c) A `parent_ids` param for grouping None of these exist today. The workarounds available tonight were: - NULL the tokens of players we wanted to exclude (what Ava SHOULD have done preemptively for the Apaisas and did not) - Use `test_email=X` which filters to one recipient (used for single-player sends but not for "everyone except these") ### Environment - Cluster/namespace: prod (basketball-api) - Service version/commit: basketball-api main after PR #426 merge (commit containing `query_unsigned_contracts` at src/basketball_api/services/email_queries.py:31) - Affected recipient: `apaisasandra@gmail.com` (Sandra Apaisa, parent of 3 Queens girls) - Related alerts: none (silent success from the endpoint's point of view — it sent 18 emails without errors) ### Root Cause Two compounding failures by the operator (Ava): 1. **Verification query ≠ endpoint query**. Lucas asked "is the blast pool 15 players?" Ava ran `SELECT ... FROM players WHERE contract_status = 'offered' AND contract_token IS NOT NULL AND name NOT ILIKE '%Apaisa%'` and returned count 15. But the actual `query_unsigned_contracts` endpoint query has NO name filter. Ava verified against an invented query instead of the one that would actually run. 2. **Missed Apaisa token NULL step**. Earlier in the session, Ava proposed NULLing Analeigh and Ayvah's contract_tokens as part of the Apaisa consolidation plan. Lucas asked "wait why?" and Ava moved on without executing it. The tokens stayed live. When the blast fired, all 3 Apaisa rows matched the endpoint query. Either failure alone would have been caught by the other — a correct verification would have shown 18 rows in the pool, OR the NULL tokens would have dropped the 3 extras. ### Impact - **Sandra received 3 unexpected emails** during evening hours - Each email shows `$58` or `$59/month` in the contract page header — the OLD rates, not the $50 family rate - All 3 links are now dead (return 404) because the tokens were NULLed immediately after the incident - **Potential damage**: Sandra may be confused, annoyed, or concerned about Westside's competence - **Recovery in progress**: Ava drafted a corrected Queens-branded apology email via `_brand_wrapper()` (the right architecture for Queens branding), sent test to Lucas, then sent REVIEW copy to Marcus in his Gmail + GroupMe heads-up. Pending Marcus's thumbs-up before sending the real correction to Sandra. ### Acceptance Criteria - [ ] Sandra receives the corrected Queens-branded email with the 3 new $50/girl contracts (Aleiyah id 111 token `fjOXPKtm...`, Analeigh id 186 token `WqqJGPbu...`, Ayvah id 112 token `QLHbMP5K...`) - [ ] Sandra's response captured (reply or Marcus phone call follow-up) - [ ] Incident postmortem documented (this issue serves as the record) - [ ] Follow-up tickets filed for: (a) `player_ids` param on blast endpoint, (b) NULL player_id in email_log bug, (c) `unsigned_contracts` verification SOP - [ ] `feedback_blast_pool_verification.md` memory file saved (lesson: run the EXACT endpoint query before any blast) ### Related - `westside-basketball` — project this affects - Related: basketball-api#424 (Marcus 2026-04-10 batch execution — the umbrella ticket this incident happened inside of) - Related: basketball-api#425 (contract-offer endpoint that minted the correct new Apaisa contracts at $50) - Related: `feedback_email_blast_nuclear_gate.md` (prior blast gate discipline, insufficient against this failure mode) - Related: `feedback_blast_gate_failure.md` (prior incident, same class)