Real-Stripe integration tests leave orphan test-mode Price/Product objects #503

New issue

Open

opened 2026-04-18 06:08:23 +00:00 by forgejo_admin · 0 comments

forgejo_admin commented

2026-04-18 06:08:23 +00:00

Owner

Type

Bug

Lineage

Standalone — discovered 2026-04-18 during QA review of PR #499. QA noted the new tests/test_monthly_payment_link_real_stripe.py (and its older sibling tests/test_payment_link_real_stripe.py from #494) do not clean up the test-mode Stripe objects they create.

Repo

forgejo_admin/basketball-api

What Broke

Every CI run that executes a real-Stripe integration test leaves behind permanent artifacts in the test-mode Stripe account: a Product, a Price, a PaymentLink. They are not deactivated or archived after the assertion passes. Over weeks of CI runs the test-mode account accumulates thousands of orphans, which:

slows down the Stripe dashboard (test-mode Products list)
makes it hard to distinguish real test fixtures from CI debris
eventually could hit Stripe API list-pagination limits if we build any test-mode audit tooling on top

Repro Steps

git checkout main
Set BASKETBALL_STRIPE_TEST_API_KEY to the test-mode key
python -m pytest tests/test_monthly_payment_link_real_stripe.py tests/test_payment_link_real_stripe.py
Open https://dashboard.stripe.com/test/products — new entries from this run remain
Repeat; orphans accumulate

Expected Behavior

After each real-Stripe integration test passes (or fails), a teardown fixture archives the created Product, deactivates the Price, and deactivates the PaymentLink. Test-mode account stays clean.

Environment

stripe==14.4.1
Tests run in CI via Woodpecker pipeline test step (python:3.12-slim) with BASKETBALL_STRIPE_TEST_API_KEY repo secret
Tests also runnable locally with same env var

Blast Radius

Test-only. No prod impact. But the orphans cost us dashboard clarity and will grow without bound.

Scope / Fix Direction

Add a pytest fixture (likely tests/conftest.py or a per-file fixture) that tracks created Stripe ids and runs teardown in finally.
Teardown: stripe.PaymentLink.modify(id, active=False), stripe.Price.modify(id, active=False), stripe.Product.modify(id, active=False) (Products can't be hard-deleted if ever referenced; archiving via active=False is the supported path).
Apply to both existing real-Stripe integration tests.
One-off script to archive existing orphans (optional; can be a separate ticket if scope creeps).

Acceptance Criteria

Both real-Stripe integration tests archive Product/Price/PaymentLink on teardown
Teardown runs even on test failure (pytest fixture yield + finally pattern, not try/except in test body)
Added assertion in teardown: subsequent Retrieve returns active: false for the archived resources
CI still passes end-to-end

Constraints

Do NOT modify any non-test code.
Do NOT delete-vs-archive — Stripe Products with referenced Prices can't be deleted.
Do NOT couple the fixture to a specific helper — it should work for any Stripe id the test registers.

Checklist

PR opened
CI green
Existing orphans triaged (spawn cleanup ticket if blast radius is big)

forgejo_admin/basketball-api #494 — tournament real-Stripe test
forgejo_admin/basketball-api #498 — monthly real-Stripe test
QA review comment on PR #499 — test cleanup concern

### Type Bug ### Lineage Standalone — discovered 2026-04-18 during QA review of PR #499. QA noted the new `tests/test_monthly_payment_link_real_stripe.py` (and its older sibling `tests/test_payment_link_real_stripe.py` from #494) do not clean up the test-mode Stripe objects they create. ### Repo `forgejo_admin/basketball-api` ### What Broke Every CI run that executes a real-Stripe integration test leaves behind permanent artifacts in the test-mode Stripe account: a Product, a Price, a PaymentLink. They are not deactivated or archived after the assertion passes. Over weeks of CI runs the test-mode account accumulates thousands of orphans, which: - slows down the Stripe dashboard (test-mode Products list) - makes it hard to distinguish real test fixtures from CI debris - eventually could hit Stripe API list-pagination limits if we build any test-mode audit tooling on top ### Repro Steps 1. `git checkout main` 2. Set `BASKETBALL_STRIPE_TEST_API_KEY` to the test-mode key 3. `python -m pytest tests/test_monthly_payment_link_real_stripe.py tests/test_payment_link_real_stripe.py` 4. Open https://dashboard.stripe.com/test/products — new entries from this run remain 5. Repeat; orphans accumulate ### Expected Behavior After each real-Stripe integration test passes (or fails), a teardown fixture archives the created Product, deactivates the Price, and deactivates the PaymentLink. Test-mode account stays clean. ### Environment - `stripe==14.4.1` - Tests run in CI via Woodpecker pipeline test step (`python:3.12-slim`) with `BASKETBALL_STRIPE_TEST_API_KEY` repo secret - Tests also runnable locally with same env var ### Blast Radius Test-only. No prod impact. But the orphans cost us dashboard clarity and will grow without bound. ### Scope / Fix Direction 1. Add a pytest fixture (likely `tests/conftest.py` or a per-file fixture) that tracks created Stripe ids and runs teardown in `finally`. 2. Teardown: `stripe.PaymentLink.modify(id, active=False)`, `stripe.Price.modify(id, active=False)`, `stripe.Product.modify(id, active=False)` (Products can't be hard-deleted if ever referenced; archiving via active=False is the supported path). 3. Apply to both existing real-Stripe integration tests. 4. One-off script to archive existing orphans (optional; can be a separate ticket if scope creeps). ### Acceptance Criteria - [ ] Both real-Stripe integration tests archive Product/Price/PaymentLink on teardown - [ ] Teardown runs even on test failure (pytest fixture `yield` + `finally` pattern, not try/except in test body) - [ ] Added assertion in teardown: subsequent `Retrieve` returns `active: false` for the archived resources - [ ] CI still passes end-to-end ### Constraints - Do NOT modify any non-test code. - Do NOT delete-vs-archive — Stripe Products with referenced Prices can't be deleted. - Do NOT couple the fixture to a specific helper — it should work for any Stripe id the test registers. ### Checklist - [ ] PR opened - [ ] CI green - [ ] Existing orphans triaged (spawn cleanup ticket if blast radius is big) ### Related - `forgejo_admin/basketball-api #494` — tournament real-Stripe test - `forgejo_admin/basketball-api #498` — monthly real-Stripe test - QA review comment on PR #499 — test cleanup concern