forgejo_admin/pal-e-platform
1
0
Fork 1
Code Issues 35 Pull requests Projects Releases Packages Wiki Activity Actions

Add Telegram receiver to Alertmanager as default route #42

New issue
Closed
opened 2026-03-14 14:50:30 +00:00 by forgejo_admin · 0 comments
forgejo_admin commented 2026-03-14 14:50:30 +00:00
Owner
Copy link

Lineage

plan-pal-e-platform → Phase 3 (Alerting) → Phase 3a (Telegram Alerting)

Repo

forgejo_admin/pal-e-platform

User Story

As the platform operator
I want Alertmanager to push alerts to a Telegram group
So that the entire team gets notified on their phones instantly when something breaks (MTTR)

Context

Phase 3 (PR #35, merged) deployed PrometheusRules, Alertmanager routing with a dormant Slack receiver, and an Alertmanager Tailscale funnel. Alerts fire and are visible in the Alertmanager UI, but they don't push to anyone's phone yet.

Telegram is the chosen push channel because it has native Alertmanager support (telegram_configs), zero infrastructure overhead, and reliable mobile push notifications. A Telegram group "Pal-E Alerts" has been created with the bot @personal_assistant_ldraney_bot. The group chat ID is the alerting target — adding a new developer means adding them to the Telegram group. No infrastructure changes per developer.

Key technical facts:

  • PR #35 already deployed Alertmanager config with a default null receiver and conditional Slack receiver
  • The Alertmanager config is in the kube-prometheus-stack Helm values under alertmanager.config in terraform/main.tf
  • The route.receiver is currently "default" (null — alerts go nowhere except the UI)
  • Bot token and group chat ID are available as TF variables
  • Alertmanager natively supports telegram_configs — this is a first-class receiver type

File Targets

Files to modify:

  • terraform/main.tf — add telegram receiver with telegram_configs to the Alertmanager config in kube-prometheus-stack Helm values. Change route.receiver from "default" to "telegram". Add set_sensitive blocks for bot token and chat ID.
  • terraform/variables.tf — add telegram_bot_token (sensitive) and telegram_chat_id (sensitive) variables

Files NOT to touch:

  • .woodpecker.yaml — unrelated
  • terraform/dashboards/ — unrelated
  • salt/ — unrelated

Acceptance Criteria

  • telegram receiver added to Alertmanager config with telegram_configs:
    • bot_token from TF variable (injected via set_sensitive, NOT in yamlencode block)
    • chat_id from TF variable (injected via set_sensitive)
    • parse_mode: "HTML" for rich formatting
    • send_resolved: true (notify when alert clears)
  • route.receiver changed from "default" to "telegram"
  • "default" null receiver kept as fallback
  • Slack receiver remains available but dormant (no route points to it)
  • telegram_bot_token variable: type string, sensitive, no default
  • telegram_chat_id variable: type string, sensitive, no default
  • tofu validate passes
  • tofu fmt applied

Test Expectations

  • tofu plan shows changes to kube-prometheus-stack Helm release (Alertmanager config update)
  • Post-apply: Alertmanager UI shows telegram receiver as default route
  • Post-apply: trigger a test alert, Telegram group receives notification

Constraints

  • Bot token MUST use set_sensitive (like grafana.adminPassword) — never put it in the yamlencode block where it would appear in plan output
  • Chat ID should also use set_sensitive for consistency
  • The set_sensitive path for telegram_configs follows the Helm chart structure: alertmanager.config.receivers[N].telegram_configs[0].bot_token — verify the correct index N (it depends on where in the concat the telegram receiver is placed)
  • Keep the existing receiver ordering: default (index 0), telegram (add as new), slack conditional (if present)
  • Follow the same dynamic "set_sensitive" pattern used for the Slack webhook URL in PR #35

Checklist

  • PR opened with Closes #42 in body
  • tofu plan output included in PR description
  • tofu fmt and tofu validate pass
  • No unrelated changes

Related

  • project-pal-e-platform — project
  • phase-observability-3a-telegram-alerting — phase note in pal-e-docs
  • PR #35 — parent work (PrometheusRules + Alertmanager baseline)
### Lineage `plan-pal-e-platform` → Phase 3 (Alerting) → Phase 3a (Telegram Alerting) ### Repo `forgejo_admin/pal-e-platform` ### User Story As the platform operator I want Alertmanager to push alerts to a Telegram group So that the entire team gets notified on their phones instantly when something breaks (MTTR) ### Context Phase 3 (PR #35, merged) deployed PrometheusRules, Alertmanager routing with a dormant Slack receiver, and an Alertmanager Tailscale funnel. Alerts fire and are visible in the Alertmanager UI, but they don't push to anyone's phone yet. Telegram is the chosen push channel because it has native Alertmanager support (`telegram_configs`), zero infrastructure overhead, and reliable mobile push notifications. A Telegram group "Pal-E Alerts" has been created with the bot `@personal_assistant_ldraney_bot`. The group chat ID is the alerting target — adding a new developer means adding them to the Telegram group. No infrastructure changes per developer. Key technical facts: - PR #35 already deployed Alertmanager config with a `default` null receiver and conditional Slack receiver - The Alertmanager config is in the kube-prometheus-stack Helm values under `alertmanager.config` in `terraform/main.tf` - The `route.receiver` is currently `"default"` (null — alerts go nowhere except the UI) - Bot token and group chat ID are available as TF variables - Alertmanager natively supports `telegram_configs` — this is a first-class receiver type ### File Targets Files to modify: - `terraform/main.tf` — add `telegram` receiver with `telegram_configs` to the Alertmanager config in kube-prometheus-stack Helm values. Change `route.receiver` from `"default"` to `"telegram"`. Add `set_sensitive` blocks for bot token and chat ID. - `terraform/variables.tf` — add `telegram_bot_token` (sensitive) and `telegram_chat_id` (sensitive) variables Files NOT to touch: - `.woodpecker.yaml` — unrelated - `terraform/dashboards/` — unrelated - `salt/` — unrelated ### Acceptance Criteria - [ ] `telegram` receiver added to Alertmanager config with `telegram_configs`: - `bot_token` from TF variable (injected via `set_sensitive`, NOT in yamlencode block) - `chat_id` from TF variable (injected via `set_sensitive`) - `parse_mode: "HTML"` for rich formatting - `send_resolved: true` (notify when alert clears) - [ ] `route.receiver` changed from `"default"` to `"telegram"` - [ ] `"default"` null receiver kept as fallback - [ ] Slack receiver remains available but dormant (no route points to it) - [ ] `telegram_bot_token` variable: type string, sensitive, no default - [ ] `telegram_chat_id` variable: type string, sensitive, no default - [ ] `tofu validate` passes - [ ] `tofu fmt` applied ### Test Expectations - [ ] `tofu plan` shows changes to kube-prometheus-stack Helm release (Alertmanager config update) - [ ] Post-apply: Alertmanager UI shows `telegram` receiver as default route - [ ] Post-apply: trigger a test alert, Telegram group receives notification ### Constraints - Bot token MUST use `set_sensitive` (like `grafana.adminPassword`) — never put it in the `yamlencode` block where it would appear in plan output - Chat ID should also use `set_sensitive` for consistency - The `set_sensitive` path for telegram_configs follows the Helm chart structure: `alertmanager.config.receivers[N].telegram_configs[0].bot_token` — verify the correct index N (it depends on where in the `concat` the telegram receiver is placed) - Keep the existing receiver ordering: default (index 0), telegram (add as new), slack conditional (if present) - Follow the same `dynamic "set_sensitive"` pattern used for the Slack webhook URL in PR #35 ### Checklist - [ ] PR opened with `Closes #42` in body - [ ] `tofu plan` output included in PR description - [ ] `tofu fmt` and `tofu validate` pass - [ ] No unrelated changes ### Related - `project-pal-e-platform` — project - `phase-observability-3a-telegram-alerting` — phase note in pal-e-docs - PR #35 — parent work (PrometheusRules + Alertmanager baseline)
forgejo_admin 2026-03-14 14:55:55 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
188-cross-repo-worktree-isolation-for-parall
111-fix-keycloak-probe
86-fix-rotate-woodpecker-api-token-in-salt
71-feat-deploy-pyrra-slo-manager-with-servi
64-hotfix-woodpecker-oauth-login-broken-for
18-fix-grafana-duplicate-default-datasource-v2
18-fix-grafana-duplicate-default-datasource
13-fix-cnpg-all-parameters
No results found.
Labels
Clear labels   
domain:backend

   
domain:devops

   
domain:frontend

   
status:approved

QA passed, awaiting merge approval

  
status:in-progress

Dev agent is actively working

  
status:needs-fix

QA found issues, back to dev

  
status:qa

PR submitted, awaiting QA review

  
type:bug

Bug fix

  
type:devops

Infrastructure/CI/config work

  
type:feature

New functionality

No labels
domain:backend
domain:devops
domain:frontend
status:approved
status:in-progress
status:needs-fix
status:qa
type:bug
type:devops
type:feature
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
forgejo_admin/pal-e-platform#42
No description provided.