Add Telegram receiver as default Alertmanager route #40

New issue

Closed

opened 2026-03-14 14:07:31 +00:00 by forgejo_admin · 0 comments

forgejo_admin commented

2026-03-14 14:07:31 +00:00

Owner

Lineage

plan-pal-e-platform → Phase 3 (Alerting) → Phase 3a (Telegram Alerting)

Repo

forgejo_admin/pal-e-platform

User Story

As the platform operator
I want Alertmanager to push alerts to my phone via Telegram
So that I detect incidents immediately without checking a dashboard (MTTR)

Context

Phase 3 (PR #35) deployed PrometheusRules and an Alertmanager Tailscale funnel with a dormant Slack receiver. This issue adds Telegram as the primary notification channel.

The Telegram bot already exists and is verified:

Bot: @personal_assistant_ldraney_bot ("Personal Assistant")
Bot token: stored at ~/secrets/telegram/bot_token
Chat ID: stored at ~/secrets/telegram/chat_id (value: 5892513378)
Test message sent and received successfully

Alertmanager natively supports Telegram via telegram_configs — this is a first-class receiver, not a webhook hack.

Current Alertmanager config (from PR #35) in kube-prometheus-stack Helm values:

route.receiver: "default" (null receiver)
Conditional Slack route (only if slack_webhook_url is non-empty — currently unused)
group_by: ["alertname", "namespace"]
group_wait: 30s, group_interval: 5m, repeat_interval: 12h

File Targets

Files to modify:

terraform/main.tf — modify the alertmanager config section in kube-prometheus-stack Helm values: add telegram receiver with telegram_configs, change route.receiver from "default" to "telegram", keep default receiver as fallback
terraform/variables.tf — add telegram_bot_token (sensitive, no default) and telegram_chat_id (sensitive, no default) variables

Files NOT to touch:

terraform/dashboards/ — unrelated
.woodpecker.yaml — just merged, unrelated
salt/ — not relevant

Acceptance Criteria

telegram_bot_token variable added (type string, sensitive, no default)
telegram_chat_id variable added (type string, sensitive, no default)

Telegram receiver added to Alertmanager config in kube-prometheus-stack Helm values:

receivers:
  - name: telegram
    telegram_configs:
      - chat_id: <from variable>
        parse_mode: HTML
        send_resolved: true
  - name: default
    (existing null receiver)
  - name: slack
    (existing conditional receiver)

telegram_bot_token injected via set_sensitive (same pattern as Slack webhook URL — must NOT appear in yamlencode)
Default route changed: route.receiver: "telegram" (was "default")
Existing Slack conditional receiver and route remain unchanged
group_wait: 30s, group_interval: 5m, repeat_interval: 4h (changed from 12h — more appropriate for Telegram push notifications)
tofu validate passes
tofu fmt applied

Test Expectations

tofu plan shows changes to kube-prometheus-stack Helm values and new Alertmanager config
Post-apply: Alertmanager UI shows telegram as the default receiver
Post-apply: trigger a test alert, verify Telegram push notification arrives

Constraints

The telegram_bot_token MUST be injected via set_sensitive — it cannot appear in the yamlencode() block or it will be exposed in plan output. Follow the same dynamic "set_sensitive" pattern used for slack_webhook_url in PR #35.
The telegram_chat_id can safely go in yamlencode() since it's just a number (not a secret), but marking it sensitive in the variable definition keeps it out of plan output as a precaution
Keep the existing default null receiver — it's the fallback if Telegram config fails
Keep the existing conditional Slack receiver — it remains available but unused
send_resolved: true so you get notified when alerts clear (not just when they fire)

Checklist

PR opened with Closes #40 in body
tofu plan output included in PR description
tofu fmt and tofu validate pass
No unrelated changes

project-pal-e-platform — project
phase-observability-3a-telegram-alerting — phase note in pal-e-docs
PR #35 — parent (PrometheusRules + Alertmanager funnel + Slack)

### Lineage `plan-pal-e-platform` → Phase 3 (Alerting) → Phase 3a (Telegram Alerting) ### Repo `forgejo_admin/pal-e-platform` ### User Story As the platform operator I want Alertmanager to push alerts to my phone via Telegram So that I detect incidents immediately without checking a dashboard (MTTR) ### Context Phase 3 (PR #35) deployed PrometheusRules and an Alertmanager Tailscale funnel with a dormant Slack receiver. This issue adds Telegram as the **primary** notification channel. The Telegram bot already exists and is verified: - Bot: `@personal_assistant_ldraney_bot` ("Personal Assistant") - Bot token: stored at `~/secrets/telegram/bot_token` - Chat ID: stored at `~/secrets/telegram/chat_id` (value: `5892513378`) - Test message sent and received successfully Alertmanager natively supports Telegram via `telegram_configs` — this is a first-class receiver, not a webhook hack. Current Alertmanager config (from PR #35) in kube-prometheus-stack Helm values: - `route.receiver: "default"` (null receiver) - Conditional Slack route (only if `slack_webhook_url` is non-empty — currently unused) - `group_by: ["alertname", "namespace"]` - `group_wait: 30s`, `group_interval: 5m`, `repeat_interval: 12h` ### File Targets Files to modify: - `terraform/main.tf` — modify the `alertmanager` config section in kube-prometheus-stack Helm values: add `telegram` receiver with `telegram_configs`, change `route.receiver` from `"default"` to `"telegram"`, keep `default` receiver as fallback - `terraform/variables.tf` — add `telegram_bot_token` (sensitive, no default) and `telegram_chat_id` (sensitive, no default) variables Files NOT to touch: - `terraform/dashboards/` — unrelated - `.woodpecker.yaml` — just merged, unrelated - `salt/` — not relevant ### Acceptance Criteria - [ ] `telegram_bot_token` variable added (type string, sensitive, no default) - [ ] `telegram_chat_id` variable added (type string, sensitive, no default) - [ ] Telegram receiver added to Alertmanager config in kube-prometheus-stack Helm values: ``` receivers: - name: telegram telegram_configs: - chat_id: <from variable> parse_mode: HTML send_resolved: true - name: default (existing null receiver) - name: slack (existing conditional receiver) ``` - [ ] `telegram_bot_token` injected via `set_sensitive` (same pattern as Slack webhook URL — must NOT appear in yamlencode) - [ ] Default route changed: `route.receiver: "telegram"` (was `"default"`) - [ ] Existing Slack conditional receiver and route remain unchanged - [ ] `group_wait: 30s`, `group_interval: 5m`, `repeat_interval: 4h` (changed from 12h — more appropriate for Telegram push notifications) - [ ] `tofu validate` passes - [ ] `tofu fmt` applied ### Test Expectations - [ ] `tofu plan` shows changes to kube-prometheus-stack Helm values and new Alertmanager config - [ ] Post-apply: Alertmanager UI shows `telegram` as the default receiver - [ ] Post-apply: trigger a test alert, verify Telegram push notification arrives ### Constraints - The `telegram_bot_token` MUST be injected via `set_sensitive` — it cannot appear in the `yamlencode()` block or it will be exposed in plan output. Follow the same `dynamic "set_sensitive"` pattern used for `slack_webhook_url` in PR #35. - The `telegram_chat_id` can safely go in `yamlencode()` since it's just a number (not a secret), but marking it sensitive in the variable definition keeps it out of plan output as a precaution - Keep the existing `default` null receiver — it's the fallback if Telegram config fails - Keep the existing conditional Slack receiver — it remains available but unused - `send_resolved: true` so you get notified when alerts clear (not just when they fire) ### Checklist - [ ] PR opened with `Closes #40` in body - [ ] `tofu plan` output included in PR description - [ ] `tofu fmt` and `tofu validate` pass - [ ] No unrelated changes ### Related - `project-pal-e-platform` — project - `phase-observability-3a-telegram-alerting` — phase note in pal-e-docs - PR #35 — parent (PrometheusRules + Alertmanager funnel + Slack)