Add Hetzner edge node module for custom domain reverse proxy #420

Merged

ldraney merged 4 commits from hetzner-edge-node into main 2026-06-13 08:08:45 +00:00

Conversation 1 Commits 4 Files changed 15 +336 -46

ldraney commented 2026-06-13 07:29:28 +00:00

Owner

Copy link

Summary

• Add hetzner-edge Terraform module to provision a Hetzner VPS as a reverse proxy for custom domains (palinks.app, landscaping-assistant.app, westsidekingsandqueens.com)
• VPS runs Caddy (auto-TLS) + Tailscale (mesh to k3s) via cloud-init
• Wire hcloud provider, secrets registry, and Makefile pipeline

Closes #419

Changes

• terraform/modules/hetzner-edge/ — new module: server, SSH key, firewall, cloud-init
• terraform/versions.tf — add hetznercloud/hcloud provider ~> 1.49
• terraform/providers.tf — add hcloud provider block
• terraform/variables.tf — add hetzner_api_token, edge_ssh_public_key, edge_tailscale_auth_key
• terraform/main.tf — wire hetzner_edge module
• Makefile — add 3 new vars to TF_SECRET_VARS
• salt/pillar/secrets_registry.sls — add metadata for 3 new secrets
• docs/hetzner-edge.md — architecture doc with mermaid diagram
• README.md — add module to table, add docs section

Test Plan

• tofu fmt -check passes
• tofu validate passes (after tofu init with new provider)
• tofu plan shows 3 resources to create (server, SSH key, firewall)
• No regressions in existing modules

Review Checklist

• Passed automated review-fix loop
• No secrets committed
• No unnecessary file changes
• Commit messages are descriptive
• Feature flag needed? No — infrastructure provisioning

Related Notes

• ldraney/pal-e-platform #419 — the Forgejo issue this PR implements
• project-palinks — primary consumer (palinks.app custom domain)

## Summary - Add `hetzner-edge` Terraform module to provision a Hetzner VPS as a reverse proxy for custom domains (palinks.app, landscaping-assistant.app, westsidekingsandqueens.com) - VPS runs Caddy (auto-TLS) + Tailscale (mesh to k3s) via cloud-init - Wire hcloud provider, secrets registry, and Makefile pipeline Closes #419 ## Changes - `terraform/modules/hetzner-edge/` — new module: server, SSH key, firewall, cloud-init - `terraform/versions.tf` — add hetznercloud/hcloud provider ~> 1.49 - `terraform/providers.tf` — add hcloud provider block - `terraform/variables.tf` — add hetzner_api_token, edge_ssh_public_key, edge_tailscale_auth_key - `terraform/main.tf` — wire hetzner_edge module - `Makefile` — add 3 new vars to TF_SECRET_VARS - `salt/pillar/secrets_registry.sls` — add metadata for 3 new secrets - `docs/hetzner-edge.md` — architecture doc with mermaid diagram - `README.md` — add module to table, add docs section ## Test Plan - [ ] `tofu fmt -check` passes - [ ] `tofu validate` passes (after `tofu init` with new provider) - [ ] `tofu plan` shows 3 resources to create (server, SSH key, firewall) - [ ] No regressions in existing modules ## Review Checklist - [ ] Passed automated review-fix loop - [ ] No secrets committed - [ ] No unnecessary file changes - [ ] Commit messages are descriptive - [ ] Feature flag needed? No — infrastructure provisioning ## Related Notes - `ldraney/pal-e-platform #419` — the Forgejo issue this PR implements - `project-palinks` — primary consumer (palinks.app custom domain)

ldraney added 1 commit 2026-06-13 07:29:28 +00:00

Add Hetzner edge node module for custom domain reverse proxy (#419) ... 71f4330a13

Terraform module provisions a Hetzner VPS (CAX11 ARM) with Caddy and
Tailscale via cloud-init. Serves as reverse proxy for palinks.app,
landscaping-assistant.app, and westsidekingsandqueens.com.

- hcloud provider added to versions.tf and providers.tf
- hetzner-edge module: server, SSH key, firewall (22/80/443)
- Three new secrets in Salt pillar registry
- Makefile updated for tofu-secrets pipeline
- docs/hetzner-edge.md with architecture diagram
- README updated with module table and docs link

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

ldraney commented 2026-06-13 07:31:43 +00:00

Author

Owner

Copy link

PR #420 Review

DOMAIN REVIEW

Tech stack: Terraform (hcloud provider), cloud-init YAML, Salt pillar (secrets registry), Makefile, Markdown docs.

Terraform module structure: The new hetzner-edge module follows the established pattern of main.tf + variables.tf + outputs.tf. However, it deviates in several places from existing module conventions observed in keycloak, monitoring, networking, etc.

---

BLOCKERS

1. Tailscale auth key persisted in Hetzner API via cloud-init user_data (Security)
terraform/modules/hetzner-edge/cloud-init.yaml line 11:

tailscale up --authkey=${tailscale_auth_key} --hostname=edge-proxy

This auth key is interpolated into user_data, which Hetzner stores and exposes via hcloud server describe and the Hetzner Cloud Console. Anyone with Hetzner API or console access can retrieve the plaintext auth key after provisioning.

Mitigation options (pick one):

• Use an ephemeral auth key with short TTL (the secrets_registry notes "Reusable, ephemeral" -- good, but confirm the TTL is short enough that it expires before the next tofu plan run would reveal it)
• Provision the auth key via Salt post-boot instead of cloud-init
• Use hcloud_server lifecycle ignore_changes = [user_data] and rotate the key immediately after first boot

2. domains variable declared but never used
terraform/modules/hetzner-edge/variables.tf lines 24-32 declare a domains variable with a map of domain-to-upstream mappings. This variable is:

• Not passed in terraform/main.tf module block (only ssh_public_key and tailscale_auth_key are passed)
• Not referenced in cloud-init.yaml
• Not referenced anywhere in the module's main.tf

Dead code in Terraform modules is a BLOCKER because tofu validate will warn, and it misleads future readers into thinking domain routing is configured when it is not. Either wire it into the Caddyfile template (via cloud-init) or remove it and defer to the follow-up SaltStack work documented in docs/hetzner-edge.md.

3. secrets.auto.tfvars.example not updated
Existing convention: every new sensitive variable gets a CHANGEME placeholder in secrets.auto.tfvars.example. This PR adds 3 new root-level variables (hetzner_api_token, edge_ssh_public_key, edge_tailscale_auth_key) but does not update the example file. This breaks the onboarding workflow -- a fresh clone won't know these variables exist without reading the diff.

---

NITS

4. SSH port 22 open to 0.0.0.0/0 and ::/0
terraform/modules/hetzner-edge/main.tf lines 8-12: SSH is open to the entire internet. Since this VPS joins the Tailscale mesh, SSH access could be restricted to the Tailscale CGNAT range (100.64.0.0/10) or to known IPs. Not a blocker since the VPS uses key-based auth, but it widens the attack surface unnecessarily.

5. No versions.tf in the module directory
Existing modules don't all have per-module versions.tf files, but the root versions.tf pins hcloud ~> 1.49. This is fine for now. Nit: consider adding a module-level required_providers block if the module is ever extracted for reuse.

6. Root variable descriptions could be more detailed
Existing conventions include generation instructions and consumer context in descriptions. For example:

• hetzner_api_token: could note "Hetzner Cloud Console > project > API Tokens > Read/Write"
• edge_tailscale_auth_key: could note "Generate ephemeral, reusable key from Tailscale admin > Settings > Keys"

The secrets_registry entries have this detail, which is good, but the Terraform variable descriptions are the first thing operators see during tofu plan.

7. edge_ssh_public_key naming convention
Other SSH-related variables in the codebase don't exist yet (this is the first), but the edge_ prefix is module-specific while other variables use the service name as prefix (e.g., harbor_admin_password, forgejo_admin_password). Consider hetzner_edge_ssh_public_key for consistency with the module name.

8. Cloud-init uses curl | sh pattern
Line 10: curl -fsSL https://tailscale.com/install.sh | sh -- standard for Tailscale but inherently trusting the remote script. Pinning to a specific Tailscale version or using apt repo + version pin would be more reproducible.

9. hetzner_api_token rotation_days: 0
The secrets_registry entry sets rotation_days: 0 (no rotation). While Hetzner API tokens don't expire, a rotation policy (e.g., 365 days) is good hygiene for tokens with read/write access to infrastructure.

---

SOP COMPLIANCE

• PR body has: Summary, Changes, Test Plan, Related -- all present and well-structured
• No secrets committed -- all secrets flow through Salt GPG pillar
• Commit messages are descriptive (single commit, clear title)
• No .env files or credentials in the diff
• Scope is focused on the stated objective
• secrets.auto.tfvars.example not updated (convention violation -- listed as BLOCKER #3)

---

PROCESS OBSERVATIONS

• Deployment risk: Low. This creates new infrastructure (3 Hetzner resources) with no impact on existing k3s/Terraform resources. tofu plan should show only additions.
• Rollback: Clean -- tofu destroy -target=module.hetzner_edge removes everything.
• Follow-up tracking: The docs/hetzner-edge.md Follow-Up Work section lists 4 items (Salt states, DNS automation, monitoring, failover). These should become Forgejo issues if they aren't already.
• Documentation: The architecture doc with mermaid diagram is thorough and a good addition.

---

VERDICT: NOT APPROVED

Three blockers must be resolved:

1. Cloud-init user_data exposes the Tailscale auth key in Hetzner API -- add mitigation
2. domains variable is dead code -- remove or wire it
3. secrets.auto.tfvars.example must be updated with the 3 new variables

## PR #420 Review ### DOMAIN REVIEW **Tech stack**: Terraform (hcloud provider), cloud-init YAML, Salt pillar (secrets registry), Makefile, Markdown docs. **Terraform module structure**: The new `hetzner-edge` module follows the established pattern of `main.tf` + `variables.tf` + `outputs.tf`. However, it deviates in several places from existing module conventions observed in `keycloak`, `monitoring`, `networking`, etc. --- ### BLOCKERS **1. Tailscale auth key persisted in Hetzner API via cloud-init user_data (Security)** `terraform/modules/hetzner-edge/cloud-init.yaml` line 11: ``` tailscale up --authkey=${tailscale_auth_key} --hostname=edge-proxy ``` This auth key is interpolated into `user_data`, which Hetzner stores and exposes via `hcloud server describe` and the Hetzner Cloud Console. Anyone with Hetzner API or console access can retrieve the plaintext auth key after provisioning. **Mitigation options** (pick one): - Use an ephemeral auth key with short TTL (the secrets_registry notes "Reusable, ephemeral" -- good, but confirm the TTL is short enough that it expires before the next `tofu plan` run would reveal it) - Provision the auth key via Salt post-boot instead of cloud-init - Use `hcloud_server` lifecycle `ignore_changes = [user_data]` and rotate the key immediately after first boot **2. `domains` variable declared but never used** `terraform/modules/hetzner-edge/variables.tf` lines 24-32 declare a `domains` variable with a map of domain-to-upstream mappings. This variable is: - Not passed in `terraform/main.tf` module block (only `ssh_public_key` and `tailscale_auth_key` are passed) - Not referenced in `cloud-init.yaml` - Not referenced anywhere in the module's `main.tf` Dead code in Terraform modules is a BLOCKER because `tofu validate` will warn, and it misleads future readers into thinking domain routing is configured when it is not. Either wire it into the Caddyfile template (via cloud-init) or remove it and defer to the follow-up SaltStack work documented in `docs/hetzner-edge.md`. **3. `secrets.auto.tfvars.example` not updated** Existing convention: every new sensitive variable gets a `CHANGEME` placeholder in `secrets.auto.tfvars.example`. This PR adds 3 new root-level variables (`hetzner_api_token`, `edge_ssh_public_key`, `edge_tailscale_auth_key`) but does not update the example file. This breaks the onboarding workflow -- a fresh clone won't know these variables exist without reading the diff. --- ### NITS **4. SSH port 22 open to `0.0.0.0/0` and `::/0`** `terraform/modules/hetzner-edge/main.tf` lines 8-12: SSH is open to the entire internet. Since this VPS joins the Tailscale mesh, SSH access could be restricted to the Tailscale CGNAT range (`100.64.0.0/10`) or to known IPs. Not a blocker since the VPS uses key-based auth, but it widens the attack surface unnecessarily. **5. No `versions.tf` in the module directory** Existing modules don't all have per-module `versions.tf` files, but the root `versions.tf` pins `hcloud ~> 1.49`. This is fine for now. Nit: consider adding a module-level `required_providers` block if the module is ever extracted for reuse. **6. Root variable descriptions could be more detailed** Existing conventions include generation instructions and consumer context in descriptions. For example: - `hetzner_api_token`: could note "Hetzner Cloud Console > project > API Tokens > Read/Write" - `edge_tailscale_auth_key`: could note "Generate ephemeral, reusable key from Tailscale admin > Settings > Keys" The secrets_registry entries have this detail, which is good, but the Terraform variable descriptions are the first thing operators see during `tofu plan`. **7. `edge_ssh_public_key` naming convention** Other SSH-related variables in the codebase don't exist yet (this is the first), but the `edge_` prefix is module-specific while other variables use the service name as prefix (e.g., `harbor_admin_password`, `forgejo_admin_password`). Consider `hetzner_edge_ssh_public_key` for consistency with the module name. **8. Cloud-init uses `curl | sh` pattern** Line 10: `curl -fsSL https://tailscale.com/install.sh | sh` -- standard for Tailscale but inherently trusting the remote script. Pinning to a specific Tailscale version or using apt repo + version pin would be more reproducible. **9. `hetzner_api_token` rotation_days: 0** The secrets_registry entry sets `rotation_days: 0` (no rotation). While Hetzner API tokens don't expire, a rotation policy (e.g., 365 days) is good hygiene for tokens with read/write access to infrastructure. --- ### SOP COMPLIANCE - [x] PR body has: Summary, Changes, Test Plan, Related -- all present and well-structured - [x] No secrets committed -- all secrets flow through Salt GPG pillar - [x] Commit messages are descriptive (single commit, clear title) - [x] No `.env` files or credentials in the diff - [x] Scope is focused on the stated objective - [ ] `secrets.auto.tfvars.example` not updated (convention violation -- listed as BLOCKER #3) --- ### PROCESS OBSERVATIONS - **Deployment risk**: Low. This creates new infrastructure (3 Hetzner resources) with no impact on existing k3s/Terraform resources. `tofu plan` should show only additions. - **Rollback**: Clean -- `tofu destroy -target=module.hetzner_edge` removes everything. - **Follow-up tracking**: The `docs/hetzner-edge.md` Follow-Up Work section lists 4 items (Salt states, DNS automation, monitoring, failover). These should become Forgejo issues if they aren't already. - **Documentation**: The architecture doc with mermaid diagram is thorough and a good addition. --- ### VERDICT: NOT APPROVED Three blockers must be resolved: 1. Cloud-init user_data exposes the Tailscale auth key in Hetzner API -- add mitigation 2. `domains` variable is dead code -- remove or wire it 3. `secrets.auto.tfvars.example` must be updated with the 3 new variables

ldraney added 1 commit 2026-06-13 07:32:48 +00:00

Fix review blockers: dead code, user_data exposure, example file ... 8ae5a0208d

- Remove unused `domains` variable (Caddy config is follow-up work)
- Add lifecycle ignore_changes on user_data to prevent re-exposing
  Tailscale auth key on subsequent applies
- Document auth key must be ephemeral + single-use
- Add Hetzner vars to secrets.auto.tfvars.example

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

ldraney added 1 commit 2026-06-13 07:37:47 +00:00

Pin hcloud provider source in hetzner-edge module ... 7b1aed50dd

Without explicit required_providers in the module, tofu resolves to
hashicorp/hcloud (legacy namespace) instead of hetznercloud/hcloud.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

ldraney added 1 commit 2026-06-13 07:41:37 +00:00

Generate Tailscale auth key via provider instead of manual secret ... 0887b6a27a

Use tailscale_tailnet_key resource to auto-generate an ephemeral,
single-use auth key from the existing OAuth client credentials.
Eliminates edge_tailscale_auth_key from variables, Salt pillar,
Makefile, and example file.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

ldraney merged commit 1010ee1960 into main 2026-06-13 08:08:45 +00:00

ldraney deleted branch hetzner-edge-node 2026-06-13 08:08:45 +00:00

ldraney referenced this pull request from a commit 2026-06-13 08:08:46 +00:00

Add Hetzner edge node module for custom domain reverse proxy (#420)

ldraney referenced this pull request from a commit 2026-06-13 08:25:02 +00:00

Add Hetzner edge node secrets to Salt GPG pillar

ldraney referenced this pull request 2026-06-13 08:25:28 +00:00

Add Hetzner edge node secrets to Salt GPG pillar #421

ldraney referenced this pull request 2026-06-13 12:43:32 +00:00

Add tag:edge to Tailscale ACL, rotate OAuth client, fix edge server type #424

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

domain:backend

  

domain:devops

  

domain:frontend

  

status:approved

QA passed, awaiting merge approval

  

status:in-progress

Dev agent is actively working

  

status:needs-fix

QA found issues, back to dev

  

status:qa

PR submitted, awaiting QA review

  

type:bug

Bug fix

  

type:devops

Infrastructure/CI/config work

  

type:feature

New functionality

No labels

domain:backend

domain:devops

domain:frontend

status:approved

status:in-progress

status:needs-fix

status:qa

type:bug

type:devops

type:feature

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

ldraney/pal-e-platform!420

No description provided.