pal-e-platform

Bootstrap repo that deploys the minimum viable self-hosting stack onto an existing k3s cluster. Once Forgejo is running with observability, all future infrastructure work moves to self-hosted repos. GitHub is the disaster recovery entry point.

Definition of done: tofu apply produces a working Forgejo instance with CI, container registry, and observability.

Bootstrap Dependency Chain

k3s (exists)
  → Tailscale operator (networking/ingress via funnels)
    → Prometheus + Grafana + Loki (observability)
      → Forgejo (git hosting)
        → Woodpecker CI (build pipelines)
      → Harbor (container registry)

Status: Complete

All milestones delivered. This repo is in maintenance mode. Active development happens in pal-e-services and individual app repos on Forgejo.

See #8 (Bootstrap platform epic) for the full history.

Tech Stack

Component	Tool	Purpose
IaC	OpenTofu	Declarative infrastructure management
Cluster	k3s	Lightweight Kubernetes (pre-existing)
Ingress + TLS	Tailscale funnels	Zero-config ingress and TLS via tailnet
Metrics	Prometheus + Grafana	Collection and dashboards
Logs	Loki	Lightweight log aggregation
Git hosting	Forgejo	Self-hosted git hosting and code review
CI	Woodpecker CI	Container-native CI, Forgejo-integrated
Registry	Harbor	CNCF container registry with vulnerability scanning

What Moves to Forgejo

These are explicitly not in scope for this repo. Once Forgejo is running, pal-e-services (hosted on Forgejo) handles:

• Service onboarding (image → registry → deploy → monitor)
• MinIO (S3-compatible object storage)
• ArgoCD (GitOps app deployments, if needed)
• Application deployments (openclaw import, pal-e-assets, etc.)
• Production workload management

Terraform Layout

terraform/
├── main.tf              # Bootstrap resources
├── providers.tf         # k3s, Helm providers
├── variables.tf
├── outputs.tf
├── k3s.tfvars           # Actual values (gitignored)
├── k3s.tfvars.example   # Placeholder template (committed)
└── versions.tf

Single root module. No sub-modules until complexity justifies them.

State Management

State is stored in-cluster using the kubernetes backend. Each repo gets its own secret in the tofu-state namespace:

backend "kubernetes" {
  secret_suffix = "pal-e-platform"   # creates secret: tfstate-pal-e-platform
  config_path   = "~/.kube/config"
  namespace     = "tofu-state"
}

Other repos (e.g. pal-e-services) follow the same convention with their own secret_suffix.

Secrets Architecture

Secrets flow through a GPG-encrypted Salt pillar pipeline. No plaintext secrets in git.

Salt pillar (GPG-encrypted .sls)
  → make tofu-secrets (sudo salt-call → decrypt → render)
    → terraform/secrets.auto.tfvars (gitignored, plaintext on disk)
      → tofu plan / tofu apply

Components

File	Purpose	Committed
salt/pillar/secrets/platform.sls	GPG-encrypted secret values	Yes (encrypted)
salt/pillar/secrets_registry.sls	Metadata: origin, rotation schedule, notes	Yes (no secrets)
Makefile (TF_SECRET_VARS)	Allowlist of secrets rendered to tfvars	Yes
terraform/secrets.auto.tfvars	Rendered plaintext for tofu (gitignored)	No
~/secrets/	Plaintext backup on local NVMe	No

Adding a New Secret

1. Generate or obtain the secret value
2. GPG-encrypt: echo -n 'VALUE' | gpg --encrypt --armor --recipient 81A03D1CF874DC90
3. Add the encrypted block to salt/pillar/secrets/platform.sls
4. Add metadata to salt/pillar/secrets_registry.sls
5. Add the variable name to TF_SECRET_VARS in Makefile
6. Add a variable block in terraform/variables.tf
7. Run make tofu-secrets to verify rendering
8. Back up plaintext to ~/secrets/ (local NVMe)

GPG Key

• Key ID: 81A03D1CF874DC90
• Identity: Salt Master (pal-e-platform) <salt@pal-e.local>
• Algorithm: RSA 4096
• Backup: Private GitHub repo ldraney/secrets

Current Secrets (15)

Platform (12): tailscale oauth (2), grafana, forgejo, woodpecker (3), harbor (2), minio, keycloak, paledocs DB. Observability (3): slack webhook (dormant), telegram bot token, telegram chat ID.

Architecture Principles

1. Bootstrap-first. Get the self-hosting stack running before optimizing anything. Perfect is the enemy of deployed.
2. Repos have an end. This repo's job is done when Forgejo + CI + observability are running. Future work lives in pal-e-services.
3. In-cluster by default. Every infrastructure need is met by a self-hosted service. Cloud-managed services are opt-in when scale justifies cost.
4. Portable across substrates. Same Helm charts run on k3s or EKS. Environment differences live in .tfvars files, not in code.
5. Observable from day one. Metrics and logs are platform concerns, not afterthoughts.