Salt Phase 1: bootstrap + Makefile

forgejo_admin commented

2026-02-27 07:45:54 +00:00

Owner

Summary

Introduce SaltStack host configuration management alongside existing Terraform cluster management (Phase 1 of salt host management plan)
Add unified Makefile CLI with salt-* and tofu-* targets for all platform operations
Create idempotent bootstrap script that installs salt-onedir from AUR, configures master+minion on localhost, and verifies connectivity

Changes

Makefile: unified CLI with 8 targets (salt-bootstrap, salt-test, salt-apply, salt-uninstall, tofu-init, tofu-plan, tofu-apply, tofu-fmt)
salt/bootstrap.sh: idempotent install of salt-onedir via paru, generates /etc/salt/master.d/roots.conf with dynamic repo path, starts (not enables) services, accepts minion key, verifies test.ping. Supports --remove for clean uninstall.
salt/master.conf: listens on 127.0.0.1 (localhost only), warning log level. file_roots/pillar_roots are NOT here -- generated dynamically by bootstrap.sh so repo path is resolved at install time.
salt/minion.conf: master 127.0.0.1, minion ID "archbox", warning log level
salt/states/top.sls: empty state tree (Phase 2 populates with package/service/kernel/user states)
salt/pillar/top.sls: empty pillar tree (Phase 2 populates with GPG-encrypted secrets)

Design decisions:

salt-onedir over salt package: system Python is 3.14 which breaks Salt dependencies; salt-onedir bundles its own Python at /opt/salt
master.d/roots.conf generated at bootstrap: avoids hardcoding repo path, different worktrees/clones resolve correctly
Start but don't enable services: operator confirms everything works before enabling auto-start
--local flag on salt-call in Makefile: local file client since master+minion are colocated

No changes to terraform/ or any existing files. Purely additive.

Test Plan

Run make salt-bootstrap on the Arch box
Verify salt-call test.ping returns True
Verify make salt-test runs dry-run highstate successfully
Verify make tofu-plan still works (existing Terraform unbroken)
Verify make salt-uninstall cleanly removes Salt

Review Checklist

Passed automated review-fix loop
No secrets committed
No unnecessary file changes
Commit messages are descriptive

issue-pal-e-platform-salt-bootstrap -- the issue this PR addresses
plan-2026-02-26-salt-host-management -- Phase 1 of the salt host management plan
host-inventory-archbox -- complete host audit that informs Phase 2 state definitions

## Summary - Introduce SaltStack host configuration management alongside existing Terraform cluster management (Phase 1 of salt host management plan) - Add unified Makefile CLI with salt-* and tofu-* targets for all platform operations - Create idempotent bootstrap script that installs salt-onedir from AUR, configures master+minion on localhost, and verifies connectivity ## Changes - `Makefile`: unified CLI with 8 targets (salt-bootstrap, salt-test, salt-apply, salt-uninstall, tofu-init, tofu-plan, tofu-apply, tofu-fmt) - `salt/bootstrap.sh`: idempotent install of salt-onedir via paru, generates /etc/salt/master.d/roots.conf with dynamic repo path, starts (not enables) services, accepts minion key, verifies test.ping. Supports --remove for clean uninstall. - `salt/master.conf`: listens on 127.0.0.1 (localhost only), warning log level. file_roots/pillar_roots are NOT here -- generated dynamically by bootstrap.sh so repo path is resolved at install time. - `salt/minion.conf`: master 127.0.0.1, minion ID "archbox", warning log level - `salt/states/top.sls`: empty state tree (Phase 2 populates with package/service/kernel/user states) - `salt/pillar/top.sls`: empty pillar tree (Phase 2 populates with GPG-encrypted secrets) Design decisions: - salt-onedir over salt package: system Python is 3.14 which breaks Salt dependencies; salt-onedir bundles its own Python at /opt/salt - master.d/roots.conf generated at bootstrap: avoids hardcoding repo path, different worktrees/clones resolve correctly - Start but don't enable services: operator confirms everything works before enabling auto-start - --local flag on salt-call in Makefile: local file client since master+minion are colocated No changes to terraform/ or any existing files. Purely additive. ## Test Plan - [ ] Run `make salt-bootstrap` on the Arch box - [ ] Verify `salt-call test.ping` returns True - [ ] Verify `make salt-test` runs dry-run highstate successfully - [ ] Verify `make tofu-plan` still works (existing Terraform unbroken) - [ ] Verify `make salt-uninstall` cleanly removes Salt ## Review Checklist - [ ] Passed automated review-fix loop - [ ] No secrets committed - [ ] No unnecessary file changes - [ ] Commit messages are descriptive ## Related Notes - `issue-pal-e-platform-salt-bootstrap` -- the issue this PR addresses - `plan-2026-02-26-salt-host-management` -- Phase 1 of the salt host management plan - `host-inventory-archbox` -- complete host audit that informs Phase 2 state definitions

forgejo_admin added 1 commit

2026-02-27 07:45:54 +00:00

Add Salt bootstrap, configuration, and unified Makefile 6d155f616f

Introduces SaltStack host configuration management alongside the existing
Terraform cluster management. Salt manages everything outside the cluster
(packages, firewall, services, k3s, secrets); Terraform manages everything
inside it.

New files:
- Makefile: unified CLI with salt-* and tofu-* targets
- salt/bootstrap.sh: idempotent install of salt-onedir from AUR via paru,
  configures master+minion on localhost, accepts key, verifies connectivity
- salt/master.conf: listen on 127.0.0.1, warning log level
- salt/minion.conf: connect to localhost, minion ID "archbox"
- salt/states/top.sls: empty state tree (Phase 2 populates)
- salt/pillar/top.sls: empty pillar tree (Phase 2 populates)

Plan: plan-2026-02-26-salt-host-management (Phase 1)
Issue: issue-pal-e-platform-salt-bootstrap

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

forgejo_admin added 1 commit

2026-02-27 07:46:46 +00:00

fix: drop --local from salt-call, use restart for idempotency 006854c53e

Two issues found in review:

1. salt-call --local reads /etc/salt/minion for config, but file_roots
   and pillar_roots are set in the master config (/etc/salt/master.d/
   roots.conf). Without --local, salt-call communicates with the local
   master which has the correct roots. Drop --local since we set up a
   full master+minion in bootstrap.

2. Use systemctl restart instead of start in bootstrap.sh so config
   changes are picked up on idempotent re-runs. start is a no-op if
   the service is already running.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

forgejo_admin commented

2026-02-27 07:46:51 +00:00

Author

Owner

Review round 1 — 2 issues found and fixed:

Blocking: --local flag on salt-call in Makefile — salt-call --local reads config from /etc/salt/minion, but file_roots and pillar_roots are set in the master config (/etc/salt/master.d/roots.conf). This means make salt-test and make salt-apply would not find the state tree. Fix: dropped --local so salt-call communicates with the local master which has the correct roots.
Improvement: systemctl start → restart in bootstrap.sh — On idempotent re-runs where config may have changed, start is a no-op if the service is already running. Changed to restart so config changes are always picked up.

**Review round 1 — 2 issues found and fixed:** 1. **Blocking: `--local` flag on `salt-call` in Makefile** — `salt-call --local` reads config from `/etc/salt/minion`, but `file_roots` and `pillar_roots` are set in the *master* config (`/etc/salt/master.d/roots.conf`). This means `make salt-test` and `make salt-apply` would not find the state tree. Fix: dropped `--local` so salt-call communicates with the local master which has the correct roots. 2. **Improvement: `systemctl start` → `restart` in bootstrap.sh** — On idempotent re-runs where config may have changed, `start` is a no-op if the service is already running. Changed to `restart` so config changes are always picked up.

forgejo_admin commented

2026-02-27 07:47:16 +00:00

Author

Owner

Review round 2 -- clean pass.

All 6 files reviewed. Zero issues found. The --local fix from round 1 is correct. Bootstrap script is idempotent, handles both install and uninstall paths, and the Makefile provides all 8 required targets.

Ready for user review and manual testing on the Arch box.

**Review round 2 -- clean pass.** All 6 files reviewed. Zero issues found. The `--local` fix from round 1 is correct. Bootstrap script is idempotent, handles both install and uninstall paths, and the Makefile provides all 8 required targets. Ready for user review and manual testing on the Arch box.

forgejo_admin commented

2026-02-27 07:48:56 +00:00

Author

Owner

Design Decision Correction

The PR body mentions a design decision about using --local flag on salt-call in Makefile: local file client since master+minion are colocated. However, this was superseded by commit 006854c, which removed --local from the salt-call invocation.

The Makefile now correctly uses salt-call without --local, allowing it to properly read file_roots and pillar_roots from the master configuration. This is the correct approach since the master config is the source of truth for these paths.

The code is correct — this comment is just clarifying that the PR body's design decision should reflect the final implementation.

## Design Decision Correction The PR body mentions a design decision about using `--local flag on salt-call in Makefile: local file client since master+minion are colocated`. However, this was superseded by commit 006854c, which removed `--local` from the salt-call invocation. The Makefile now correctly uses `salt-call` **without** `--local`, allowing it to properly read `file_roots` and `pillar_roots` from the master configuration. This is the correct approach since the master config is the source of truth for these paths. The code is correct — this comment is just clarifying that the PR body's design decision should reflect the final implementation.

forgejo_admin commented

2026-02-27 08:11:53 +00:00

Author

Owner

QA Review Fixes — All 6 Warnings Addressed

Pushed commit 794c352 with fixes for all QA warnings:

W1. PR body mentions `--local` but code doesn't use it

No code change needed. Confirmed grep shows zero --local references in any code or comments. The PR body "Design decisions" section is inaccurate — the Makefile salt-call commands correctly omit --local because salt-call already defaults to local mode when contacting the master on localhost. The previous commit (006854c) already dropped --local from the code. The PR body should be read with this correction in mind.

W2. `sudo -u paru` nesting could prompt for password

Fixed. Added sudo -u "${BUILD_USER}" sudo -v warmup before the paru call in install_salt(). If the cache refresh fails, a warning is emitted but execution continues (paru's --noconfirm handles the rest).

W3. `do_remove()` doesn't clean up the minion key

Fixed. Added salt-key --delete="${MINION_ID}" --yes at the top of do_remove(), before stopping services. This prevents stale accepted keys from causing silent auth failures after uninstall+reinstall. Also reordered service stops: minion first, then master (minion needs master running to process the key delete).

W4. `do_remove()` doesn't clean up `/etc/salt/master` and `/etc/salt/minion`

Fixed. Added rm -f /etc/salt/master and rm -f /etc/salt/minion alongside the existing roots.conf cleanup. The /etc/salt/ directory itself is still preserved (comment retained).

W5. `sleep 2` after starting salt-master is fragile

Fixed. Replaced with a port poll loop that checks ss -tlnp for port 4506 (salt-master ret port), polling every 1s for up to 15s. If it still isn't listening after 15s, a warning is emitted and execution continues rather than hard-failing.

W6. No `.gitignore` entries for Salt cache/runtime artifacts

Fixed. Added to existing .gitignore:

# Salt runtime artifacts
salt/states/__pycache__/
salt/pillar/__pycache__/
*.pyc

## QA Review Fixes — All 6 Warnings Addressed Pushed commit `794c352` with fixes for all QA warnings: ### W1. PR body mentions `--local` but code doesn't use it **No code change needed.** Confirmed grep shows zero `--local` references in any code or comments. The PR body "Design decisions" section is inaccurate — the Makefile `salt-call` commands correctly omit `--local` because salt-call already defaults to local mode when contacting the master on localhost. The previous commit (`006854c`) already dropped `--local` from the code. The PR body should be read with this correction in mind. ### W2. `sudo -u paru` nesting could prompt for password **Fixed.** Added `sudo -u "${BUILD_USER}" sudo -v` warmup before the `paru` call in `install_salt()`. If the cache refresh fails, a warning is emitted but execution continues (paru's `--noconfirm` handles the rest). ### W3. `do_remove()` doesn't clean up the minion key **Fixed.** Added `salt-key --delete="${MINION_ID}" --yes` at the **top** of `do_remove()`, before stopping services. This prevents stale accepted keys from causing silent auth failures after uninstall+reinstall. Also reordered service stops: minion first, then master (minion needs master running to process the key delete). ### W4. `do_remove()` doesn't clean up `/etc/salt/master` and `/etc/salt/minion` **Fixed.** Added `rm -f /etc/salt/master` and `rm -f /etc/salt/minion` alongside the existing `roots.conf` cleanup. The `/etc/salt/` directory itself is still preserved (comment retained). ### W5. `sleep 2` after starting salt-master is fragile **Fixed.** Replaced with a port poll loop that checks `ss -tlnp` for port 4506 (salt-master ret port), polling every 1s for up to 15s. If it still isn't listening after 15s, a warning is emitted and execution continues rather than hard-failing. ### W6. No `.gitignore` entries for Salt cache/runtime artifacts **Fixed.** Added to existing `.gitignore`: ``` # Salt runtime artifacts salt/states/__pycache__/ salt/pillar/__pycache__/ *.pyc ```

forgejo_admin added 1 commit

2026-02-27 08:28:46 +00:00

fix: address QA review warnings on Salt bootstrap PR 794c3522bd

W2: Add sudo credential cache warmup before paru call to prevent
    password prompt when sudo cache is expired inside root shell.
W3: Delete minion key in do_remove() before stopping master to
    prevent stale accepted key causing silent auth failure on reinstall.
W4: Remove /etc/salt/master and /etc/salt/minion in do_remove()
    since bootstrap.sh is responsible for placing them.
W5: Replace fragile sleep 2 with port poll loop (up to 15s) that
    waits for salt-master to listen on port 4506.
W6: Add Salt runtime artifacts (__pycache__, *.pyc) to .gitignore.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

forgejo_admin commented

2026-02-27 08:29:12 +00:00

Author

Owner

W1 clarification: The PR description mentions --local flag on salt-call in Makefile under Design decisions. This was removed during review — the Makefile correctly uses salt-call state.apply (master-minion mode, not masterless --local). The PR body is stale on this point; the code is correct.

**W1 clarification:** The PR description mentions `--local flag on salt-call in Makefile` under Design decisions. This was removed during review — the Makefile correctly uses `salt-call state.apply` (master-minion mode, not masterless `--local`). The PR body is stale on this point; the code is correct.