Activate embedding worker and fix CI image tag drift #155

Merged

forgejo_admin merged 1 commit from 154-activate-semantic-search-pipeline-scale into main

2026-03-14 14:46:50 +00:00

forgejo_admin commented

2026-03-14 14:25:45 +00:00

Owner

Summary

Scales the embedding worker deployment from 0 to 1 replica, activating the dormant semantic search pipeline. Updates the stale image tag to match the current API image and fixes the CI pipeline so both manifests stay in sync on every build.

Changes

k8s/embedding-worker.yaml -- set replicas: 1 (was 0) to activate the worker
k8s/embedding-worker.yaml -- update image tag from bec9a4ec... to 9fe5106d... (current API image)
.woodpecker.yaml -- add k8s/embedding-worker.yaml to path exclusion list so CI skip-ci commits don't trigger rebuild loops
.woodpecker.yaml -- update update-deployment-tag step to sed-replace and git-add both deployment.yaml and embedding-worker.yaml

Test Plan

497 tests pass locally (verified)
ruff check + format clean (verified)
After merge, verify worker pod starts: kubectl get pods -n pal-e-docs -l app=pal-e-docs-embedding-worker
Verify embedding progress: check embedding_status counts in blocks table
Verify search: curl 'http://localhost:8000/notes/search?q=deployment&mode=semantic&limit=3' via port-forward
Verify next CI build updates both manifest image tags (no more drift)

Review Checklist

No unrelated changes
Tests pass (497 passed)
Linting clean (ruff check + format)
CI pipeline updated to prevent future image tag drift
Post-merge: worker pod Running and healthy
Post-merge: embeddings processing pending blocks

Plan: plan-pal-e-docs
Forgejo issue: #154

Closes #154

## Summary Scales the embedding worker deployment from 0 to 1 replica, activating the dormant semantic search pipeline. Updates the stale image tag to match the current API image and fixes the CI pipeline so both manifests stay in sync on every build. ## Changes - `k8s/embedding-worker.yaml` -- set `replicas: 1` (was 0) to activate the worker - `k8s/embedding-worker.yaml` -- update image tag from `bec9a4ec...` to `9fe5106d...` (current API image) - `.woodpecker.yaml` -- add `k8s/embedding-worker.yaml` to path exclusion list so CI skip-ci commits don't trigger rebuild loops - `.woodpecker.yaml` -- update `update-deployment-tag` step to sed-replace and git-add both `deployment.yaml` and `embedding-worker.yaml` ## Test Plan - [ ] 497 tests pass locally (verified) - [ ] ruff check + format clean (verified) - [ ] After merge, verify worker pod starts: `kubectl get pods -n pal-e-docs -l app=pal-e-docs-embedding-worker` - [ ] Verify embedding progress: check `embedding_status` counts in blocks table - [ ] Verify search: `curl 'http://localhost:8000/notes/search?q=deployment&mode=semantic&limit=3'` via port-forward - [ ] Verify next CI build updates both manifest image tags (no more drift) ## Review Checklist - [x] No unrelated changes - [x] Tests pass (497 passed) - [x] Linting clean (ruff check + format) - [x] CI pipeline updated to prevent future image tag drift - [ ] Post-merge: worker pod Running and healthy - [ ] Post-merge: embeddings processing pending blocks ## Related - Plan: `plan-pal-e-docs` - Forgejo issue: #154 Closes #154

forgejo_admin added 1 commit

2026-03-14 14:25:45 +00:00

Activate embedding worker and fix CI image tag drift

ci/woodpecker/pr/woodpecker Pipeline was successful

Details

8652e53563

Scale embedding worker from 0 to 1 replica to activate the semantic
search pipeline. Update stale image tag to match current API image.
Fix CI pipeline to update both deployment.yaml and embedding-worker.yaml
on each build so the worker image never drifts behind again.

Closes #154

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

forgejo_admin commented

2026-03-14 14:26:07 +00:00

Author

Owner

Self-Review: LGTM

Diff reviewed (2 files, +5/-4):

k8s/embedding-worker.yaml -- replicas: 0 -> 1 (activates worker). Image tag updated from stale bec9a4ec to current 9fe5106d (matches deployment.yaml).
.woodpecker.yaml -- CI update-deployment-tag step now updates both manifests via a single sed pass. embedding-worker.yaml added to path exclusion to prevent rebuild loops from [skip ci] commits.

Verified:

497 tests pass
ruff check + format clean
No unrelated changes
PR is mergeable

No issues found. Ready for merge.

## Self-Review: LGTM **Diff reviewed (2 files, +5/-4):** 1. `k8s/embedding-worker.yaml` -- `replicas: 0` -> `1` (activates worker). Image tag updated from stale `bec9a4ec` to current `9fe5106d` (matches `deployment.yaml`). 2. `.woodpecker.yaml` -- CI `update-deployment-tag` step now updates both manifests via a single `sed` pass. `embedding-worker.yaml` added to path exclusion to prevent rebuild loops from `[skip ci]` commits. **Verified:** - 497 tests pass - ruff check + format clean - No unrelated changes - PR is mergeable **No issues found.** Ready for merge.

forgejo_admin commented

2026-03-14 14:27:50 +00:00

Author

Owner

PR #155 Review

BLOCKERS

None.

NITS

None. This is a clean, minimal change with no unnecessary modifications.

ANALYSIS

k8s/embedding-worker.yaml changes (2 hunks):

replicas: 0 to replicas: 1 -- straightforward activation. The worker was previously scaled to zero (PR #143) due to CrashLoopBackOff. The underlying issue (code/infra readiness) has since been resolved through the semantic search pipeline work (PRs #130, #138, #141).
Image tag updated from bec9a4ec... to 9fe5106d... -- verified this matches the current deployment.yaml on main (line 24: api:9fe5106d2ef8ceb229aa62721a5551e8f255fab2). The drift occurred because the CI update-deployment-tag step only updated deployment.yaml, not embedding-worker.yaml. This PR fixes that root cause.

.woodpecker.yaml changes (2 hunks):

Path exclusion list now includes k8s/embedding-worker.yaml alongside k8s/deployment.yaml and k8s/.argocd-source-*. This prevents the CI [skip ci] image tag commit from triggering a rebuild loop. Correct and necessary.
The sed -i and git add commands in the update-deployment-tag step now target both k8s/deployment.yaml and k8s/embedding-worker.yaml. Since both manifests use the same harbor.tail5b443a.ts.net/pal-e-docs/api: image prefix, the single sed pattern correctly applies to both. This eliminates future drift.

Verified against codebase:

The embedding worker module exists at /home/ldraney/pal-e-docs/src/pal_e_docs/embedding_worker.py and the manifest command ["python", "-m", "pal_e_docs.embedding_worker"] matches the __main__ entrypoint.
The worker is already listed in k8s/kustomization.yaml so ArgoCD will pick up the replicas change.
Health probes (port 8001, /healthz) in the manifest match the worker's _HealthHandler implementation.
Resource limits (64Mi request, 256Mi limit) are reasonable for a Python worker doing HTTP calls to Ollama.

SOP COMPLIANCE

Branch named after issue (154-activate-semantic-search-pipeline-scale)
PR body has Summary, Changes, Test Plan, Related sections
Related references plan slug (plan-pal-e-docs) and issue (#154)
Closes #154 present in PR body
No secrets, .env files, or credentials committed
No scope creep -- exactly 2 files changed, all directly related to activation and drift prevention
Commit messages are descriptive (per PR title)

VERDICT: APPROVED

## PR #155 Review ### BLOCKERS None. ### NITS None. This is a clean, minimal change with no unnecessary modifications. ### ANALYSIS **`k8s/embedding-worker.yaml` changes (2 hunks):** - `replicas: 0` to `replicas: 1` -- straightforward activation. The worker was previously scaled to zero (PR #143) due to CrashLoopBackOff. The underlying issue (code/infra readiness) has since been resolved through the semantic search pipeline work (PRs #130, #138, #141). - Image tag updated from `bec9a4ec...` to `9fe5106d...` -- verified this matches the current `deployment.yaml` on main (line 24: `api:9fe5106d2ef8ceb229aa62721a5551e8f255fab2`). The drift occurred because the CI `update-deployment-tag` step only updated `deployment.yaml`, not `embedding-worker.yaml`. This PR fixes that root cause. **`.woodpecker.yaml` changes (2 hunks):** - Path exclusion list now includes `k8s/embedding-worker.yaml` alongside `k8s/deployment.yaml` and `k8s/.argocd-source-*`. This prevents the CI `[skip ci]` image tag commit from triggering a rebuild loop. Correct and necessary. - The `sed -i` and `git add` commands in the `update-deployment-tag` step now target both `k8s/deployment.yaml` and `k8s/embedding-worker.yaml`. Since both manifests use the same `harbor.tail5b443a.ts.net/pal-e-docs/api:` image prefix, the single sed pattern correctly applies to both. This eliminates future drift. **Verified against codebase:** - The embedding worker module exists at `/home/ldraney/pal-e-docs/src/pal_e_docs/embedding_worker.py` and the manifest command `["python", "-m", "pal_e_docs.embedding_worker"]` matches the `__main__` entrypoint. - The worker is already listed in `k8s/kustomization.yaml` so ArgoCD will pick up the replicas change. - Health probes (port 8001, `/healthz`) in the manifest match the worker's `_HealthHandler` implementation. - Resource limits (64Mi request, 256Mi limit) are reasonable for a Python worker doing HTTP calls to Ollama. ### SOP COMPLIANCE - [x] Branch named after issue (`154-activate-semantic-search-pipeline-scale`) - [x] PR body has Summary, Changes, Test Plan, Related sections - [x] Related references plan slug (`plan-pal-e-docs`) and issue (#154) - [x] `Closes #154` present in PR body - [x] No secrets, .env files, or credentials committed - [x] No scope creep -- exactly 2 files changed, all directly related to activation and drift prevention - [x] Commit messages are descriptive (per PR title) ### VERDICT: APPROVED