Add semantic search API endpoint (GET /notes/semantic-search) #138

Merged

forgejo_admin merged 2 commits from 137-semantic-search-endpoint into main

2026-03-09 14:17:19 +00:00

forgejo_admin commented

2026-03-09 14:02:23 +00:00

Owner

Summary

Adds GET /notes/semantic-search endpoint that performs cosine similarity search over block embeddings using pgvector. Query text is embedded via Ollama, then matched against blocks.embedding using the <=> operator with HNSW index. Returns ranked blocks with note context, content snippets, and similarity scores.

Changes

src/pal_e_docs/schemas.py -- Added SemanticSearchResult response model with slug, title, note_type, status, project, anchor_id, block_type, content_snippet, and similarity fields
src/pal_e_docs/routes/notes.py -- Added GET /notes/semantic-search route mirroring the search_notes pattern: same parameter style, raw SQL, filter logic (note_type, project, status, tags). Uses query instruction prefix for embeddings, returns 501 on SQLite, 503 on Ollama errors. Reuses extract_block_text from embedding_worker for content snippets
tests/test_semantic_search.py -- 12 tests: 501 on SQLite, validation (empty/missing query, limit bounds), Ollama error handling (ConnectError, TimeoutException, HTTPStatusError, empty embeddings), query instruction prefix verification, model name verification, cosine operator usage, filter passthrough

Test Plan

pytest tests/ -k test_semantic -- 12 tests pass
Full test suite (excluding browser): 586 passed, 7 failed (pre-existing test_template_renderer failures on main)
ruff check and ruff format pass on all changed files

Review Checklist

No unrelated changes
Tests pass (pytest tests/ -k test_semantic)
Lint passes (ruff check, ruff format)
Follows existing search_notes pattern (raw SQL, same filters, same parameter style)
No schema migrations needed (uses existing blocks.embedding column)
PR description includes Closes #137

Plan: plan-2026-02-26-tf-modularize-postgres (traceability)
Forgejo issue: #137

Closes #137

## Summary Adds `GET /notes/semantic-search` endpoint that performs cosine similarity search over block embeddings using pgvector. Query text is embedded via Ollama, then matched against `blocks.embedding` using the `<=>` operator with HNSW index. Returns ranked blocks with note context, content snippets, and similarity scores. ## Changes - `src/pal_e_docs/schemas.py` -- Added `SemanticSearchResult` response model with slug, title, note_type, status, project, anchor_id, block_type, content_snippet, and similarity fields - `src/pal_e_docs/routes/notes.py` -- Added `GET /notes/semantic-search` route mirroring the `search_notes` pattern: same parameter style, raw SQL, filter logic (note_type, project, status, tags). Uses query instruction prefix for embeddings, returns 501 on SQLite, 503 on Ollama errors. Reuses `extract_block_text` from `embedding_worker` for content snippets - `tests/test_semantic_search.py` -- 12 tests: 501 on SQLite, validation (empty/missing query, limit bounds), Ollama error handling (ConnectError, TimeoutException, HTTPStatusError, empty embeddings), query instruction prefix verification, model name verification, cosine operator usage, filter passthrough ## Test Plan - `pytest tests/ -k test_semantic` -- 12 tests pass - Full test suite (excluding browser): 586 passed, 7 failed (pre-existing `test_template_renderer` failures on main) - `ruff check` and `ruff format` pass on all changed files ## Review Checklist - [x] No unrelated changes - [x] Tests pass (`pytest tests/ -k test_semantic`) - [x] Lint passes (`ruff check`, `ruff format`) - [x] Follows existing `search_notes` pattern (raw SQL, same filters, same parameter style) - [x] No schema migrations needed (uses existing `blocks.embedding` column) - [x] PR description includes `Closes #137` ## Related - Plan: `plan-2026-02-26-tf-modularize-postgres` (traceability) - Forgejo issue: #137 Closes #137

forgejo_admin added 1 commit

2026-03-09 14:02:23 +00:00

Add GET /notes/semantic-search endpoint for vector similarity search

ci/woodpecker/pr/woodpecker Pipeline failed

Details

836a9e4a6b

Implements pgvector cosine similarity search over block embeddings.
Query text is embedded via Ollama, then matched against blocks.embedding
using the <=> operator. Returns ranked blocks with note context and
similarity scores.

Includes 12 tests covering 501 on SQLite, validation, Ollama error
handling (503 for connect/timeout/HTTP errors), query instruction
prefix verification, cosine operator usage, filter passthrough, and
limit bounds.

Closes #137

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

forgejo_admin referenced this pull request

2026-03-09 14:04:37 +00:00

6d: Add semantic search API endpoint (GET /notes/semantic-search) #137

forgejo_admin referenced this pull request from a commit

2026-03-09 14:12:20 +00:00

Fix 5 QA nits on semantic search endpoint

forgejo_admin added 1 commit

2026-03-09 14:12:20 +00:00

Fix 5 QA nits on semantic search endpoint

ci/woodpecker/pr/woodpecker Pipeline failed

Details

7cbfe7bde8

1. Pass note_title to extract_block_text so heading snippets include
   the parent note title (e.g. "My Note: Section Title")
2. Reword misleading test docstring — the test only verifies filter
   params are accepted, not that they reach SQL
3. Consolidate _EMBEDDING_MODEL into settings.embedding_model (single
   source of truth in config.py, used by both notes.py and
   embedding_worker.py)
4. Remove unused b.id from SELECT clause
5. Add comment explaining asymmetric query/document instruction prefixes

Closes #138

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

forgejo_admin referenced this pull request

2026-03-09 14:14:19 +00:00

6d: Add semantic search API endpoint (GET /notes/semantic-search) #137

forgejo_admin referenced this pull request from a commit

2026-03-09 14:17:13 +00:00

Fix 5 QA nits on semantic search endpoint

forgejo_admin force-pushed 137-semantic-search-endpoint from 7cbfe7bde8

ci/woodpecker/pr/woodpecker Pipeline failed

Details

to 831d56ee38

ci/woodpecker/pr/woodpecker Pipeline failed

Details