ldraney/landscaping-assistant
1
0
Fork 0
Code Issues 32 Pull requests 1 Projects Releases Packages Wiki Activity Actions 4

Schedule image digestion: Claude Vision OCR to WorkQueueItems #204

New issue
Closed
opened 2026-06-13 21:26:38 +00:00 by ldraney · 4 comments
ldraney commented 2026-06-13 21:26:38 +00:00
Owner
Copy link

Type

Feature

Lineage

Standalone -- extends the existing Upload tab (#33, done) with AI-powered schedule parsing.

Repo

ldraney/landscaping-assistant

User Story

As a crew lead
I want to photograph my boss's printed weekly schedule and have the app parse it into WorkQueueItems
So that I don't have to manually re-enter 50+ property names across 5 days every week

Context

The boss prints a weekly schedule as a spreadsheet with columns for Monday through Friday. Each column lists property/client names to visit that day. The schedule has quirks that make naive OCR insufficient:

Column-overflow pattern: When one day's list is too long for its column, the boss continues it in the adjacent column below a visual gap. For example, Wednesday's overflow appears in Thursday's column space -- the bottom names in Thursday actually belong to Wednesday. The boss groups properties by geographic proximity, not strict column alignment.

Handwritten additions: Sometimes extra names are handwritten at the bottom of the page, outside the printed table structure.

Existing infrastructure:

  • Upload model with Active Storage (MinIO in prod, local in dev) -- already deployed and working
  • Property model with client_name, latitude, longitude
  • WorkQueueItem model with property_id, work_date, position
  • Anthropic API key in ~/secrets/anthropic/credentials.env
  • Example schedule images already in MinIO (3 photos from iPhone 12, dated May 2026)

Zero-bloat constraint: Unknown names (not matching any existing Property) must NEVER be inserted into the database. Only properties that fuzzy-match an existing Property.client_name get WorkQueueItems created. Unmatched names are displayed for informational awareness but discarded.

File Targets

Files the agent should create:

  • app/services/schedule_digester.rb -- orchestrates the Vision API call and property matching
  • app/views/uploads/digest.html.erb -- confirmation/review UI showing matched vs unmatched names
  • db/migrate/TIMESTAMP_add_week_start_to_uploads.rb -- associate an upload with the week it covers

Files the agent should modify:

  • Gemfile -- add gem "anthropic"
  • app/controllers/uploads_controller.rb -- add digest and confirm_digest actions
  • app/models/upload.rb -- add week_start date field (no AR associations to WorkQueueItem; the service creates items directly, the relationship is implicit through week_start date range)
  • config/routes.rb -- add digest routes under uploads
  • app/views/uploads/show.html.erb -- add "Digest Schedule" button

Files the agent should NOT touch:

  • app/models/property.rb -- no schema changes to properties
  • app/controllers/weeks_controller.rb -- existing week view stays unchanged
  • Any seed data or migration that alters the properties table

Feature Flag

Flag: none
This is an extension of the existing upload flow, not a new user-facing surface. The button only appears on the upload show page for lead+ roles who already have access.

Acceptance Criteria

  • "Digest Schedule" button on upload show page sends image to Claude Vision API
  • Claude returns structured JSON with day-keyed property name arrays
  • Column-overflow is correctly handled (names reassigned to actual day, not column day)
  • Each extracted name is fuzzy-matched against Property.client_name (case-insensitive, tolerant of minor spelling differences)
  • Confirmation screen shows: matched properties (green, with day), unmatched names (gray, skipped), ambiguous matches (yellow, selectable)
  • On confirm, WorkQueueItems are created for the target week (Mon-Fri dates derived from a date picker or "this week" default)
  • No new Property records are ever created -- only existing properties get WorkQueueItems
  • Duplicate detection: if a WorkQueueItem already exists for that property+date, skip it (don't error)
  • When Claude Vision API is unavailable or returns malformed/unparseable JSON, the user sees a clear error message and can retry; no partial WorkQueueItems are created

Test Expectations

  • Unit test: ScheduleDigester parses structured Claude response and matches against properties
  • Unit test: fuzzy matching handles case differences, minor typos, whitespace
  • Unit test: malformed API responses are handled gracefully (no unrescued exceptions)
  • Request spec: digest action requires lead+ role
  • Request spec: confirm creates correct WorkQueueItems for matched properties
  • Run command: bundle exec rspec spec/services/schedule_digester_spec.rb spec/requests/uploads_spec.rb

Constraints

  • Use the anthropic Ruby gem for API calls
  • Claude model: use claude-sonnet-4-20250514 for vision (cost-effective for OCR)
  • API key sourced from ENV["ANTHROPIC_API_KEY"] (already in secrets)
  • The Vision prompt must explicitly describe the column-overflow pattern so Claude can reassign names correctly
  • Fuzzy matching: use Levenshtein distance or trigram similarity -- keep it simple, no external gems if possible
  • Property matching is against client_name only (not address)

Checklist

  • PR opened
  • Tests pass
  • No unrelated changes

Related

  • landscaping-assistant -- project this affects
  • Issue #33 (done) -- Upload tab: snap and store schedule paper photo in MinIO
### Type Feature ### Lineage Standalone -- extends the existing Upload tab (#33, done) with AI-powered schedule parsing. ### Repo `ldraney/landscaping-assistant` ### User Story As a crew lead I want to photograph my boss's printed weekly schedule and have the app parse it into WorkQueueItems So that I don't have to manually re-enter 50+ property names across 5 days every week ### Context The boss prints a weekly schedule as a spreadsheet with columns for Monday through Friday. Each column lists property/client names to visit that day. The schedule has quirks that make naive OCR insufficient: **Column-overflow pattern:** When one day's list is too long for its column, the boss continues it in the adjacent column below a visual gap. For example, Wednesday's overflow appears in Thursday's column space -- the bottom names in Thursday actually belong to Wednesday. The boss groups properties by geographic proximity, not strict column alignment. **Handwritten additions:** Sometimes extra names are handwritten at the bottom of the page, outside the printed table structure. **Existing infrastructure:** - Upload model with Active Storage (MinIO in prod, local in dev) -- already deployed and working - Property model with `client_name`, `latitude`, `longitude` - WorkQueueItem model with `property_id`, `work_date`, `position` - Anthropic API key in `~/secrets/anthropic/credentials.env` - Example schedule images already in MinIO (3 photos from iPhone 12, dated May 2026) **Zero-bloat constraint:** Unknown names (not matching any existing Property) must NEVER be inserted into the database. Only properties that fuzzy-match an existing `Property.client_name` get WorkQueueItems created. Unmatched names are displayed for informational awareness but discarded. ### File Targets Files the agent should create: - `app/services/schedule_digester.rb` -- orchestrates the Vision API call and property matching - `app/views/uploads/digest.html.erb` -- confirmation/review UI showing matched vs unmatched names - `db/migrate/TIMESTAMP_add_week_start_to_uploads.rb` -- associate an upload with the week it covers Files the agent should modify: - `Gemfile` -- add `gem "anthropic"` - `app/controllers/uploads_controller.rb` -- add `digest` and `confirm_digest` actions - `app/models/upload.rb` -- add `week_start` date field (no AR associations to WorkQueueItem; the service creates items directly, the relationship is implicit through week_start date range) - `config/routes.rb` -- add digest routes under uploads - `app/views/uploads/show.html.erb` -- add "Digest Schedule" button Files the agent should NOT touch: - `app/models/property.rb` -- no schema changes to properties - `app/controllers/weeks_controller.rb` -- existing week view stays unchanged - Any seed data or migration that alters the properties table ### Feature Flag Flag: none This is an extension of the existing upload flow, not a new user-facing surface. The button only appears on the upload show page for lead+ roles who already have access. ### Acceptance Criteria - [ ] "Digest Schedule" button on upload show page sends image to Claude Vision API - [ ] Claude returns structured JSON with day-keyed property name arrays - [ ] Column-overflow is correctly handled (names reassigned to actual day, not column day) - [ ] Each extracted name is fuzzy-matched against `Property.client_name` (case-insensitive, tolerant of minor spelling differences) - [ ] Confirmation screen shows: matched properties (green, with day), unmatched names (gray, skipped), ambiguous matches (yellow, selectable) - [ ] On confirm, WorkQueueItems are created for the target week (Mon-Fri dates derived from a date picker or "this week" default) - [ ] No new Property records are ever created -- only existing properties get WorkQueueItems - [ ] Duplicate detection: if a WorkQueueItem already exists for that property+date, skip it (don't error) - [ ] When Claude Vision API is unavailable or returns malformed/unparseable JSON, the user sees a clear error message and can retry; no partial WorkQueueItems are created ### Test Expectations - [ ] Unit test: `ScheduleDigester` parses structured Claude response and matches against properties - [ ] Unit test: fuzzy matching handles case differences, minor typos, whitespace - [ ] Unit test: malformed API responses are handled gracefully (no unrescued exceptions) - [ ] Request spec: digest action requires lead+ role - [ ] Request spec: confirm creates correct WorkQueueItems for matched properties - Run command: `bundle exec rspec spec/services/schedule_digester_spec.rb spec/requests/uploads_spec.rb` ### Constraints - Use the `anthropic` Ruby gem for API calls - Claude model: use `claude-sonnet-4-20250514` for vision (cost-effective for OCR) - API key sourced from `ENV["ANTHROPIC_API_KEY"]` (already in secrets) - The Vision prompt must explicitly describe the column-overflow pattern so Claude can reassign names correctly - Fuzzy matching: use Levenshtein distance or trigram similarity -- keep it simple, no external gems if possible - Property matching is against `client_name` only (not address) ### Checklist - [ ] PR opened - [ ] Tests pass - [ ] No unrelated changes ### Related - `landscaping-assistant` -- project this affects - Issue #33 (done) -- Upload tab: snap and store schedule paper photo in MinIO
ldraney commented 2026-06-13 21:50:00 +00:00
Author
Owner
Copy link

Scope Review: NEEDS_REFINEMENT

Review note: review-1450-2026-06-13
Well-structured feature issue with all template sections present. Four items need attention before this is READY:

  • [BODY] Upload-to-WorkQueueItem association unclear -- issue says "has_many :work_queue_items through digest" but no digest join model is defined. Clarify whether this is a direct FK association or an implicit relationship (service creates items, no AR association on Upload).
  • [BODY] Missing error-handling acceptance criterion -- what happens when Claude Vision API is unavailable or returns malformed JSON?
  • [SCOPE] The schedule-upload user story on project-landscaping-assistant needs updating -- currently says "Snap a photo for later reference" but this ticket extends it to AI-powered parsing into WorkQueueItems.
  • [SCOPE] No arch-rails-app architecture note exists in pal-e-docs. Create one for traceability completeness.
## Scope Review: NEEDS_REFINEMENT Review note: `review-1450-2026-06-13` Well-structured feature issue with all template sections present. Four items need attention before this is READY: - **[BODY]** Upload-to-WorkQueueItem association unclear -- issue says "has_many :work_queue_items through digest" but no digest join model is defined. Clarify whether this is a direct FK association or an implicit relationship (service creates items, no AR association on Upload). - **[BODY]** Missing error-handling acceptance criterion -- what happens when Claude Vision API is unavailable or returns malformed JSON? - **[SCOPE]** The `schedule-upload` user story on project-landscaping-assistant needs updating -- currently says "Snap a photo for later reference" but this ticket extends it to AI-powered parsing into WorkQueueItems. - **[SCOPE]** No `arch-rails-app` architecture note exists in pal-e-docs. Create one for traceability completeness.
ldraney commented 2026-06-13 21:53:56 +00:00
Author
Owner
Copy link

Scope Review: APPROVED (re-review)

Review note: review-1450-2026-06-13-r2

Both [BODY] findings from the initial review have been resolved:

  • Upload-to-WorkQueueItem relationship is now clearly documented as implicit (no AR associations, service creates items directly)
  • Error handling AC added (AC9) with matching test expectation for malformed API responses

Two [SCOPE] items remain but are project-level shared gaps, not blocking:

  • User story on project page still says "photo for later reference" -- needs update to reflect AI-powered parsing
  • arch-rails-app note does not exist in pal-e-docs (shared gap across many tickets)

Ticket is ready to move to next_up.

## Scope Review: APPROVED (re-review) Review note: `review-1450-2026-06-13-r2` Both [BODY] findings from the initial review have been resolved: - Upload-to-WorkQueueItem relationship is now clearly documented as implicit (no AR associations, service creates items directly) - Error handling AC added (AC9) with matching test expectation for malformed API responses Two [SCOPE] items remain but are project-level shared gaps, not blocking: - User story on project page still says "photo for later reference" -- needs update to reflect AI-powered parsing - arch-rails-app note does not exist in pal-e-docs (shared gap across many tickets) Ticket is ready to move to next_up.
ldraney commented 2026-06-14 02:39:12 +00:00
Author
Owner
Copy link

Validation: PARTIAL

Tiers executed: Tier 1 (code review), Tier 3 (CI pipeline + dev server)
Validation note: validation-204-2026-06-13

12 checks: 11 PASS, 1 FAIL

Passing checks

  • anthropic gem installed (v1.48.1) and locked
  • ScheduleDigester service (267 lines) with Claude Vision API, Levenshtein fuzzy matching, error handling
  • week_start migration exists and reflected in schema
  • digest + confirm_digest routes present
  • "Digest Schedule" button on upload show page
  • Confirmation UI with matched (green), fuzzy (yellow), unmatched (gray/skipped) display
  • Controller error handling (ApiError, ParseError rescued with user flash messages)
  • Duplicate detection (skips existing WorkQueueItem for same property+date)
  • No new Property records created (zero-bloat enforced)
  • Test files exist (schedule_digester_spec.rb, uploads_spec.rb)
  • Woodpecker pipeline #515: all 6 steps green

Failing check

  • Dev server routes return HTTP 500 — ActiveRecord::PendingMigrationError blocks ALL routes. Two migrations pending: add_completed_by_other_to_work_queue_items (different PR) + add_week_start_to_uploads (this PR). Fix: run bin/rails db:migrate on dev server.

Verdict rationale

PARTIAL because the code is correct and CI passes, but we cannot verify the UI or end-to-end flow on the live dev server until migrations are applied. This is an ops gap, not a code defect.

## Validation: PARTIAL Tiers executed: Tier 1 (code review), Tier 3 (CI pipeline + dev server) Validation note: `validation-204-2026-06-13` **12 checks**: 11 PASS, 1 FAIL ### Passing checks - anthropic gem installed (v1.48.1) and locked - ScheduleDigester service (267 lines) with Claude Vision API, Levenshtein fuzzy matching, error handling - week_start migration exists and reflected in schema - digest + confirm_digest routes present - "Digest Schedule" button on upload show page - Confirmation UI with matched (green), fuzzy (yellow), unmatched (gray/skipped) display - Controller error handling (ApiError, ParseError rescued with user flash messages) - Duplicate detection (skips existing WorkQueueItem for same property+date) - No new Property records created (zero-bloat enforced) - Test files exist (schedule_digester_spec.rb, uploads_spec.rb) - Woodpecker pipeline #515: all 6 steps green ### Failing check - **Dev server routes return HTTP 500** — `ActiveRecord::PendingMigrationError` blocks ALL routes. Two migrations pending: `add_completed_by_other_to_work_queue_items` (different PR) + `add_week_start_to_uploads` (this PR). Fix: run `bin/rails db:migrate` on dev server. ### Verdict rationale PARTIAL because the code is correct and CI passes, but we cannot verify the UI or end-to-end flow on the live dev server until migrations are applied. This is an ops gap, not a code defect.
ldraney commented 2026-06-14 02:45:01 +00:00
Author
Owner
Copy link

Validation: PASS (r2)

Tiers executed: Tier 1 (CI test suite), Tier 3 (prod deployment verification)
Validation note: validation-204-2026-06-13-r2

12 checks: 12 PASS, 0 FAIL

Re-validation after migration fix. All checks that previously blocked (HTTP 500s on dev) now pass cleanly:

  • Woodpecker pipeline #515 green (371 specs, 0 failures)
  • Pod image 058fa49 matches merge commit
  • ArgoCD: Synced + Healthy
  • /up returns 200 on both prod and dev funnels
  • Digest routes (POST /uploads/:id/digest, POST /uploads/:id/confirm_digest) present
  • ScheduleDigester service deployed, anthropic gem (1.48.1) installed
  • week_start migration applied
  • All existing routes healthy, no regressions
## Validation: PASS (r2) Tiers executed: Tier 1 (CI test suite), Tier 3 (prod deployment verification) Validation note: `validation-204-2026-06-13-r2` 12 checks: **12 PASS, 0 FAIL** Re-validation after migration fix. All checks that previously blocked (HTTP 500s on dev) now pass cleanly: - Woodpecker pipeline #515 green (371 specs, 0 failures) - Pod image `058fa49` matches merge commit - ArgoCD: Synced + Healthy - `/up` returns 200 on both prod and dev funnels - Digest routes (`POST /uploads/:id/digest`, `POST /uploads/:id/confirm_digest`) present - `ScheduleDigester` service deployed, `anthropic` gem (1.48.1) installed - `week_start` migration applied - All existing routes healthy, no regressions
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
docs/update-app-architecture
revert-213
move-other-crew-bottom-remove-schedule
feature/today-view-improvements
154-enable-assume-ssl
revert-145-person-icon
docs/user-stories-auth
docs/update-infra-pipeline-post-incident
fix-cache-step-blocking-deploy
fix-kaniko-registry-pull
deactivate-from-today
feature/quick-address-entry
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
ldraney/landscaping-assistant#204
No description provided.