feat(hints): bring the contour strategy to macOS with a capture scope (#1550)

Author: y3owk1n

configs/default-config.toml

@@ -64,7 +64,8 @@ text = "#E8EEFF"
 # See https://github.com/y3owk1n/neru/blob/main/docs/CONFIGURATION.md#hints
 [hints]
 enabled = true
-strategy = "axtree"                         # Element detection: "axtree" (AX API) or "vision" (Vision Framework)
+strategy = "axtree"                         # Element detection: "axtree" (AX API), "vision" (screen recognition) or "contour" (edge detection)
+capture_scope = "window"                    # Region the vision and contour strategies scan: "window" (focused window) or "screen"
 hint_characters = "asdfghjkl"               # Characters used for hint labels
 label_direction = "normal"                  # Hint label algorithm: "normal" (default) or "reverse"
 max_depth = 50                              # Max accessibility tree depth (0 = unlimited)

docs/CLI.md

@@ -302,7 +302,8 @@ nothing.
 | `--hide-on-empty-search` |  | none | `hints` | Hide all hints when search query is empty (requires --search) |
 | `--role` |  | value, repeatable | `hints` | Filter by element role (comma-separated: button,link — the hints.clickable_roles vocabulary, see 'neru roles'). Repeat the flag to add more |
 | `--text` |  | value, repeatable | `hints` | Filter elements by text content (comma-separated, case-insensitive substring match). Repeat the flag to add more |
-| `--strategy` |  | value | `hints` | Element detection strategy: axtree (the platform accessibility tree), vision (screen recognition: the Vision framework on macOS, tesseract OCR on Linux), or wl-kbptr (contour detection via embedded C) |
+| `--strategy` |  | value | `hints` | Element detection strategy: axtree (the platform accessibility tree), vision (screen recognition: the Vision framework on macOS, tesseract OCR on Linux), or contour (edge and contour analysis of the window pixels, ported from wl-kbptr) |
+| `--capture-scope` |  | value | `hints` | Region the vision and contour strategies scan: window (the focused window) or screen (the whole active screen) |
 | `--label-direction` |  | value | `hints` | Hint label enumeration: normal (default, prefix-avoidance, prefers shorter labels) or reverse (spreads labels across the alphabet) |
 | `--split-word` |  | none | `hints` | Split detected text into word-level regions (requires vision strategy) |
 | `--zoom-to-depth` |  | value | `recursive_grid` | Auto-zoom to the given depth (a non-negative integer) in recursive-grid at the current cursor position |

docs/CONFIGURATION.md

@@ -765,7 +765,12 @@ Explicit component colors override theme derivation. Omitted colors inherit from
 
 ## [hints]
 
-Labels clickable UI elements with short overlay labels. By default uses the platform accessibility tree (`axtree` strategy). Optionally uses on-screen recognition (`vision` strategy) for apps whose accessibility tree is too thin to hint from — detects elements from a screen capture scoped to the focused window, through the Vision framework on macOS (text plus rectangles) and tesseract OCR on Linux (text only). On Linux, the `wl-kbptr` strategy detects buttons, icons, and text by running contour detection directly on the captured window buffer without external library dependencies.
+Labels clickable UI elements with short overlay labels. By default uses the platform accessibility tree (`axtree` strategy). Two screen-capture strategies exist for apps whose accessibility tree is too thin to hint from; both scan the focused window by default (`capture_scope` widens that to the whole screen), both add the system surfaces the `include_*` options ask for from the accessibility tree, and neither is available on Windows, which has no capture backend:
+
+- `vision`: on-screen recognition. The Vision framework on macOS (text plus rectangles), tesseract OCR on Linux (text only). Detected text becomes the element's title, so hint search (`--search`) and `--split-word` work. Costs an ML or OCR pass per activation, and on Linux needs tesseract installed.
+- `contour`: edge and contour analysis of the window pixels, an algorithm ported from [wl-kbptr](https://github.com/moverest/wl-kbptr). Finds anything with a visible outline (buttons, icons, toolbar items, text runs) in a few milliseconds with no external dependency. Elements carry no text, so search and word splitting do not apply, and `hints.vision.*` is not read.
+
+Pick `vision` when you want to type what you see, or the app is text-heavy. Pick `contour` when latency matters, the targets are icons rather than words, or OCR is not installed. Both are overridable per-app.
 
 Press `/` to text-search elements. `Space` for multi-word queries. `Return` confirms filtered hints (first is auto-selected). `Escape` cancels search.
 
@@ -776,7 +781,8 @@ Start with search visible: `neru hints --search` (see [CLI.md](CLI.md#neru-hints
 | Option                             | Type         | Default                 | Description                                                                                                                                                                                                                                                                                                                          |
 | ---------------------------------- | ------------ | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
 | `enabled`                          | bool         | `true`                  | Enable/disable hints mode                                                                                                                                                                                                                                                                                                            |
-| `strategy`                         | string       | `"axtree"`              | Element detection strategy: `"axtree"` (the platform accessibility tree), `"vision"` (screen recognition — Vision framework on macOS, tesseract OCR on Linux, unavailable on Windows), or `"wl-kbptr"` (Linux only contour detection replicated from wl-kbptr). Vision and wl-kbptr modes detect the frontmost window content from a screen capture while still using the accessibility tree for system elements (menubar, dock, NC). Overridable per-app via `[hints.app_configs]`. |
+| `strategy`                         | string       | `"axtree"`              | Element detection strategy: `"axtree"` (the platform accessibility tree), `"vision"` (screen recognition — Vision framework on macOS, tesseract OCR on Linux, unavailable on Windows), or `"contour"` (edge and contour analysis ported from wl-kbptr, macOS and Linux, unavailable on Windows). Both capture strategies detect the focused window content from a screen capture; see the section intro for when to pick which. Overridable per-app via `[hints.app_configs]`. |
+| `capture_scope`                    | string       | `"window"`              | Region the `vision` and `contour` strategies scan: `"window"` (the focused window, or the whole screen when nothing is focused) or `"screen"` (the whole active screen, so notifications, panels and adjacent tiled windows get hints too, at the cost of a bigger capture). Ignored by `axtree`. Overridable per-app via `[hints.app_configs]` and per-activation via `neru hints --capture-scope`. |
 | `hint_characters`                  | string       | `"asdfghjkl"`           | Characters used for labels                                                                                                                                                                                                                                                                                                           |
 | `label_direction`                  | string       | `"normal"`              | Hint label algorithm: `"normal"` (default, prefix-avoidance greedy) or `"reverse"` (reverse-order tiers). Empty value defaults to `"normal"`. Overridable per-app via `[hints.app_configs]` and per-activation via the `neru hints --label-direction` CLI flag. See [Choosing a label direction](#choosing-a-label-direction) below. |
 | `max_depth`                        | int          | `50`                    | Max accessibility tree depth (0 = unlimited)                                                                                                                                                                                                                                                                                         |
@@ -1023,7 +1029,8 @@ You can also mix directions per-app via `[hints.app_configs]` or per-activation
 | Field                        | Type   | Description                                                                                                                                                                               |
 | ---------------------------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
 | `bundle_id`                  | string | App bundle ID                                                                                                                                                                             |
-| `strategy`                   | string | Override element detection strategy for this app (`"axtree"`, `"vision"`, or `"wl-kbptr"`). Empty string = use global `hints.strategy`.                                                                  |
+| `strategy`                   | string | Override element detection strategy for this app (`"axtree"`, `"vision"`, or `"contour"`). Empty string = use global `hints.strategy`.                                                                  |
+| `capture_scope`              | string | Override the region the `vision` and `contour` strategies scan for this app (`"window"` or `"screen"`). Empty string = use global `hints.capture_scope`. |
 | `label_direction`            | string | Override hint label algorithm for this app (`"normal"` or `"reverse"`). Empty string = use global `hints.label_direction`. See [Choosing a label direction](#choosing-a-label-direction). |
 | `additional_clickable_roles` | array  | Extra roles to treat as clickable, same vocabulary as [`clickable_roles`](#clickable-roles)                                                                                              |
 | `ignore_clickable_check`     | bool   | Skip clickability heuristic for this app                                                                                                                                                  |

docs/CROSS_PLATFORM.md

@@ -346,9 +346,10 @@ places those pixels.
 ⁷ Linux `vision` is **text-only**, and permanently so. macOS runs three Vision
 requests — text recognition, rectangle detection and saliency — and an OCR
 engine answers the first. `hints.vision.detect_rectangles` and the four
-`rectangle_*` options are therefore declared macOS-only rather than met with a
-contour-detection library, which would be a heavy new required dependency for a
-sub-feature of a non-default strategy
+`rectangle_*` options are therefore declared macOS-only: they tune the Vision
+framework's rectangle request, which has no OCR equivalent. The `contour`
+strategy is a separate, dependency-free detector, not an implementation of
+`detect_rectangles`
 ([ADR 0013](./adr/0013-parity-is-measured-in-words-not-subsystems.md)). The
 other fourteen `hints.vision.*` options are read on Linux exactly as they are on
 macOS.
@@ -1058,12 +1059,18 @@ green in every cell while an option means nothing, which is exactly how
 | `recursive_grid.app_configs.strategy = vision` | option | ✅ | ✅ | ❌ | the vision strategy needs an element-detection engine, which macOS has in the Vision framework and Linux in tesseract; Windows has neither, so it finds nothing there and none of its settings are read; use axtree |
 | `scroll.app_configs.strategy = vision` | option | ✅ | ✅ | ❌ | the vision strategy needs an element-detection engine, which macOS has in the Vision framework and Linux in tesseract; Windows has neither, so it finds nothing there and none of its settings are read; use axtree |
 | `app_configs.strategy = vision` | option | ✅ | ✅ | ❌ | the vision strategy needs an element-detection engine, which macOS has in the Vision framework and Linux in tesseract; Windows has neither, so it finds nothing there and none of its settings are read; use axtree |
-| `hints.strategy = wl-kbptr` | option | ❌ | ✅ | ❌ | the wl-kbptr strategy detects UI elements via contour analysis of screen captures on Linux |
-| `hints.app_configs.strategy = wl-kbptr` | option | ❌ | ✅ | ❌ | the wl-kbptr strategy detects UI elements via contour analysis of screen captures on Linux |
-| `grid.app_configs.strategy = wl-kbptr` | option | ❌ | ✅ | ❌ | the wl-kbptr strategy detects UI elements via contour analysis of screen captures on Linux |
-| `recursive_grid.app_configs.strategy = wl-kbptr` | option | ❌ | ✅ | ❌ | the wl-kbptr strategy detects UI elements via contour analysis of screen captures on Linux |
-| `scroll.app_configs.strategy = wl-kbptr` | option | ❌ | ✅ | ❌ | the wl-kbptr strategy detects UI elements via contour analysis of screen captures on Linux |
-| `app_configs.strategy = wl-kbptr` | option | ❌ | ✅ | ❌ | the wl-kbptr strategy detects UI elements via contour analysis of screen captures on Linux |
+| `hints.strategy = contour` | option | ✅ | ✅ | ❌ | the contour strategy runs edge detection over a screen capture, which macOS and Linux can take and Windows cannot; there it finds nothing, use axtree |
+| `hints.app_configs.strategy = contour` | option | ✅ | ✅ | ❌ | the contour strategy runs edge detection over a screen capture, which macOS and Linux can take and Windows cannot; there it finds nothing, use axtree |
+| `grid.app_configs.strategy = contour` | option | ✅ | ✅ | ❌ | the contour strategy runs edge detection over a screen capture, which macOS and Linux can take and Windows cannot; there it finds nothing, use axtree |
+| `recursive_grid.app_configs.strategy = contour` | option | ✅ | ✅ | ❌ | the contour strategy runs edge detection over a screen capture, which macOS and Linux can take and Windows cannot; there it finds nothing, use axtree |
+| `scroll.app_configs.strategy = contour` | option | ✅ | ✅ | ❌ | the contour strategy runs edge detection over a screen capture, which macOS and Linux can take and Windows cannot; there it finds nothing, use axtree |
+| `app_configs.strategy = contour` | option | ✅ | ✅ | ❌ | the contour strategy runs edge detection over a screen capture, which macOS and Linux can take and Windows cannot; there it finds nothing, use axtree |
+| `hints.capture_scope` | option | ✅ | ✅ | ❌ | capture_scope only shapes the vision and contour strategies, and Windows has no capture backend for either |
+| `hints.app_configs.capture_scope` | option | ✅ | ✅ | ❌ | capture_scope only shapes the vision and contour strategies, and Windows has no capture backend for either |
+| `grid.app_configs.capture_scope` | option | ✅ | ✅ | ❌ | capture_scope only shapes the vision and contour strategies, and Windows has no capture backend for either |
+| `recursive_grid.app_configs.capture_scope` | option | ✅ | ✅ | ❌ | capture_scope only shapes the vision and contour strategies, and Windows has no capture backend for either |
+| `scroll.app_configs.capture_scope` | option | ✅ | ✅ | ❌ | capture_scope only shapes the vision and contour strategies, and Windows has no capture backend for either |
+| `app_configs.capture_scope` | option | ✅ | ✅ | ❌ | capture_scope only shapes the vision and contour strategies, and Windows has no capture backend for either |
 | `recursive_grid.animation.enabled` | option | ✅ | ✅ | ❌ | the Windows overlay backend has no grid transition animation |
 | `recursive_grid.animation.duration_ms` | option | ✅ | ✅ | ❌ | the Windows overlay backend has no grid transition animation |
 | `monitor_select.enabled` | option | ✅ | ✅ | ❌ | monitor_select needs the optional MonitorSelector overlay extension, which the Windows backend does not implement |
@@ -1098,7 +1105,8 @@ green in every cell while an option means nothing, which is exactly how
 | `smooth_scroll.duration_per_pixel` | option | ✅ | ✅ | ❌ | the Windows scroll is injected in one step; macOS and Linux animate it, and on X11 the steps are whole wheel notches because X has no smaller scroll to send |
 | `--split-word` | mode flag | ✅ | ✅ | ❌ | splitting detected text into words needs the vision strategy, which Windows has no engine for; there the flag is refused rather than ignored |
 | `--strategy=vision` | mode flag | ✅ | ✅ | ❌ | the vision strategy needs an element-detection engine, which macOS has in the Vision framework and Linux in tesseract; Windows has neither, so detection returns nothing and no hints appear; use axtree |
-| `--strategy=wl-kbptr` | mode flag | ❌ | ✅ | ❌ | the wl-kbptr strategy detects UI elements via contour analysis of screen captures on Linux |
+| `--strategy=contour` | mode flag | ✅ | ✅ | ❌ | the contour strategy runs edge detection over a screen capture, which macOS and Linux can take and Windows cannot; there it finds nothing, use axtree |
+| `--capture-scope` | mode flag | ✅ | ✅ | ❌ | capture_scope only shapes the vision and contour strategies, and Windows has no capture backend for either |
 | `hide_cursor` | action | ✅ | ❌ | ❌ | a Wayland client may not hide another client's cursor, and the blessed Linux stack is Wayland; Windows has no equivalent either |
 | `show_cursor` | action | ✅ | ❌ | ❌ | a Wayland client may not hide another client's cursor, and the blessed Linux stack is Wayland; Windows has no equivalent either |
 | `scroll_left` | action | ✅ | ✅ | ❌ | the Windows wheel event carries no horizontal delta, so a sideways scroll injects nothing |

internal/adapter/platform/darwin/vision.h

@@ -37,5 +37,10 @@ VisionResult *NeruDetectElements(CGRect detectionRect, NeruVisionConfig config);
 // Returns a CGImageRef (caller must CFRelease).
 CGImageRef NeruCaptureScreen(void);
 
+// Captures the display containing rect (global top-left-origin points) and
+// reports that display's bounds in the same coordinates through outBounds.
+// Returns a CGImageRef (caller must CFRelease), or NULL when capture fails.
+CGImageRef NeruCaptureDisplayContaining(CGRect rect, CGRect *outBounds);
+
 // Frees a VisionResult previously returned by NeruDetectElements.
 void NeruFreeVisionResult(VisionResult *result);