Skip to content

Commit 54782e3

Browse files
y3owk1ny3owk1n
authored
feat(linux): recognise on-screen text for the vision hint strategy (#1480)
1 parent 9ed69c4 commit 54782e3

42 files changed

Lines changed: 3214 additions & 420 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/ci.yml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -86,7 +86,9 @@ jobs:
8686
libxkbcommon-dev \
8787
wayland-protocols \
8888
libei-dev \
89-
liboeffis-dev
89+
liboeffis-dev \
90+
libtesseract-dev \
91+
tesseract-ocr-eng
9092
9193
# Install Devbox on macOS only
9294
- name: Install Devbox

‎.github/workflows/publish-artifacts.yml‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -214,7 +214,9 @@ jobs:
214214
libxkbcommon-dev \
215215
wayland-protocols \
216216
libei-dev \
217-
liboeffis-dev
217+
liboeffis-dev \
218+
libtesseract-dev \
219+
tesseract-ocr-eng
218220
219221
- uses: actions/setup-go@v7
220222
with:
@@ -274,7 +276,9 @@ jobs:
274276
libxkbcommon-dev \
275277
wayland-protocols \
276278
libei-dev \
277-
liboeffis-dev
279+
liboeffis-dev \
280+
libtesseract-dev \
281+
tesseract-ocr-eng
278282
279283
- uses: actions/setup-go@v7
280284
with:

‎README.md‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -220,7 +220,7 @@ More modes, more engines, more platforms — and it's free. If you've been payin
220220
| Grid | ✅ | ✅ | ✅ |
221221
| Vim-Style Scroll | ✅ | ✅ | ✅ |
222222
| Hints (Accessibility API) | ✅ | 🔵 | 🔵 |
223-
| Hints (Vision OCR) | ✅ | 🔲 | 🔲 |
223+
| Hints (Vision OCR) | ✅ | 🔵 | 🔲 |
224224
| Direct Mouse Injection | ✅ | ✅ | ✅ |
225225
| Global Hotkeys | ✅ | ✅ | ✅ |
226226
| Native Overlays | ✅ | ✅ | ✅ |
@@ -230,10 +230,12 @@ More modes, more engines, more platforms — and it's free. If you've been payin
230230

231231
**Hints caveats.** On **Linux**, hints work through AT-SPI, so coverage depends
232232
on the app exposing an accessibility tree (GTK/Qt do; Chromium and Electron apps
233-
need `--force-renderer-accessibility`). On **Windows**, UI Automation coverage is
234-
initial and the tree walk is shallow, and per-app config does not re-apply when
235-
you change windows. **Linux requires X11 or Wayland on wlroots/KWin — GNOME
236-
Wayland is not supported**; use a GNOME X11 session.
233+
need `--force-renderer-accessibility`). Where that tree is too thin, Vision OCR
234+
is the fallback — tesseract, text only, and not on KDE, which has no screen
235+
capture Neru can use. On **Windows**, UI Automation coverage is initial and the
236+
tree walk is shallow, per-app config does not re-apply when you change windows,
237+
and there is no OCR fallback at all. **Linux requires X11 or Wayland on
238+
wlroots/KWin — GNOME Wayland is not supported**; use a GNOME X11 session.
237239

238240
→ [Roadmap](docs/ROADMAP.md) · [Cross-platform details](docs/CROSS_PLATFORM.md)
239241

‎devbox.json‎

Lines changed: 81 additions & 76 deletions
Original file line numberDiff line numberDiff line change
@@ -1,78 +1,83 @@
11
{
2-
"$schema": "https://raw.githubusercontent.com/jetify-com/devbox/0.17.3/.schema/devbox.schema.json",
3-
"packages": {
4-
"gotools": "0.44.0",
5-
"gofumpt": "0.10.0",
6-
"golines": "0.15.0",
7-
"nixfmt": "1.3.1",
8-
"gopls": "0.22.0",
9-
"just": "1.54.0",
10-
"go": "1.26.4",
11-
"nixd": "2.9.1",
12-
"golangci-lint": "2.12.2",
13-
"golangci-lint-langserver": "0.0.12",
14-
"just-lsp": "0.4.7",
15-
"clang-tools": "21.1.8",
16-
"pkg-config": {
17-
"version": "0.29.2",
18-
"platforms": ["x86_64-linux", "aarch64-linux"]
19-
},
20-
"cairo": {
21-
"version": "1.18.4",
22-
"platforms": ["x86_64-linux", "aarch64-linux"]
23-
},
24-
"wayland": {
25-
"version": "1.22.0",
26-
"platforms": ["x86_64-linux", "aarch64-linux"]
27-
},
28-
"xorg.libX11": {
29-
"version": "1.8.12",
30-
"platforms": ["x86_64-linux", "aarch64-linux"]
31-
},
32-
"xorg.libXrender": {
33-
"version": "0.9.12",
34-
"platforms": ["x86_64-linux", "aarch64-linux"]
35-
},
36-
"xorg.libXext": {
37-
"version": "1.3.6",
38-
"platforms": ["x86_64-linux", "aarch64-linux"]
39-
},
40-
"xorg.libXtst": {
41-
"version": "1.2.5",
42-
"platforms": ["x86_64-linux", "aarch64-linux"]
43-
},
44-
"xorg.libXrandr": {
45-
"version": "1.5.4",
46-
"platforms": ["x86_64-linux", "aarch64-linux"]
47-
},
48-
"xorg.libXinerama": {
49-
"version": "1.1.5",
50-
"platforms": ["x86_64-linux", "aarch64-linux"]
51-
},
52-
"xorg.libXfixes": {
53-
"version": "6.0.1",
54-
"platforms": ["x86_64-linux", "aarch64-linux"]
55-
},
56-
"libxkbcommon": {
57-
"version": "1.13.1",
58-
"platforms": ["x86_64-linux", "aarch64-linux"]
59-
},
60-
"wayland-protocols": {
61-
"version": "1.33",
62-
"platforms": ["x86_64-linux", "aarch64-linux"]
63-
},
64-
"xorg.libXi": {
65-
"version": "1.8.2",
66-
"platforms": ["x86_64-linux", "aarch64-linux"]
67-
},
68-
"libei": {
69-
"version": "1.5.0",
70-
"platforms": ["x86_64-linux", "aarch64-linux"]
71-
}
72-
},
73-
"shell": {
74-
"init_hook": [
75-
"if [[ \"$(uname)\" == \"Linux\" ]]; then export PKG_CONFIG_PATH_FOR_TARGET=\"$(find /nix/store -maxdepth 5 -type d -name pkgconfig 2>/dev/null | tr '\\n' ':')\" && export CGO_CFLAGS=\"-I$(pkg-config --variable=includedir cairo) $(pkg-config --cflags xi) $CGO_CFLAGS\"; fi"
76-
]
77-
}
2+
"$schema": "https://raw.githubusercontent.com/jetify-com/devbox/0.17.3/.schema/devbox.schema.json",
3+
"packages": {
4+
"gotools": "0.44.0",
5+
"gofumpt": "0.10.0",
6+
"golines": "0.15.0",
7+
"nixfmt": "1.3.1",
8+
"gopls": "0.22.0",
9+
"just": "1.54.0",
10+
"go": "1.26.4",
11+
"nixd": "2.9.1",
12+
"golangci-lint": "2.12.2",
13+
"golangci-lint-langserver": "0.0.12",
14+
"just-lsp": "0.4.7",
15+
"clang-tools": "21.1.8",
16+
"pkg-config": {
17+
"version": "0.29.2",
18+
"platforms": ["x86_64-linux", "aarch64-linux"]
19+
},
20+
"cairo": {
21+
"version": "1.18.4",
22+
"platforms": ["x86_64-linux", "aarch64-linux"]
23+
},
24+
"wayland": {
25+
"version": "1.22.0",
26+
"platforms": ["x86_64-linux", "aarch64-linux"]
27+
},
28+
"xorg.libX11": {
29+
"version": "1.8.12",
30+
"platforms": ["x86_64-linux", "aarch64-linux"]
31+
},
32+
"xorg.libXrender": {
33+
"version": "0.9.12",
34+
"platforms": ["x86_64-linux", "aarch64-linux"]
35+
},
36+
"xorg.libXext": {
37+
"version": "1.3.6",
38+
"platforms": ["x86_64-linux", "aarch64-linux"]
39+
},
40+
"xorg.libXtst": {
41+
"version": "1.2.5",
42+
"platforms": ["x86_64-linux", "aarch64-linux"]
43+
},
44+
"xorg.libXrandr": {
45+
"version": "1.5.4",
46+
"platforms": ["x86_64-linux", "aarch64-linux"]
47+
},
48+
"xorg.libXinerama": {
49+
"version": "1.1.5",
50+
"platforms": ["x86_64-linux", "aarch64-linux"]
51+
},
52+
"xorg.libXfixes": {
53+
"version": "6.0.1",
54+
"platforms": ["x86_64-linux", "aarch64-linux"]
55+
},
56+
"libxkbcommon": {
57+
"version": "1.13.1",
58+
"platforms": ["x86_64-linux", "aarch64-linux"]
59+
},
60+
"wayland-protocols": {
61+
"version": "1.33",
62+
"platforms": ["x86_64-linux", "aarch64-linux"]
63+
},
64+
"xorg.libXi": {
65+
"version": "1.8.2",
66+
"platforms": ["x86_64-linux", "aarch64-linux"]
67+
},
68+
"libei": {
69+
"version": "1.5.0",
70+
"platforms": ["x86_64-linux", "aarch64-linux"]
71+
},
72+
"tesseract": {
73+
"version": "5.5.2",
74+
"platforms": ["x86_64-linux", "aarch64-linux"]
75+
}
76+
},
77+
"shell": {
78+
"init_hook": [
79+
"if [[ \"$(uname)\" == \"Linux\" ]]; then export PKG_CONFIG_PATH_FOR_TARGET=\"$(find /nix/store -maxdepth 5 -type d -name pkgconfig 2>/dev/null | tr '\\n' ':')\" && export CGO_CFLAGS=\"-I$(pkg-config --variable=includedir cairo) $(pkg-config --cflags xi) $CGO_CFLAGS\"; fi",
80+
"if [[ \"$(uname)\" == \"Linux\" && -z \"${TESSDATA_PREFIX:-}\" ]]; then export TESSDATA_PREFIX=\"$(pkg-config --variable=prefix tesseract 2>/dev/null)/share/tessdata\"; fi"
81+
]
82+
}
7883
}

‎devbox.lock‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -897,6 +897,54 @@
897897
}
898898
}
899899
},
900+
"tesseract@5.5.2": {
901+
"last_modified": "2026-08-01T16:34:20Z",
902+
"resolved": "github:NixOS/nixpkgs/a5cbcfe954791221bfffe2307f7d1a1bf61a871e#tesseract",
903+
"source": "devbox-search",
904+
"version": "5.5.2",
905+
"systems": {
906+
"aarch64-darwin": {
907+
"outputs": [
908+
{
909+
"name": "out",
910+
"path": "/nix/store/pajzicgg31c5ghs9l4q2nkcndhf17id7-tesseract-5.5.2",
911+
"default": true
912+
}
913+
],
914+
"store_path": "/nix/store/pajzicgg31c5ghs9l4q2nkcndhf17id7-tesseract-5.5.2"
915+
},
916+
"aarch64-linux": {
917+
"outputs": [
918+
{
919+
"name": "out",
920+
"path": "/nix/store/xcc6aa5dwa7n24z6djzncwrjghnnj4ln-tesseract-5.5.2",
921+
"default": true
922+
}
923+
],
924+
"store_path": "/nix/store/xcc6aa5dwa7n24z6djzncwrjghnnj4ln-tesseract-5.5.2"
925+
},
926+
"x86_64-darwin": {
927+
"outputs": [
928+
{
929+
"name": "out",
930+
"path": "/nix/store/ygxv2nig79j0l0nbgyl6660hxlb2vq3r-tesseract-5.5.2",
931+
"default": true
932+
}
933+
],
934+
"store_path": "/nix/store/ygxv2nig79j0l0nbgyl6660hxlb2vq3r-tesseract-5.5.2"
935+
},
936+
"x86_64-linux": {
937+
"outputs": [
938+
{
939+
"name": "out",
940+
"path": "/nix/store/f87ns5vl166f9xwzj8711lw2p855ljgw-tesseract-5.5.2",
941+
"default": true
942+
}
943+
],
944+
"store_path": "/nix/store/f87ns5vl166f9xwzj8711lw2p855ljgw-tesseract-5.5.2"
945+
}
946+
}
947+
},
900948
"wayland-protocols@1.33": {
901949
"last_modified": "2024-02-24T23:06:34Z",
902950
"resolved": "github:NixOS/nixpkgs/9a9dae8f6319600fa9aebde37f340975cab4b8c0#wayland-protocols",

‎docs/CLI.md‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -103,8 +103,9 @@ Accepted by every command.
103103

104104
¹ Element discovery quality differs by platform: a full accessibility tree on
105105
macOS, an AT-SPI walk on Linux whose coverage depends on the application, and an
106-
initial shallow UI Automation walk on Windows. The `vision` strategy is macOS
107-
only. See [Accessibility and hints](CROSS_PLATFORM.md#accessibility-and-hints).
106+
initial shallow UI Automation walk on Windows. The `vision` strategy is the
107+
fallback where that tree is thin, on macOS and Linux; Windows has no engine for
108+
it. See [Accessibility and hints](CROSS_PLATFORM.md#accessibility-and-hints).
108109

109110
² Two action subcommands are limited: `hide_cursor` and `show_cursor` are macOS
110111
only, and `scroll_left` / `scroll_right` have no effect on Windows. See
@@ -301,7 +302,7 @@ nothing.
301302
| `--hide-on-empty-search` | | none | `hints` | Hide all hints when search query is empty (requires --search) |
302303
| `--role` | | value, repeatable | `hints` | Filter by element role (comma-separated: button,link — the hints.clickable_roles vocabulary, see 'neru roles'). Repeat the flag to add more |
303304
| `--text` | | value, repeatable | `hints` | Filter elements by text content (comma-separated, case-insensitive substring match). Repeat the flag to add more |
304-
| `--strategy` | | value | `hints` | Element detection strategy: axtree (macOS AX API) or vision (Vision Framework) |
305+
| `--strategy` | | value | `hints` | Element detection strategy: axtree (the platform accessibility tree) or vision (screen recognition: the Vision framework on macOS, tesseract OCR on Linux) |
305306
| `--label-direction` | | value | `hints` | Hint label enumeration: normal (default, prefix-avoidance, prefers shorter labels) or reverse (spreads labels across the alphabet) |
306307
| `--split-word` | | none | `hints` | Split detected text into word-level regions (requires vision strategy) |
307308
| `--zoom-to-depth` | | value | `recursive_grid` | Auto-zoom to the given depth (a non-negative integer) in recursive-grid at the current cursor position |
@@ -321,7 +322,9 @@ nothing.
321322
any of them.
322323
- `--label-direction` is explained under
323324
[Choosing a label direction](CONFIGURATION.md#choosing-a-label-direction).
324-
- `--strategy vision` and `--split-word` are macOS only. See
325+
- `--strategy vision` and `--split-word` work on macOS and Linux, and do
326+
nothing on Windows, which has no element-detection engine. On Linux the
327+
strategy is text-only. See
325328
[Accessibility and hints](CROSS_PLATFORM.md#accessibility-and-hints).
326329

327330
**Where the defaults come from**

0 commit comments

Comments
 (0)