Skip to content

Commit 015e742

Browse files
authored
fix: six real bugs found by the full-feature test campaign; release 0.12.1 (#242)
A 30+-test suite exercising every ProbeScript feature against a real app on both a fresh Android emulator and a fresh iOS simulator (AI family via a local LM Studio Gemma 4 model) surfaced six genuine bugs: 1. toggle was a double no-op: the agent's toggle DeviceAction has always been 'break;', and the parser rejected #id selectors, leaving them to misparse as junk recipe calls. toggle now parses a full selector and dispatches it as a real tap. 2. A recipe named with a filler word (recipe "add and verify") was unreachable: stripping applied to call names only. Both sides now normalize identically. 3. Android set location never worked: it ran the emulator-console command 'emu' inside the device shell. Now uses adb emu correctly. 4. iOS permission verbs hung the session: simctl privacy grant silently terminates the app while the WS lingers half-open. All four permission verbs now eagerly relaunch + reconnect on iOS simulators. 5. wait N seconds was an agent RPC, so kill-the-app followed by a wait burned the step timeout in doomed reconnects. Duration waits now sleep CLI-side. 6. close keyboard printed 'close the app' (cosmetic). All fixes live-verified against real devices, each with regression tests. Release 0.12.1 per the versioning policy.
1 parent 78bfe7d commit 015e742

14 files changed

Lines changed: 278 additions & 18 deletions

File tree

CHANGELOG.md

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,37 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
66

77
## [Unreleased]
88

9+
## [0.12.1] - 2026-08-15
10+
11+
### Fixed
12+
- **`toggle` never actually toggled anything, and `toggle #id` didn't even parse.** Found by the
13+
full-feature test campaign: the agent's `toggle` DeviceAction has always been a no-op
14+
(`case 'toggle': break;`), so every `toggle` step "passed" without touching the device — and the
15+
parser only accepted a bare ident or quoted string, leaving `#id` selectors dangling to misparse
16+
as a junk recipe call (the PT-26/R-5 class). `toggle` now parses a full selector (text, `#id`,
17+
ordinal) and dispatches it as a real tap — which is how a Switch actually toggles.
18+
- **A recipe whose own name contains a filler word (e.g. `recipe "add and verify"`) was
19+
unreachable by its exact written name.** Filler stripping was applied to the call name only —
20+
`add and verify "x"` stripped to "add verify", which matched nothing because the definition
21+
kept its "and". Both sides now normalize identically before matching.
22+
- **Android `set location` had never worked.** It ran `adb shell emu geo fix ...` — but `emu` is
23+
an emulator *console* command (`adb emu ...`), not a device-shell binary, so every call failed
24+
with "/system/bin/sh: emu: inaccessible or not found". Live-verified fixed against a real
25+
emulator.
26+
- **iOS permission verbs left the session hanging until the step timeout.** `simctl privacy
27+
grant/revoke/reset` silently terminates the target app (confirmed live via launchctl), but the
28+
WebSocket lingered half-open with no error, so the next RPC hung for the full step timeout
29+
instead of failing fast enough to auto-reconnect. All four permission verbs
30+
(`allow`/`deny`/`grant all`/`revoke all`) now eagerly relaunch and reconnect on iOS simulators,
31+
mirroring `restart the app`. Live-verified: grant → wait → assert now completes in ~4s.
32+
- **`wait N seconds` round-tripped through the agent as an RPC**, so `kill the app` followed by
33+
any duration wait hit the dead connection and burned the whole step timeout in doomed reconnect
34+
attempts (through an adb forward, "nothing listening" surfaces as accept-then-EOF rather than
35+
ECONNREFUSED, so the PT-18 relaunch heuristic never fired either). Duration waits now sleep
36+
CLI-side — waiting for wall-clock time needs no device. Live-verified: kill → wait → open now
37+
completes in ~20s end-to-end.
38+
- **`close keyboard` printed "close the app" in progress output** (cosmetic).
39+
940
## [0.12.0] - 2026-08-15
1041

1142
### Added

VERSION

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
0.12.0
1+
0.12.1

docs/wiki/Home.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ Welcome to the FlutterProbe wiki. This documentation covers architecture details
1818

1919
## Project Status
2020

21-
FlutterProbe is in active development. Current version: **0.12.0**.
21+
FlutterProbe is in active development. Current version: **0.12.1**.
2222

2323
### Repository Structure
2424

internal/parser/parser.go

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -833,12 +833,16 @@ func (p *Parser) parseActionToggle() (Step, error) {
833833
line := p.peek().Line
834834
p.advance() // toggle
835835
p.skipFillers()
836-
name := ""
837-
if p.peek().Type == TOKEN_IDENT || p.peek().Type == TOKEN_STRING {
838-
name = p.advance().Literal
839-
}
836+
// Full-feature campaign finding: this used to accept only a bare ident or
837+
// quoted string into Name — `toggle #id` left the #id token dangling to
838+
// misparse as a junk recipe call (the PT-26/R-5 dangling-token class),
839+
// and Name was then sent to the agent's `toggle` DeviceAction, which is
840+
// a no-op — so even the accepted forms never actually toggled anything.
841+
// Parse a real selector (text, #id, ordinal, ...) instead; the executor
842+
// dispatches it as a tap, which is how a Switch actually toggles.
843+
sel := p.parseSelector()
840844
p.consumeNewline()
841-
return ActionStep{Verb: VerbToggle, Name: name, Line: line}, nil
845+
return ActionStep{Verb: VerbToggle, Sel: &sel, Line: line}, nil
842846
}
843847

844848
// ---- Assert ----

internal/parser/parser_test.go

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2148,3 +2148,39 @@ test "signal"
21482148
t.Errorf("value: got %q, want %q", a1.Text, "abc123")
21492149
}
21502150
}
2151+
2152+
// ---- Full-feature campaign findings (2026-08-15) ----
2153+
2154+
// TestParser_ToggleByID covers the campaign finding that `toggle #id` left
2155+
// the #id token dangling to misparse as a junk recipe call (the PT-26/R-5
2156+
// dangling-token class) — toggle now parses a real selector.
2157+
func TestParser_ToggleByID(t *testing.T) {
2158+
src := `test "t"
2159+
toggle #unit_system_toggle
2160+
`
2161+
prog := mustParse(t, src)
2162+
assertStepCount(t, prog.Tests[0].Body, 1)
2163+
a := firstAction(t, prog.Tests[0].Body)
2164+
if a.Verb != parser.VerbToggle {
2165+
t.Errorf("verb: got %q, want toggle", a.Verb)
2166+
}
2167+
if a.Sel == nil || a.Sel.Kind != parser.SelectorID || a.Sel.Text != "#unit_system_toggle" {
2168+
t.Errorf("selector: got %+v, want ID selector #unit_system_toggle", a.Sel)
2169+
}
2170+
}
2171+
2172+
// TestParser_ToggleByText is the regression guard for the pre-existing
2173+
// quoted-text form.
2174+
func TestParser_ToggleByText(t *testing.T) {
2175+
src := `test "t"
2176+
toggle "Dark Mode"
2177+
`
2178+
prog := mustParse(t, src)
2179+
a := firstAction(t, prog.Tests[0].Body)
2180+
if a.Verb != parser.VerbToggle {
2181+
t.Errorf("verb: got %q, want toggle", a.Verb)
2182+
}
2183+
if a.Sel == nil || a.Sel.Text != "Dark Mode" {
2184+
t.Errorf("selector: got %+v, want text Dark Mode", a.Sel)
2185+
}
2186+
}

internal/runner/device_context.go

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -532,7 +532,12 @@ func (dc *DeviceContext) SetLocation(ctx context.Context, lat, lng string) error
532532
fmt.Printf(" \033[36m📍\033[0m Setting location to %s, %s\n", lat, lng)
533533
switch dc.Platform {
534534
case device.PlatformAndroid:
535-
if _, err := dc.Manager.ADB().Shell(ctx, dc.Serial, "emu", "geo", "fix", lng, lat); err != nil {
535+
// Campaign finding: this used to run `adb shell emu geo fix ...` —
536+
// but `emu` is an *emulator console* command (`adb emu ...`, no
537+
// shell), not a binary that exists in the device shell, so Android
538+
// set-location had never actually worked: every call failed with
539+
// "/system/bin/sh: emu: inaccessible or not found".
540+
if _, err := dc.Manager.ADB().Run(ctx, dc.Serial, "emu", "geo", "fix", lng, lat); err != nil {
536541
return fmt.Errorf("set location: %w", err)
537542
}
538543
case device.PlatformIOS:

internal/runner/dispatch_step_test.go

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -202,3 +202,81 @@ func TestDispatchStep_ActionAndAssert(t *testing.T) {
202202
t.Errorf("assert executions: seeCalls=%d, want 1", client.seeCalls)
203203
}
204204
}
205+
206+
// ---- Full-feature campaign findings (2026-08-15) ----
207+
208+
// TestRunStep_ToggleDispatchesAsTap covers the campaign finding that the
209+
// agent's `toggle` DeviceAction was a silent no-op — toggle now dispatches
210+
// its selector as a real tap (a Switch toggles on tap).
211+
func TestRunStep_ToggleDispatchesAsTap(t *testing.T) {
212+
client := &scriptedClient{fakeAIClient: &fakeAIClient{}, tapAlwaysOK: true}
213+
e := newScriptedExecutor(client)
214+
215+
step := parser.ActionStep{Verb: parser.VerbToggle, Sel: &parser.Selector{Kind: parser.SelectorID, Text: "#unit_system_toggle"}}
216+
if err := e.RunStep(context.Background(), step); err != nil {
217+
t.Fatalf("unexpected error: %v", err)
218+
}
219+
if client.tapCalls != 1 {
220+
t.Errorf("toggle should dispatch exactly one tap, got %d", client.tapCalls)
221+
}
222+
}
223+
224+
// TestRunRecipeCall_DefinitionWithFillerWordMatches covers the campaign
225+
// finding that filler-word stripping was applied to the call name only:
226+
// `recipe "add and verify"` was unreachable because the call stripped its
227+
// "and" while the definition kept it. Both sides now normalize the same way.
228+
func TestRunRecipeCall_DefinitionWithFillerWordMatches(t *testing.T) {
229+
client := &scriptedClient{fakeAIClient: &fakeAIClient{}, tapAlwaysOK: true}
230+
e := newScriptedExecutor(client)
231+
e.RegisterRecipe(parser.RecipeDef{
232+
Name: "add and verify",
233+
Params: []string{"label"},
234+
Body: []parser.Step{
235+
parser.ActionStep{Verb: parser.VerbTap, Sel: &parser.Selector{Kind: parser.SelectorText, Text: "X"}},
236+
},
237+
})
238+
239+
// The parser turns `add and verify "250 ml"` into this call shape.
240+
call := parser.RecipeCall{Name: "add and verify <arg>", Args: []string{"250 ml"}}
241+
if err := e.RunStep(context.Background(), call); err != nil {
242+
t.Fatalf("recipe with filler word in its definition name should be reachable: %v", err)
243+
}
244+
if client.tapCalls != 1 {
245+
t.Errorf("recipe body should have executed, tapCalls=%d", client.tapCalls)
246+
}
247+
}
248+
249+
// TestStepDescription_CloseKeyboard covers the cosmetic campaign finding
250+
// that `close keyboard` printed "close the app" in progress output.
251+
func TestStepDescription_CloseKeyboard(t *testing.T) {
252+
e := newScriptedExecutor(&scriptedClient{fakeAIClient: &fakeAIClient{}})
253+
kb := e.stepDescription(parser.ActionStep{Verb: parser.VerbClose, Name: "keyboard"})
254+
if kb != "close keyboard" {
255+
t.Errorf("close keyboard description: got %q", kb)
256+
}
257+
app := e.stepDescription(parser.ActionStep{Verb: parser.VerbClose})
258+
if app != "close the app" {
259+
t.Errorf("close app description: got %q", app)
260+
}
261+
}
262+
263+
// TestRunWait_DurationIsCLISide covers the campaign finding that
264+
// `wait N seconds` round-tripped through the agent: after `kill the app`,
265+
// the wait hit the dead connection and burned the step timeout in doomed
266+
// reconnect attempts. Duration waits now sleep CLI-side — no RPC at all.
267+
func TestRunWait_DurationIsCLISide(t *testing.T) {
268+
client := &scriptedClient{fakeAIClient: &fakeAIClient{}}
269+
e := newScriptedExecutor(client)
270+
271+
start := time.Now()
272+
err := e.RunStep(context.Background(), parser.WaitStep{Kind: parser.WaitDuration, Duration: 0.2})
273+
if err != nil {
274+
t.Fatalf("unexpected error: %v", err)
275+
}
276+
if elapsed := time.Since(start); elapsed < 180*time.Millisecond {
277+
t.Errorf("duration wait returned too fast (%v) — did it actually sleep?", elapsed)
278+
}
279+
if client.waitRPCs != 0 {
280+
t.Errorf("duration wait must not call the agent, got %d Wait RPCs", client.waitRPCs)
281+
}
282+
}

internal/runner/executor.go

Lines changed: 101 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ import (
1818

1919
"github.com/alphawavesystems/flutter-probe/internal/ai"
2020
"github.com/alphawavesystems/flutter-probe/internal/config"
21+
"github.com/alphawavesystems/flutter-probe/internal/device"
2122
"github.com/alphawavesystems/flutter-probe/internal/parser"
2223
"github.com/alphawavesystems/flutter-probe/internal/probelink"
2324
"github.com/alphawavesystems/flutter-probe/internal/redact"
@@ -407,6 +408,11 @@ func (e *Executor) stepDescription(step parser.Step) string {
407408
case parser.VerbOpen:
408409
return "open the app"
409410
case parser.VerbClose:
411+
// Campaign finding: `close keyboard` used to print "close the
412+
// app" in progress output — Name distinguishes the two forms.
413+
if s.Name == "keyboard" {
414+
return "close keyboard"
415+
}
410416
return "close the app"
411417
case parser.VerbGoBack:
412418
return "go back"
@@ -439,6 +445,11 @@ func (e *Executor) stepDescription(step parser.Step) string {
439445
if s.Sel != nil {
440446
return fmt.Sprintf("long press %q", s.Sel.Text)
441447
}
448+
case parser.VerbToggle:
449+
if s.Sel != nil {
450+
return fmt.Sprintf("toggle %q", s.Sel.Text)
451+
}
452+
return fmt.Sprintf("toggle %q", s.Name)
442453
case parser.VerbKill:
443454
return "kill the app"
444455
case parser.VerbCopyClipboard:
@@ -632,7 +643,19 @@ func (e *Executor) runAction(ctx context.Context, a parser.ActionStep) error {
632643
return e.client.DeviceAction(ctx, "rotate", a.Name)
633644

634645
case parser.VerbToggle:
635-
return e.client.DeviceAction(ctx, "toggle", a.Name)
646+
// Campaign finding: the agent's `toggle` DeviceAction has always
647+
// been a no-op (`case 'toggle': break;` in executor.dart) — every
648+
// `toggle` step "passed" without touching the device. A Switch
649+
// toggles on tap, so dispatch the parsed selector as a real tap.
650+
// Name is the legacy pre-selector field, kept as a text-selector
651+
// fallback for any step built programmatically the old way.
652+
if a.Sel != nil {
653+
return e.client.Tap(ctx, toSelectorParam(e.resolveSelector(*a.Sel)))
654+
}
655+
if a.Name != "" {
656+
return e.client.Tap(ctx, probelink.SelectorParam{Kind: "text", Text: e.resolve(a.Name)})
657+
}
658+
return fmt.Errorf("toggle: missing selector at line %d", a.Line)
636659

637660
case parser.VerbShake:
638661
return e.client.DeviceAction(ctx, "shake", "")
@@ -765,25 +788,37 @@ func (e *Executor) runAction(ctx context.Context, a parser.ActionStep) error {
765788
// Cloud mode: permissions auto-granted via Appium capabilities
766789
return nil
767790
}
768-
return e.deviceCtx.AllowPermission(ctx, a.Name)
791+
if err := e.deviceCtx.AllowPermission(ctx, a.Name); err != nil {
792+
return err
793+
}
794+
return e.relaunchAfterIOSPermissionChange(ctx)
769795

770796
case parser.VerbDenyPermission:
771797
if e.deviceCtx == nil {
772798
return nil
773799
}
774-
return e.deviceCtx.DenyPermission(ctx, a.Name)
800+
if err := e.deviceCtx.DenyPermission(ctx, a.Name); err != nil {
801+
return err
802+
}
803+
return e.relaunchAfterIOSPermissionChange(ctx)
775804

776805
case parser.VerbGrantAllPerms:
777806
if e.deviceCtx == nil {
778807
return nil
779808
}
780-
return e.deviceCtx.GrantAllPermissions(ctx)
809+
if err := e.deviceCtx.GrantAllPermissions(ctx); err != nil {
810+
return err
811+
}
812+
return e.relaunchAfterIOSPermissionChange(ctx)
781813

782814
case parser.VerbRevokeAllPerms:
783815
if e.deviceCtx == nil {
784816
return nil
785817
}
786-
return e.deviceCtx.RevokeAllPermissions(ctx)
818+
if err := e.deviceCtx.RevokeAllPermissions(ctx); err != nil {
819+
return err
820+
}
821+
return e.relaunchAfterIOSPermissionChange(ctx)
787822

788823
case parser.VerbKill:
789824
if e.deviceCtx == nil {
@@ -900,6 +935,36 @@ func (e *Executor) runAction(ctx context.Context, a parser.ActionStep) error {
900935
return fmt.Errorf("unknown action verb %q at line %d", a.Verb, a.Line)
901936
}
902937

938+
// relaunchAfterIOSPermissionChange restores the session after any of the
939+
// permission verbs runs on an iOS simulator. Campaign finding (full-feature
940+
// run, 2026-08-15): `simctl privacy grant/revoke/reset` silently TERMINATES
941+
// the target app — confirmed live via launchctl after a grant — but the
942+
// CLI's WebSocket can linger half-open with no error, so the *next* RPC
943+
// hangs until its step timeout instead of failing fast enough to trigger
944+
// auto-reconnect. Relaunch and reconnect eagerly instead, mirroring what
945+
// `restart the app` already does. No-op on Android and physical iOS (their
946+
// permission paths don't kill the app).
947+
func (e *Executor) relaunchAfterIOSPermissionChange(ctx context.Context) error {
948+
dc := e.deviceCtx
949+
if dc == nil || dc.Platform != device.PlatformIOS || dc.IsPhysical {
950+
return nil
951+
}
952+
e.client.Close()
953+
if err := dc.RestartApp(ctx); err != nil {
954+
return fmt.Errorf("relaunch after permission change: %w", err)
955+
}
956+
newClient, err := dc.Reconnect(ctx)
957+
if err != nil {
958+
return fmt.Errorf("reconnect after permission change: %w", err)
959+
}
960+
e.client = newClient
961+
e.clientGen.Add(1)
962+
if e.onReconnect != nil {
963+
e.onReconnect(newClient)
964+
}
965+
return nil
966+
}
967+
903968
// runAssertNative handles `see native "..."` / `don't see native "..."`:
904969
// a native (non-Flutter) UI element matched by uiautomator's text or
905970
// resource-id, dispatched through DeviceContext instead of the Dart agent.
@@ -1097,8 +1162,23 @@ func redactSelector(raw string) parser.Selector {
10971162
// ---- Wait execution ----
10981163

10991164
func (e *Executor) runWait(ctx context.Context, w parser.WaitStep) error {
1165+
// Campaign finding: a plain `wait N seconds` used to round-trip through
1166+
// the agent as an RPC — so `kill the app` followed by `wait 2 seconds`
1167+
// hit the dead connection, triggered auto-reconnect against an
1168+
// intentionally-killed app, and burned the whole step timeout (through
1169+
// an adb forward, "nothing listening" surfaces as accept-then-EOF, not
1170+
// ECONNREFUSED, so PT-18's relaunch heuristic never fired either).
1171+
// Waiting for wall-clock time needs no device at all — sleep CLI-side.
1172+
if w.Kind == parser.WaitDuration {
1173+
select {
1174+
case <-time.After(time.Duration(w.Duration * float64(time.Second))):
1175+
return nil
1176+
case <-ctx.Done():
1177+
return ctx.Err()
1178+
}
1179+
}
1180+
11001181
kindStr := map[parser.WaitKind]string{
1101-
parser.WaitDuration: "duration",
11021182
parser.WaitAppears: "appears",
11031183
parser.WaitDisappears: "disappears",
11041184
parser.WaitPageLoad: "page_load",
@@ -1216,6 +1296,21 @@ func (e *Executor) runRecipeCall(ctx context.Context, rc parser.RecipeCall) erro
12161296
stripped = stripRecipeCallArgs(rc.Name)
12171297
recipe, ok = e.recipes[stripped]
12181298
}
1299+
if !ok {
1300+
// Campaign finding: stripping was applied to the CALL name only,
1301+
// never the DEFINITION name — so a recipe whose own name contains a
1302+
// filler word, e.g. `recipe "add and verify" (x)`, was unreachable
1303+
// by its exact written name: the call `add and verify "v"` parses as
1304+
// "add and verify <arg>", strips to "add verify", and "add verify"
1305+
// matches nothing because the definition kept its "and". Normalize
1306+
// both sides the same way before comparing.
1307+
for defName, def := range e.recipes {
1308+
if stripRecipeCallArgs(defName) == stripped {
1309+
recipe, ok = def, true
1310+
break
1311+
}
1312+
}
1313+
}
12191314
if !ok {
12201315
// PT-02(a): an unrecognized recipe call used to silently no-op ("may
12211316
// be a filler line"), which masked genuine typos and broken recipe

0 commit comments

Comments
 (0)