Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
- [x] `[Finally]` section + `later` assert modifier — Send a signal to background processes at file-end and assert exit code + output. `later` keyword defers assert evaluation to file-end. Execution order: entries → later asserts → [Finally] LIFO → @defer LIFO. See #19.
- [x] `[Prompts]` — Interactive prompt/response section: match stdout patterns and send responses via stdin. Pipe-based (no PTY). Supports substring and regex matching, multiplier syntax, ambiguity detection, and unmatched prompt failure. Default timeout 30s.
- [x] `--no-color` — Disable ANSI color codes in output. Also respects `NO_COLOR` env var (https://no-color.org/) and auto-disables when stdout is not a TTY.
- [x] Strict directive validation — reject malformed `@timeout`, `@poll`, `@env` at parse time with line-numbered errors; file-level `@timeout` support [#43](https://github.com/sleipi/cli-t/issues/43)
- [x] `--fail-fast` — Stop execution on the first test failure instead of running all entries
- [x] Refactor `cmd/clitest/` package structure — Extracted display, resolve, filter, vars, and executor logic into dedicated `internal/` packages. Reduced `cmd/clitest/` from 17 to 7 files.
- [x] Linting — Introduced `golangci-lint` with CI integration and resolved all issues
Expand Down Expand Up @@ -48,7 +49,6 @@
- [ ] Register Domain
- [ ] Publish VS Code extension to Marketplace (requires publisher account)
- [ ] Syntax overhaul before 1.0 release [#40](https://github.com/sleipi/cli-t/issues/40)
- [ ] Strict directive validation — reject malformed directives at parse time [#43](https://github.com/sleipi/cli-t/issues/43)

## Won't Do

Expand Down
37 changes: 35 additions & 2 deletions SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -499,7 +499,7 @@ EXIT 0
|-----------|-------|--------|-------------|
| `@group` | file, entry | `@group tag1 tag2 ...` | Space-separated tags for filtering |
| `@skip` | file, entry | `@skip [reason]` | Skip this entry/file (reason optional) |
| `@timeout` | entry | `@timeout MS` | Max time to wait (required for `EXIT NEVER`) |
| `@timeout` | file, entry | `@timeout MS` | Max time to wait; file-level sets default for all entries |
| `@poll` | entry | `@poll MS` | Polling interval for `EXIT NEVER` asserts (default: 100ms) |
| `@defer` | entry | `@defer` | Marks entry as cleanup — runs at file end (LIFO), always, not a test |
| `@workdir` | file, entry | `@workdir <path>` | Run command in specified directory |
Expand Down Expand Up @@ -604,7 +604,7 @@ Sets a real environment variable on the child process. Supported at both file-le

**Scoping:** Entry-level env vars are isolated to that entry only — they do NOT leak to subsequent entries. File-level env vars apply to all entries in the file, including `@defer` entries.

**Values:** The value is everything after the first `=`. Values may contain `=`, spaces, and special characters. Empty values (`@env KEY=`) are valid. Malformed directives without `=` are silently ignored.
**Values:** The value is everything after the first `=`. Values may contain `=`, spaces, and special characters. Empty values (`@env KEY=`) are valid. Malformed directives without `=` are rejected at parse time.

**Variable substitution:** `--var` placeholders (`{{name}}`) in values are expanded (since substitution runs before parsing).

Expand All @@ -628,6 +628,39 @@ EXIT 0
stdout == "false"
```

#### Directive Validation

All directives are validated at parse time. If any directive has an invalid value, clitest reports the error(s) and does not execute the file. Multiple errors may be reported at once.

**Error format:** `<file>:line <N>: @<directive>: <message>`

| Directive | Valid values | Notes |
|-----------|-------------|-------|
| `@timeout` | Integer >= 0 (milliseconds) | `0` means no timeout (infinite wait). No duration suffixes. |
| `@poll` | Integer > 0 (milliseconds) | Must be positive. No duration suffixes. |
| `@env` | `KEY=VALUE` with non-empty key | Empty value after `=` is valid. Missing `=` is rejected. |

Unknown directives are silently ignored.

#### File-level `@timeout`

`@timeout` may appear in frontmatter to set a default timeout for all entries in the file. Entry-level `@timeout` overrides the file-level value.

```
---
@timeout 5000
---

# This entry inherits @timeout 5000
echo "quick"
EXIT 0

# This entry overrides to 10s
@timeout 10000
echo "slow"
EXIT 0
```

### Planned Directives (roadmap)

| Directive | Description |
Expand Down
32 changes: 29 additions & 3 deletions cmd/clitest/run.go
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
package main

import (
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"sync/atomic"

"github.com/sleipi/cli-t/internal/display"
Expand Down Expand Up @@ -219,15 +221,27 @@ func loadAndParse(path string, v map[string]string) (*types.File, error) {
return nil, fmt.Errorf("reading %s: %w", path, err)
}
input := vars.Substitute(string(content), v)
f, err := parser.ParseFile(input)
if err != nil {
return nil, fmt.Errorf("parsing %s: %w", path, err)
f, errs := parser.ParseFile(input)
if len(errs) > 0 {
// Check if these are DirectiveErrors (validation) or structural parse errors
var de *parser.DirectiveError
if errors.As(errs[0], &de) {
// Validation errors: format with file prefix
msgs := make([]string, len(errs))
for i, e := range errs {
msgs[i] = fmt.Sprintf("%s:%s", path, e.Error())
}
return nil, fmt.Errorf("%s", strings.Join(msgs, "\n"))
}
// Structural parse errors: use legacy format
return nil, fmt.Errorf("parsing %s: %w", path, errs[0])
}
f.Path = path
if err := resolveWorkdirs(f); err != nil {
return nil, fmt.Errorf("resolving workdir in %s: %w", path, err)
}
resolveEnvs(f)
resolveTimeouts(f)
return f, nil
}

Expand Down Expand Up @@ -295,3 +309,15 @@ func resolveEnvs(f *types.File) {
}
}
}

// resolveTimeouts inherits file-level @timeout into entries that don't define their own.
func resolveTimeouts(f *types.File) {
if f.Directives.Timeout == nil {
return
}
for i := range f.Entries {
if f.Entries[i].Directives.Timeout == nil {
f.Entries[i].Directives.Timeout = f.Directives.Timeout
}
}
}
16 changes: 10 additions & 6 deletions internal/executor/background.go
Original file line number Diff line number Diff line change
Expand Up @@ -50,13 +50,17 @@ func backgroundEntry(entry types.Entry, cmd string, captures map[string]string)

keepAlive := hasLaterAsserts(entry) || entry.Finally != nil

timeout := entry.Directives.Timeout
if timeout <= 0 {
timeout = DefaultBackgroundTimeout
timeout := DefaultBackgroundTimeout
if entry.Directives.Timeout != nil {
if *entry.Directives.Timeout == 0 {
timeout = 0 // explicitly no timeout
} else {
timeout = *entry.Directives.Timeout
}
}
poll := entry.Directives.Poll
if poll <= 0 {
poll = DefaultPollInterval
poll := DefaultPollInterval
if entry.Directives.Poll != nil {
poll = *entry.Directives.Poll
}

deadline := time.Now().Add(time.Duration(timeout) * time.Millisecond)
Expand Down
26 changes: 14 additions & 12 deletions internal/executor/background_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,16 @@ import (
"github.com/sleipi/cli-t/internal/types"
)

func intPtr(v int) *int { return &v }

func TestBackgroundEntry_PassingAsserts(t *testing.T) {
entry := types.Entry{
Command: `sh -c 'echo ready; sleep 10'`,
ExitNever: true,
Asserts: []types.Assert{{Query: "stdout", Predicate: "contains", Value: "ready"}},
Directives: types.EntryDirectives{
Timeout: 3000,
Poll: 50,
Timeout: intPtr(3000),
Poll: intPtr(50),
},
}
captures := map[string]string{}
Expand All @@ -37,8 +39,8 @@ func TestBackgroundEntry_KeepAliveWithLater(t *testing.T) {
{Query: "stdout", Predicate: "contains", Value: "later_output", Later: true},
},
Directives: types.EntryDirectives{
Timeout: 3000,
Poll: 50,
Timeout: intPtr(3000),
Poll: intPtr(50),
},
}
captures := map[string]string{}
Expand All @@ -59,8 +61,8 @@ func TestBackgroundEntry_KeepAliveWithFinally(t *testing.T) {
Asserts: []types.Assert{{Query: "stdout", Predicate: "contains", Value: "ready"}},
Finally: &types.Finally{Signal: "KILL", ExitCode: 137, Timeout: 1000},
Directives: types.EntryDirectives{
Timeout: 3000,
Poll: 50,
Timeout: intPtr(3000),
Poll: intPtr(50),
},
}
captures := map[string]string{}
Expand All @@ -80,8 +82,8 @@ func TestBackgroundEntry_Timeout(t *testing.T) {
ExitNever: true,
Asserts: []types.Assert{{Query: "stdout", Predicate: "contains", Value: "never_appears"}},
Directives: types.EntryDirectives{
Timeout: 200,
Poll: 50,
Timeout: intPtr(200),
Poll: intPtr(50),
},
}
captures := map[string]string{}
Expand Down Expand Up @@ -110,8 +112,8 @@ func TestBackgroundEntry_UnexpectedExit(t *testing.T) {
ExitNever: true,
Asserts: []types.Assert{{Query: "stdout", Predicate: "contains", Value: "ready"}},
Directives: types.EntryDirectives{
Timeout: 3000,
Poll: 50,
Timeout: intPtr(3000),
Poll: intPtr(50),
},
}
captures := map[string]string{}
Expand Down Expand Up @@ -140,8 +142,8 @@ func TestBackgroundEntry_CapturesStored(t *testing.T) {
Asserts: []types.Assert{{Query: "stdout", Predicate: "contains", Value: "ready"}},
Captures: []types.Capture{{Name: "bgpid", Query: "pid"}},
Directives: types.EntryDirectives{
Timeout: 3000,
Poll: 50,
Timeout: intPtr(3000),
Poll: intPtr(50),
},
}
captures := map[string]string{}
Expand Down
10 changes: 7 additions & 3 deletions internal/executor/executor.go
Original file line number Diff line number Diff line change
Expand Up @@ -30,9 +30,13 @@ func Entry(entry types.Entry, captures map[string]string) Result {

// promptEntry runs a command with interactive prompts.
func promptEntry(entry types.Entry, cmd string, captures map[string]string) Result {
timeout := entry.Directives.Timeout
if timeout <= 0 {
timeout = 30000 // default 30s
timeout := 30000 // default 30s
if entry.Directives.Timeout != nil {
if *entry.Directives.Timeout == 0 {
timeout = 0 // explicitly no timeout
} else {
timeout = *entry.Directives.Timeout
}
}

prompts := make([]runner.PromptDef, len(entry.Prompts))
Expand Down
2 changes: 1 addition & 1 deletion internal/executor/executor_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,7 @@ func TestEntry_PromptResponds(t *testing.T) {
Prompts: []types.Prompt{
{Pattern: "Enter name:", IsRegex: false, Response: "Alice", Repeat: 1},
},
Directives: types.EntryDirectives{Timeout: 3000},
Directives: types.EntryDirectives{Timeout: func() *int { v := 3000; return &v }()},
}
captures := map[string]string{}
er := Entry(entry, captures)
Expand Down
Loading