survey: Desktop test automation orchestrator

[aoirint]

Warning

This issue was created with assistance from LLMs.

Summary

• Record an anonymized investigation into a generic desktop test automation orchestrator.
• The planned system targets Windows desktop applications, games, GUI tools, CLI tools, and mod-capable applications, while avoiding any dependency on a specific product, user, host, IP address, or private project.
• This issue is an investigation record only; no remaining implementation work is tracked here.

Goal

• Design a reusable test foundation that can automatically launch, operate, observe, collect artifacts from, analyze, and report on desktop software in local or remote Windows environments.
• Keep the model generic enough for multiple application families while still supporting practical desktop automation concerns such as process checks, screenshots, logs, deployment, teardown, and artifact collection.
• Make configuration strict and modular so automated code-assistance tools can safely edit small parts of the system without rewriting the whole orchestration layer.

Core Design Principles

• Use a Test Plan as the central configuration unit.
• Separate the execution environment from the test target.
• Treat local execution and remote execution through the same abstraction.
• Represent targets as a set of traits instead of a single fixed application type.
• Model scenarios as phase-based lifecycles rather than a flat list of steps.
• Keep interactions and observations separate.
• Store run outputs through an artifact manifest.
• Prefer rule-based analysis for standard pass, warning, and failure conditions.
• Centralize extension points through a plugin registry.
• Keep schemas strict enough for safe automated editing and review.

Test Plan Model

A Test Plan answers the following questions:

• Where should the test run?
• What target should be tested?
• How should the environment be prepared?
• How should the target be launched?
• How should the target be operated?
• What should be observed?
• What artifacts should be collected?
• How should results be judged?
• How should the run be reported?

The plan is composed from these major areas:

• Executor: environment and command/file operation abstraction.
• Target: application profile and traits.
• Scenario: lifecycle phases and steps.
• Artifact Collection: files, logs, screenshots, dumps, and process data.
• Analysis: rules that evaluate collected observations and artifacts.
• Report: human-readable and machine-readable summaries.

Architecture Outline

• A user or automation assistant invokes a Python CLI/API.
• The CLI/API loads and validates a Test Plan.
• The plan combines executor, target, scenario, artifact collection, analysis, and reporting configuration.
• The runner executes scenario phases through the selected executor.
• Observations and artifacts are written into a run-specific artifact directory.
• Analyzers evaluate the collected outputs.
• Reporters generate Markdown, JSON, and potentially other report formats.

Executor Design

The Executor abstracts where and how commands and file operations run. It should not know about specific application ecosystems, engines, stores, plugin loaders, or mod systems.

Planned executor types:

• local_windows
• ssh_windows
• Future candidates: hypervisor_windows, self_hosted_runner

Initial implementation focus:

• local_windows
• ssh_windows

Expected executor capabilities:

• run(command)
• run_powershell(script)
• upload(local_path, remote_path)
• download(remote_path, local_path)
• exists(path)
• mkdir(path)
• read_file(path)
• write_file(path, content)
• list_processes()
• kill_process(name)
• capture_screenshot()
• resolve_path(path)

Target Design

The Target describes the application under test. It should define identity and launch information, but should not contain executor-specific behavior or analysis logic.

Target identity fields:

• name
• version
• process_name

Target launch fields:

• command
• working_dir
• env

Target traits are composable booleans or structured capabilities, for example:

• windows_gui
• steam
• unity
• unreal
• electron
• dotnet
• modded
• plugin_loader

Traits can be used to attach default log paths, default artifact collectors, and default analysis rules without hard-coding behavior into the executor.

Scenario Design

A Scenario represents a lifecycle with explicit phases. The plan avoids treating scenarios as a single flat step list because desktop tests need reliable preparation, launch, observation, collection, analysis, and cleanup behavior.

Planned phases:

• setup
• prepare
• deploy
• pre_launch
• launch
• interact
• observe
• collect
• analyze
• teardown

Scenario step controls should include:

• always
• continue_on_error
• timeout_seconds
• retry

Important lifecycle rule:

• teardown must still run when earlier phases fail, so launched processes and temporary state do not remain behind.

Interaction And Observation

Scenario steps are split into two conceptual groups.

Interaction steps act on the environment or target:

• launch_target
• terminate_process
• run_powershell
• run_script
• send_keys
• click
• run_automation_script
• upload
• download

Observation steps inspect the environment or target state:

• wait_for_process
• wait_for_window
• wait_for_log_pattern
• screenshot
• check_file_exists
• sample_cpu_memory
• check_exit_code
• collect_process_snapshot

This separation is intended to make later additions such as GUI automation, image checks, log monitoring, OCR, and external visual automation integrations easier to add without mixing concerns.

Artifact Model

Each test run should create a unique run directory under an artifacts area. A manifest.json should be the durable index for everything collected or produced by the run.

Representative artifact layout:

artifacts/
  runs/
    <run-id>/
      manifest.json
      report.md
      report.json
      stdout.txt
      stderr.txt
      screenshots/
      logs/
      dumps/
      process/

The manifest should record at least:

• run ID
• status
• executor metadata
• target metadata
• start/end timestamps
• duration
• artifact references
• analysis summary
• applied analysis rules

Example artifact types:

• screenshots
• application logs
• stdout/stderr
• crash dumps
• process snapshots
• generated Markdown/JSON reports

Analyzer Design

Analyzers evaluate observations and artifacts and produce pass, warning, or failure results. Standard checks should be declarative and rule-based where possible, while unusual checks can be added through plugins.

Planned standard analyzers:

• log_pattern
• file_exists
• file_count
• exit_code
• process_lifetime
• screenshot_basic
• json_assert
• custom_python

Representative analysis rules:

• Fail when fatal or unhandled-exception log patterns appear.
• Pass only if a target process remains alive for a minimum duration.
• Fail when crash dump files are produced.
• Check exit codes for CLI-style targets.
• Re-run analysis against an existing artifact directory without repeating the test run.

Plugin Registry

Extension points should be registered in a central plugin registry so the core runner can remain small and stable.

Plugin categories:

• executor
• step
• target_trait
• artifact_collector
• analyzer
• reporter

Examples of built-in registrations:

• executors: local_windows, ssh_windows
• steps: wait, launch_target, screenshot, run_powershell, terminate_process
• analyzers: log_pattern, file_count, process_lifetime, exit_code
• reporters: markdown, json, junit

The intended benefit is that new execution environments, target traits, scenario steps, artifact collectors, analyzers, and report formats can be added without broad changes to the core runner.

Configuration Shape

Configuration should be split into reusable files and composed by plan files.

Representative layout:

configs/
  executors/
    local.toml
    sidecar-example.toml
  targets/
    example-windows-app.toml
    example-steam-app.toml
    example-electron-app.toml
  scenarios/
    smoke.toml
    launch-only.toml
    gui-basic.toml
  analysis/
    generic-gui-app.toml
    generic-cli-app.toml
  plans/
    example-local.toml
    example-sidecar.toml

A plan file should include reusable executor, target, scenario, and analysis configuration. This allows the same target and scenario to run locally or remotely by swapping executor configuration.

CLI Shape

Planned CLI commands:

• dto validate --plan <plan>: validate configuration files.
• dto doctor --executor <executor>: check executor prerequisites.
• dto run --plan <plan>: execute a full test plan.
• dto analyze --run <run-id>: re-run analysis against existing artifacts.
• dto report --run <run-id>: regenerate reports from a completed run.
• dto list-runs: list previous runs.
• dto show-run <run-id>: inspect one run.

Internal Model

Representative internal models:

• RunContext: current run ID, loaded plan, selected executor, artifact store, variables, and logger.
• StepResult: step ID, step type, status, timestamps, message, and produced artifacts.
• RunResult: run ID, final status, step results, artifact references, and analysis results.

These models provide a stable boundary between the runner, steps, analyzers, artifact storage, and reporters.

Suggested Repository Layout

Representative implementation layout:

desktop-test-orchestrator/
  pyproject.toml
  README.md
  configs/
  scenarios/
  src/
    dto/
      cli.py
      config_loader.py
      schema.py
      registry.py
      core/
      executors/
      steps/
      targets/
      artifacts/
      analyzers/
      reporters/
      plugins/
  tests/
    unit/
    integration/
    fixtures/
  artifacts/
    .gitkeep

Roadmap Captured In The Plan

• Milestone 0: project skeleton, CLI, include loading, schemas, plugin registry, run context/results, artifact manifest.
• Milestone 1: local Windows executor, wait/run PowerShell/launch/terminate/screenshot/wait-for-process steps, Markdown/JSON reporting.
• Milestone 2: SSH Windows executor, upload/download, remote work directory handling, remote screenshot/process list, doctor command.
• Milestone 3: artifact collection and analyzers, including glob collection, log patterns, file counts, process lifetime, exit codes, and manifest improvements.
• Milestone 4: deployment-oriented steps such as upload, download, clean directory, copy artifacts, and build output placement.
• Milestone 5: GUI automation steps such as automation scripts, key input, click, and window waiting.
• Milestone 6: target traits for common runtime/application families and plugin-loader style capabilities.
• Milestone 7: advanced observation such as screenshot analysis, image similarity, OCR, and external visual automation integration.

Initial Scope

The first useful implementation should include:

• configuration schema
• plugin registry
• local Windows executor
• SSH Windows executor
• phase-based scenario model
• artifact manifest
• Markdown report
• log-pattern analyzer

Explicit Non-Goals For The Initial Work

The plan intentionally defers:

• GUI application frontend
• complex image recognition
• distributed scheduling
• large custom DSL
• broad target-trait catalog
• advanced visual automation-agent integration

Privacy Notes

The investigation intentionally avoids including:

• specific game names
• specific commercial application names
• personal names
• usernames
• email addresses
• real IP addresses
• real hostnames
• personal local paths
• API keys
• SSH private keys
• internal organization names
• unpublished project names

Examples should use placeholders such as:

• Example Application
• ExampleApp.exe
• C:\Path\To\ExampleApp
• sidecar-example
• <run-id>

Verification

• Reviewed the supplied sanitized plan and expanded it into an anonymized issue summary.
• No code changes, reproduction steps, or test runs were performed.

Outcome

• The desktop test automation orchestrator investigation is recorded in enough detail to understand the planned architecture, configuration model, extensibility model, artifact strategy, analyzer strategy, and staged roadmap.
• No follow-up task is currently requested or tracked by this issue.