OSWorld Experiment Manager

English | 한국어

A cross-platform Python/ttkbootstrap desktop GUI for configuring, queuing, monitoring, validating, comparing, and reporting OSWorld experiments, with provider-aware API cost tracking.

English interface in the Dark theme. The same workspace can switch to Korean without restarting.

Highlights

• Generate safe OSWorld commands from text, dropdown, numeric, Boolean, file, and folder controls.
• Queue multiple experiments and monitor live completed/total and success/fail/total task counts.
• Preserve the OSWorld checkout and Python interpreter per Queue item; legacy entries infer their original checkout from the result path when possible.
• Browse, search, filter, and multi-select the 369 tasks registered in the official OSWorld task set without entering UUIDs manually.
• Pause after the current experiment, edit the pending Queue, and resume from the updated list.
• Run a configurable smoke-test subset before the full experiment and stop the Queue only when rule-based checks detect an execution-blocking failure.
• Apply paper-aligned or OSWorld-default model presets while keeping every option manually editable.
• Store OpenAI, Google, OpenRouter, LM Studio, and RunPod credentials in the operating system's secure credential store.
• Check provider connectivity and load the server's live model list directly into the editable model dropdown.
• Track per-task and cumulative API costs with actual/estimated labels, saved price snapshots, manual overrides, and optional experiment budgets.
• Export rule-based result analysis as styled Word reports.
• Save a versioned, secret-free experiment record during every run and reconstruct comparable records from existing result folders.
• Browse saved runs in Analysis / Report, filter the history, and inspect one experiment's environment, task outcomes, commands, and errors.
• Send completion or failure notifications through cross-platform SMTP email, with optional macOS Notification Center and Messages integrations.
• Switch the interface between English and Korean without losing the current form values, Queue, or console history.
• Switch between the Nord Light and Dark themes without restarting the application; the selected theme is restored on the next launch.
• Save portable, secret-free settings profiles and choose a persistent startup default.
• Launch without a preinstalled OSWorld checkout, then locate an existing copy or clone and prepare the official repository from OSWorld Setup.

Repository layout

The GUI is maintained independently at:

./OSWorld-gui-tool

OSWorld-gui-tool/
├── docs/screenshots/
├── pyproject.toml
├── run_gui.py
├── src/osworld_gui/
└── tests/

Standalone setup

OSWorld does not need to be installed before the manager starts. Use Python 3.10 or newer with Tk support, then install the standalone GUI:

git clone https://github.com/issumer-ku/OSWorld-gui-tool.git
cd ./OSWorld-gui-tool
python -m pip install -e .
osworld-gui

You can also run the source checkout without installing it:

cd ./OSWorld-gui-tool
python run_gui.py

To select an OSWorld checkout explicitly:

OSWORLD_ROOT=../OSWorld osworld-gui

Use a Python distribution with Tk support if tkinter is unavailable in your current environment.

On first launch, the manager automatically opens OSWorld Setup when no valid checkout is configured. Enter a path, use Auto, or choose Install OSWorld. The installer clones the official xlang-ai/OSWorld repository, creates an isolated Python 3.10 environment, and installs requirements.txt there. Experiments automatically use that interpreter even when the GUI itself runs on a newer Python release. Clone and dependency-install output is streamed to Debug Console. VMware, Docker, and other host runtimes remain separate OSWorld prerequisites.

Platform support

The manager itself is not macOS-only. Its core Python/ttkbootstrap features run on macOS, Windows, and desktop Linux, subject to the OSWorld runner and virtualization backend being available on that host.

Capability	macOS	Windows	Linux
GUI, Queue, debug console, reports	Yes	Yes	Yes
Secure credential storage	Keychain	Credential Locker	Secret Service / system keyring
SMTP email notifications	Yes	Yes	Yes
Native desktop notification	Notification Center	Not currently integrated	Not currently integrated
Messages / iMessage	Yes	No	No

Linux credential storage requires a usable Secret Service or other keyring backend in the desktop session. Native Windows/Linux notification integrations can be added independently; they are not required for experiment execution.

Typical workflow

1. If OSWorld is not connected, open OSWorld Setup to locate an existing checkout or install the official repository. The selected path is restored on later launches.
2. Open API / Secrets, select a provider from the dropdown, and save its API key and optional Base URL.
3. In Experiment Setup / Queue, choose a model preset and adjust the environment, model, task, agent, and execution options. Every preset starts with VMware as the local desktop environment.
4. Select the VM, task metadata, configuration, and result paths with the file and folder dialogs.
5. Add the experiment to the Queue. Double-click an item, or use Load / edit selected, to revise it.
6. Review the generated command with Preview command.
7. Start the Queue and follow task progress in the table or the live Debug Console. While an experiment is running, Run Queue becomes Pause. Pausing lets the current experiment finish before the manager waits; Resume then re-reads the current Queue and continues with the latest pending entries, including edits, additions, and removals.
8. The manager first runs the configured smoke-test tasks. It stops the Queue and sends a notification if it finds a fatal failure.
9. When all remaining tasks finish, the manager can quietly save a styled Word report without opening an external application. Control this per Queue item with Auto-Generate Report under Experiment.

Settings profiles and startup defaults

The toolbar above Experiment provides Save Settings File, Load Settings File, Reset, and Set Current as Default. Reset restores the experiment form, runner options, additional JSON, notification inputs, API entry fields, and debug-console font size to program defaults. It clears an explicit startup default but deliberately preserves Queue entries and API keys in the operating-system credential store. A portable JSON profile stores the current form, every runner option, additional JSON, Queue, notification fields, UI language, Light/Dark theme, and provider Base URLs. API keys and SMTP passwords are deliberately excluded and remain in the operating system's credential store.

Use the Theme dropdown in the header to switch between Nord Light and Dark without restarting. The selected theme is restored on the next launch and is included in portable settings profiles.

The current form is also saved on a normal program exit, so it is restored on the next launch instead of resetting to a model preset. Set current as default creates an explicit startup template; once set, that template takes priority over the most recently used form until it is replaced or a profile without a default is loaded.

Live debug console

The Debug Console displays the generated command, runner stdout/stderr, process ID, exit code, Queue progress, and lifecycle status with timestamps. Commands, status messages, warnings, and errors use distinct colors.

• Disable Auto-scroll to inspect earlier output while an experiment runs.
• Use Clear console, Copy all, or Save log… to manage the current session output.
• Use the Font size −/+ controls to resize console text immediately; the selected size is restored on the next launch.
• Console history survives an English/Korean UI switch.
• Python child-process buffering is disabled so output appears as promptly as the runner emits it.

Analysis / Report

The Analysis / Report tab appears immediately after Experiment Setup / Queue. Its upper pane lists automatically saved and reconstructed experiment records with local start time, status, provider, model, success/fail/total, success rate, record source, and result folder. Search matches experiment name, provider, model, and result path; the adjacent filter limits the list by running, completed, failed, stopped, or interrupted status. Select one record for its complete details, or select two or three records for A/B/C comparison. Click any history column heading to sort ascending or descending; the active column displays an arrow indicating the current direction.

Selecting one record fills the detail tabs:

• Overview shows timestamps, elapsed time, counts and percentages, runner environment, platform details, record provenance, and configuration/task-set fingerprints.
• Task Outcomes shows domain-qualified task IDs, full instructions, score, failure reason, and termination metadata. Its own text and status filters can narrow large task sets.
• Configuration / Commands / Errors shows the secret-masked runner options, executed commands, recorded errors, terminal message, and recovery metadata.
• API Cost shows requests, tokens, pricing basis, per-task cost, anomalies, and the saved price source. With multiple records selected, it compares the original cost snapshots side by side.

Collapse the history or selected-details block with − and restore it with +; the visible block uses the released space. Open Result Folder opens the selected run in Finder, File Explorer, or the Linux file manager. The list refreshes after background restoration and while Queue progress is being recorded.

With two or three records selected, Comparison shows success/fail/total and success/failure percentages side by side in a bordered row-and-column grid. A configuration-difference table identifies provider, model, runner, runner-option, operating-system, architecture, and Python changes. Task identity uses domain/task-id; a task JSON hash mismatch is reported separately from experiment-setting changes.

The task-change table uses an A/B/C status pattern and can filter common successes, common failures, regressions, improvements, changed outcomes, missing or incomplete tasks, task-definition changes, A-success/B-failure, A-failure/B-success, and failures from each selected run. Tasks absent from one run's original plan are shown as Not planned, rather than being conflated with a missing result.

Choose the run whose settings should seed a rerun, then use Prepare selected rerun or Prepare all shown. The manager transfers the chosen domain-qualified tasks and the source run's non-secret options to Experiment Setup / Queue without starting them. Review the generated name, active OSWorld path, VM, model, and other options before adding the experiment to the Queue. Credential placeholders are never copied from history records.

Use Word report with one selected record to create a detailed report, or with two or three records to create a comparison report. Comparison reports include the side-by-side summary, environment and configuration differences, task-level transitions, and each run's successful and failed task lists. The report language follows the current UI language.

Experiment records can be moved between computers as .oswexp packages. Portable export contains the versioned records used for comparison; Full diagnostic additionally includes sanitized event history and task termination details. Both modes exclude credentials, screenshots, videos, and general runtime logs. Import validates package paths and checksums, indexes new records locally, and skips records whose experiment ID already exists.

Smoke-test decisions and reports

Smoke-test evaluation is automatic and rule-based. The manager checks:

• nonzero runner exit codes and process crashes;
• missing or invalid result.txt files;
• adapter exceptions and API authentication/connection failures recorded in termination.json or runtime.log;
• fatal runtime patterns such as uncaught tracebacks and segmentation faults.

Valid 0.0 scores, stagnation, and max_steps are task-performance outcomes. They do not stop the Queue, even when a runner records stagnation under the broad model_error provenance.

The report generator aggregates completion counts, mean success rate, missing results, termination causes, detected errors, commands, and complete successful and failed task lists into a styled python-docx document. The document is saved directly without launching Microsoft Word or another external application.

Auto-Generate Report is stored with each Queue item. Reports created from Analysis / Report use the current UI language and are also saved quietly.

Provider connectivity and model discovery

In the Model runner-options tab, Check next to API Base URL performs a read-only request against the provider's models endpoint. Load Models populates the editable model dropdown, while Refresh requests a fresh list from the server. Changing the AI provider updates the Base URL to its saved or official default endpoint. Connection success or failure and model-list results appear in the bordered Status box instead of an OK popup.

Button results throughout the manager use the same Status box whenever possible, including Queue validation, duplication/removal, reports, defaults, notifications, and provider operations. File open/save workflows retain their native dialogs and completion/error popups because the selected path is part of the result; command preview remains a dedicated non-modal window.

OpenAI, OpenRouter, LM Studio, and RunPod vLLM use their OpenAI-compatible /models endpoint. Google uses the paginated Gemini v1beta/models endpoint and keeps only models that support generateContent. Requests read API keys from the operating-system credential store; keys are never added to URLs, logs, exported settings, or source files.

The RunPod · vLLM Serverless preset selects vllm, fills https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1, and keeps the OSWorld desktop on local VMware. Replace ENDPOINT_ID, save the RunPod account API key in the vllm (RunPod Serverless) provider, and click Load Models to replace the example model ID with the model actually served by the endpoint. RunPod checks use a longer timeout to accommodate a Serverless cold start. See RunPod's OpenAI compatibility documentation.

The RunPod · vLLM Pod preset selects the separate vllm_pod provider and fills https://POD_ID-8000.proxy.runpod.net/v1. Expose HTTP port 8000, start vLLM on 0.0.0.0:8000 with --api-key, replace POD_ID, and save that vLLM server key under the separate vllm (RunPod Pod) provider. The two RunPod credentials are intentionally independent. RunPod's HTTP proxy has a 100-second request limit; direct TCP exposure can be used with a Base URL such as http://PUBLIC_IP:MAPPED_PORT/v1 when longer requests are required. See the RunPod port exposure documentation.

The Custom · Lab vLLM Server preset selects vllm_custom, keeps the local OSWorld desktop on VMware, and fills the private-network example http://192.168.0.100:8000/v1. Replace the address with the lab server's IP or DNS name, then use Load Models to select the exact model ID returned by /models. Its vllm (Custom / Lab) credential is independent from both RunPod rows. The API key is optional for an unauthenticated internal endpoint; when the server starts with vllm serve --api-key, save that same key here. Use HTTPS or another protected network path when traffic crosses an untrusted network.

API cost tracking

The status row includes a live API Cost panel for the current task, the current Queue, and a full-experiment forecast when enough usage data is available. Cost Settings stores model-specific token, request, image, or GPU-hour rates. Auto Fill identifies the active provider and model and uses a documented machine-readable source when available; Refresh requests that source again. OpenRouter prices are loaded from its Models API. Providers without a stable pricing endpoint can use a built-in snapshot or manual rate. LM Studio and lab vLLM default to zero API charge, while an optional GPU-hour rate can represent compute cost.

Selecting a model in Runner Options also checks and saves available pricing automatically. This background price lookup does not read the operating system's credential store. Buttons and interactive controls provide hover tooltips, and English UI labels use Title Case consistently.

Each Queue item keeps a credential-free pricing snapshot. During execution the manager incrementally reads model_usage from OSWorld traj.jsonl files and the optional .osworld-manager/api-usage.jsonl journal. Actual provider cost is preferred when recorded; otherwise token or compute usage is evaluated against the snapshot. Missing usage remains not configured rather than being reported as zero. If the projected total exceeds an experiment budget, the Queue pauses after the current experiment.

The API Cost page under Analysis / Report shows request and token counts, cost basis, cost per task, cost per successful task, the most expensive task, and tasks above twice the median cost. Selecting up to three experiments compares these values and their original pricing sources. Cost data is included in .oswexp packages, but API keys are never included.

Adaptive options

Controls react to the selected configuration:

• Selecting docker disables the VMware .vmx file field.
• Auto beside the VMware VM field searches the active OSWorld checkout, sibling OSWorld checkouts, and standard user virtual-machine folders for .vmx files. It selects the active checkout's first match and reports the result in the Status box.
• Selecting vmware fixes parallel environments at 1, because controlling one personal VM from multiple processes can cause VM or snapshot conflicts.
• A model preset limits the AI-provider dropdown to compatible providers.
• The LM Studio preset permits only lm_studio; the RunPod Serverless preset permits only vllm, the RunPod Pod preset permits only vllm_pod, and the custom lab preset permits only vllm_custom.
• Manual task IDs accept UUID, UUID.json, or domain/UUID.json separated by commas, whitespace, or newlines and take precedence over the task-list JSON.
• The domain dropdown is populated from the directories under the active task configuration folder's examples directory. The current official task set includes chrome, gimp, libreoffice_calc, libreoffice_impress, libreoffice_writer, multi_apps, os, thunderbird, vlc, and vs_code, plus all.
• A repeat count of two or more creates independent Queue entries and result folders such as experiment-run-1 and experiment-run-2.

Official task picker

The Tasks runner-options tab keeps the manual-ID field for advanced use and adds Select tasks… for guided selection. The resizable picker reads only the task IDs registered in the active checkout's official evaluation_examples/test_all.json. JSON files that exist only in the examples folder are ignored. The current official task set exposes 369 tasks.

• Search matches domain, UUID, and full instruction text.
• The domain filter, Select all shown, and extended single/multi-selection support fast bulk selection.
• Rows display [domain] — shortened instruction; hovering a row or selecting it reveals the full instruction.
• Add selected stages tasks without duplicates. Staged tasks can be removed individually or cleared before submission.
• Add to Queue creates one experiment containing all staged tasks. For example, 12 staged tasks create one Queue row whose progress runs from 0/12 to 12/12; repeat count duplicates that complete 12-task experiment.
• The generated domain/UUID list remains visible in the manual field and is preserved by settings files and startup-state persistence. Reopening the picker restores all official tasks found in that field.

Hover over a label, input, checkbox, or browse button to see concise help in the currently selected language. Runner-specific JSON now lives in the Additional options runner-options tab, so it does not consume Queue space unless that tab is selected. The editor preserves at least three visible lines.

Results and Queue progress

The selected result path is a base folder. The experiment name is appended automatically:

selected-results-folder/
└── Qwen25VL72B_Enhanced_TestSmall_Screenshot_v4/
    ├── .osworld-manager/
    │   ├── Qwen25VL72B_Enhanced_TestSmall_Screenshot_v4/
    │   │   ├── execution-metadata.json
    │   │   └── selected-tasks.json
    │   └── history/<experiment-id>/
    │       ├── experiment-record.json
    │       └── events.jsonl
    ├── pyautogui/
    └── osworld-report-*.docx

The Queue calculates Done/Total from the selected domain and task IDs. During execution it periodically checks for new result.txt files and displays an animated running indicator. Click a path in the Result folder column to open that directory in the operating system's file manager. A vertical scrollbar appears automatically when the Queue contains more rows than the visible area can show.

The adjacent Success/Fail/Total column is updated by the same live scan. A finite score greater than 0 counts as success; 0, negative, non-finite, or unreadable scores count as failure. Tasks without result.txt remain pending and appear only in the total.

Every Queue run receives a unique experiment ID. Its versioned experiment-record.json is updated atomically at start, task-plan creation, progress changes, command launch, and final completion, failure, stop, or interruption. It stores timestamps, provider/model/runner settings, a configuration fingerprint, operating-system and Python details, masked commands and errors, and domain-qualified task outcomes with instructions and task-file hashes. Re-running the same experiment name creates a separate record instead of replacing an earlier run. API keys, passwords, tokens, and other credential-like option values are redacted.

The manager maintains a rebuildable local index at ~/.osworld-experiment-manager/experiment-history.sqlite3. The portable JSON inside the result folder remains the source of truth. At startup, known result locations are scanned in the background: existing records are re-indexed, stale running records are finalized as interrupted, and older result folders are reconstructed from selected-tasks.json, result.txt, and termination.json. Reconstructed records explicitly identify whether the original planned task list was available or only observed task outputs could be recovered.

Stop uses the same native platform style as the other buttons. While the Queue is active, Run Queue becomes Pause. Pause never terminates the experiment currently in progress; it waits after that experiment finishes. While paused, pending Queue entries can be edited, added, or removed. Resume re-reads the visible Queue and runs the latest pending entries. Clicking Stop while idle only displays a status message.

At the start of every experiment, execution-metadata.json records workspace_root, git_branch, git_commit, and dirty_state for the exact OSWorld checkout being executed. dirty_state is clean, dirty (including untracked files), or unavailable when Git metadata cannot be read. A UTC capture timestamp is stored alongside these fields.

Credentials and notifications

API keys and SMTP app passwords are stored through Python keyring in the operating system's secure credential store, never in the settings JSON or Queue. This maps to macOS Keychain, Windows Credential Locker, or a Linux Secret Service/system keyring backend.

The API / Secrets tab opens as a fresh-entry form and never reads the operating-system credential store during application startup, tab creation, or provider selection. Select a provider and click Load saved values only when you want to authorize a credential-store lookup. Existing values are then placed into the regular input fields. The key remains masked until Show is pressed, and the adjacent Copy button copies it to the clipboard. Show changes to Hide while the key is visible. Load success, missing values, and credential-store errors are reported inline below the inputs without a popup. There is no separate persistent-values panel. Exported settings still contain Base URLs only and never include API keys.

• SMTP email works across platforms using the configured host, port, user, recipients, and app password. Separate multiple addresses with commas or semicolons; each recipient receives an individual message.
• macOS Notification Center is attempted only on macOS.
• macOS Messages accepts comma- or semicolon-separated phone numbers and Apple IDs and is ignored on other platforms.

Email and Messages are sent independently. An empty recipient field is silently ignored.

Completion messages include success, failure, and total task counts, both percentages, and the result folder. Completion emails additionally include the AI provider, model, absolute agent-runner path, actual commands, and complete successful and failed task lists. Stopped or failed Messages include up to three representative error entries; email includes every error entry captured for that execution. Credential values such as API keys, tokens, and client passwords are redacted from commands written to notifications and reports.

Direct voice calls are not included. A paid telephony provider such as Twilio would require a separate integration and credentials.

Tests

cd ./OSWorld-gui-tool
python -m pytest -q