OpenCastor is a universal runtime for embodied AI (~270 Python files, 140+ test files, 3431 tests).
OpenCastor/
├── castor/ # Main Python package
│ ├── __init__.py # Version string (__version__)
│ ├── __main__.py # Package entry point
│ ├── cli.py # Unified CLI entry point (48+ commands)
│ ├── main.py # Core runtime: perception-action loop
│ ├── api.py # FastAPI gateway server (all REST endpoints)
│ ├── api_errors.py # Structured JSON error handling for API
│ ├── auth.py # Unified auth manager (providers + channels)
│ ├── wizard.py # Interactive setup wizard
│ ├── web_wizard.py # Web-based configuration wizard
│ ├── setup_catalog.py # Shared setup metadata (providers/profiles/stacks/presets)
│ ├── setup_service.py # Shared setup orchestration (CLI + web APIs)
│ ├── dashboard.py # Streamlit web UI
│ ├── dashboard_tui.py # Terminal UI dashboard (tmux-based, preferred)
│ ├── config_validation.py # RCAN config validation (fail-fast on startup)
│ ├── connectivity.py # Internet & provider reachability checks
│ ├── offline_fallback.py # Auto-switch to local provider on connectivity loss
│ ├── provider_fallback.py # Auto-switch on quota/credit errors
│ ├── tiered_brain.py # Multi-model switching by latency budget
│ ├── prompt_cache.py # LLM response caching (reduces API cost)
│ ├── healthcheck.py # Component health checks
│ ├── hardware_detect.py # Auto-detect cameras and drivers
│ ├── hailo_vision.py # Hailo-8 edge accelerator integration
│ ├── registry.py # Component registry
│ ├── crash.py # Crash handler
│ ├── watchdog.py # System health monitor + crash recovery
│ ├── telemetry.py # Performance metrics, memory usage tracking
│ ├── runtime_stats.py # Runtime statistics
│ ├── battery.py # Battery monitoring
│ ├── geofence.py # Geofencing utilities
│ ├── peripherals.py # Peripheral device management
│ ├── fleet.py # Multi-robot fleet management
│ ├── hub.py # Model hub integration
│ ├── plugins.py # Plugin system
│ ├── profiles.py # User profile management
│ ├── daemon.py # systemd service management
│ ├── audit.py # Audit logging
│ ├── approvals.py # Work approval workflow
│ ├── privacy.py # Privacy / data deletion utilities
│ ├── schedule.py # Task scheduling
│ ├── network.py # Network utilities
│ ├── backup.py / restore.py # Config backup & restore
│ ├── export.py # Config bundle export
│ ├── migrate.py # RCAN config migration
│ ├── diff.py # Config diff viewer
│ ├── lint.py # Deep config validation
│ ├── conformance.py # RCAN conformance checking
│ ├── configure.py # Configuration CLI helpers
│ ├── upgrade.py # Self-update + doctor
│ ├── fix.py # Auto-fix common issues
│ ├── update_check.py # Version update checking
│ ├── record.py # Episode recording
│ ├── learn.py # Interactive learning tutorial
│ ├── demo.py # Cinematic terminal demo
│ ├── repl.py # Python REPL with robot objects
│ ├── shell.py # Interactive command shell
│ ├── watch.py # Live Rich TUI telemetry
│ ├── logs.py # Log viewing utilities
│ ├── benchmark.py # Performance profiling
│ ├── calibrate.py # Interactive hardware calibration
│ ├── test_hardware.py # Hardware testing CLI
│ ├── memory.py # SQLite episode store (EpisodeMemory)
│ ├── metrics.py # Prometheus-compatible metrics (MetricsRegistry)
│ ├── tools.py # LLM tool calling registry (ToolRegistry)
│ ├── memory_search.py # Memory search utilities
│ ├── claude_proxy.py # Claude API proxy
│ ├── depth.py # OAK-D depth overlay + obstacle zone detection
│ ├── nav.py # WaypointNav dead-reckoning navigation
│ ├── behaviors.py # BehaviorRunner (YAML step sequences)
│ ├── auth_jwt.py # Multi-user JWT auth (OPENCASTOR_USERS env var)
│ ├── usage.py # UsageTracker (SQLite token/cost tracking)
│ ├── camera.py # CameraManager (multi-camera support)
│ ├── stream.py # WebRTC stream (aiortc optional)
│ ├── recorder.py # VideoRecorder: MP4 capture via OpenCV
│ ├── webhooks.py # WebhookDispatcher: outbound POST on robot events
│ ├── gestures.py # GestureController: MediaPipe hand gesture → action
│ ├── response_cache.py # SQLite LRU cache keyed by SHA-256(instruction+image)
│ ├── avoidance.py # ReactiveAvoidance: LiDAR+depth obstacle stop/slow
│ ├── pointcloud.py # PointCloudCapture: 3D PLY export from OAK-D
│ ├── detection.py # ObjectDetector: YOLOv8/DETR real-time detection
│ ├── sim_bridge.py # SimBridge: MuJoCo/Gazebo/Webots export+import
│ ├── episode_search.py # EpisodeSearchIndex: BM25 memory search
│ ├── voice_loop.py # VoiceLoop: wake-word + STT + brain pipeline
│ ├── workspace.py # WorkspaceManager: multi-robot namespace isolation
│ ├── personalities.py # PersonalityManager: tone-injection profiles
│ ├── finetune.py # FineTuneExporter: JSONL export for OpenAI/Anthropic
│ ├── slam.py # SLAM mapper: occupancy grid via wheel odometry + LiDAR
│ ├── ina219.py # Battery monitor: INA219 I2C sensor (voltage/current/SOC)
│ ├── privacy_mode.py # PrivacyMode: zero-cloud-egress enforcement
│ ├── rcan_generator.py # RCAN config generator: NL description → YAML via LLM
│ │
│ ├── commands/ # CLI sub-commands
│ │ ├── __init__.py
│ │ ├── swarm.py # castor swarm status/command/stop/sync
│ │ ├── hub.py # castor hub list/search/install/publish
│ │ ├── update.py # castor update (git pull / pip upgrade + swarm SSH)
│ │ ├── benchmark.py # castor benchmark --providers comparison
│ │ └── deploy.py # castor deploy (SSH-push config + restart)
│ │
│ ├── providers/ # LLM provider adapters
│ │ ├── __init__.py # get_provider() factory
│ │ ├── base.py # BaseProvider ABC + Thought + ProviderQuotaError
│ │ ├── google_provider.py # Google Gemini
│ │ ├── openai_provider.py # OpenAI GPT-4.1 (also OpenRouter proxy)
│ │ ├── anthropic_provider.py # Anthropic Claude
│ │ ├── ollama_provider.py # Local Ollama
│ │ ├── huggingface_provider.py # HuggingFace Hub
│ │ ├── llamacpp_provider.py # llama.cpp local inference
│ │ ├── mlx_provider.py # Apple MLX acceleration
│ │ ├── apple_provider.py # Apple Foundation Models provider (on-device)
│ │ ├── apple_preflight.py # Apple platform/SDK/model readiness checks
│ │ ├── vertex_provider.py # Google Vertex AI (google-genai SDK)
│ │ ├── openrouter_provider.py # OpenRouter (100+ models, OPENROUTER_API_KEY)
│ │ ├── groq_provider.py # Groq LPU-accelerated inference (GROQ_API_KEY)
│ │ ├── sentence_transformers_provider.py # Sentence Transformers embeddings
│ │ ├── vla_provider.py # Vision-Language-Action (OpenVLA/Octo/pi0)
│ │ ├── onnx_provider.py # ONNX Runtime on-device inference (ONNX_MODEL_PATH)
│ │ ├── kimi_provider.py # Moonshot AI Kimi (MOONSHOT_API_KEY)
│ │ ├── minimax_provider.py # MiniMax (MINIMAX_API_KEY)
│ │ └── qwen_provider.py # Qwen3 local via Ollama
│ │
│ ├── drivers/ # Hardware driver implementations
│ │ ├── __init__.py
│ │ ├── base.py # DriverBase ABC (move/stop/close/health_check)
│ │ ├── pca9685.py # I2C PWM motor driver (Amazon/Adafruit kits)
│ │ ├── dynamixel.py # Robotis Dynamixel servo (Protocol 2.0)
│ │ ├── composite.py # CompositeDriver: routes action keys to sub-drivers
│ │ ├── ros2_driver.py # ROS2 bridge driver (rclpy, mock mode)
│ │ ├── imu_driver.py # IMU driver: MPU6050/BNO055/ICM-42688 (smbus2)
│ │ ├── lidar_driver.py # 2D LiDAR driver: RPLidar A1/A2/C1/S2
│ │ ├── stepper_driver.py # Stepper motor: NEMA 17/23 via DRV8825/TMC2209/A4988
│ │ ├── gpio_driver.py # GPIO driver: RPi.GPIO + gpiod pin control
│ │ ├── odrive_driver.py # Brushless motor: ODrive / VESC (CAN bus)
│ │ └── simulation_driver.py # Simulation driver: Gazebo/Webots/mock
│ │
│ ├── channels/ # Messaging channel integrations
│ │ ├── __init__.py # Channel registry + create_channel() factory
│ │ ├── base.py # BaseChannel ABC
│ │ ├── session.py # Session management
│ │ ├── whatsapp_neonize.py # WhatsApp via neonize (QR code scan)
│ │ ├── whatsapp_twilio.py # WhatsApp via Twilio (legacy)
│ │ ├── telegram_channel.py # Telegram Bot (long-polling)
│ │ ├── discord_channel.py # Discord Bot
│ │ ├── slack_channel.py # Slack Bot (Socket Mode)
│ │ ├── mqtt_channel.py # MQTT (paho-mqtt)
│ │ ├── homeassistant_channel.py # Home Assistant (websocket)
│ │ ├── teams_channel.py # Microsoft Teams (incoming webhook + Bot Framework)
│ │ └── matrix_channel.py # Matrix/Element (matrix-nio)
│ │
│ ├── fs/ # Virtual Filesystem (Unix-inspired)
│ │ ├── __init__.py # CastorFS facade class
│ │ ├── namespace.py # Hierarchical namespace (/dev, /etc, /proc, etc.)
│ │ ├── permissions.py # PermissionTable, ACL, Cap (capabilities)
│ │ ├── safety.py # SafetyLayer (bounds, rate limiting, e-stop)
│ │ ├── memory.py # MemoryStore (episodic, semantic, procedural)
│ │ ├── context.py # ContextWindow (multi-turn), Pipeline (Unix pipes)
│ │ └── proc.py # ProcFS (read-only runtime introspection)
│ │
│ ├── safety/ # Safety & authorization subsystem
│ │ ├── __init__.py
│ │ ├── anti_subversion.py # Input scanning (check_input_safety, ScanVerdict)
│ │ ├── authorization.py # WorkAuthority, WorkOrder, audit log
│ │ ├── bounds.py # BoundsChecker (joint, force, workspace)
│ │ ├── monitor.py # Continuous safety monitoring
│ │ ├── protocol.py # Safety protocol definitions
│ │ └── state.py # SafetyStateSnapshot, SafetyTelemetry
│ │
│ ├── rcan/ # RCAN protocol implementation
│ │ ├── __init__.py
│ │ ├── ruri.py # RURI addressing (rcan://domain.name.id)
│ │ ├── message.py # RCANMessage envelope, MessageType, Priority
│ │ ├── rbac.py # RCANRole (CREATOR→GUEST), Scope, RCANPrincipal
│ │ ├── router.py # MessageRouter (dispatch RCAN messages)
│ │ ├── capabilities.py # Capability, CapabilityRegistry
│ │ ├── jwt_auth.py # RCANTokenManager (JWT sign/verify)
│ │ └── mdns.py # mDNS robot discovery (optional)
│ │
│ ├── agents/ # Multi-agent framework
│ │ ├── __init__.py
│ │ ├── base.py # BaseAgent ABC, AgentStatus
│ │ ├── shared_state.py # SharedState (pub/sub event bus)
│ │ ├── registry.py # AgentRegistry (lifecycle management)
│ │ ├── observer.py # ObserverAgent (scene understanding)
│ │ ├── navigator.py # NavigatorAgent (path planning)
│ │ ├── manipulator_agent.py # ManipulatorAgent (arm/gripper)
│ │ ├── communicator.py # CommunicatorAgent (NL intent routing)
│ │ ├── guardian.py # GuardianAgent (safety meta-agent, veto + e-stop)
│ │ └── orchestrator.py # OrchestratorAgent (master, single RCAN output)
│ │
│ ├── specialists/ # Task specialist agents
│ │ ├── __init__.py
│ │ ├── base_specialist.py # BaseSpecialist ABC, Task, TaskResult
│ │ ├── scout.py # ScoutSpecialist (visual exploration)
│ │ ├── manipulator.py # ManipulatorSpecialist (grasping)
│ │ ├── dock.py # DockSpecialist (docking/charging)
│ │ ├── responder.py # ResponderSpecialist (alert responses)
│ │ └── task_planner.py # TaskPlanner (decompose → typed tasks)
│ │
│ ├── learner/ # Self-improving loop (Sisyphus pattern)
│ │ ├── __init__.py
│ │ ├── episode.py # Episode (observation/action/outcome)
│ │ ├── episode_store.py # EpisodeStore (persistent JSON storage)
│ │ ├── sisyphus.py # SisyphusLoop + ImprovementResult + SisyphusStats
│ │ ├── pm_stage.py # PMStage (analyze episodes, find failures)
│ │ ├── dev_stage.py # DevStage (propose patches)
│ │ ├── qa_stage.py # QAStage (validate patches)
│ │ ├── apply_stage.py # ApplyStage (deploy approved patches)
│ │ ├── patches.py # Patch, ConfigPatch, PromptPatch, BehaviorPatch
│ │ └── alma.py # ALMAConsolidation (swarm patch aggregation)
│ │
│ └── swarm/ # Multi-robot coordination
│ ├── __init__.py
│ ├── peer.py # SwarmPeer (remote robot proxy)
│ ├── coordinator.py # SwarmCoordinator (task distribution)
│ ├── consensus.py # SwarmConsensus (majority-vote protocol)
│ ├── events.py # SwarmEvent (pub/sub envelope)
│ ├── shared_memory.py # SharedMemory (distributed key-value)
│ └── patch_sync.py # PatchSync (incremental config sync)
│
├── config/
│ ├── swarm.yaml # Swarm node registry (name/host/ip/port/token/tags)
│ ├── hub_index.json # Model hub index (16 presets, GitHub raw URLs)
│ └── presets/ # 18 hardware preset RCAN configs
│ ├── amazon_kit_generic.rcan.yaml
│ ├── adeept_generic.rcan.yaml
│ ├── waveshare_alpha.rcan.yaml
│ ├── sunfounder_picar.rcan.yaml
│ ├── dynamixel_arm.rcan.yaml
│ ├── rpi_rc_car.rcan.yaml
│ ├── arduino_l298n.rcan.yaml
│ ├── esp32_generic.rcan.yaml
│ ├── cytron_maker_pi.rcan.yaml
│ ├── elegoo_tumbller.rcan.yaml
│ ├── freenove_4wd.rcan.yaml
│ ├── lego_mindstorms_ev3.rcan.yaml
│ ├── lego_spike_prime.rcan.yaml
│ ├── makeblock_mbot.rcan.yaml
│ ├── vex_iq.rcan.yaml
│ ├── yahboom_rosmaster.rcan.yaml
│ ├── groq_rover.rcan.yaml # Groq LPU-accelerated rover
│ └── oak4_pro.rcan.yaml # OAK-4 Pro with depth+IMU
│
├── sdk/
│ └── js/ # JavaScript/TypeScript client SDK
│ ├── src/index.ts # CastorClient: command/stream/status/stop/health
│ ├── package.json
│ └── tsconfig.json
│
├── tests/ # 140+ test files, 3431 tests (0 failures)
│ ├── test_api_endpoints.py # FastAPI gateway (133 tests)
│ ├── test_config_validation.py # Config validation
│ ├── test_offline_fallback.py # OfflineFallbackManager
│ ├── test_learner/ # Sisyphus loop (12 test files)
│ ├── test_agents/ # Agent framework (11 test files)
│ ├── test_swarm/ # Multi-robot swarm (6 test files)
│ ├── test_fs/ # Virtual filesystem
│ ├── test_safety/ # Safety subsystem
│ ├── test_rcan/ # RCAN protocol
│ ├── test_channels/ # Messaging channels
│ ├── test_providers/ # AI providers
│ └── test_drivers/ # Hardware drivers
│
├── scripts/
│ ├── install.sh / install.ps1 # One-line installers (Linux/Windows)
│ ├── install-check.sh / .ps1 # Install verification
│ ├── uninstall.sh # Uninstaller
│ ├── start_dashboard.sh # Kiosk mode tmux launcher
│ └── sync-version.py # Keep version strings in sync
│
├── site/ # Static landing page (Cloudflare Pages)
├── brand/ # Brand assets (logos, badges)
├── .github/
│ └── workflows/
│ ├── ci.yml # Tests + lint + type check
│ ├── validate_rcan.yml # RCAN schema validation
│ ├── install-test.yml # Multi-platform install verification (scheduled)
│ ├── release.yml # PyPI release automation (on tag push)
│ ├── deploy-pages.yml # Cloudflare Pages deploy (on push to main)
│ ├── auto-label.yml # PR auto-labeling
│ ├── create_backlog_issues.yml # Backlog maintenance (scheduled)
│ └── stale.yml # Stale issue/PR management (scheduled)
├── .env.example
├── pyproject.toml
├── requirements.txt
├── Dockerfile # Container with health check
├── docker-compose.yml # Gateway + runtime + dashboard services
├── CONTRIBUTING.md
├── wrangler.toml # Cloudflare Pages config
└── README.md
| Subsystem | Path | Purpose |
|---|---|---|
| Providers | castor/providers/ |
17 LLM adapters (Gemini, GPT-4.1, Claude, Ollama, HuggingFace, llama.cpp, MLX, Apple Foundation Models, Vertex, OpenRouter, Groq, VLA, ONNX, Kimi, MiniMax, Qwen, SentenceTransformers); _caps+_robot_name injected by api.py for prompt context |
| Drivers | castor/drivers/ |
Hardware: PCA9685, Dynamixel, CompositeDriver, ROS2 bridge |
| Channels | castor/channels/ |
Messaging: WhatsApp (neonize/Twilio, group_jids/group_name_filter routing), Telegram, Discord, Slack, MQTT, Home Assistant |
| VFS | castor/fs/ |
Unix-inspired virtual filesystem with capabilities, memory tiers, e-stop |
| Safety | castor/safety/ |
Anti-subversion, BoundsChecker, WorkAuthority, GuardianAgent |
| RCAN | castor/rcan/ |
Protocol: RURI addressing, RBAC (5 roles), JWT auth, mDNS discovery |
| Agents | castor/agents/ |
Multi-agent: Observer, Navigator, Manipulator, Communicator, Guardian, Orchestrator |
| Specialists | castor/specialists/ |
Task agents: Scout, Responder, Dock, Manipulator, TaskPlanner |
| Learner | castor/learner/ |
Sisyphus self-improving loop: PM→Dev→QA→Apply |
| Swarm | castor/swarm/ |
Multi-robot: coordinator, consensus, shared memory, patch sync |
| API | castor/api.py |
FastAPI gateway; 50+ REST endpoints + WebSocket telemetry |
| Memory | castor/memory.py |
SQLite episode store; 10k episode FIFO; CASTOR_MEMORY_DB |
| Metrics | castor/metrics.py |
Stdlib Prometheus; GET /api/metrics; 13 pre-registered metrics |
| Tools | castor/tools.py |
LLM tool calling; 4 built-ins; positional-only name param |
| Nav | castor/nav.py |
Dead-reckoning WaypointNav; reads RCAN physics block |
| Behaviors | castor/behaviors.py |
YAML step sequences; BehaviorRunner dispatch table |
| Camera | castor/camera.py |
Multi-camera CameraManager; tile/primary/depth_overlay modes |
| Stream | castor/stream.py |
WebRTC via aiortc (optional); fallback to MJPEG |
| Usage | castor/usage.py |
SQLite token/cost tracker; GET /api/usage |
| Depth | castor/depth.py |
OAK-D JET colormap overlay; obstacle zone detection |
Required top-level keys: rcan_version, metadata, agent, physics, drivers, network, rcan_protocol
metadata.robot_name— requiredagent.model— requireddrivers— must be non-empty list
Validated by castor/config_validation.py on gateway startup. 16 presets in config/presets/.
| Workflow | Trigger | Purpose |
|---|---|---|
ci.yml |
Push, PR | Tests + ruff lint + type check |
validate_rcan.yml |
Push/PR on *.rcan.yaml |
JSON schema validation |
install-test.yml |
Scheduled | Multi-platform install test |
release.yml |
Tag push | PyPI release automation |
deploy-pages.yml |
Push to main | Cloudflare Pages deploy |
stale.yml |
Scheduled | Stale issue/PR cleanup |