diff --git a/.cursor/rules/jep-process.mdc b/.cursor/rules/jep-process.mdc index 9abcd5e87..02d709caa 100644 --- a/.cursor/rules/jep-process.mdc +++ b/.cursor/rules/jep-process.mdc @@ -12,9 +12,9 @@ This rule helps with creating Jumpstarter Enhancement Proposals (JEPs). ## Creating a JEP -1. **Choose the next JEP number**: Look at existing files in `python/docs/source/internal/jeps/` and pick the next available incrementing integer. JEP-0000 through JEP-0009 are reserved for process/meta-JEPs, so start from JEP-0010 for regular proposals. +1. **Choose the next JEP number**: Look at existing files in `python/docs/source/contributing/jeps/` and pick the next available incrementing integer. JEP-0000 through JEP-0009 are reserved for process/meta-JEPs, so start from JEP-0010 for regular proposals. -2. **Create the file**: Copy the template from `python/docs/source/internal/jeps/JEP-NNNN-template.md` to `python/docs/source/internal/jeps/JEP-NNNN-short-title.md`, replacing `NNNN` with the zero-padded number and `short-title` with a descriptive slug. +2. **Create the file**: Copy the template from `python/docs/source/contributing/jeps/JEP-NNNN-template.md` to `python/docs/source/contributing/jeps/JEP-NNNN-short-title.md`, replacing `NNNN` with the zero-padded number and `short-title` with a descriptive slug. 3. **Fill in the metadata table**: - Set the JEP number (incrementing integer, NOT the PR number) @@ -45,8 +45,8 @@ JEPs use this format for individual decisions: **Alternatives considered:** -1. **Option A** — Brief description. -2. **Option B** — Brief description. +1. **Option A** - Brief description. +2. **Option B** - Brief description. **Decision:** Option A. @@ -56,7 +56,7 @@ JEPs use this format for individual decisions: ## Key Rules - JEP numbers are incrementing integers, NOT derived from PR numbers -- JEPs live in `python/docs/source/internal/jeps/` +- JEPs live in `python/docs/source/contributing/jeps/` - All JEPs should be merged as PRs so the documentation is part of the Jumpstarter docs/source - Rejected JEPs are normally not merged, but can be merged with "Rejected" status if there is an architectural reason to preserve them - The license for all documents is Apache-2.0 diff --git a/.cursor/rules/releasing-operator.mdc b/.cursor/rules/releasing-operator.mdc index 7ce00c072..cddbf5c5b 100644 --- a/.cursor/rules/releasing-operator.mdc +++ b/.cursor/rules/releasing-operator.mdc @@ -18,7 +18,7 @@ The operator bundle version is driven by variables in `controller/deploy/operato - **`IMAGE_TAG_BASE`**: Registry prefix (`quay.io/jumpstarter-dev/jumpstarter-operator`). - **`BUNDLE_IMG`** / **`CATALOG_IMG`**: Also derived from `IMAGE_TAG_BASE` and `VERSION` automatically. -The `config/manifests/bases/jumpstarter-operator.clusterserviceversion.yaml` uses placeholder version `0.0.0` -- the real version is injected by `make bundle`. +The `config/manifests/bases/jumpstarter-operator.clusterserviceversion.yaml` uses placeholder version `0.0.0` - the real version is injected by `make bundle`. ## Image Tag Convention @@ -34,7 +34,7 @@ When preparing a new release version X.Y.Z, update these files together: ### 1. `controller/deploy/operator/Makefile` - `VERSION ?= X.Y.Z` -- `REPLACES ?= jumpstarter-operator.vPREVIOUS` (the most recent version published to the OLM channel -- including RCs. E.g. if `v0.8.1-rc.1` was published, REPLACES must be `jumpstarter-operator.v0.8.1-rc.1`, not `v0.8.0`) +- `REPLACES ?= jumpstarter-operator.vPREVIOUS` (the most recent version published to the OLM channel - including RCs. E.g. if `v0.8.1-rc.1` was published, REPLACES must be `jumpstarter-operator.v0.8.1-rc.1`, not `v0.8.0`) - Optionally update `OPENSHIFT_VERSIONS` if the supported range changes ### 2. `controller/deploy/operator/api/v1alpha1/jumpstarter_types.go` diff --git a/.cursor/skills/propose-jep/SKILL.md b/.cursor/skills/propose-jep/SKILL.md index ea177b20f..bdff04031 100644 --- a/.cursor/skills/propose-jep/SKILL.md +++ b/.cursor/skills/propose-jep/SKILL.md @@ -10,7 +10,7 @@ You are helping the user create a new Jumpstarter Enhancement Proposal (JEP). ## Context -JEPs are design documents for substantial changes to the Jumpstarter project — changes that affect multiple components, alter public APIs or protocols, or require community consensus. Read `.cursor/rules/jep-process.mdc` for the full process definition. +JEPs are design documents for substantial changes to the Jumpstarter project - changes that affect multiple components, alter public APIs or protocols, or require community consensus. Read `.cursor/rules/jep-process.mdc` for the full process definition. JEP topic: $ARGUMENTS @@ -18,22 +18,22 @@ JEP topic: $ARGUMENTS ### 1. Determine the next JEP number -List existing files in `python/docs/source/internal/jeps/` and pick the next available incrementing integer. JEP-0000 through JEP-0009 are reserved for process/meta-JEPs, so start from JEP-0010 for regular proposals. +List existing files in `python/docs/source/contributing/jeps/` and pick the next available incrementing integer. JEP-0000 through JEP-0009 are reserved for process/meta-JEPs, so start from JEP-0010 for regular proposals. ### 2. Gather information Before writing the JEP, ask the user clarifying questions to understand: -- **What problem does this solve?** — The motivation section needs a concrete problem description. -- **Who is affected?** — Which components, drivers, or user workflows are impacted? -- **What are the alternatives?** — Each design decision needs at least two alternatives considered. -- **What are the compatibility implications?** — Does this break existing APIs, protocols, or workflows? +- **What problem does this solve?** - The motivation section needs a concrete problem description. +- **Who is affected?** - Which components, drivers, or user workflows are impacted? +- **What are the alternatives?** - Each design decision needs at least two alternatives considered. +- **What are the compatibility implications?** - Does this break existing APIs, protocols, or workflows? If the user provided a description in `$ARGUMENTS`, use it as a starting point but still ask about gaps. ### 3. Create the JEP file -Copy the template from `python/docs/source/internal/jeps/JEP-NNNN-template.md` and create a new file at `python/docs/source/internal/jeps/JEP-NNNN-short-title.md` where: +Copy the template from `python/docs/source/contributing/jeps/JEP-NNNN-template.md` and create a new file at `python/docs/source/contributing/jeps/JEP-NNNN-short-title.md` where: - `NNNN` is the zero-padded next number - `short-title` is a descriptive slug derived from the proposal title @@ -46,9 +46,9 @@ Fill in: ### 4. Update the JEP index -Add the new JEP to the appropriate table in `python/docs/source/internal/jeps/README.md` (Process, Standards Track, or Informational). +Add the new JEP to the appropriate table in `python/docs/source/contributing/jeps/index.md` (Process, Standards Track, or Informational). -Add the new JEP file to the `{toctree}` directive at the bottom of `python/docs/source/internal/jeps/README.md`. +Add the new JEP file to the `{toctree}` directive at the bottom of `python/docs/source/contributing/jeps/index.md`. ### 5. Present the result diff --git a/python/.devcontainer/Dockerfile b/.devcontainer/Dockerfile similarity index 90% rename from python/.devcontainer/Dockerfile rename to .devcontainer/Dockerfile index 2d14439da..c5091e503 100644 --- a/python/.devcontainer/Dockerfile +++ b/.devcontainer/Dockerfile @@ -7,7 +7,7 @@ WORKDIR /opt COPY --from=ghcr.io/astral-sh/uv:latest /uv /bin/uv COPY --from=ghcr.io/astral-sh/uv:latest /uvx /bin/uvx -COPY ./.python-version ./ +COPY ./python/.python-version ./ # Install required tools for development RUN apt-get update && apt-get install -y iperf3 libusb-dev \ No newline at end of file diff --git a/python/.devcontainer/devcontainer.json b/.devcontainer/devcontainer.json similarity index 92% rename from python/.devcontainer/devcontainer.json rename to .devcontainer/devcontainer.json index 7d041f647..ef039270c 100644 --- a/python/.devcontainer/devcontainer.json +++ b/.devcontainer/devcontainer.json @@ -4,8 +4,8 @@ "context": "..", "dockerfile": "Dockerfile" }, - "postCreateCommand": "make sync", - "postStartCommand": "uv python pin 3.12 && uv run pre-commit install", + "postCreateCommand": "cd python && make sync", + "postStartCommand": "cd python && uv python pin 3.12 && uv run pre-commit install", "remoteUser": "vscode", // Mount USB devices to devcontainer for tests "mounts": [ diff --git a/python/.devfile.yaml b/.devfile.yaml similarity index 82% rename from python/.devfile.yaml rename to .devfile.yaml index 57b5f7fe7..39e3ccd40 100644 --- a/python/.devfile.yaml +++ b/.devfile.yaml @@ -22,17 +22,21 @@ commands: - id: serve-docs exec: component: runtime + workingDir: ${PROJECT_SOURCE}/python commandLine: make docs-serve DOC_LISTEN="--host 0.0.0.0" - id: sync exec: component: runtime + workingDir: ${PROJECT_SOURCE}/python commandLine: make sync - id: clean exec: component: runtime + workingDir: ${PROJECT_SOURCE}/python commandLine: make clean - id: test exec: component: runtime + workingDir: ${PROJECT_SOURCE}/python commandLine: make test diff --git a/.github/workflows/documentation.yaml b/.github/workflows/documentation.yaml index 9f8206170..46a42338d 100644 --- a/.github/workflows/documentation.yaml +++ b/.github/workflows/documentation.yaml @@ -96,6 +96,28 @@ jobs: - name: Build the documentation for the current version (no warnings allowed) run: make sync && make docs + linkcheck: + runs-on: ubuntu-latest + timeout-minutes: 15 + steps: + - uses: actions/checkout@v4 + with: + fetch-depth: 0 + fetch-tags: true + + - name: Install the latest version of uv + uses: astral-sh/setup-uv@803947b9bd8e9f986429fa0c5a41c367cd732b41 # v7.2.1 + with: + version: "latest" + + - name: Install Python + run: | + uv python pin 3.12 + uv python install + + - name: Check documentation links + run: make sync && make docs-linkcheck + # Deployment job deploy: environment: diff --git a/.github/workflows/e2e.yaml b/.github/workflows/e2e.yaml index 862a8fcbb..a95c39172 100644 --- a/.github/workflows/e2e.yaml +++ b/.github/workflows/e2e.yaml @@ -61,7 +61,7 @@ jobs: - name: Install Go uses: actions/setup-go@v5 with: - go-version: "1.22" + go-version: "1.24" - name: Cache controller image id: cache @@ -104,7 +104,7 @@ jobs: - name: Install Go uses: actions/setup-go@v5 with: - go-version: "1.22" + go-version: "1.24" - name: Cache operator artifacts id: cache @@ -189,7 +189,7 @@ jobs: - name: Install Go uses: actions/setup-go@v5 with: - go-version: "1.22" + go-version: "1.24" - name: Download controller image uses: actions/download-artifact@v4 @@ -259,7 +259,7 @@ jobs: - name: Install Go uses: actions/setup-go@v5 with: - go-version: "1.22" + go-version: "1.24" - name: Setup compat environment (old controller v0.8.1) run: make e2e-compat-setup COMPAT_SCENARIO=old-controller @@ -287,7 +287,7 @@ jobs: - name: Install Go uses: actions/setup-go@v5 with: - go-version: "1.22" + go-version: "1.24" - name: Download controller image uses: actions/download-artifact@v4 diff --git a/e2e/LICENSE b/LICENSE similarity index 100% rename from e2e/LICENSE rename to LICENSE diff --git a/README.md b/README.md index f50fba910..2cad82f7b 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,9 @@ A free, open source tool for automated testing on real and virtual hardware with CI/CD integration. Simplify device automation with consistent rules across local -and distributed environments. +and distributed environments. Every interface is programmatic - there is no GUI +wall - so human developers, test scripts, CI pipelines, and AI agents interact +with hardware through the same APIs. ## Highlights @@ -17,7 +19,7 @@ and distributed environments. - 🐍 **Python-Powered** - Leverage Python's testing ecosystem - 🔌 **Hardware Abstraction** - Simplify complex hardware interfaces with drivers - 🌐 **Collaborative** - Share test hardware globally -- ⚙️ **CI/CD Ready** - Works with cloud native developer environments and pipelines +- ⚙️ **Automation Ready** - Same APIs for humans, test scripts, CI pipelines, and AI agents - 💻 **Cross-Platform** - Supports Linux and macOS ## Repository Structure @@ -115,7 +117,7 @@ make e2e-clean ### Prerequisites - Python 3.11+ (for Python components) -- Go 1.22+ (for controller) +- Go 1.24+ (for controller) - Docker/Podman (for container builds) - kubectl (for Kubernetes deployment) @@ -123,12 +125,12 @@ make e2e-clean ```shell # Build all components -make all +make build # Build specific components -make python # Python packages -make controller # Controller binary -make protocol # Generate protocol code +make build-python # Python packages +make build-controller # Controller binary +make build-protocol # Generate protocol code # Run tests make test @@ -139,20 +141,13 @@ make e2e # Run tests make e2e-clean # Clean up ``` -### Running Locally - -```shell -# Start a local development environment -make dev -``` - ## Documentation Jumpstarter's documentation is available at [jumpstarter.dev](https://jumpstarter.dev). - [Getting Started](https://jumpstarter.dev/main/getting-started/) - [User Guide](https://jumpstarter.dev/main/introduction/) -- [API Reference](https://jumpstarter.dev/main/api/) +- [API Reference](https://jumpstarter.dev/main/reference/) - [Contributing Guide](https://jumpstarter.dev/main/contributing.html) ## Contributing diff --git a/controller/README.md b/controller/README.md index 121715874..1624eb910 100644 --- a/controller/README.md +++ b/controller/README.md @@ -1,119 +1,24 @@ -# jumpstarter-controller +# Jumpstarter Controller -[![Build and push container image](https://github.com/jumpstarter-dev/jumpstarter-controller/actions/workflows/build.yaml/badge.svg)](https://github.com/jumpstarter-dev/jumpstarter-controller/actions/workflows/build.yaml) -![GitHub Release](https://img.shields.io/github/v/release/jumpstarter-dev/jumpstarter-controller) -![GitHub Downloads (all assets, all releases)](https://img.shields.io/github/downloads/jumpstarter-dev/jumpstarter-controller/total) -[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/jumpstarter-dev/jumpstarter-controller) +The Jumpstarter controller is the Kubernetes-native service component of +[Jumpstarter](https://jumpstarter.dev). It implements the server-side gRPC +services defined by the [Jumpstarter Protocol](../protocol/), running as a +Kubernetes operator that manages Custom Resources for clients, exporters, and +leases. The controller enables distributed hardware sharing by routing traffic +between clients and exporters, handling lease negotiation, and enforcing access +policies. -// TODO(user): Add simple overview of use/purpose - -## Description -// TODO(user): An in-depth paragraph about your project and overview of use - -## Getting Started - -### Prerequisites -- go version v1.22.0+ -- kubectl version v1.11.3+. -- Access to a Kubernetes v1.11.3+ cluster. - -### To Deploy on the cluster -**Build and push your image to the location specified by `IMG`:** +## Development ```sh make docker-push IMG=/jumpstarter-controller:tag -``` - -**NOTE:** This image ought to be published in the personal registry you specified. -And it is required to have access to pull the image from the working environment. -Make sure you have the proper permission to the registry if the above commands don’t work. - -**Install the CRDs into the cluster:** - -```sh make install -``` - -**Deploy the Manager to the cluster with the image specified by `IMG`:** - -```sh -make deploy IMG=/jumpstarter-router:tag -``` - -> **NOTE**: If you encounter RBAC errors, you may need to grant yourself cluster-admin -privileges or be logged in as admin. - -**Create instances of your solution** -You can apply the samples (examples) from the config/sample: - -```sh -kubectl apply -k config/samples/ -``` - ->**NOTE**: Ensure that the samples has default values to test it out. - -### To Uninstall -**Delete the instances (CRs) from the cluster:** - -```sh -kubectl delete -k config/samples/ -``` - -**Delete the APIs(CRDs) from the cluster:** - -```sh -make uninstall -``` - -**UnDeploy the controller from the cluster:** +make deploy IMG=/jumpstarter-controller:tag -```sh make undeploy +make uninstall ``` -## Project Distribution - -Following are the steps to build the installer and distribute this project to users. - -1. Build the installer for the image built and published in the registry: - -```sh -make build-installer IMG=/jumpstarter-router:tag -``` - -NOTE: The makefile target mentioned above generates an 'install.yaml' -file in the dist directory. This file contains all the resources built -with Kustomize, which are necessary to install this project without -its dependencies. - -2. Using the installer - -Users can just run kubectl apply -f to install the project, i.e.: - -```sh -kubectl apply -f https://raw.githubusercontent.com//jumpstarter-router//dist/install.yaml -``` - -## Contributing -// TODO(user): Add detailed information on how you would like others to contribute to this project - -**NOTE:** Run `make help` for more information on all potential `make` targets - -More information can be found via the [Kubebuilder Documentation](https://book.kubebuilder.io/introduction.html) - -## License - -Copyright 2024. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. - +For production deployment, see the +[Service Installation](https://jumpstarter.dev/main/getting-started/installation/index.html) +documentation. diff --git a/controller/api/v1alpha1/client_types.go b/controller/api/v1alpha1/client_types.go index 8e2481b9a..4fc2d81d3 100644 --- a/controller/api/v1alpha1/client_types.go +++ b/controller/api/v1alpha1/client_types.go @@ -21,15 +21,12 @@ import ( metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" ) -// EDIT THIS FILE! THIS IS SCAFFOLDING FOR YOU TO OWN! -// NOTE: json tags are required. Any new fields you add must have json tags for the fields to be serialized. - -// ClientSpec defines the desired state of Identity +// ClientSpec defines the desired state of Client type ClientSpec struct { Username *string `json:"username,omitempty"` } -// ClientStatus defines the observed state of Identity +// ClientStatus defines the observed state of Client type ClientStatus struct { // Status field for the clients Credential *corev1.LocalObjectReference `json:"credential,omitempty"` @@ -39,11 +36,11 @@ type ClientStatus struct { // +kubebuilder:object:root=true // +kubebuilder:subresource:status -// Client is the Schema for the identities API +// Client is the Schema for the clients API type Client struct { // The Client in the Jumpstarter controller represents a user that can access the Jumpstarter Controller. // Clients can be associated to external identity OIDC providers by providing Username, i.e. - // Spec.Username: "kc:user-name-in-keycloak" + // Spec.Username: "oidc:user@example.com" metav1.TypeMeta `json:",inline"` metav1.ObjectMeta `json:"metadata,omitempty"` @@ -53,7 +50,7 @@ type Client struct { // +kubebuilder:object:root=true -// ClientList contains a list of Identity +// ClientList contains a list of Client type ClientList struct { metav1.TypeMeta `json:",inline"` metav1.ListMeta `json:"metadata,omitempty"` diff --git a/controller/api/v1alpha1/exporter_types.go b/controller/api/v1alpha1/exporter_types.go index 1b4a9b8ef..9bc6cfbe7 100644 --- a/controller/api/v1alpha1/exporter_types.go +++ b/controller/api/v1alpha1/exporter_types.go @@ -21,9 +21,6 @@ import ( metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" ) -// EDIT THIS FILE! THIS IS SCAFFOLDING FOR YOU TO OWN! -// NOTE: json tags are required. Any new fields you add must have json tags for the fields to be serialized. - // ExporterSpec defines the desired state of Exporter type ExporterSpec struct { Username *string `json:"username,omitempty"` diff --git a/controller/api/v1alpha1/exporteraccesspolicy_types.go b/controller/api/v1alpha1/exporteraccesspolicy_types.go index 43b0efc80..a2b98ac6f 100644 --- a/controller/api/v1alpha1/exporteraccesspolicy_types.go +++ b/controller/api/v1alpha1/exporteraccesspolicy_types.go @@ -20,9 +20,6 @@ import ( metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" ) -// EDIT THIS FILE! THIS IS SCAFFOLDING FOR YOU TO OWN! -// NOTE: json tags are required. Any new fields you add must have json tags for the fields to be serialized. - type From struct { ClientSelector metav1.LabelSelector `json:"clientSelector,omitempty"` } diff --git a/controller/api/v1alpha1/lease_types.go b/controller/api/v1alpha1/lease_types.go index fe4c704dc..d91a5bc08 100644 --- a/controller/api/v1alpha1/lease_types.go +++ b/controller/api/v1alpha1/lease_types.go @@ -86,7 +86,7 @@ const ( // +kubebuilder:printcolumn:JSONPath=".spec.clientRef.name",name=Client,type=string // +kubebuilder:printcolumn:JSONPath=".status.exporterRef.name",name=Exporter,type=string -// Lease is the Schema for the exporters API +// Lease is the Schema for the leases API type Lease struct { // Lease is the schema for the Leases API. Leases represent a // request for a specific exporter by a client. The lease is diff --git a/controller/deploy/microshift-bootc/README.md b/controller/deploy/microshift-bootc/README.md deleted file mode 100644 index b35d92344..000000000 --- a/controller/deploy/microshift-bootc/README.md +++ /dev/null @@ -1,420 +0,0 @@ -# MicroShift Bootc Deployment - -This directory contains the configuration and scripts to build a bootable container (bootc) image with MicroShift and the Jumpstarter operator pre-installed. - -> **⚠️ Community Edition Disclaimer** -> -> This MicroShift-based deployment is a **community-supported edition** intended for development, testing, and evaluation scenarios. It is **not officially supported** for production use, although it can be OK for small labs. -> -> **For production deployments**, we strongly recommend using the official Jumpstarter Controller deployment on Kubernetes or OpenShift clusters with proper high availability, security, and support. See the [official installation documentation](https://jumpstarter.dev/main/getting-started/installation/service/index.html) for production deployment guides. - -## Overview - -This community edition deployment provides a lightweight, all-in-one solution ideal for: -- **Edge devices** with limited resources -- **Development and testing** environments -- **Proof-of-concept** deployments -- **Local experimentation** with Jumpstarter - -**Features:** -- **MicroShift 4.20 (OKD)** - Lightweight Kubernetes distribution -- **Jumpstarter Operator** - Pre-installed and ready to use -- **TopoLVM CSI** - Dynamic storage provisioning using LVM -- **Configuration Web UI** - Easy setup and management at port 8880 -- **Pod Monitoring** - Real-time pod status dashboard - -## Prerequisites - -- **Fedora/RHEL-based system** (tested on Fedora 42) -- **Podman** installed and configured -- **Root/sudo access** required for privileged operations -- **At least 4GB RAM** and 20GB disk space recommended - -## Quick Start - -### 1. Build the Bootc Image - -```bash -make bootc-build -``` - -This builds a container image with MicroShift and all dependencies. - -### 2. Run as Container (Development/Testing) - -```bash -make bootc-run -``` - -This will: -- Create a 1GB LVM disk image at `/var/lib/microshift-okd/lvmdisk.image` -- Start MicroShift in a privileged container -- Set up LVM volume groups inside the container for TopoLVM -- Wait for MicroShift to be ready - -**Output example:** -``` -MicroShift is running in a bootc container -Hostname: jumpstarter.10.0.2.2.nip.io -Container: jumpstarter-microshift-okd -LVM disk: /var/lib/microshift-okd/lvmdisk.image -VG name: myvg1 -Ports: HTTP:80, HTTPS:443, Config Service:8880 -``` - -### 3. Access the Services - -#### Configuration Web UI -- URL: `http://localhost:8880` -- Login: `root` / `jumpstarter` (default - you'll be required to change it) -- Features: - - Configure hostname and base domain - - Set controller image version - - Change root password (required on first use) - - Download kubeconfig - - Monitor pod status - -#### MicroShift API -- URL: `https://jumpstarter..nip.io:6443` -- Download kubeconfig from the web UI or extract from container - -#### Pod Monitoring Dashboard -- URL: `http://localhost:8880/pods` -- Auto-refreshes every 5 seconds -- Shows all pods across all namespaces - -## Container Management - -### View Running Pods - -```bash -sudo podman exec -it jumpstarter-microshift-okd oc get pods -A -``` - -### Open Shell in Container - -```bash -make bootc-sh -``` - -### Stop Container - -```bash -make bootc-stop -``` - -### Remove Container - -```bash -make bootc-rm -``` - -This will: -- Stop the container -- Remove the container -- Clean up LVM volume groups (myvg1) -- Detach loop devices - -**Note:** The LVM disk image (`/var/lib/microshift-okd/lvmdisk.image`) is preserved. To remove it completely, use `make clean`. - -### Complete Rebuild - -```bash -make bootc-rm bootc-build bootc-run -``` - -This stops, removes, rebuilds, and restarts the container with the latest changes. - -## Creating a Bootable QCOW2 Image - -For production deployments, you can create a bootable QCOW2 disk image that can be: -- Installed on bare metal -- Used in virtual machines (KVM/QEMU, OpenStack, etc.) -- Deployed to edge devices - -### Build QCOW2 Image - -```bash -make build-image -``` - -This will: -1. Clean up any existing LVM resources to avoid conflicts -2. Build the bootc container image (if not already built) -3. Use `bootc-image-builder` to create a bootable QCOW2 image -4. Output the image to `./output/qcow2/disk.qcow2` - -**Note:** This process takes several minutes and requires significant disk space (20GB+). - -**Important:** If you're running the container (`make bootc-run`) and want to build the image, stop the container first with `make bootc-rm` to avoid LVM conflicts. - -### Configuration - -The QCOW2 image is configured via `config.toml`: -- **LVM partitioning:** Creates `myvg1` volume group with 20GB minimum -- **Root filesystem:** XFS on LVM (10GB minimum) -- **Default password:** `root:jumpstarter` (change via web UI on first boot) - -### Using the QCOW2 Image - -#### In a Virtual Machine (KVM/QEMU) - -```bash -qemu-system-x86_64 \ - -m 4096 \ - -smp 2 \ - -drive file=output/qcow2/disk.qcow2,format=qcow2 \ - -net nic -net user,hostfwd=tcp::8880-:8880,hostfwd=tcp::443-:443 -``` - -#### Convert to Other Formats - -```bash -# Convert to raw disk image -qemu-img convert -f qcow2 -O raw output/qcow2/disk.qcow2 output/disk.raw - -# Convert to VirtualBox VDI -qemu-img convert -f qcow2 -O vdi output/qcow2/disk.qcow2 output/disk.vdi -``` - -## Architecture - -### Components - -``` -┌─────────────────────────────────────────────┐ -│ Bootc Container / Image │ -├─────────────────────────────────────────────┤ -│ • Fedora CoreOS 9 base │ -│ • MicroShift 4.20 (OKD) │ -│ • Jumpstarter Operator │ -│ • TopoLVM CSI (storage) │ -│ • Configuration Service (Python/Flask) │ -│ • Firewalld (ports 22, 80, 443, 8880) │ -└─────────────────────────────────────────────┘ -``` - -### Storage Setup - -When running as a container: -1. Script creates `/var/lib/microshift-okd/lvmdisk.image` (1GB) -2. Image is copied into the container -3. Loop device is created inside container -4. LVM volume group `myvg1` is created -5. TopoLVM uses `myvg1` for dynamic PV provisioning - -When deployed from QCOW2: -1. Bootc image builder creates proper disk partitioning -2. LVM volume group `myvg1` is set up on disk -3. Root filesystem uses part of the VG -4. Remaining space available for TopoLVM - -## Customization - -### Change Default Image - -```bash -BOOTC_IMG=quay.io/your-org/microshift-bootc:v1.0 make bootc-build -``` - -### Modify Manifests - -Add Kubernetes manifests to `/etc/microshift/manifests.d/002-jumpstarter/` by editing: -- `kustomization.yaml` - Kustomize configuration -- Additional YAML files will be automatically applied - -### Update Configuration Service - -Edit `config-svc/app.py` and rebuild: - -```bash -make bootc-build -``` - -For live testing without rebuild: - -```bash -make bootc-reload-app -``` - -## Troubleshooting - -### LVM/TopoLVM Issues - -Check if volume group exists in container: - -```bash -sudo podman exec jumpstarter-microshift-okd vgs -sudo podman exec jumpstarter-microshift-okd pvs -``` - -If TopoLVM pods are crashing, recreate the LVM setup: - -```bash -make bootc-rm # Automatically cleans up VG and loop devices -make clean # Remove the disk image for a fresh start -make bootc-run -``` - -### MicroShift Not Starting - -Check logs: - -```bash -sudo podman logs jumpstarter-microshift-okd -sudo podman exec jumpstarter-microshift-okd journalctl -u microshift -f -``` - -### Configuration Service Issues - -Check service status: - -```bash -sudo podman exec jumpstarter-microshift-okd systemctl status config-svc -sudo podman exec jumpstarter-microshift-okd journalctl -u config-svc -f -``` - -### Port Conflicts - -If ports 80, 443, or 8880 are in use, modify `run-microshift.sh`: - -```bash -HTTP_PORT=8080 -HTTPS_PORT=8443 -CONFIG_SVC_PORT=9880 -``` - -### Bootc Image Builder Fails - -Ensure sufficient disk space and clean up: - -```bash -sudo podman system prune -a -sudo rm -rf output/ -``` - -## Makefile Targets - -| Target | Description | -|--------|-------------| -| `make help` | Display all available targets | -| `make bootc-build` | Build the bootc container image | -| `make bootc-run` | Run MicroShift in a container | -| `make bootc-stop` | Stop the running container | -| `make bootc-rm` | Remove container and clean up LVM resources | -| `make bootc-sh` | Open shell in container | -| `make bootc-reload-app` | Reload config service without rebuild (dev mode) | -| `make build-image` | Create bootable QCOW2 image | -| `make bootc-push` | Push image to registry | -| `make clean` | Clean up images, artifacts, and LVM disk | - -## Files - -| File | Description | -|------|-------------| -| `Containerfile` | Container build definition | -| `config.toml` | Bootc image builder configuration | -| `run-microshift.sh` | Container startup script | -| `kustomization.yaml` | Kubernetes manifests configuration | -| `config-svc/app.py` | Configuration web UI service | -| `config-svc/config-svc.service` | Systemd service definition | - -## Network Configuration - -### Hostname Resolution - -The system uses `nip.io` for automatic DNS resolution: -- Default: `jumpstarter..nip.io` -- Example: `jumpstarter.10.0.2.2.nip.io` resolves to `10.0.2.2` - -### Firewall Ports - -| Port | Service | Description | -|------|---------|-------------| -| 80 | HTTP | MicroShift ingress | -| 443 | HTTPS | MicroShift API and ingress | -| 8880 | Config UI | Web configuration interface | -| 6443 | API Server | Kubernetes API (internal) | - -## Security Notes - -⚠️ **Important Security Considerations:** - -1. **Default Password:** The system ships with `root:jumpstarter` as the default password - - **Console login:** You will be forced to change the password on first SSH/console login - - **Web UI:** You must change the password before accessing the configuration interface -2. **TLS Certificates:** MicroShift uses self-signed certs by default -3. **Privileged Container:** Required for systemd, LVM, and networking -4. **Authentication:** Web UI uses PAM authentication with root credentials -5. **Production Use:** Consider additional hardening for production deployments - -## Development Workflow - -Typical development cycle: - -```bash -# 1. Make changes to code/configuration -vim config-svc/app.py - -# 2. Quick reload (no rebuild needed) -make bootc-reload-app - -# 3. Access and test -curl http://localhost:8880 - -# 4. Check logs if issues -make bootc-sh -journalctl -u config-svc -f - -# 5. For major changes, do full rebuild -make bootc-rm bootc-build bootc-run -``` - -## Production Deployment - -1. **Build QCOW2 image:** - ```bash - make build-image - ``` - -2. **Copy image to target system:** - ```bash - scp output/qcow2/disk.qcow2 target-host:/var/lib/libvirt/images/ - ``` - -3. **Create VM or write to disk:** - ```bash - # For VM - virt-install --name jumpstarter \ - --memory 4096 \ - --vcpus 2 \ - --disk path=/var/lib/libvirt/images/disk.qcow2 \ - --import \ - --os-variant fedora39 - - # For bare metal - dd if=output/qcow2/disk.qcow2 of=/dev/sdX bs=4M status=progress - ``` - -4. **First boot:** - - Console login will require password change from default `jumpstarter` - - Access web UI at `http://:8880` and set new password - -## Resources - -### Jumpstarter Documentation -- [Official Installation Guide](https://jumpstarter.dev/main/getting-started/installation/service/index.html) - **Recommended for production** -- [Jumpstarter Project](https://github.com/jumpstarter-dev/jumpstarter) - -### Technology Stack -- [MicroShift Documentation](https://microshift.io/) -- [Bootc Documentation](https://containers.github.io/bootc/) -- [TopoLVM Documentation](https://github.com/topolvm/topolvm) - -## Support - -For issues and questions: -- File issues on the Jumpstarter GitHub repository -- Check container logs: `sudo podman logs jumpstarter-microshift-okd` -- Review systemd journals: `make bootc-sh` then `journalctl -xe` - diff --git a/controller/deploy/microshift-bootc/README.md b/controller/deploy/microshift-bootc/README.md new file mode 120000 index 000000000..6c71dda07 --- /dev/null +++ b/controller/deploy/microshift-bootc/README.md @@ -0,0 +1 @@ +../../../python/docs/source/getting-started/installation/service/bootc.md \ No newline at end of file diff --git a/controller/deploy/microshift-bootc/config-svc/README.md b/controller/deploy/microshift-bootc/config-svc/README.md index e3ea32bd1..2a151f92c 100644 --- a/controller/deploy/microshift-bootc/config-svc/README.md +++ b/controller/deploy/microshift-bootc/config-svc/README.md @@ -165,13 +165,4 @@ mypy app.py auth.py system.py api.py routes.py - Hostname validation per RFC 1123 - Default password change enforcement -## License - -Apache License 2.0 - -## Links - -- Homepage: https://jumpstarter.dev -- Documentation: https://docs.jumpstarter.dev -- Repository: https://github.com/jumpstarter-dev/jumpstarter-controller diff --git a/controller/deploy/operator/README.md b/controller/deploy/operator/README.md index f03cb1359..11f790eb6 100644 --- a/controller/deploy/operator/README.md +++ b/controller/deploy/operator/README.md @@ -1,117 +1,21 @@ -# jumpstarter-operator -// TODO(user): Add simple overview of use/purpose +# Jumpstarter Operator -## Description -// TODO(user): An in-depth paragraph about your project and overview of use +The Jumpstarter operator manages the lifecycle of Jumpstarter controller +components on Kubernetes using the Operator Lifecycle Manager (OLM). It +packages the controller, router, and associated resources into a single +installable unit. -## Getting Started - -### Prerequisites -- go version v1.24.0+ -- docker version 17.03+. -- kubectl version v1.11.3+. -- Access to a Kubernetes v1.11.3+ cluster. - -### To Deploy on the cluster -**Build and push your image to the location specified by `IMG`:** +## Development ```sh make docker-build docker-push IMG=/jumpstarter-operator:tag -``` - -**NOTE:** This image ought to be published in the personal registry you specified. -And it is required to have access to pull the image from the working environment. -Make sure you have the proper permission to the registry if the above commands don’t work. - -**Install the CRDs into the cluster:** - -```sh make install -``` - -**Deploy the Manager to the cluster with the image specified by `IMG`:** - -```sh make deploy IMG=/jumpstarter-operator:tag -``` - -> **NOTE**: If you encounter RBAC errors, you may need to grant yourself cluster-admin -privileges or be logged in as admin. -**Create instances of your solution** -You can apply the samples (examples) from the config/sample: - -```sh -kubectl apply -k config/samples/ -``` - ->**NOTE**: Ensure that the samples has default values to test it out. - -### To Uninstall -**Delete the instances (CRs) from the cluster:** - -```sh -kubectl delete -k config/samples/ -``` - -**Delete the APIs(CRDs) from the cluster:** - -```sh -make uninstall -``` - -**UnDeploy the controller from the cluster:** - -```sh make undeploy +make uninstall ``` -## Project Distribution - -Following the options to release and provide this solution to the users. - -### By providing a bundle with all YAML files - -1. Build the installer for the image built and published in the registry: - -```sh -make build-installer IMG=/jumpstarter-operator:tag -``` - -**NOTE:** The makefile target mentioned above generates an 'install.yaml' -file in the dist directory. This file contains all the resources built -with Kustomize, which are necessary to install this project without its -dependencies. - -2. Using the installer - -Users can just run 'kubectl apply -f ' to install -the project, i.e.: - -```sh -kubectl apply -f https://raw.githubusercontent.com//jumpstarter-operator//dist/install.yaml -``` - -## Contributing -// TODO(user): Add detailed information on how you would like others to contribute to this project - -**NOTE:** Run `make help` for more information on all potential `make` targets - -More information can be found via the [Kubebuilder Documentation](https://book.kubebuilder.io/introduction.html) - -## License - -Copyright 2025. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. - +For production deployment, see the +[Service Installation](https://jumpstarter.dev/main/getting-started/installation/index.html) +documentation. diff --git a/controller/deploy/operator/api/v1alpha1/jumpstarter_types.go b/controller/deploy/operator/api/v1alpha1/jumpstarter_types.go index 5f87b4eb3..371d8afd2 100644 --- a/controller/deploy/operator/api/v1alpha1/jumpstarter_types.go +++ b/controller/deploy/operator/api/v1alpha1/jumpstarter_types.go @@ -320,7 +320,7 @@ type GRPCKeepaliveConfig struct { // Timeout for keepalive ping acknowledgment. // If a ping is not acknowledged within this time, the connection is considered broken. - // The default is high to avoid issues when the network on a exporter is overloaded, i.e. + // The default is high to avoid issues when the network on an exporter is overloaded, i.e. // during flashing. // +kubebuilder:default="180s" Timeout *metav1.Duration `json:"timeout,omitempty"` @@ -411,7 +411,7 @@ type K8sAuthConfig struct { type TLSConfig struct { // Name of the Kubernetes secret containing the TLS certificate and private key. // The secret must contain 'tls.crt' and 'tls.key' keys. - // If useCertManager is enabled, this secret will be automatically managed and + // If spec.certManager.enabled is true, this secret will be automatically managed and // configured by cert-manager. // +kubebuilder:validation:Pattern=^[a-z0-9]([a-z0-9\-\.]*[a-z0-9])?$ CertSecret string `json:"certSecret,omitempty"` @@ -540,7 +540,7 @@ type NodePortConfig struct { Enabled bool `json:"enabled,omitempty"` // NodePort port number to expose on each node. - // Must be in the range 30000-32767 for most Kubernetes clusters. + // Must be a valid port number (1-65535). Kubernetes typically allocates NodePorts in the range 30000-32767 by default. // +kubebuilder:validation:Minimum=1 // +kubebuilder:validation:Maximum=65535 Port int32 `json:"port,omitempty"` @@ -617,7 +617,7 @@ type ServerCertConfig struct { // Reference an existing cert-manager Issuer or ClusterIssuer. // Use this to integrate with existing PKI infrastructure (ACME, Vault, etc.). - // This overrides SelfSigned.Enabled = true which is the default setting + // This overrides the default selfSigned.enabled=true setting. IssuerRef *IssuerReference `json:"issuerRef,omitempty"` } diff --git a/controller/deploy/operator/config/crd/bases/jumpstarter.dev_clients.yaml b/controller/deploy/operator/config/crd/bases/jumpstarter.dev_clients.yaml index fb6bfc155..d6db9d4a2 100644 --- a/controller/deploy/operator/config/crd/bases/jumpstarter.dev_clients.yaml +++ b/controller/deploy/operator/config/crd/bases/jumpstarter.dev_clients.yaml @@ -17,7 +17,7 @@ spec: - name: v1alpha1 schema: openAPIV3Schema: - description: Client is the Schema for the identities API + description: Client is the Schema for the clients API properties: apiVersion: description: |- @@ -37,13 +37,13 @@ spec: metadata: type: object spec: - description: ClientSpec defines the desired state of Identity + description: ClientSpec defines the desired state of Client properties: username: type: string type: object status: - description: ClientStatus defines the observed state of Identity + description: ClientStatus defines the observed state of Client properties: credential: description: Status field for the clients diff --git a/controller/deploy/operator/config/crd/bases/jumpstarter.dev_leases.yaml b/controller/deploy/operator/config/crd/bases/jumpstarter.dev_leases.yaml index 3d53f4386..e27ad1aa0 100644 --- a/controller/deploy/operator/config/crd/bases/jumpstarter.dev_leases.yaml +++ b/controller/deploy/operator/config/crd/bases/jumpstarter.dev_leases.yaml @@ -27,7 +27,7 @@ spec: name: v1alpha1 schema: openAPIV3Schema: - description: Lease is the Schema for the exporters API + description: Lease is the Schema for the leases API properties: apiVersion: description: |- diff --git a/controller/deploy/operator/config/crd/bases/operator.jumpstarter.dev_jumpstarters.yaml b/controller/deploy/operator/config/crd/bases/operator.jumpstarter.dev_jumpstarters.yaml index c0ed0adc3..09b2c925c 100644 --- a/controller/deploy/operator/config/crd/bases/operator.jumpstarter.dev_jumpstarters.yaml +++ b/controller/deploy/operator/config/crd/bases/operator.jumpstarter.dev_jumpstarters.yaml @@ -474,7 +474,7 @@ spec: description: |- Reference an existing cert-manager Issuer or ClusterIssuer. Use this to integrate with existing PKI infrastructure (ACME, Vault, etc.). - This overrides SelfSigned.Enabled = true which is the default setting + This overrides the default selfSigned.enabled=true setting. properties: caBundle: description: |- @@ -706,7 +706,7 @@ spec: port: description: |- NodePort port number to expose on each node. - Must be in the range 30000-32767 for most Kubernetes clusters. + Must be a valid port number (1-65535). Kubernetes typically allocates NodePorts in the range 30000-32767 by default. format: int32 maximum: 65535 minimum: 1 @@ -789,7 +789,7 @@ spec: description: |- Timeout for keepalive ping acknowledgment. If a ping is not acknowledged within this time, the connection is considered broken. - The default is high to avoid issues when the network on a exporter is overloaded, i.e. + The default is high to avoid issues when the network on an exporter is overloaded, i.e. during flashing. type: string type: object @@ -804,7 +804,7 @@ spec: description: |- Name of the Kubernetes secret containing the TLS certificate and private key. The secret must contain 'tls.crt' and 'tls.key' keys. - If useCertManager is enabled, this secret will be automatically managed and + If spec.certManager.enabled is true, this secret will be automatically managed and configured by cert-manager. pattern: ^[a-z0-9]([a-z0-9\-\.]*[a-z0-9])?$ type: string @@ -978,7 +978,7 @@ spec: port: description: |- NodePort port number to expose on each node. - Must be in the range 30000-32767 for most Kubernetes clusters. + Must be a valid port number (1-65535). Kubernetes typically allocates NodePorts in the range 30000-32767 by default. format: int32 maximum: 65535 minimum: 1 @@ -1248,7 +1248,7 @@ spec: port: description: |- NodePort port number to expose on each node. - Must be in the range 30000-32767 for most Kubernetes clusters. + Must be a valid port number (1-65535). Kubernetes typically allocates NodePorts in the range 30000-32767 by default. format: int32 maximum: 65535 minimum: 1 @@ -1292,7 +1292,7 @@ spec: description: |- Name of the Kubernetes secret containing the TLS certificate and private key. The secret must contain 'tls.crt' and 'tls.key' keys. - If useCertManager is enabled, this secret will be automatically managed and + If spec.certManager.enabled is true, this secret will be automatically managed and configured by cert-manager. pattern: ^[a-z0-9]([a-z0-9\-\.]*[a-z0-9])?$ type: string @@ -1466,7 +1466,7 @@ spec: port: description: |- NodePort port number to expose on each node. - Must be in the range 30000-32767 for most Kubernetes clusters. + Must be a valid port number (1-65535). Kubernetes typically allocates NodePorts in the range 30000-32767 by default. format: int32 maximum: 65535 minimum: 1 @@ -1549,7 +1549,7 @@ spec: description: |- Timeout for keepalive ping acknowledgment. If a ping is not acknowledged within this time, the connection is considered broken. - The default is high to avoid issues when the network on a exporter is overloaded, i.e. + The default is high to avoid issues when the network on an exporter is overloaded, i.e. during flashing. type: string type: object @@ -1564,7 +1564,7 @@ spec: description: |- Name of the Kubernetes secret containing the TLS certificate and private key. The secret must contain 'tls.crt' and 'tls.key' keys. - If useCertManager is enabled, this secret will be automatically managed and + If spec.certManager.enabled is true, this secret will be automatically managed and configured by cert-manager. pattern: ^[a-z0-9]([a-z0-9\-\.]*[a-z0-9])?$ type: string diff --git a/controller/deploy/operator/config/manifests/bases/jumpstarter-operator.clusterserviceversion.yaml b/controller/deploy/operator/config/manifests/bases/jumpstarter-operator.clusterserviceversion.yaml index d5fa1b039..f4156f8ab 100644 --- a/controller/deploy/operator/config/manifests/bases/jumpstarter-operator.clusterserviceversion.yaml +++ b/controller/deploy/operator/config/manifests/bases/jumpstarter-operator.clusterserviceversion.yaml @@ -61,7 +61,7 @@ metadata: description: Jumpstarter is a cloud-native framework for Hardware-in-the-Loop (HIL) testing and development operators.operatorframework.io/internal-objects: '["clients.jumpstarter.dev","exporters.jumpstarter.dev","leases.jumpstarter.dev","exporteraccesspolicies.jumpstarter.dev"]' - repository: https://github.com/jumpstarter-dev/jumpstarter-controller + repository: https://github.com/jumpstarter-dev/jumpstarter support: The Jumpstarter Community labels: operatorframework.io/arch.amd64: supported diff --git a/controller/internal/service/login/service.go b/controller/internal/service/login/service.go index 4ea8e4b3a..dbddd2e7d 100644 --- a/controller/internal/service/login/service.go +++ b/controller/internal/service/login/service.go @@ -137,7 +137,7 @@ func (s *Service) Start(ctx context.Context) error { // Otherwise treat it as a bare port and prepend ":". if port != "" { if _, _, err := net.SplitHostPort(port); err != nil { - // Not a valid host:port — assume bare port (e.g. "8086") + // Not a valid host:port - assume bare port (e.g. "8086") if port[0] != ':' { port = ":" + port } diff --git a/protocol/LICENSE b/protocol/LICENSE deleted file mode 100644 index 9b5e4019d..000000000 --- a/protocol/LICENSE +++ /dev/null @@ -1,202 +0,0 @@ - - Apache License - Version 2.0, January 2004 - http://www.apache.org/licenses/ - - TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION - - 1. Definitions. - - "License" shall mean the terms and conditions for use, reproduction, - and distribution as defined by Sections 1 through 9 of this document. - - "Licensor" shall mean the copyright owner or entity authorized by - the copyright owner that is granting the License. - - "Legal Entity" shall mean the union of the acting entity and all - other entities that control, are controlled by, or are under common - control with that entity. For the purposes of this definition, - "control" means (i) the power, direct or indirect, to cause the - direction or management of such entity, whether by contract or - otherwise, or (ii) ownership of fifty percent (50%) or more of the - outstanding shares, or (iii) beneficial ownership of such entity. - - "You" (or "Your") shall mean an individual or Legal Entity - exercising permissions granted by this License. - - "Source" form shall mean the preferred form for making modifications, - including but not limited to software source code, documentation - source, and configuration files. - - "Object" form shall mean any form resulting from mechanical - transformation or translation of a Source form, including but - not limited to compiled object code, generated documentation, - and conversions to other media types. - - "Work" shall mean the work of authorship, whether in Source or - Object form, made available under the License, as indicated by a - copyright notice that is included in or attached to the work - (an example is provided in the Appendix below). - - "Derivative Works" shall mean any work, whether in Source or Object - form, that is based on (or derived from) the Work and for which the - editorial revisions, annotations, elaborations, or other modifications - represent, as a whole, an original work of authorship. For the purposes - of this License, Derivative Works shall not include works that remain - separable from, or merely link (or bind by name) to the interfaces of, - the Work and Derivative Works thereof. - - "Contribution" shall mean any work of authorship, including - the original version of the Work and any modifications or additions - to that Work or Derivative Works thereof, that is intentionally - submitted to Licensor for inclusion in the Work by the copyright owner - or by an individual or Legal Entity authorized to submit on behalf of - the copyright owner. For the purposes of this definition, "submitted" - means any form of electronic, verbal, or written communication sent - to the Licensor or its representatives, including but not limited to - communication on electronic mailing lists, source code control systems, - and issue tracking systems that are managed by, or on behalf of, the - Licensor for the purpose of discussing and improving the Work, but - excluding communication that is conspicuously marked or otherwise - designated in writing by the copyright owner as "Not a Contribution." - - "Contributor" shall mean Licensor and any individual or Legal Entity - on behalf of whom a Contribution has been received by Licensor and - subsequently incorporated within the Work. - - 2. Grant of Copyright License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - copyright license to reproduce, prepare Derivative Works of, - publicly display, publicly perform, sublicense, and distribute the - Work and such Derivative Works in Source or Object form. - - 3. Grant of Patent License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - (except as stated in this section) patent license to make, have made, - use, offer to sell, sell, import, and otherwise transfer the Work, - where such license applies only to those patent claims licensable - by such Contributor that are necessarily infringed by their - Contribution(s) alone or by combination of their Contribution(s) - with the Work to which such Contribution(s) was submitted. If You - institute patent litigation against any entity (including a - cross-claim or counterclaim in a lawsuit) alleging that the Work - or a Contribution incorporated within the Work constitutes direct - or contributory patent infringement, then any patent licenses - granted to You under this License for that Work shall terminate - as of the date such litigation is filed. - - 4. Redistribution. You may reproduce and distribute copies of the - Work or Derivative Works thereof in any medium, with or without - modifications, and in Source or Object form, provided that You - meet the following conditions: - - (a) You must give any other recipients of the Work or - Derivative Works a copy of this License; and - - (b) You must cause any modified files to carry prominent notices - stating that You changed the files; and - - (c) You must retain, in the Source form of any Derivative Works - that You distribute, all copyright, patent, trademark, and - attribution notices from the Source form of the Work, - excluding those notices that do not pertain to any part of - the Derivative Works; and - - (d) If the Work includes a "NOTICE" text file as part of its - distribution, then any Derivative Works that You distribute must - include a readable copy of the attribution notices contained - within such NOTICE file, excluding those notices that do not - pertain to any part of the Derivative Works, in at least one - of the following places: within a NOTICE text file distributed - as part of the Derivative Works; within the Source form or - documentation, if provided along with the Derivative Works; or, - within a display generated by the Derivative Works, if and - wherever such third-party notices normally appear. The contents - of the NOTICE file are for informational purposes only and - do not modify the License. You may add Your own attribution - notices within Derivative Works that You distribute, alongside - or as an addendum to the NOTICE text from the Work, provided - that such additional attribution notices cannot be construed - as modifying the License. - - You may add Your own copyright statement to Your modifications and - may provide additional or different license terms and conditions - for use, reproduction, or distribution of Your modifications, or - for any such Derivative Works as a whole, provided Your use, - reproduction, and distribution of the Work otherwise complies with - the conditions stated in this License. - - 5. Submission of Contributions. Unless You explicitly state otherwise, - any Contribution intentionally submitted for inclusion in the Work - by You to the Licensor shall be under the terms and conditions of - this License, without any additional terms or conditions. - Notwithstanding the above, nothing herein shall supersede or modify - the terms of any separate license agreement you may have executed - with Licensor regarding such Contributions. - - 6. Trademarks. This License does not grant permission to use the trade - names, trademarks, service marks, or product names of the Licensor, - except as required for reasonable and customary use in describing the - origin of the Work and reproducing the content of the NOTICE file. - - 7. Disclaimer of Warranty. Unless required by applicable law or - agreed to in writing, Licensor provides the Work (and each - Contributor provides its Contributions) on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or - implied, including, without limitation, any warranties or conditions - of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A - PARTICULAR PURPOSE. You are solely responsible for determining the - appropriateness of using or redistributing the Work and assume any - risks associated with Your exercise of permissions under this License. - - 8. Limitation of Liability. In no event and under no legal theory, - whether in tort (including negligence), contract, or otherwise, - unless required by applicable law (such as deliberate and grossly - negligent acts) or agreed to in writing, shall any Contributor be - liable to You for damages, including any direct, indirect, special, - incidental, or consequential damages of any character arising as a - result of this License or out of the use or inability to use the - Work (including but not limited to damages for loss of goodwill, - work stoppage, computer failure or malfunction, or any and all - other commercial damages or losses), even if such Contributor - has been advised of the possibility of such damages. - - 9. Accepting Warranty or Additional Liability. While redistributing - the Work or Derivative Works thereof, You may choose to offer, - and charge a fee for, acceptance of support, warranty, indemnity, - or other liability obligations and/or rights consistent with this - License. However, in accepting such obligations, You may act only - on Your own behalf and on Your sole responsibility, not on behalf - of any other Contributor, and only if You agree to indemnify, - defend, and hold each Contributor harmless for any liability - incurred by, or claims asserted against, such Contributor by reason - of your accepting any such warranty or additional liability. - - END OF TERMS AND CONDITIONS - - APPENDIX: How to apply the Apache License to your work. - - To apply the Apache License to your work, attach the following - boilerplate notice, with the fields enclosed by brackets "[]" - replaced with your own identifying information. (Don't include - the brackets!) The text should be enclosed in the appropriate - comment syntax for the file format. We also recommend that a - file or class name and description of purpose be included on the - same "printed page" as the copyright notice for easier - identification within third-party archives. - - Copyright [yyyy] [name of copyright owner] - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. \ No newline at end of file diff --git a/protocol/README.md b/protocol/README.md index 6b3773a69..cb9d09bda 100644 --- a/protocol/README.md +++ b/protocol/README.md @@ -1,40 +1,26 @@ # Jumpstarter Protocol -The Jumpstarter Protocol defines the gRPC-based communication layer for the [Jumpstarter](https://jumpstarter.dev) Hardware-in-the-Loop (HiL) ecosystem. It enables seamless, secure, and scalable interaction between clients, the Jumpstarter Service, and exporters—whether they are interfacing with physical or virtual hardware, locally or remotely. - -## Overview -Jumpstarter Protocol provides a unified gRPC interface for: - -- **Clients** to control and monitor remote/local hardware -- **Exporters** to expose hardware interfaces over gRPC -- **Jumpstarter Service** to route and manage connections - -Thanks to gRPC’s support for HTTP/2, streaming, and tunneling, the protocol works efficiently across enterprise networks, VPNs, and cloud environments. It appears as standard HTTPS traffic, making it compatible with existing security infrastructure. - -## Features -- 🔌 **Unified Interface:** Interact with virtual or physical hardware through a consistent API. -- 🔐 **Secure by Design:** Leverages gRPC over HTTPS for encrypted communication. -- 🌐 **Flexible Topology:** Supports direct or routed connections via the Jumpstarter Router. -- 📡 **Tunneling Support:** Can tunnel Unix sockets, TCP, and UDP connections over gRPC streams. - -## Related Projects - -- [**Jumpstarter Python:**](https://github.com/jumpstarter-dev/jumpstarter) The Python implementation of this protocol for clients and exporters. -- [**Jumpstarter Service:**](https://github.com/jumpstarter-dev/jumpstarter-controller) The Go implementation of this protocol as a Kubernetes controller. - - -## Documentation - -Jumpstarter's documentation is available at -[jumpstarter.dev](https://jumpstarter.dev). - -## Contributing - -Jumpstarter welcomes contributors of all levels of experience and would love to -see you involved in the project. See the [contributing -guide](https://jumpstarter.dev/contributing/) to get started. - -## License - -Jumpstarter is licensed under the Apache 2.0 License ([LICENSE](LICENSE) or -[https://www.apache.org/licenses/LICENSE-2.0](https://www.apache.org/licenses/LICENSE-2.0)). +The Jumpstarter Protocol defines the gRPC-based communication layer for the +[Jumpstarter](https://jumpstarter.dev) Hardware-in-the-Loop (HiL) ecosystem. It +enables seamless, secure, and scalable interaction between clients, the +Jumpstarter Service, and exporters - whether they are interfacing with physical +or virtual hardware, locally or remotely. + +The protocol provides a unified gRPC interface for clients to control and monitor +hardware, exporters to expose hardware interfaces, and the Jumpstarter Service to +route and manage connections. Thanks to gRPC's support for HTTP/2, streaming, and +tunneling, the protocol works efficiently across enterprise networks, VPNs, and +cloud environments. + +## Code Generation + +The protobuf definitions live under `proto/`. Downstream consumers generate +language-specific bindings using [Buf](https://buf.build/). Both the controller +(Go) and the Python packages maintain their own `buf.gen.yaml` to generate stubs +from these definitions. + +## Development + +```sh +make lint +``` diff --git a/python/.gitignore b/python/.gitignore index f771a16b7..3c793c7e2 100644 --- a/python/.gitignore +++ b/python/.gitignore @@ -71,6 +71,11 @@ instance/ # Sphinx documentation docs/build/ docs/build_all/ +docs/source/reference/crds/client.md +docs/source/reference/crds/exporter.md +docs/source/reference/crds/exporteraccesspolicy.md +docs/source/reference/crds/jumpstarter.md +docs/source/reference/crds/lease.md # PyBuilder .pybuilder/ diff --git a/python/LICENSE b/python/LICENSE deleted file mode 100644 index 7a4a3ea24..000000000 --- a/python/LICENSE +++ /dev/null @@ -1,202 +0,0 @@ - - Apache License - Version 2.0, January 2004 - http://www.apache.org/licenses/ - - TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION - - 1. Definitions. - - "License" shall mean the terms and conditions for use, reproduction, - and distribution as defined by Sections 1 through 9 of this document. - - "Licensor" shall mean the copyright owner or entity authorized by - the copyright owner that is granting the License. - - "Legal Entity" shall mean the union of the acting entity and all - other entities that control, are controlled by, or are under common - control with that entity. For the purposes of this definition, - "control" means (i) the power, direct or indirect, to cause the - direction or management of such entity, whether by contract or - otherwise, or (ii) ownership of fifty percent (50%) or more of the - outstanding shares, or (iii) beneficial ownership of such entity. - - "You" (or "Your") shall mean an individual or Legal Entity - exercising permissions granted by this License. - - "Source" form shall mean the preferred form for making modifications, - including but not limited to software source code, documentation - source, and configuration files. - - "Object" form shall mean any form resulting from mechanical - transformation or translation of a Source form, including but - not limited to compiled object code, generated documentation, - and conversions to other media types. - - "Work" shall mean the work of authorship, whether in Source or - Object form, made available under the License, as indicated by a - copyright notice that is included in or attached to the work - (an example is provided in the Appendix below). - - "Derivative Works" shall mean any work, whether in Source or Object - form, that is based on (or derived from) the Work and for which the - editorial revisions, annotations, elaborations, or other modifications - represent, as a whole, an original work of authorship. For the purposes - of this License, Derivative Works shall not include works that remain - separable from, or merely link (or bind by name) to the interfaces of, - the Work and Derivative Works thereof. - - "Contribution" shall mean any work of authorship, including - the original version of the Work and any modifications or additions - to that Work or Derivative Works thereof, that is intentionally - submitted to Licensor for inclusion in the Work by the copyright owner - or by an individual or Legal Entity authorized to submit on behalf of - the copyright owner. For the purposes of this definition, "submitted" - means any form of electronic, verbal, or written communication sent - to the Licensor or its representatives, including but not limited to - communication on electronic mailing lists, source code control systems, - and issue tracking systems that are managed by, or on behalf of, the - Licensor for the purpose of discussing and improving the Work, but - excluding communication that is conspicuously marked or otherwise - designated in writing by the copyright owner as "Not a Contribution." - - "Contributor" shall mean Licensor and any individual or Legal Entity - on behalf of whom a Contribution has been received by Licensor and - subsequently incorporated within the Work. - - 2. Grant of Copyright License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - copyright license to reproduce, prepare Derivative Works of, - publicly display, publicly perform, sublicense, and distribute the - Work and such Derivative Works in Source or Object form. - - 3. Grant of Patent License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - (except as stated in this section) patent license to make, have made, - use, offer to sell, sell, import, and otherwise transfer the Work, - where such license applies only to those patent claims licensable - by such Contributor that are necessarily infringed by their - Contribution(s) alone or by combination of their Contribution(s) - with the Work to which such Contribution(s) was submitted. If You - institute patent litigation against any entity (including a - cross-claim or counterclaim in a lawsuit) alleging that the Work - or a Contribution incorporated within the Work constitutes direct - or contributory patent infringement, then any patent licenses - granted to You under this License for that Work shall terminate - as of the date such litigation is filed. - - 4. Redistribution. You may reproduce and distribute copies of the - Work or Derivative Works thereof in any medium, with or without - modifications, and in Source or Object form, provided that You - meet the following conditions: - - (a) You must give any other recipients of the Work or - Derivative Works a copy of this License; and - - (b) You must cause any modified files to carry prominent notices - stating that You changed the files; and - - (c) You must retain, in the Source form of any Derivative Works - that You distribute, all copyright, patent, trademark, and - attribution notices from the Source form of the Work, - excluding those notices that do not pertain to any part of - the Derivative Works; and - - (d) If the Work includes a "NOTICE" text file as part of its - distribution, then any Derivative Works that You distribute must - include a readable copy of the attribution notices contained - within such NOTICE file, excluding those notices that do not - pertain to any part of the Derivative Works, in at least one - of the following places: within a NOTICE text file distributed - as part of the Derivative Works; within the Source form or - documentation, if provided along with the Derivative Works; or, - within a display generated by the Derivative Works, if and - wherever such third-party notices normally appear. The contents - of the NOTICE file are for informational purposes only and - do not modify the License. You may add Your own attribution - notices within Derivative Works that You distribute, alongside - or as an addendum to the NOTICE text from the Work, provided - that such additional attribution notices cannot be construed - as modifying the License. - - You may add Your own copyright statement to Your modifications and - may provide additional or different license terms and conditions - for use, reproduction, or distribution of Your modifications, or - for any such Derivative Works as a whole, provided Your use, - reproduction, and distribution of the Work otherwise complies with - the conditions stated in this License. - - 5. Submission of Contributions. Unless You explicitly state otherwise, - any Contribution intentionally submitted for inclusion in the Work - by You to the Licensor shall be under the terms and conditions of - this License, without any additional terms or conditions. - Notwithstanding the above, nothing herein shall supersede or modify - the terms of any separate license agreement you may have executed - with Licensor regarding such Contributions. - - 6. Trademarks. This License does not grant permission to use the trade - names, trademarks, service marks, or product names of the Licensor, - except as required for reasonable and customary use in describing the - origin of the Work and reproducing the content of the NOTICE file. - - 7. Disclaimer of Warranty. Unless required by applicable law or - agreed to in writing, Licensor provides the Work (and each - Contributor provides its Contributions) on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or - implied, including, without limitation, any warranties or conditions - of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A - PARTICULAR PURPOSE. You are solely responsible for determining the - appropriateness of using or redistributing the Work and assume any - risks associated with Your exercise of permissions under this License. - - 8. Limitation of Liability. In no event and under no legal theory, - whether in tort (including negligence), contract, or otherwise, - unless required by applicable law (such as deliberate and grossly - negligent acts) or agreed to in writing, shall any Contributor be - liable to You for damages, including any direct, indirect, special, - incidental, or consequential damages of any character arising as a - result of this License or out of the use or inability to use the - Work (including but not limited to damages for loss of goodwill, - work stoppage, computer failure or malfunction, or any and all - other commercial damages or losses), even if such Contributor - has been advised of the possibility of such damages. - - 9. Accepting Warranty or Additional Liability. While redistributing - the Work or Derivative Works thereof, You may choose to offer, - and charge a fee for, acceptance of support, warranty, indemnity, - or other liability obligations and/or rights consistent with this - License. However, in accepting such obligations, You may act only - on Your own behalf and on Your sole responsibility, not on behalf - of any other Contributor, and only if You agree to indemnify, - defend, and hold each Contributor harmless for any liability - incurred by, or claims asserted against, such Contributor by reason - of your accepting any such warranty or additional liability. - - END OF TERMS AND CONDITIONS - - APPENDIX: How to apply the Apache License to your work. - - To apply the Apache License to your work, attach the following - boilerplate notice, with the fields enclosed by brackets "[]" - replaced with your own identifying information. (Don't include - the brackets!) The text should be enclosed in the appropriate - comment syntax for the file format. We also recommend that a - file or class name and description of purpose be included on the - same "printed page" as the copyright notice for easier - identification within third-party archives. - - Copyright [yyyy] [name of copyright owner] - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. \ No newline at end of file diff --git a/python/Makefile b/python/Makefile index 5fb033fe2..a29d564b0 100644 --- a/python/Makefile +++ b/python/Makefile @@ -41,25 +41,28 @@ help: default: help -docs-singlehtml: +docs-singlehtml: docs-generate-crds uv run --isolated --all-packages --group docs $(MAKE) -C docs singlehtml -docs: +docs-generate-crds: + uv run --isolated --all-packages --group docs python3 docs/source/reference/generate-crd-docs.py + +docs: docs-generate-crds uv run --isolated --all-packages --group docs $(MAKE) -C docs html SPHINXOPTS="-W --keep-going -n" -docs-all: +docs-all: docs-generate-crds uv run --isolated --all-packages --group docs $(MAKE) -C docs multiversion -docs-serve: clean-docs +docs-serve: clean-docs docs-generate-crds uv run --isolated --all-packages --group docs $(MAKE) -C docs serve docs-serve-all: clean-docs docs-all uv run --isolated --all-packages --group docs $(MAKE) -C docs serve-multiversion -docs-test: +docs-test: docs-generate-crds uv run --isolated --all-packages --group docs $(MAKE) -C docs doctest -docs-linkcheck: +docs-linkcheck: docs-generate-crds uv run --isolated --all-packages --group docs $(MAKE) -C docs linkcheck pkg-test-%: packages/% diff --git a/python/README.md b/python/README.md index e39bad73c..48868a09c 100644 --- a/python/README.md +++ b/python/README.md @@ -1,70 +1,16 @@ -# ![bolt](assets/bolt.svg) Jumpstarter +# Jumpstarter Python -[![Matrix](https://img.shields.io/matrix/jumpstarter%3Amatrix.org?color=blue)](https://matrix.to/#/#jumpstarter:matrix.org) -[![Etherpad](https://img.shields.io/badge/Etherpad-Notes-blue?logo=etherpad)](https://etherpad.jumpstarter.dev/pad-lister) -[![Community Meeting](https://img.shields.io/badge/Weekly%20Meeting-Google%20Meet-blue?logo=google-meet)](https://meet.google.com/gzd-hhbd-hpu) -![GitHub Release](https://img.shields.io/github/v/release/jumpstarter-dev/jumpstarter) -![PyPI - Version](https://img.shields.io/pypi/v/jumpstarter) -![GitHub Downloads (all assets, all releases)](https://img.shields.io/github/downloads/jumpstarter-dev/jumpstarter/total) -[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/jumpstarter-dev/jumpstarter) +The Python implementation of [Jumpstarter](https://jumpstarter.dev): client +libraries, the `jmp` CLI, hardware drivers, and the testing framework. This +directory is managed as a [uv workspace](https://docs.astral.sh/uv/concepts/workspaces/). -[![E2E Tests](https://github.com/jumpstarter-dev/jumpstarter/actions/workflows/e2e.yaml/badge.svg)](https://github.com/jumpstarter-dev/jumpstarter/actions/workflows/e2e.yaml) -[![Tests](https://github.com/jumpstarter-dev/jumpstarter/actions/workflows/pytest.yaml/badge.svg)](https://github.com/jumpstarter-dev/jumpstarter/actions/workflows/pytest.yaml) -[![documentation](https://github.com/jumpstarter-dev/jumpstarter/actions/workflows/documentation.yaml/badge.svg)](https://github.com/jumpstarter-dev/jumpstarter/actions/workflows/documentation.yaml)
-[![Wheels](https://github.com/jumpstarter-dev/jumpstarter/actions/workflows/publish.yaml/badge.svg)](https://github.com/jumpstarter-dev/jumpstarter/actions/workflows/publish.yaml) -[![Flashing bundles](https://github.com/jumpstarter-dev/jumpstarter/actions/workflows/build_oci_bundle.yaml/badge.svg)](https://github.com/jumpstarter-dev/jumpstarter/actions/workflows/build_oci_bundle.yaml) -[![Containers](https://github.com/jumpstarter-dev/jumpstarter/actions/workflows/build.yaml/badge.svg)](https://github.com/jumpstarter-dev/jumpstarter/actions/workflows/build.yaml) +## Development -A free, open source tool for automated testing on real and virtual hardware with -CI/CD integration. Simplify device automation with consistent rules across local -and distributed environments. +```sh +make build-python +make test +make lint-fix -## Highlights - -- 🧪 **Unified Testing** - One tool for local, virtual, and remote hardware -- 🐍 **Python-Powered** - Leverage Python's testing ecosystem -- 🔌 **Hardware Abstraction** - Simplify complex hardware interfaces with - drivers -- 🌐 **Collaborative** - Share test hardware globally -- ⚙️ **CI/CD Ready** - Works with cloud native developer environments and - pipelines -- 💻 **Cross-Platform** - Supports Linux and macOS - -## Installation - -Install all the Jumpstarter Python components: - -```shell -pip install --extra-index-url https://pkg.jumpstarter.dev/ jumpstarter-all +uv run ruff check . +uv run ruff format . ``` - -Or, just install the `jmp` CLI tool: - -```shell -pip install --extra-index-url https://pkg.jumpstarter.dev/ jumpstarter-cli -``` - -To install the [Jumpstarter -Service](https://jumpstarter.dev/main/introduction/service.html) in your Kubernetes -cluster, see the [Service -Installation](https://jumpstarter.dev/main/getting-started/installation/index.html) -documentation. - -## Documentation - -Jumpstarter's documentation is available at -[jumpstarter.dev](https://jumpstarter.dev). - -Additionally, the command line reference documentation can be viewed with `jmp ---help`. - -## Contributing - -Jumpstarter welcomes contributors of all levels of experience and would love to -see you involved in the project. See the [contributing -guide](https://jumpstarter.dev/main/contributing.html) to get started. - -## License - -Jumpstarter is licensed under the Apache 2.0 License ([LICENSE](LICENSE) or -[https://www.apache.org/licenses/LICENSE-2.0](https://www.apache.org/licenses/LICENSE-2.0)). diff --git a/python/__templates__/create_driver.sh b/python/__templates__/create_driver.sh index 5f61dea8a..45b467c8c 100755 --- a/python/__templates__/create_driver.sh +++ b/python/__templates__/create_driver.sh @@ -73,9 +73,29 @@ export: # Add required config parameters here ``` + + + + ## API Reference -Add API documentation here. +```{eval-rst} +.. autoclass:: jumpstarter_driver_${DRIVER_MODULE_NAME}.driver.${DRIVER_CLASS}() +``` + + EOF # Need to expand variables after EOF to prevent early expansion $sed_cmd "s/\${DRIVER_CLASS}/${DRIVER_CLASS}/g; s/\${DRIVER_NAME}/${DRIVER_NAME}/g; s/\${DRIVER_MODULE_NAME}/${DRIVER_MODULE_NAME}/g" "${README_FILE}" diff --git a/python/docs/source/_static/css/custom.css b/python/docs/source/_static/css/custom.css index fc690789e..420293054 100644 --- a/python/docs/source/_static/css/custom.css +++ b/python/docs/source/_static/css/custom.css @@ -7,6 +7,11 @@ display: none !important; } +/* Align sidebar logo with content heading */ +.sidebar-drawer { + padding-top: 1.5rem; +} + /* Fix version name overflow in sidebar */ .sidebar-brand .sidebar-brand-text { white-space: normal; @@ -18,4 +23,33 @@ max-width: 100%; overflow: hidden; text-overflow: ellipsis; -} \ No newline at end of file +} + + +.glossary-term { + border-bottom: 1px dotted var(--color-background-border); +} + +.glossary-term[data-tooltip] { + position: relative; +} + +.glossary-term[data-tooltip]:hover::after, +.glossary-term[data-tooltip].tooltip-active::after { + content: attr(data-tooltip); + position: absolute; + left: 0; + bottom: 100%; + background: var(--color-background-secondary); + color: var(--color-foreground-primary); + border: 1px solid var(--color-foreground-muted); + padding: 0.4em 0.6em; + border-radius: 4px; + font-size: 0.85em; + min-width: 200px; + max-width: 400px; + white-space: normal; + z-index: 1000; + pointer-events: none; +} + diff --git a/python/docs/source/_static/js/glossary-tooltips.js b/python/docs/source/_static/js/glossary-tooltips.js new file mode 100644 index 000000000..5507a0028 --- /dev/null +++ b/python/docs/source/_static/js/glossary-tooltips.js @@ -0,0 +1,73 @@ +(function () { + var definitions = {}; + var isTouch = !window.matchMedia("(pointer: fine)").matches; + + function fetchGlossary() { + var links = document.querySelectorAll('a.reference.internal[href*="glossary.html#term-"]'); + if (links.length === 0) return; + + var href = links[0].getAttribute("href"); + var glossaryUrl = href.split("#")[0]; + var base = window.location.pathname.replace(/[^/]*$/, ""); + var url = new URL(glossaryUrl, window.location.origin + base).href; + + fetch(url) + .then(function (r) { return r.text(); }) + .then(function (html) { + var parser = new DOMParser(); + var doc = parser.parseFromString(html, "text/html"); + doc.querySelectorAll("dl.glossary dt[id]").forEach(function (dt) { + var id = dt.getAttribute("id"); + var dd = dt.nextElementSibling; + if (!dd || dd.tagName !== "DD") { + dd = dt.parentElement.querySelector("dd"); + } + if (dd) { + definitions[id] = dd.textContent.trim(); + } + }); + applyTooltips(); + }) + .catch(function () {}); + } + + function applyTooltips() { + document.querySelectorAll('a.reference.internal[href*="glossary.html#term-"]').forEach(function (a) { + var href = a.getAttribute("href"); + var termId = href.split("#")[1]; + var def = definitions[termId]; + if (def) { + var span = document.createElement("span"); + span.className = "glossary-term"; + span.setAttribute("data-tooltip", def); + span.innerHTML = a.innerHTML; + a.parentNode.replaceChild(span, a); + + if (isTouch) { + span.addEventListener("click", function (e) { + e.preventDefault(); + var wasActive = span.classList.contains("tooltip-active"); + document.querySelectorAll(".glossary-term.tooltip-active").forEach(function (el) { + el.classList.remove("tooltip-active"); + }); + if (!wasActive) { + span.classList.add("tooltip-active"); + } + }); + } + } + }); + + if (isTouch) { + document.addEventListener("click", function (e) { + if (!e.target.closest(".glossary-term")) { + document.querySelectorAll(".glossary-term.tooltip-active").forEach(function (el) { + el.classList.remove("tooltip-active"); + }); + } + }); + } + } + + document.addEventListener("DOMContentLoaded", fetchGlossary); +})(); diff --git a/python/docs/source/_static/js/mermaid-theme.js b/python/docs/source/_static/js/mermaid-theme.js new file mode 100644 index 000000000..ffbf3acef --- /dev/null +++ b/python/docs/source/_static/js/mermaid-theme.js @@ -0,0 +1,46 @@ +(function () { + function getEffectiveTheme() { + var theme = document.body ? document.body.getAttribute("data-theme") || "auto" : "auto"; + if (theme === "auto") { + return window.matchMedia("(prefers-color-scheme: dark)").matches ? "dark" : "light"; + } + return theme; + } + + function getMermaidTheme() { + return getEffectiveTheme() === "dark" ? "dark" : "default"; + } + + function renderMermaid() { + if (typeof mermaid === "undefined") return; + mermaid.initialize({ startOnLoad: false, theme: getMermaidTheme() }); + document.querySelectorAll("pre.mermaid").forEach(function (el) { + var code = el.getAttribute("data-original") || el.textContent; + el.setAttribute("data-original", code); + el.removeAttribute("data-processed"); + el.textContent = code; + }); + mermaid.run({ querySelector: "pre.mermaid" }); + } + + document.addEventListener("DOMContentLoaded", function () { + renderMermaid(); + + var observer = new MutationObserver(function (mutations) { + mutations.forEach(function (m) { + if (m.attributeName === "data-theme") renderMermaid(); + }); + }); + if (document.body) { + observer.observe(document.body, { attributes: true }); + } + + var mq = window.matchMedia("(prefers-color-scheme: dark)"); + if (mq.addEventListener) { + mq.addEventListener("change", function () { + var theme = document.body.getAttribute("data-theme"); + if (theme === "auto" || !theme) renderMermaid(); + }); + } + }); +})(); diff --git a/python/docs/source/_templates/page.html b/python/docs/source/_templates/page.html index 692be9f7a..5c68ffb56 100644 --- a/python/docs/source/_templates/page.html +++ b/python/docs/source/_templates/page.html @@ -3,17 +3,6 @@ {% include "head.html" %} {% endblock %} -{% block content %} -{% if pagename != 'index' %} -
-

Warning

-

This documentation is actively being updated as the project evolves and may not be complete in all areas.

-
-{% endif %} - -{{ super() }} -{% endblock %} - {% block footer %} {% include "footer.html" %} -{% endblock %} \ No newline at end of file +{% endblock %} diff --git a/python/docs/source/_templates/warning.html b/python/docs/source/_templates/warning.html deleted file mode 100644 index c5bc4009e..000000000 --- a/python/docs/source/_templates/warning.html +++ /dev/null @@ -1,4 +0,0 @@ -
-

Warning

-

This documentation is actively being updated as the project evolves and may not be complete in all areas.

-
\ No newline at end of file diff --git a/python/docs/source/conf.py b/python/docs/source/conf.py index fbe972c2b..ae532cdb7 100644 --- a/python/docs/source/conf.py +++ b/python/docs/source/conf.py @@ -3,7 +3,7 @@ # For the full list of built-in configuration values, see the documentation: # https://www.sphinx-doc.org/en/master/usage/configuration.html -# -- Project information ----------------------------------------------------- +# - Project information ----------------------------------------------------- # https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information import asyncio @@ -17,10 +17,10 @@ sys.path.insert(0, os.path.abspath("../..")) project = "jumpstarter" -copyright = "2025, Jumpstarter Contributors" +copyright = "2026, Jumpstarter Contributors" author = "Jumpstarter Contributors" -# -- General configuration --------------------------------------------------- +# - General configuration --------------------------------------------------- # https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration extensions = [ @@ -39,13 +39,13 @@ exclude_patterns = [] mermaid_version = "10.9.1" +mermaid_init_js = "" suppress_warnings = [ - "ref.class", # suppress unresolved Python class references (external references - # are warnings otherwise) + "ref.class", ] -# -- Options for HTML output ------------------------------------------------- +# - Options for HTML output ------------------------------------------------- # https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output html_theme = "furo" @@ -88,7 +88,7 @@ def get_index_url(): doctest_test_doctest_blocks = "" -html_js_files = ["js/theme-toggle.js"] +html_js_files = ["js/theme-toggle.js", "js/mermaid-theme.js", "js/glossary-tooltips.js"] html_static_path = ["_static"] html_css_files = ["css/custom.css"] html_sidebars = { diff --git a/python/docs/source/contributing.md b/python/docs/source/contributing.md index 869af2209..ef7014ec8 100644 --- a/python/docs/source/contributing.md +++ b/python/docs/source/contributing.md @@ -3,6 +3,15 @@ Thank you for your interest in contributing to Jumpstarter, we are an open community and we welcome contributions. +- [How to Contribute](contributing/how-to-contribute.md): How to set up, make + changes, and submit a pull request +- [Development Environment](contributing/development-environment.md): Setting up + your local environment for Python and Go development +- [Guidelines](contributing/guidelines.md): Code style, documentation practices, + and AI assistant configuration +- [Jumpstarter Enhancement Proposals](contributing/jeps/index.md): Process for + proposing significant changes to the project + ## Getting Help - **Matrix**: [Community](https://matrix.to/#/#jumpstarter:matrix.org) @@ -11,92 +20,12 @@ community and we welcome contributions. - **Weekly Meeting**: [Google Meet](https://meet.google.com/gzd-hhbd-hpu) - **Etherpad**: [Docs](https://etherpad.jumpstarter.dev/pad-lister) -## Getting Started - -0. Get familiar with [Jumpstarter Internals](./contributing/internals.md) -1. Follow our [dev setup guide](./contributing/development-environment.md) -2. Make changes on a new branch -3. Test your changes thoroughly -4. Submit a pull request - -If you have questions, reach out in our Matrix chat or open an issue on GitHub. - -## Contribution Guidelines - -### Making Changes - -- Focus on a single issue. -- Follow code style (validate with `make lint`, fix with `make lint-fix`) -- Perform static type checking with (`make ty-pkg-${package_name}`) -- Add tests and update documentation. New drivers/features need tests and docs. -- Verify all tests pass (`make pkg-test-${package_name}` or `make test`) - -### Commit Messages - -- Use clear, descriptive messages -- Reference issue numbers when applicable -- Follow conventional commit format when possible - -### Pull Requests - -- Provide a clear description -- Link to relevant issues -- Ensure all tests pass - -## Types of Contributions - -### Code Contributions - -We welcome bug fixes, features, and improvements to the core codebase. - - -## AI Assistants - -This project accepts contributions from AI assistants, although you should be careful when creating code from AI assistants, -and figure out if the code you are submitting could infringe any licensing, for example, reusing code from other incompatible -GPL licenses, you should do your due diligence. - -This project includes cursor rules to help Cursor AI understand our codebase and development patterns. When working with Cursor AI: - -- **Driver Creation**: If asked to create a new driver, Cursor will guide you through the process using our `create_driver.sh` script -- **Code Style**: Cursor will follow our established patterns and conventions -- **Testing**: Cursor will remind you to add tests and run our test suite - -The cursor rules are located in `.cursor/rules/` directory, with specific guidance for driver creation in `.cursor/rules/creating-new-drivers.mdc`. - - -### Contributing Drivers - -To create a new driver scaffold: - -```console -$ ./__templates__/create_driver.sh driver_package DriverClass "Your Name" "your.email@example.com" -``` - -For private drivers, consider forking our -[jumpstarter-driver-template](https://github.com/jumpstarter-dev/jumpstarter-driver-template). - -Test your driver: `make pkg-test-${package_name}` - -### Contributing Documentation - -Jumpstarter uses Sphinx with Markdown. Build and preview locally: - -```console -$ make docs-serve -``` - -Documentation recommended practices: - -- Use clear, concise language -- Include practical examples -- Break up text with headers, lists, and code blocks -- Target both beginners and advanced users - ```{toctree} :maxdepth: 1 :hidden: +contributing/how-to-contribute.md contributing/development-environment.md -contributing/internals.md +contributing/guidelines.md +contributing/jeps/index.md ``` diff --git a/python/docs/source/contributing/development-environment.md b/python/docs/source/contributing/development-environment.md index 70a2b8872..1a7e75d9e 100644 --- a/python/docs/source/contributing/development-environment.md +++ b/python/docs/source/contributing/development-environment.md @@ -1,10 +1,10 @@ # Development Environment You can use -[devspaces](https://github.com/jumpstarter-dev/jumpstarter/blob/main/python/.devfile.yaml), -[devcontainers](https://github.com/jumpstarter-dev/jumpstarter/tree/main/python/.devcontainer), -or your favorite OS/distro to develop Jumpstarter, however the following -examples are for Fedora 43. +[Eclipse Che](https://github.com/jumpstarter-dev/jumpstarter/blob/main/.devfile.yaml), +[devcontainers](https://github.com/jumpstarter-dev/jumpstarter/tree/main/.devcontainer), +or your favorite OS/distro to develop Jumpstarter. The following examples +are for Fedora. Jumpstarter is programmed in Python and Go, the Jumpstarter controller is written in Go, while the core and drivers are written in Python. @@ -27,19 +27,20 @@ Then you can clone the project and build the virtual environment with: ```console $ git clone https://github.com/jumpstarter-dev/jumpstarter.git $ cd jumpstarter -$ make sync +$ make build-python ``` At this point you can run any of the jumpstarter commands prefixing them with -`uv run`: +`uv run` from the `python/` directory: ```console +$ cd python $ uv run jmp ``` ### Running the Tests -To run the tests, you can use the `make` command: +To run the tests, you can use `make` from the repository root: ```console $ make test ``` @@ -52,9 +53,8 @@ $ make pkg-test-${package_name} ## Go Environment -The Jumpstarter controller lives in the -[jumpstarter-controller](https://github.com/jumpstarter-dev/jumpstarter-controller) -repository. +The Jumpstarter controller lives in the `controller/` directory within the +[jumpstarter](https://github.com/jumpstarter-dev/jumpstarter) monorepo. To install the basic set of dependencies, run the following commands: @@ -62,11 +62,10 @@ To install the basic set of dependencies, run the following commands: $ sudo dnf install -y git make golang kubectl ``` -Then you can clone the project and build the project with: +Then you can build the controller from the repository root: ```console -$ git clone https://github.com/jumpstarter-dev/jumpstarter-controller.git -$ cd jumpstarter-controller +$ cd controller $ make build ``` diff --git a/python/docs/source/contributing/guidelines.md b/python/docs/source/contributing/guidelines.md new file mode 100644 index 000000000..43e73783b --- /dev/null +++ b/python/docs/source/contributing/guidelines.md @@ -0,0 +1,50 @@ +# Guidelines + +## Documentation + +- Use clear, concise language +- Include practical examples +- Break up text with headers, lists, and code blocks +- Target both beginners and advanced users +- For third-party tools (`pytest`, `kubectl`, `cert-manager`, etc.), link to the + official documentation on first mention rather than defining them inline +- The [glossary](../glossary.md) is reserved for Jumpstarter-specific terms only + (entities, concepts, CLI commands). Do not add well-known industry terms or + third-party project names to it +- Use ASCII hyphens (`-`) instead of en-dash or em-dash characters + +## AI Assistants + +This project accepts contributions from AI assistants, although you should be +careful when creating code from AI assistants, and figure out if the code you +are submitting could infringe any licensing, for example, reusing code from +other incompatible GPL licenses, you should do your due diligence. + +### Cursor AI + +This project includes cursor rules to help Cursor AI understand our codebase +and development patterns. When working with Cursor AI: + +- **Driver Creation**: If asked to create a new driver, Cursor will guide you + through the process using our `create_driver.sh` script +- **Code Style**: Cursor will follow our established patterns and conventions +- **Testing**: Cursor will remind you to add tests and run our test suite + +The cursor rules are located in `.cursor/rules/` directory, with specific +guidance for driver creation in `.cursor/rules/creating-new-drivers.mdc`. + +### Claude Code + +This project also includes Claude Code configuration in the `.claude/` +directory. When working with Claude Code: + +- **Project Rules**: The `.claude/rules/` directory contains rules for project + structure, driver creation, {term}`operator` releases, and the {term}`JEP` process. Claude + Code loads these automatically. +- **CLAUDE.md**: The root `CLAUDE.md` provides project-level instructions + including key commands for testing (`make pkg-test-`), linting + (`make lint-fix`), and type checking (`make pkg-ty-`). +- **Code Style**: Claude Code follows TDD practices - writing failing tests + first, then minimal implementation code. +- **Driver Creation**: When asked to create a new driver, Claude Code follows + the guidelines in `.claude/rules/creating-new-drivers.md`. diff --git a/python/docs/source/contributing/how-to-contribute.md b/python/docs/source/contributing/how-to-contribute.md new file mode 100644 index 000000000..f000ac722 --- /dev/null +++ b/python/docs/source/contributing/how-to-contribute.md @@ -0,0 +1,62 @@ +# How to Contribute + +1. Get familiar with the [Introduction](../introduction/index.md) +2. Follow the [development environment](development-environment.md) setup +3. Make changes on a new branch +4. Test your changes thoroughly +5. Submit a pull request + +If you have questions, reach out in our Matrix chat or open an issue on GitHub. + +## Making Changes + +- Focus on a single issue +- Follow code style (validate with `make lint`, fix with `make lint-fix`) +- Perform static type checking with (`make pkg-ty-${package_name}`) +- Add tests and update documentation +- Verify all tests pass (`make pkg-test-${package_name}` or `make test`) + +## Commit Messages + +- Use clear, descriptive messages +- Reference issue numbers when applicable +- Follow conventional commit format when possible + +## Pull Requests + +- Provide a clear description +- Link to relevant issues +- Ensure all tests pass + +## Types of Contributions + +### Code + +We welcome bug fixes, features, and improvements to the core codebase. + +### Drivers + +To create a new driver scaffold: + +```console +$ ./__templates__/create_driver.sh driver_package DriverClass "Your Name" "your.email@example.com" +``` + +For private drivers, consider forking our +[jumpstarter-driver-template](https://github.com/jumpstarter-dev/jumpstarter-driver-template). + +Test your driver: `make pkg-test-${package_name}` + +### Documentation + +Jumpstarter uses Sphinx with Markdown. Build and preview locally: + +```console +$ make docs-serve +``` + +### Jumpstarter Enhancement Proposals + +For significant changes that affect multiple components, change public APIs, or +require community consensus, follow the +[{term}`JEP` process](jeps/index.md). diff --git a/python/docs/source/contributing/images/architecture.png b/python/docs/source/contributing/images/architecture.png deleted file mode 100644 index c60ecdb0c..000000000 Binary files a/python/docs/source/contributing/images/architecture.png and /dev/null differ diff --git a/python/docs/source/contributing/images/router.png b/python/docs/source/contributing/images/router.png deleted file mode 100644 index be77ad1b1..000000000 Binary files a/python/docs/source/contributing/images/router.png and /dev/null differ diff --git a/python/docs/source/contributing/images/rpc.png b/python/docs/source/contributing/images/rpc.png deleted file mode 100644 index 1919e431b..000000000 Binary files a/python/docs/source/contributing/images/rpc.png and /dev/null differ diff --git a/python/docs/source/contributing/internals.md b/python/docs/source/contributing/internals.md deleted file mode 100644 index 61d6517e2..000000000 --- a/python/docs/source/contributing/internals.md +++ /dev/null @@ -1,23 +0,0 @@ -# Internals - -## Architecture - -Jumpstarter consists of primarily three components, the control plane (`Controller` and `Router`) running inside a kubernetes cluster, the `Exporter` running on dedicated `Exporter Hosts` or developer machines (for local development workflow), and the `Client` interacting with the `Exporter`. - -The `Controller` handles inventory/lease management and access control, and stores its states as kubernetes CRDs. The `Router` provides a rendezvous point for clients to connect to exporters not on the local network. THe `Exporter` interacts with the `Device Under Test` with a set of `Drivers`, and exposes the methods provided by the `Drivers` over the network. The `Client` connects to the `Exporter` either directly, or over the `Router`, and calls the methods provided by the `Drivers` to perform actions on the `Device Under Test`. - -![Architecture](./images/architecture.png) - -## RPC - -Jumpstarter in its essence, is a RPC framework for `Clients` to call methods provided by `Drivers`. `Drivers` can expose three styles of RPCs, `Unary`, `Server streaming` and `Bidirectional streaming`, which are implemented with their counterparts in `gRPC`, see [RPC life cycle](https://grpc.io/docs/what-is-grpc/core-concepts/#rpc-life-cycle) for an in depth introduction to these RPC types. - -![RPC](./images/rpc.png) - -On top of `Bidirectional streaming` RPC, Jumpstarter also implements a generic byte stream interface, similar to TCP, for tunneling existing protocol (e.g. SSH) over Jumpstarter. - -## Router - -The Jumpstarter `Router` is just like ngrok or Cloudflare Tunnel, it allows for the `Client` to connect to `Exporters` without public IP addresses or behind NATs/firewalls, by tunneling a byte stream over Bidirectional streaming gRPC. - -![Router](./images/router.png) diff --git a/python/docs/source/internal/jeps/JEP-0000-jep-process.md b/python/docs/source/contributing/jeps/JEP-0000-jep-process.md similarity index 92% rename from python/docs/source/internal/jeps/JEP-0000-jep-process.md rename to python/docs/source/contributing/jeps/JEP-0000-jep-process.md index be10275eb..e3dd3e6d8 100644 --- a/python/docs/source/internal/jeps/JEP-0000-jep-process.md +++ b/python/docs/source/contributing/jeps/JEP-0000-jep-process.md @@ -1,3 +1,7 @@ +--- +orphan: true +--- + # JEP-0000: Jumpstarter Enhancement Proposal Process | Field | Value | @@ -12,7 +16,7 @@ ## Abstract -This document defines the Jumpstarter Enhancement Proposal (JEP) process — the +This document defines the Jumpstarter Enhancement Proposal (JEP) process - the mechanism by which substantial changes to the Jumpstarter project are proposed, discussed, and decided upon. JEPs provide a consistent, transparent record of design decisions for the Jumpstarter hardware-in-the-loop (HiL) testing framework @@ -25,19 +29,19 @@ As Jumpstarter grows in contributors, drivers, and production deployments, the project needs a structured way to propose and evaluate changes that go beyond routine bug fixes and minor improvements. An informal "open a PR and see what happens" approach doesn't scale when changes touch hardware interfaces, gRPC -protocol definitions, operator CRDs, or the driver plugin architecture — areas +protocol definitions, operator CRDs, or the driver plugin architecture - areas where mistakes are expensive to reverse. The JEP process gives the community: -- **Visibility** — a single place to discover what's being proposed, what's been +- **Visibility** - a single place to discover what's being proposed, what's been decided, and why. -- **Structured discussion** — a template that forces authors to think through +- **Structured discussion** - a template that forces authors to think through motivation, hardware implications, backward compatibility, and testing before code is written. -- **Historical record** — versioned markdown files in the repository whose git +- **Historical record** - versioned markdown files in the repository whose git history captures the evolution of each proposal. -- **Inclusive governance** — a lightweight, PR-based workflow that any contributor +- **Inclusive governance** - a lightweight, PR-based workflow that any contributor can participate in, regardless of commit access. ## What Requires a JEP @@ -131,7 +135,7 @@ reviewers, and surfaces obvious concerns early. ### 2. Submit a JEP Pull Request -Create a new branch and add your JEP as a markdown file in the `python/docs/source/internal/jeps/` +Create a new branch and add your JEP as a markdown file in the `python/docs/source/contributing/jeps/` directory, following the [JEP template](JEP-NNNN-template.md). Open a pull request against the main branch. The PR-based workflow makes discussion easier through inline review comments and suggested changes. @@ -145,7 +149,7 @@ JEP: Short descriptive title The JEP number is an incrementing integer assigned sequentially (e.g., JEP-0010, JEP-0011, JEP-0012). It is not derived from the PR number. To determine the next available number, check the existing JEPs in the -`python/docs/source/internal/jeps/` directory and increment from the highest existing number. +`python/docs/source/contributing/jeps/` directory and increment from the highest existing number. Apply the `jep` label to the pull request. Fill in every section of the template. Sections marked `(Optional)` may be @@ -201,7 +205,7 @@ reused. JEP-0000 through JEP-0009 are reserved for process and meta-JEPs. ## JEP Index -The file `python/docs/source/internal/jeps/README.md` serves as the index of all JEPs. +The file `python/docs/source/contributing/jeps/index.md` serves as the index of all JEPs. Alternatively, all JEPs can be found by filtering GitHub pull requests with the `jep` label. @@ -213,16 +217,16 @@ Changes to the JEP process itself require a new Process-type JEP. This process draws inspiration from: -- [Python Enhancement Proposals (PEPs)](https://peps.python.org/pep-0001/) — +- [Python Enhancement Proposals (PEPs)](https://peps.python.org/pep-0001/) - lightweight metadata, champion model, clear status lifecycle. -- [Kubernetes Enhancement Proposals (KEPs)](https://github.com/kubernetes/enhancements/tree/master/keps) — +- [Kubernetes Enhancement Proposals (KEPs)](https://github.com/kubernetes/enhancements/tree/master/keps) - test plan requirements, graduation criteria, production readiness. -- [Rust RFCs](https://github.com/rust-lang/rfcs) — PR-based workflow, emphasis +- [Rust RFCs](https://github.com/rust-lang/rfcs) - PR-based workflow, emphasis on motivation and teaching, prior art section. -- [Architecture Decision Records (ADRs)](https://adr.github.io/) — structured +- [Architecture Decision Records (ADRs)](https://adr.github.io/) - structured decision documentation with context, alternatives, and consequences. The JEP template adopts the ADR pattern for individual design decisions. -- [GitHub SpecKit](https://github.com/github/spec-kit) — spec-driven development +- [GitHub SpecKit](https://github.com/github/spec-kit) - spec-driven development methodology with structured templates and agent-friendly document conventions. The JEP template adopts SpecKit's practice of marking sections as mandatory or optional and structuring documents for machine readability. diff --git a/python/docs/source/internal/jeps/JEP-0010-renode-integration.md b/python/docs/source/contributing/jeps/JEP-0010-renode-integration.md similarity index 85% rename from python/docs/source/internal/jeps/JEP-0010-renode-integration.md rename to python/docs/source/contributing/jeps/JEP-0010-renode-integration.md index 0a815784c..b24ee80b9 100644 --- a/python/docs/source/internal/jeps/JEP-0010-renode-integration.md +++ b/python/docs/source/contributing/jeps/JEP-0010-renode-integration.md @@ -1,3 +1,7 @@ +--- +orphan: true +--- + # JEP-0010: Renode Integration for Microcontroller Targets | Field | Value | @@ -56,11 +60,11 @@ STM32, NXP S32K, Nordic, SiFive, and other MCU platforms. The initial targets for validation are: -- **STM32F407 Discovery** (Cortex-M4F) -- opensomeip FreeRTOS/ThreadX +- **STM32F407 Discovery** (Cortex-M4F) - opensomeip FreeRTOS/ThreadX ports, Renode built-in platform -- **NXP S32K388** (Cortex-M7) -- opensomeip Zephyr port, custom +- **NXP S32K388** (Cortex-M7) - opensomeip Zephyr port, custom platform description -- **Nucleo H753ZI** (Cortex-M7) -- openbsw-zephyr, Renode built-in +- **Nucleo H753ZI** (Cortex-M7) - openbsw-zephyr, Renode built-in `stm32h743.repl` ### Constraints @@ -81,10 +85,10 @@ The `jumpstarter-driver-renode` package provides a composite driver (`Renode`) that manages a Renode simulation instance with three child drivers: -- **`RenodePower`** — controls the Renode process lifecycle (on/off) -- **`RenodeFlasher`** — handles firmware loading (`sysbus LoadELF` / +- **`RenodePower`** - controls the Renode process lifecycle (on/off) +- **`RenodeFlasher`** - handles firmware loading (`sysbus LoadELF` / `sysbus LoadBinary`) -- **`PySerial` (console)** — serial access over a PTY terminal +- **`PySerial` (console)** - serial access over a PTY terminal Users configure targets entirely through exporter YAML: @@ -115,9 +119,9 @@ A `RenodeMonitor` async client handles the telnet protocol: No gRPC protocol changes. The driver exposes standard Jumpstarter interfaces (`PowerInterface`, `FlasherInterface`) plus: -- `get_platform()`, `get_uart()`, `get_machine_name()` — read-only +- `get_platform()`, `get_uart()`, `get_machine_name()` - read-only config accessors -- `monitor_cmd(command)` — raw monitor access, gated behind +- `monitor_cmd(command)` - raw monitor access, gated behind `allow_raw_monitor: true` (default: `false`) ### Hardware Considerations @@ -129,14 +133,14 @@ system (Linux or macOS). The Renode process is managed via ## Design Decisions -### DD-1: Control Interface — Telnet Monitor +### DD-1: Control Interface - Telnet Monitor **Alternatives considered:** -1. **Telnet monitor** — Renode's built-in TCP monitor interface. +1. **Telnet monitor** - Renode's built-in TCP monitor interface. Simple socket connection, send text commands, read responses. Lightweight, no extra runtime needed. -2. **pyrenode3** — Python.NET bridge to Renode's C# internals. More +2. **pyrenode3** - Python.NET bridge to Renode's C# internals. More powerful but requires .NET runtime or Mono, heavy dependency, less stable API surface. @@ -150,14 +154,14 @@ control. The monitor client uses `anyio.connect_tcp` with `anyio.fail_after` for timeouts, consistent with `TcpNetwork` and `grpc.py` in the project. -### DD-2: UART Exposure — PTY Terminal +### DD-2: UART Exposure - PTY Terminal **Alternatives considered:** -1. **PTY** (`emulation CreateUartPtyTerminal`) — Creates a +1. **PTY** (`emulation CreateUartPtyTerminal`) - Creates a pseudo-terminal file on the host. Reuses the existing `PySerial` child driver exactly as QEMU does. Linux/macOS only. -2. **Socket** (`emulation CreateServerSocketTerminal`) — Exposes UART +2. **Socket** (`emulation CreateServerSocketTerminal`) - Exposes UART as a TCP socket. Cross-platform. Maps to `TcpNetwork` driver. Has telnet IAC negotiation bytes to handle. @@ -169,15 +173,15 @@ This reuses the same serial/pexpect/console tooling without any adaptation. Socket terminal support can be added later as a fallback for platforms without PTY support. -### DD-3: Configuration Model — Managed Mode +### DD-3: Configuration Model - Managed Mode **Alternatives considered:** -1. **Managed mode** — The driver constructs all Renode monitor +1. **Managed mode** - The driver constructs all Renode monitor commands from YAML config parameters (`platform`, `uart`, firmware path). The driver handles platform loading, UART wiring, and firmware loading programmatically. -2. **Script mode** — User provides a complete `.resc` script. The +2. **Script mode** - User provides a complete `.resc` script. The driver runs it but still manages UART terminal setup. **Decision:** Managed mode as primary, with an `extra_commands` list @@ -190,7 +194,7 @@ The `extra_commands` list covers target-specific needs like register pokes (e.g., `sysbus WriteDoubleWord 0x40090030 0x0301` for S32K388 PL011 UART enablement) and Ethernet switch setup. -### DD-4: Firmware Loading — Deferred to Flash +### DD-4: Firmware Loading - Deferred to Flash **Alternatives considered:** @@ -198,7 +202,7 @@ PL011 UART enablement) and Ethernet switch setup. simulation and starts 2. `on()` starts the simulation, `flash()` loads firmware and resets -**Decision:** Option 1 — `flash()` stores the path, `on()` loads and +**Decision:** Option 1 - `flash()` stores the path, `on()` loads and starts. **Rationale:** This matches the QEMU driver's semantic where you flash @@ -207,12 +211,12 @@ power cycles without restarting the Renode process. The `RenodeFlasher` additionally supports hot-loading: if the simulation is already running, `flash()` sends the load command and resets the machine. -### DD-5: Security — Restricted Monitor Access +### DD-5: Security - Restricted Monitor Access **Alternatives considered:** -1. **Open access** — Expose `monitor_cmd` to all authenticated clients -2. **Opt-in access** — Gate behind `allow_raw_monitor` config flag +1. **Open access** - Expose `monitor_cmd` to all authenticated clients +2. **Opt-in access** - Gate behind `allow_raw_monitor` config flag **Decision:** Opt-in with `allow_raw_monitor: false` by default. @@ -243,7 +247,7 @@ communicates via line-oriented text: 1. **Connection**: retry loop with `fail_after(timeout)`, closing leaked streams on retry 2. **Prompt detection**: matches `(monitor)` or registered machine - names only — no false positives from output like `(enabled)` + names only - no false positives from output like `(enabled)` 3. **Error detection**: per-line check against markers (`Could not find`, `Error`, `Invalid`, `Failed`, `Unknown`) 4. **Timeout**: `execute()` wraps reads in `fail_after(30)` to prevent @@ -261,27 +265,27 @@ bytes of the firmware file. If they match the ELF magic (`\x7fELF`), ### Unit Tests -- `TestRenodeMonitor` — connection retry, command execution, error +- `TestRenodeMonitor` - connection retry, command execution, error detection (per-line), disconnect, newline rejection, stream cleanup on retry, prompt matching against expected prompts only -- `TestRenodePower` — command sequence verification, extra commands +- `TestRenodePower` - command sequence verification, extra commands ordering, firmware-less boot, idempotent on/off, process termination and cleanup -- `TestRenodeFlasher` — firmware path storage, hot-load with reset, +- `TestRenodeFlasher` - firmware path storage, hot-load with reset, custom load command, invalid load command rejection, ELF magic detection, dump not-implemented -- `TestRenodeConfig` — default values, children wiring, custom config, +- `TestRenodeConfig` - default values, children wiring, custom config, PTY path construction, lifecycle ### Integration Tests -- `TestRenodeClient` — round-trip properties via `serve()`, children +- `TestRenodeClient` - round-trip properties via `serve()`, children accessibility, `monitor_cmd` disabled by default, `monitor_cmd` not running error, CLI rendering ### E2E Tests -- `test_driver_renode_e2e` — full power on/off cycle with real Renode +- `test_driver_renode_e2e` - full power on/off cycle with real Renode process, skipped when Renode is not installed ### CI @@ -314,7 +318,7 @@ meta-package includes the new driver as an optional dependency. - PTY-only UART exposure limits to Linux/macOS (acceptable since Renode itself primarily targets these platforms) - The telnet monitor protocol is text-based and less structured than - QMP's JSON — error detection requires string matching + QMP's JSON - error detection requires string matching - Full `.resc` script support is deferred; users with complex Renode setups must express their configuration as managed-mode parameters plus `extra_commands` @@ -338,13 +342,13 @@ SoCs while Renode fills the MCU gap. ## Prior Art -- **jumpstarter-driver-qemu** — The existing Jumpstarter QEMU driver +- **jumpstarter-driver-qemu** - The existing Jumpstarter QEMU driver established the composite driver pattern, `Popen`-based process management, and side-channel control protocol (QMP) that this JEP follows. -- **Renode documentation** — [Renode docs](https://renode.readthedocs.io/) +- **Renode documentation** - [Renode docs](https://renode.readthedocs.io/) for monitor commands, platform descriptions, and UART terminal types. -- **opensomeip** — [github.com/vtz/opensomeip](https://github.com/vtz/opensomeip) +- **opensomeip** - [github.com/vtz/opensomeip](https://github.com/vtz/opensomeip) provides the reference Renode targets (STM32F407, S32K388) used for validation. diff --git a/python/docs/source/internal/jeps/JEP-0011-protobuf-introspection-interface-generation.md b/python/docs/source/contributing/jeps/JEP-0011-protobuf-introspection-interface-generation.md similarity index 75% rename from python/docs/source/internal/jeps/JEP-0011-protobuf-introspection-interface-generation.md rename to python/docs/source/contributing/jeps/JEP-0011-protobuf-introspection-interface-generation.md index a43dff3e8..4afdaa039 100644 --- a/python/docs/source/internal/jeps/JEP-0011-protobuf-introspection-interface-generation.md +++ b/python/docs/source/contributing/jeps/JEP-0011-protobuf-introspection-interface-generation.md @@ -1,3 +1,7 @@ +--- +orphan: true +--- + # JEP-0011: Protobuf Introspection and Interface Generation | Field | Value | @@ -17,27 +21,27 @@ This JEP makes Jumpstarter driver interfaces discoverable to non-Python clients by introducing `.proto` files as the canonical schema artifact for each driver interface. A new **codegen CLI** introspects Python interface classes at development time and emits `.proto` source files that are committed to each driver package. A companion **interface check CLI** runs in CI to detect drift between Python interfaces and their committed `.proto` files. The existing gRPC Server Reflection service and the `DriverInstanceReport.file_descriptor_proto` field serve the same compiled descriptor set at runtime so that tools like `grpcurl`, Buf, and polyglot codegen can discover the driver API without reading Python source. -This JEP keeps the Jumpstarter wire protocol unchanged — `DriverCall` remains the transport. The `.proto` schemas serve as an advisory description layer that enables polyglot discovery and future native-gRPC migration. Proto-first workflows (defining interfaces as `.proto` files and generating Python scaffolding) are deferred to a follow-up JEP focused on non-Python codegen. +This JEP keeps the Jumpstarter wire protocol unchanged - `DriverCall` remains the transport. The `.proto` schemas serve as an advisory description layer that enables polyglot discovery and future native-gRPC migration. Proto-first workflows (defining interfaces as `.proto` files and generating Python scaffolding) are deferred to a follow-up JEP focused on non-Python codegen. ## Motivation -Today, the `DriverInstanceReport` returned by `GetReport` contains driver UUIDs, labels, parent-child relationships, and human-readable `methods_description` text. It does not include machine-readable method signatures — parameter names, types, return types, or call semantics (unary vs. streaming). This means non-Python clients cannot discover the shape of a driver's API without out-of-band knowledge, limiting Jumpstarter to a single-language ecosystem. +Today, the `DriverInstanceReport` returned by `GetReport` contains driver UUIDs, labels, parent-child relationships, and human-readable `methods_description` text. It does not include machine-readable method signatures - parameter names, types, return types, or call semantics (unary vs. streaming). This means non-Python clients cannot discover the shape of a driver's API without out-of-band knowledge, limiting Jumpstarter to a single-language ecosystem. The `@export` decorator already has access to the full method signature via `inspect.signature()`, and the interface classes already carry type annotations. However, none of this information is surfaced in a structured, interoperable format. A JVM-based test runner, a TypeScript MCP server, or a Rust flash utility all have to reverse-engineer method names, argument types, and streaming semantics from Python source code or informal documentation. -Additionally, teams that want to define interface contracts upfront — before writing any driver implementation — currently have no supported workflow. A proto-first path would let architects define the interface as a `.proto` file and generate the Python scaffolding from it, following the standard gRPC development pattern while remaining fully compatible with Jumpstarter's existing driver model. +Additionally, teams that want to define interface contracts upfront - before writing any driver implementation - currently have no supported workflow. A proto-first path would let architects define the interface as a `.proto` file and generate the Python scaffolding from it, following the standard gRPC development pattern while remaining fully compatible with Jumpstarter's existing driver model. This JEP addresses three concrete gaps: -1. **Runtime introspection** — non-Python clients have no way to discover driver APIs programmatically. -2. **Schema portability** — there is no language-neutral description of Jumpstarter driver interfaces that standard protobuf/gRPC tooling can consume. -3. **Schema stability** — there is no committed, reviewable artifact describing a driver interface. Changes to Python signatures silently change the wire contract, with no diff for reviewers and no CI signal for polyglot consumers. +1. **Runtime introspection** - non-Python clients have no way to discover driver APIs programmatically. +2. **Schema portability** - there is no language-neutral description of Jumpstarter driver interfaces that standard protobuf/gRPC tooling can consume. +3. **Schema stability** - there is no committed, reviewable artifact describing a driver interface. Changes to Python signatures silently change the wire contract, with no diff for reviewers and no CI signal for polyglot consumers. ### User Stories - **As a** Python driver developer, **I want** an opt-in linter that flags `@export` methods missing type annotations, **so that** interfaces I choose to expose to polyglot consumers are fully typed before the `.proto` file is generated. -- **As a** Java test engineer writing Android device tests, **I want to** discover all available methods on a leased device's power driver — including parameter types, return types, and streaming semantics — **so that** I can generate type-safe Kotlin stubs instead of hand-writing `DriverCall` invocations with magic string method names. +- **As a** Java test engineer writing Android device tests, **I want to** discover all available methods on a leased device's power driver - including parameter types, return types, and streaming semantics - **so that** I can generate type-safe Kotlin stubs instead of hand-writing `DriverCall` invocations with magic string method names. - **As a** tools developer building a device management dashboard, **I want to** point standard gRPC tooling (`grpcurl`, Postman, Buf Studio) at an exporter and discover every available driver interface with full type information, **so that** I can prototype interactions without reading Python source code. @@ -49,34 +53,34 @@ This JEP addresses three concrete gaps: This proposal adds three capabilities to Jumpstarter, all centered on committed `.proto` files as the canonical schema artifact: -1. **Proto Codegen CLI (Python → `.proto`)** — a developer-invoked command that introspects a `DriverInterface` class and emits a `.proto` source file. The `.proto` file is committed alongside the driver package that defines the interface. -2. **Interface check CLI (drift detection)** — runs in CI to verify the committed `.proto` file still matches the Python interface. Reports any method, parameter, return-type, or streaming-semantics mismatch as a test failure. -3. **Runtime descriptor exposure** — the exporter loads the pre-compiled descriptor set (produced by `protoc --descriptor_set_out` from the committed `.proto` files), registers the services with gRPC Server Reflection, and embeds the raw bytes in `DriverInstanceReport.file_descriptor_proto`. +1. **Proto Codegen CLI (Python → `.proto`)** - a developer-invoked command that introspects a `DriverInterface` class and emits a `.proto` source file. The `.proto` file is committed alongside the driver package that defines the interface. +2. **Interface check CLI (drift detection)** - runs in CI to verify the committed `.proto` file still matches the Python interface. Reports any method, parameter, return-type, or streaming-semantics mismatch as a test failure. +3. **Runtime descriptor exposure** - the exporter loads the pre-compiled descriptor set (produced by `protoc --descriptor_set_out` from the committed `.proto` files), registers the services with gRPC Server Reflection, and embeds the raw bytes in `DriverInstanceReport.file_descriptor_proto`. The `.proto` files are the source of truth. Introspection happens once, at development time, when the author runs the codegen CLI; it does **not** happen at exporter startup or at Python import time. This mirrors the standard gRPC development workflow and keeps the exporter's runtime free of schema-construction work. **CLI naming is intentionally deferred.** This JEP does not commit to a concrete command surface for the codegen and check tools. Whether they ship as `jmp` subcommands, a separate `jmp-devel` binary, standalone executables, or some other shape is a UX decision better made during implementation, when we can weigh how much of the developer toolchain ends up under one umbrella. Throughout this document, "the codegen CLI" and "the interface check CLI" are used as descriptive names; bash code blocks use `` and `` as placeholders for whatever the final invocation turns out to be. -Proto-first workflows — authoring `.proto` files and generating Python interface/client/driver scaffolding from them — are **out of scope for this JEP**. They are planned as a follow-up JEP once non-Python codegen is ready to consume the committed `.proto` files. +Proto-first workflows - authoring `.proto` files and generating Python interface/client/driver scaffolding from them - are **out of scope for this JEP**. They are planned as a follow-up JEP once non-Python codegen is ready to consume the committed `.proto` files. ### Wire Protocol: `DriverCall` Remains Unchanged -An important design constraint: **this JEP does not change the wire protocol.** The existing `DriverCall` and `StreamingDriverCall` RPCs — where the client sends a method name as a string and arguments as `google.protobuf.Value` — remain the actual transport mechanism. The auto-generated client code still calls `self.call("on")` and `self.streamingcall("read")` under the hood. The auto-generated driver adapter still receives dispatch through the existing `@export` decorator and `Driver` base class machinery. +An important design constraint: **this JEP does not change the wire protocol.** The existing `DriverCall` and `StreamingDriverCall` RPCs - where the client sends a method name as a string and arguments as `google.protobuf.Value` - remain the actual transport mechanism. The auto-generated client code still calls `self.call("on")` and `self.streamingcall("read")` under the hood. The auto-generated driver adapter still receives dispatch through the existing `@export` decorator and `Driver` base class machinery. -The `.proto` files and `FileDescriptorProto` descriptors serve as a **description layer** on top of the existing dispatch mechanism — they describe what methods exist, what types they use, and what streaming semantics they have. They do not replace `DriverCall` with actual protobuf-native gRPC service implementations (where `PowerInterface` would be a real gRPC service with compiled request/response message stubs). That migration would be a significant breaking change to the exporter protocol, affecting every existing client and driver, and is explicitly out of scope for this JEP. +The `.proto` files and `FileDescriptorProto` descriptors serve as a **description layer** on top of the existing dispatch mechanism - they describe what methods exist, what types they use, and what streaming semantics they have. They do not replace `DriverCall` with actual protobuf-native gRPC service implementations (where `PowerInterface` would be a real gRPC service with compiled request/response message stubs). That migration would be a significant breaking change to the exporter protocol, affecting every existing client and driver, and is explicitly out of scope for this JEP. In concrete terms: - **What the proto IS used for:** introspection (`GetReport`, gRPC reflection), compatibility checking (the interface check CLI, `buf breaking`), documentation, and polyglot codegen. - **What the proto is NOT used for:** actual RPC transport. The `DriverCall(uuid="...", method="on", args=[])` message continues to be the wire format. -A future JEP will propose adding native protobuf service implementations alongside `DriverCall` — where `protoc`-generated stubs handle serialization directly. Whether the legacy transport is eventually retired is a separate question, contingent on field experience with the dual-path implementation; this JEP does not commit to that outcome. A design sketch for this future work is included at the end of this JEP for context. +A future JEP will propose adding native protobuf service implementations alongside `DriverCall` - where `protoc`-generated stubs handle serialization directly. Whether the legacy transport is eventually retired is a separate question, contingent on field experience with the dual-path implementation; this JEP does not commit to that outcome. A design sketch for this future work is included at the end of this JEP for context. #### gRPC reflection is advisory in this JEP -gRPC reflection will advertise services described by the committed `.proto` files — for example, `jumpstarter.driver.power.v1.PowerInterface.On(Empty)`. Because the wire protocol is unchanged, **those services are not backed by native gRPC handlers in this JEP**. A client that discovers the service through reflection and attempts to invoke it directly (e.g., `grpcurl -d '{}' host:port jumpstarter.driver.power.v1.PowerInterface/On`) will receive `UNIMPLEMENTED`. +gRPC reflection will advertise services described by the committed `.proto` files - for example, `jumpstarter.driver.power.v1.PowerInterface.On(Empty)`. Because the wire protocol is unchanged, **those services are not backed by native gRPC handlers in this JEP**. A client that discovers the service through reflection and attempts to invoke it directly (e.g., `grpcurl -d '{}' host:port jumpstarter.driver.power.v1.PowerInterface/On`) will receive `UNIMPLEMENTED`. -Reflection here is deliberately **advisory** — it exposes the schema so polyglot clients, codegen pipelines, and documentation tooling can discover the driver API and generate typed stubs that drive the existing `DriverCall` transport. The follow-up native-gRPC JEP will add handlers so reflected services become directly invocable without changing the proto schema produced by this JEP. +Reflection here is deliberately **advisory** - it exposes the schema so polyglot clients, codegen pipelines, and documentation tooling can discover the driver API and generate typed stubs that drive the existing `DriverCall` transport. The follow-up native-gRPC JEP will add handlers so reflected services become directly invocable without changing the proto schema produced by this JEP. ### `FileDescriptorProto` as the Schema Format @@ -84,11 +88,11 @@ Rather than defining a custom schema message, this proposal uses protobuf's own A `FileDescriptorProto` fully describes a `.proto` file in binary form: its package name, message definitions (with field names, types, and numbers), service definitions (with method names, request/response types, and streaming semantics), and import dependencies. This is strictly more expressive than any custom schema format. -Using it means there is one descriptor format throughout the entire system — generation, runtime introspection, registry, and codegen all consume the same artifact. +Using it means there is one descriptor format throughout the entire system - generation, runtime introspection, registry, and codegen all consume the same artifact. ### Build-time introspection of `@export` methods -Introspection runs at codegen CLI invocation time, not at import or exporter startup. The `@export` decorator itself is unchanged — it still stamps markers on the function for `DriverCall` dispatch. Type information is read directly from the live class via `inspect.signature()` when the CLI tool loads the interface module: +Introspection runs at codegen CLI invocation time, not at import or exporter startup. The `@export` decorator itself is unchanged - it still stamps markers on the function for `DriverCall` dispatch. Type information is read directly from the live class via `inspect.signature()` when the CLI tool loads the interface module: ```python # inside the codegen CLI @@ -102,7 +106,7 @@ params = [ return_type = sig.return_annotation ``` -The `_infer_call_type()` helper examines both the parameter and return annotations to determine streaming semantics: `AsyncGenerator[T]` or `Generator[T]` as a return type indicates server streaming, an `AsyncGenerator` parameter indicates client streaming, and the combination indicates bidirectional streaming (as used by the TCP driver). All other signatures indicate unary calls. Methods decorated with `@exportstream` (detected via the `MARKER_STREAMCALL` attribute) are handled separately — they are raw byte stream constructors that use a `StreamData { bytes payload }` message for native gRPC bidi streaming (see "Driver Patterns and Introspection Scope" in Design Details). +The `_infer_call_type()` helper examines both the parameter and return annotations to determine streaming semantics: `AsyncGenerator[T]` or `Generator[T]` as a return type indicates server streaming, an `AsyncGenerator` parameter indicates client streaming, and the combination indicates bidirectional streaming (as used by the TCP driver). All other signatures indicate unary calls. Methods decorated with `@exportstream` (detected via the `MARKER_STREAMCALL` attribute) are handled separately - they are raw byte stream constructors that use a `StreamData { bytes payload }` message for native gRPC bidi streaming (see "Driver Patterns and Introspection Scope" in Design Details). Because introspection is build-time only, there is no per-method metadata stored on function objects, no import-time overhead, and no runtime coupling between the dispatch layer and schema description. @@ -138,7 +142,7 @@ Rather than implementing the type mapping table from scratch, the builder levera - **`TypeAdapter(T).json_schema()`** works on arbitrary types (not just models), enabling introspection of `@export` method parameter types like `list[int]`, `Optional[str]`, or `UUID`. -- **`GenerateJsonSchema`** is Pydantic's extensible schema generator with ~55 type-specific handler methods (`int_schema()`, `str_schema()`, `list_schema()`, `model_schema()`, `enum_schema()`, etc.). By subclassing it, the builder can intercept type resolution and emit protobuf `FieldDescriptorProto` / `DescriptorProto` objects instead of JSON Schema dictionaries — reusing Pydantic's type walking, generic resolution, and forward reference handling. +- **`GenerateJsonSchema`** is Pydantic's extensible schema generator with ~55 type-specific handler methods (`int_schema()`, `str_schema()`, `list_schema()`, `model_schema()`, `enum_schema()`, etc.). By subclassing it, the builder can intercept type resolution and emit protobuf `FieldDescriptorProto` / `DescriptorProto` objects instead of JSON Schema dictionaries - reusing Pydantic's type walking, generic resolution, and forward reference handling. The JSON Schema → protobuf mapping is mechanical: @@ -154,11 +158,11 @@ The JSON Schema → protobuf mapping is mechanical: | `"anyOf": [T, null]` | `optional` field | | `"enum"` | Proto `enum` type | -This approach means Pydantic handles ~80-85% of the type mapping automatically. The remaining protobuf-specific concerns — field number assignment, streaming semantics, `@exportstream` detection, `FileDescriptorProto` assembly, and package/import management — are handled by the builder's own logic. +This approach means Pydantic handles ~80-85% of the type mapping automatically. The remaining protobuf-specific concerns - field number assignment, streaming semantics, `@exportstream` detection, `FileDescriptorProto` assembly, and package/import management - are handled by the builder's own logic. ### Build-time `.proto` generation -The codegen CLI uses a `build_file_descriptor()` library function to construct a `google.protobuf.descriptor_pb2.FileDescriptorProto` from an interface class, then renders it as human-readable `.proto` source. The builder is a pure function — it is **not** called by the exporter at runtime or by any import-time hook. +The codegen CLI uses a `build_file_descriptor()` library function to construct a `google.protobuf.descriptor_pb2.FileDescriptorProto` from an interface class, then renders it as human-readable `.proto` source. The builder is a pure function - it is **not** called by the exporter at runtime or by any import-time hook. ```python from google.protobuf.descriptor_pb2 import ( @@ -222,7 +226,7 @@ This produces the same `FileDescriptorProto` that `protoc` would generate from a ### Custom Options and Doc Comments -Protobuf service and message definitions carry structure — method names, parameter types, streaming semantics — but out of the box they don't carry versioning metadata. Additionally, while the type mapping captures *what* a method does structurally, it doesn't capture *why* or *how* in human terms. This section addresses both gaps: a lightweight custom option for interface versioning, and systematic generation of proto comments from Python docstrings. +Protobuf service and message definitions carry structure - method names, parameter types, streaming semantics - but out of the box they don't carry versioning metadata. Additionally, while the type mapping captures *what* a method does structurally, it doesn't capture *why* or *how* in human terms. This section addresses both gaps: a lightweight custom option for interface versioning, and systematic generation of proto comments from Python docstrings. #### Interface Versioning @@ -233,7 +237,7 @@ This approach was chosen over a custom `interface_version` service option becaus - It follows the standard protobuf/Buf versioning convention that all gRPC tooling already understands - It avoids custom annotations and the extraction logic they require - `buf breaking` is purpose-built for detecting incompatible proto changes -- Proto contracts are either compatible or they're a new version — semver within a package version adds complexity without benefit +- Proto contracts are either compatible or they're a new version - semver within a package version adds complexity without benefit #### Custom Annotations @@ -246,20 +250,20 @@ package jumpstarter.annotations; import "google/protobuf/descriptor.proto"; extend google.protobuf.FieldOptions { - // Marks this field as a resource handle — a UUID string referencing + // Marks this field as a resource handle - a UUID string referencing // a client-negotiated stream via the Jumpstarter resource system. // See "Resource Handle Pattern" in Design Details. optional bool resource_handle = 50000; } ``` -Field number 50000 falls within the range reserved by protobuf for organization-internal use (50000–99999), avoiding collision with other projects or future protobuf additions. +Field number 50000 falls within the range reserved by protobuf for organization-internal use (50000-99999), avoiding collision with other projects or future protobuf additions. -Note that `@exportstream` methods (raw byte stream constructors) do not need a custom annotation. They are represented as bidirectional streaming RPCs with a `StreamData { bytes payload }` message type — this pattern is unambiguous and sufficient for codegen tools to infer the correct dispatch mechanism. The `StreamData` message is auto-generated into the proto package when any `@exportstream` method exists, enabling native gRPC bidi streaming for byte transport without relying on `RouterService.Stream`. +Note that `@exportstream` methods (raw byte stream constructors) do not need a custom annotation. They are represented as bidirectional streaming RPCs with a `StreamData { bytes payload }` message type - this pattern is unambiguous and sufficient for codegen tools to infer the correct dispatch mechanism. The `StreamData` message is auto-generated into the proto package when any `@exportstream` method exists, enabling native gRPC bidi streaming for byte transport without relying on `RouterService.Stream`. #### Doc comments from docstrings -Proto comments (lines starting with `//` immediately preceding a service, method, message, or field definition) are a first-class concept in the protobuf ecosystem. They're preserved in `FileDescriptorProto` source info, rendered by `protoc-gen-doc`, displayed by `grpcurl describe`, shown in Buf Schema Registry, and emitted as language-native doc comments by standard codegen plugins (`protoc-gen-java`, `protoc-gen-ts`, etc.). There's no need to duplicate them as custom options — the standard proto comment mechanism already flows through the entire toolchain. +Proto comments (lines starting with `//` immediately preceding a service, method, message, or field definition) are a first-class concept in the protobuf ecosystem. They're preserved in `FileDescriptorProto` source info, rendered by `protoc-gen-doc`, displayed by `grpcurl describe`, shown in Buf Schema Registry, and emitted as language-native doc comments by standard codegen plugins (`protoc-gen-java`, `protoc-gen-ts`, etc.). There's no need to duplicate them as custom options - the standard proto comment mechanism already flows through the entire toolchain. The `build_file_descriptor()` builder and the codegen CLI extract docstrings from Python and emit them as proto comments: @@ -349,7 +353,7 @@ message PowerReading { } ``` -The proto is clean and readable. The comments flow through standard `protoc` codegen plugins to produce language-native documentation — Javadoc for Java/Kotlin, TSDoc for TypeScript, `///` for Rust, docstrings for Python — without any custom options or annotation processing. A developer reading the `.proto` file sees a self-documenting interface contract. The package version (`v1`) provides the compatibility boundary, and `buf breaking` enforces backward-compatible evolution within a version. +The proto is clean and readable. The comments flow through standard `protoc` codegen plugins to produce language-native documentation - Javadoc for Java/Kotlin, TSDoc for TypeScript, `///` for Rust, docstrings for Python - without any custom options or annotation processing. A developer reading the `.proto` file sees a self-documenting interface contract. The package version (`v1`) provides the compatibility boundary, and `buf breaking` enforces backward-compatible evolution within a version. #### How doc comments improve codegen @@ -372,7 +376,7 @@ And `protoc-gen-ts` produces: async off(): Promise { ... } ``` -This happens for free — no custom options, no custom codegen plugins, no annotation processing. A future Jumpstarter-specific `jmp codegen` wrapper could compose these standard stubs into DeviceClass-typed wrappers, inheriting the documentation from the proto comments. +This happens for free - no custom options, no custom codegen plugins, no annotation processing. A future Jumpstarter-specific `jmp codegen` wrapper could compose these standard stubs into DeviceClass-typed wrappers, inheriting the documentation from the proto comments. #### Doc comment round-trip consistency @@ -410,9 +414,9 @@ interfaces/ network/v1/network.proto # package jumpstarter.driver.network.v1; ``` -This shape unblocks multi-language driver implementations of the same interface — every language's build tooling consumes the same `interfaces/proto/...` source, just as every language can already consume `protocol/proto/jumpstarter/v1/...` for the wire protocol — and lets standard `protoc -I interfaces/proto` import resolution work without configuration. +This shape unblocks multi-language driver implementations of the same interface - every language's build tooling consumes the same `interfaces/proto/...` source, just as every language can already consume `protocol/proto/jumpstarter/v1/...` for the wire protocol - and lets standard `protoc -I interfaces/proto` import resolution work without configuration. -**Proto package selection.** When `--proto-package` is omitted, the CLI uses the first-party convention `jumpstarter.driver.{name}.{version}` — required for in-tree interfaces. Out-of-tree authors override with `--proto-package`, e.g. `--proto-package com.example.jumpstarter.driver.abc.v1`, to publish under their own organization's reverse-domain namespace. The directory path under `interfaces/proto/` mirrors whatever namespace the author chooses, segment-for-segment. +**Proto package selection.** When `--proto-package` is omitted, the CLI uses the first-party convention `jumpstarter.driver.{name}.{version}` - required for in-tree interfaces. Out-of-tree authors override with `--proto-package`, e.g. `--proto-package com.example.jumpstarter.driver.abc.v1`, to publish under their own organization's reverse-domain namespace. The directory path under `interfaces/proto/` mirrors whatever namespace the author chooses, segment-for-segment. Implementation: loads the interface class via `importlib`, calls `build_file_descriptor()` to produce the `FileDescriptorProto`, then renders it as human-readable `.proto` source text. Python snake_case method names are converted to PascalCase RPC names (e.g., `read_data_by_identifier` → `rpc ReadDataByIdentifier`), following standard proto conventions. @@ -426,7 +430,7 @@ walks `DriverInterfaceMeta._registry` (populated at import time) to discover all ### Out-of-tree drivers -Out-of-tree driver packages — drivers maintained outside this repository — participate in the same `.proto` workflow as in-tree drivers. The maintainer runs the codegen CLI against their `DriverInterface` subclasses, **vendors** the resulting `.proto` files into their own package's `interfaces/proto/` directory (mirroring the in-tree shape), and bundles a pre-compiled descriptor set produced by `protoc --descriptor_set_out` at the package's build time. The author chooses their own reverse-domain proto namespace (e.g., `com.example.jumpstarter.driver.abc.v1`) via `--proto-package`; the directory path under `interfaces/proto/` mirrors that namespace segment-for-segment. +Out-of-tree driver packages - drivers maintained outside this repository - participate in the same `.proto` workflow as in-tree drivers. The maintainer runs the codegen CLI against their `DriverInterface` subclasses, **vendors** the resulting `.proto` files into their own package's `interfaces/proto/` directory (mirroring the in-tree shape), and bundles a pre-compiled descriptor set produced by `protoc --descriptor_set_out` at the package's build time. The author chooses their own reverse-domain proto namespace (e.g., `com.example.jumpstarter.driver.abc.v1`) via `--proto-package`; the directory path under `interfaces/proto/` mirrors that namespace segment-for-segment. ```text my-jumpstarter-drivers/ # package root @@ -439,7 +443,7 @@ my-jumpstarter-drivers/ # package root └── driver.py ``` -For multi-driver or multi-language packages, the same `interfaces/proto/` shape extends naturally — multiple interfaces under one `interfaces/proto/` tree, with sibling language directories (`python/`, `rust/`, `cpp/`) consuming the same schemas. An "interface-only" package may publish only `interfaces/proto//...` with no implementation; an "implementation-only" package omits `interfaces/` entirely and pulls the schema from a declared dependency (the same way `tonic-build` and related gRPC tooling already resolves cross-package proto imports). +For multi-driver or multi-language packages, the same `interfaces/proto/` shape extends naturally - multiple interfaces under one `interfaces/proto/` tree, with sibling language directories (`python/`, `rust/`, `cpp/`) consuming the same schemas. An "interface-only" package may publish only `interfaces/proto//...` with no implementation; an "implementation-only" package omits `interfaces/` entirely and pulls the schema from a declared dependency (the same way `tonic-build` and related gRPC tooling already resolves cross-package proto imports). ```bash \ @@ -451,14 +455,14 @@ For multi-driver or multi-language packages, the same `interfaces/proto/` shape The `jumpstarter.driver.*` namespace is reserved for first-party interfaces; out-of-tree authors must supply `--proto-package` with their own reverse-domain namespace. The CLI refuses to write to a path that overlaps the first-party namespace when an out-of-tree namespace is requested, and vice versa. -`DriverInterface` subclasses register with `DriverInterfaceMeta._registry` automatically at import time, so the codegen CLI's batch mode picks them up once the package is installed in the development environment, and the interface check CLI can run against any importable interface module — out-of-tree packages are not a special case. +`DriverInterface` subclasses register with `DriverInterfaceMeta._registry` automatically at import time, so the codegen CLI's batch mode picks them up once the package is installed in the development environment, and the interface check CLI can run against any importable interface module - out-of-tree packages are not a special case. #### Build-time automation for out-of-tree drivers Running the codegen CLI by hand and committing the result is the explicit, manual path. Out-of-tree authors who prefer not to maintain that step can hook the codegen step into their package build alongside descriptor compilation (see DD-6: *"Same hook can also handle `.proto` generation"*). With a Python build plugin wired up: ```toml -# pyproject.toml — build plugin runs codegen + protoc as part of `uv build` +# pyproject.toml - build plugin runs codegen + protoc as part of `uv build` [build-system] requires = ["hatchling", "jumpstarter-codegen-build>=1.0"] build-backend = "hatchling.build" @@ -468,7 +472,7 @@ interfaces = ["jumpstarter_driver_abc.AbcInterface"] proto-package = "com.example.jumpstarter.driver.abc.v1" ``` -`uv build` then introspects the listed interface(s), writes `.proto` source to `interfaces/proto/com/example/jumpstarter/driver/abc/v1/abc.proto`, runs `protoc --descriptor_set_out` against it, and bundles the descriptor set into the wheel — all in one step. The author writes only the `@export`-decorated Python class. They can either commit the generated `.proto` (recommended for review, `buf breaking`, and polyglot consumption) or treat it as a build artifact that lives only in the wheel; the build plugin works the same way either direction. Equivalent plugins for other build systems (`build.rs` for Rust, Gradle plugin for Kotlin/JVM, CMake module for C/C++) follow the same shape and are tracked as follow-up work alongside non-Python authoring support. +`uv build` then introspects the listed interface(s), writes `.proto` source to `interfaces/proto/com/example/jumpstarter/driver/abc/v1/abc.proto`, runs `protoc --descriptor_set_out` against it, and bundles the descriptor set into the wheel - all in one step. The author writes only the `@export`-decorated Python class. They can either commit the generated `.proto` (recommended for review, `buf breaking`, and polyglot consumption) or treat it as a build artifact that lives only in the wheel; the build plugin works the same way either direction. Equivalent plugins for other build systems (`build.rs` for Rust, Gradle plugin for Kotlin/JVM, CMake module for C/C++) follow the same shape and are tracked as follow-up work alongside non-Python authoring support. If an out-of-tree driver ships neither a committed `.proto` nor a bundled descriptor, the exporter logs a warning naming the driver and continues to load it. The driver still serves `DriverCall` traffic normally, so existing Python clients keep working. Three things degrade in that case: @@ -478,7 +482,7 @@ If an out-of-tree driver ships neither a committed `.proto` nor a bundled descri The warning text should point to the codegen CLI and recommend adding it to the package's build so polyglot clients can consume the driver. This keeps the existing "easy driver development" property intact: authors can iterate without a `.proto` and add one when they're ready to support polyglot clients. -Auto-generating descriptors for out-of-tree drivers — for example by introspecting Python interfaces at exporter startup, or by compiling shipped `.proto` source on-demand without a pre-built descriptor — is deliberately out of scope for this JEP. This JEP commits to build-time codegen as the only supported path. A future JEP may revisit runtime auto-generation as a convenience for out-of-tree drivers if real-world friction warrants it. +Auto-generating descriptors for out-of-tree drivers - for example by introspecting Python interfaces at exporter startup, or by compiling shipped `.proto` source on-demand without a pre-built descriptor - is deliberately out of scope for this JEP. This JEP commits to build-time codegen as the only supported path. A future JEP may revisit runtime auto-generation as a convenience for out-of-tree drivers if real-world friction warrants it. ### Client inheritance convention @@ -496,9 +500,9 @@ class PowerClient(PowerInterface, DriverClient): yield PowerReading.model_validate(raw, strict=True) ``` -In the current codebase, client classes inherit only from `DriverClient` (e.g., `class PowerClient(DriverClient)`). Dual inheritance gives type checkers a way to verify that every client method is actually declared on the interface — if a `DriverInterface` method is missing from the client, mypy / pyright will flag the subclass as incomplete. It also makes the client relationship to the interface explicit across languages that don't support multiple inheritance — those languages can fall back to single-inherit-from-interface with a `DriverClient` helper, but the contract is the same. +In the current codebase, client classes inherit only from `DriverClient` (e.g., `class PowerClient(DriverClient)`). Dual inheritance gives type checkers a way to verify that every client method is actually declared on the interface - if a `DriverInterface` method is missing from the client, mypy / pyright will flag the subclass as incomplete. It also makes the client relationship to the interface explicit across languages that don't support multiple inheritance - those languages can fall back to single-inherit-from-interface with a `DriverClient` helper, but the contract is the same. -**Migration:** The standard in-tree clients (PowerClient, NetworkClient, StorageMuxClient, FlasherClient, CompositeClient, and the virtual-power client) are migrated to dual inheritance alongside the `DriverInterface` migration (Phase 1b). Drivers with clients that provide client-side orchestration (e.g., `FlasherClient` with `OpendalAdapter`, `StorageMuxFlasherClient.flash()`) keep their hand-written orchestration — dual inheritance does not change the methods, only the declared bases. +**Migration:** The standard in-tree clients (PowerClient, NetworkClient, StorageMuxClient, FlasherClient, CompositeClient, and the virtual-power client) are migrated to dual inheritance alongside the `DriverInterface` migration (Phase 1b). Drivers with clients that provide client-side orchestration (e.g., `FlasherClient` with `OpendalAdapter`, `StorageMuxFlasherClient.flash()`) keep their hand-written orchestration - dual inheritance does not change the methods, only the declared bases. ### Proto-first workflow (deferred) @@ -507,7 +511,7 @@ An earlier revision of this JEP described a a proto-first codegen companion comm Rationale: - For Python-first drivers (the primary path in this repository), the proto-first adapter adds an extra inheritance layer and `@export`-on-`__method` indirection without reducing the code a driver developer writes. A driver author still writes the hardware logic in abstract methods; the adapter only relocates the `@export` decorator one class up. -- The main value of proto-first generation is for **non-Python** consumers — Kotlin, Java, TypeScript, Rust — which can already consume the committed `.proto` files via standard `protoc` plugins. A reference prototype for non-Python codegen exists and will be proposed in a follow-up JEP. +- The main value of proto-first generation is for **non-Python** consumers - Kotlin, Java, TypeScript, Rust - which can already consume the committed `.proto` files via standard `protoc` plugins. A reference prototype for non-Python codegen exists and will be proposed in a follow-up JEP. - Removing a proto-first codegen companion from this JEP shrinks the scope, unblocks the Python-first path, and avoids committing to an adapter pattern before non-Python codegen design is complete. The `.proto` schema format defined by this JEP is stable enough that the follow-up JEP can build on it without revisiting the schema. @@ -522,9 +526,9 @@ Because the `.proto` files are committed and reviewed, CI needs a way to detect --interface jumpstarter_driver_power.interface.PowerInterface ``` -The tool runs `build_file_descriptor()` against the live Python class, parses the committed `.proto` file, and reports any mismatch in method names, parameter/return types, streaming semantics, or doc comments. It runs in CI alongside `buf breaking` — `buf breaking` detects backward-incompatible changes between old and new proto revisions; the interface check CLI detects drift between the current Python interface and the current proto revision. Together they cover both classes of failure. +The tool runs `build_file_descriptor()` against the live Python class, parses the committed `.proto` file, and reports any mismatch in method names, parameter/return types, streaming semantics, or doc comments. It runs in CI alongside `buf breaking` - `buf breaking` detects backward-incompatible changes between old and new proto revisions; the interface check CLI detects drift between the current Python interface and the current proto revision. Together they cover both classes of failure. -**Discovery.** The check CLI accepts `--interface ` for single-interface use (the form shown above). For "check everything" CI runs, it walks `DriverInterfaceMeta._registry` — the same mechanism the codegen CLI's batch mode uses — so importing the package(s) under check is sufficient discovery. There is no separate yaml manifest of interfaces to keep in sync; the metaclass registry is the single source of truth. +**Discovery.** The check CLI accepts `--interface ` for single-interface use (the form shown above). For "check everything" CI runs, it walks `DriverInterfaceMeta._registry` - the same mechanism the codegen CLI's batch mode uses - so importing the package(s) under check is sufficient discovery. There is no separate yaml manifest of interfaces to keep in sync; the metaclass registry is the single source of truth. ### API / Protocol Changes @@ -547,9 +551,9 @@ message DriverInstanceReport { } ``` -This embeds the descriptor directly in the report, making `GetReport` self-describing. A Java client parses the bytes as `FileDescriptorProto`, feeds it to a `DescriptorPool`, and has full type information for every driver — method names, parameter types, return types, streaming semantics — without needing a separate gRPC reflection call. +This embeds the descriptor directly in the report, making `GetReport` self-describing. A Java client parses the bytes as `FileDescriptorProto`, feeds it to a `DescriptorPool`, and has full type information for every driver - method names, parameter types, return types, streaming semantics - without needing a separate gRPC reflection call. -**Source of the bytes.** The descriptors are loaded from a **pre-compiled descriptor set** produced by `protoc --descriptor_set_out` from the committed `.proto` files. Only the `.proto` source is committed to the repository — the compiled descriptor set is a **build artifact** generated during the package build (e.g., as a `hatchling` / `setuptools` step for Python wheels) and bundled into the distribution alongside the rest of the package's payload, the same way generated language bindings are. The exporter reads this file once at startup and indexes the `FileDescriptorProto` by driver interface class. It does **not** run introspection at startup — that work is done at development time by the codegen CLI; only the `.proto` source is committed and reviewed. +**Source of the bytes.** The descriptors are loaded from a **pre-compiled descriptor set** produced by `protoc --descriptor_set_out` from the committed `.proto` files. Only the `.proto` source is committed to the repository - the compiled descriptor set is a **build artifact** generated during the package build (e.g., as a `hatchling` / `setuptools` step for Python wheels) and bundled into the distribution alongside the rest of the package's payload, the same way generated language bindings are. The exporter reads this file once at startup and indexes the `FileDescriptorProto` by driver interface class. It does **not** run introspection at startup - that work is done at development time by the codegen CLI; only the `.proto` source is committed and reviewed. The field is `optional bytes` (not a nested message) because `FileDescriptorProto` is a well-known protobuf type that clients parse with their own language's descriptor library. Keeping it as raw bytes avoids adding `google/protobuf/descriptor.proto` as a direct dependency of the Jumpstarter protocol. @@ -578,13 +582,13 @@ def register_reflection(server, descriptor_set_path): This serves the descriptors through the standard `grpc.reflection.v1.ServerReflection` service, enabling standard tools (`grpcurl`, Postman, Java's `ProtoReflectionDescriptorDatabase`) to discover every driver interface on any exporter. -As noted in the Proposal, reflection in this JEP is **advisory**: services discovered via reflection describe the driver API but are not directly invocable — native gRPC handlers are a follow-up JEP. Standard tools can still use the reflected schema to generate typed stubs that drive `DriverCall` under the hood. +As noted in the Proposal, reflection in this JEP is **advisory**: services discovered via reflection describe the driver API but are not directly invocable - native gRPC handlers are a follow-up JEP. Standard tools can still use the reflected schema to generate typed stubs that drive `DriverCall` under the hood. -The `file_descriptor_proto` in the report and the gRPC reflection service serve the same data through different channels. The report embeds the descriptor for clients that want it inline with the driver tree. Reflection serves it through the standard gRPC mechanism for tools that expect that protocol. They are the same `FileDescriptorProto` — no duplication of schema definitions. +The `file_descriptor_proto` in the report and the gRPC reflection service serve the same data through different channels. The report embeds the descriptor for clients that want it inline with the driver tree. Reflection serves it through the standard gRPC mechanism for tools that expect that protocol. They are the same `FileDescriptorProto` - no duplication of schema definitions. ### Hardware Considerations -This JEP is a purely software-layer change. No hardware is required or affected. Introspection runs at development time inside the codegen CLI; the exporter itself reads a pre-compiled descriptor set once at startup. The `FileDescriptorProto` for a typical driver interface with 5–10 methods is approximately 1–3 KB serialized. Exporters running on resource-constrained SBCs (e.g., Raspberry Pi 4) should see no measurable runtime impact beyond one file read at startup. +This JEP is a purely software-layer change. No hardware is required or affected. Introspection runs at development time inside the codegen CLI; the exporter itself reads a pre-compiled descriptor set once at startup. The `FileDescriptorProto` for a typical driver interface with 5-10 methods is approximately 1-3 KB serialized. Exporters running on resource-constrained SBCs (e.g., Raspberry Pi 4) should see no measurable runtime impact beyond one file read at startup. ## Design Decisions @@ -592,10 +596,10 @@ This JEP is a purely software-layer change. No hardware is required or affected. **Alternatives considered:** -1. **Runtime dynamic `FileDescriptorProto` generation** — the exporter introspects `@export` methods at startup and builds descriptors on demand. -2. **Committed `.proto` files produced by the codegen CLI** — schemas are authored (via tool-assisted generation), committed to the driver package, compiled with `protoc --descriptor_set_out`, and loaded at startup. +1. **Runtime dynamic `FileDescriptorProto` generation** - the exporter introspects `@export` methods at startup and builds descriptors on demand. +2. **Committed `.proto` files produced by the codegen CLI** - schemas are authored (via tool-assisted generation), committed to the driver package, compiled with `protoc --descriptor_set_out`, and loaded at startup. -**Decision:** Option 2 — committed `.proto` files. +**Decision:** Option 2 - committed `.proto` files. **Rationale:** Committed schemas give reviewers a visible diff, CI a concrete artifact for `buf breaking`, and polyglot consumers a stable reference. Dynamic generation has no diff, couples dispatch to schema at import time, and shifts the drift-detection problem onto the exporter. An interface-check CI gate against a committed `.proto` is both simpler and more informative than runtime reconstruction. @@ -603,10 +607,10 @@ This JEP is a purely software-layer change. No hardware is required or affected. **Alternatives considered:** -1. **Mandatory at decoration time** — `@export` raises `TypeError` for any method without complete annotations. Forces the entire codebase (~111 methods across 25 packages) to be fully typed before anything builds. -2. **Opt-in via `@export(strict=True)` / `JMP_EXPORT_STRICT=1`** — `@export` in default mode emits `DeprecationWarning`. Teams enable strict mode per package. The codegen CLI always requires full annotations — enforcement moves to the tool. +1. **Mandatory at decoration time** - `@export` raises `TypeError` for any method without complete annotations. Forces the entire codebase (~111 methods across 25 packages) to be fully typed before anything builds. +2. **Opt-in via `@export(strict=True)` / `JMP_EXPORT_STRICT=1`** - `@export` in default mode emits `DeprecationWarning`. Teams enable strict mode per package. The codegen CLI always requires full annotations - enforcement moves to the tool. -**Decision:** Option 2 — opt-in. +**Decision:** Option 2 - opt-in. **Rationale:** Mandatory enforcement blocks packages that don't need polyglot exposure and couples this JEP to a 111-method mechanical fix. Opt-in lets the ecosystem migrate incrementally while still guaranteeing annotation coverage for any interface that actually publishes a `.proto`. @@ -614,50 +618,50 @@ This JEP is a purely software-layer change. No hardware is required or affected. **Alternatives considered:** -1. **Bidirectional tooling in Phase 1** — ship both the codegen CLI (Python → `.proto`) and a proto-first companion (`.proto` → Python interface + client + driver adapter). -2. **Python-first only** — ship only the codegen CLI and the interface check CLI. Proto-first is deferred to a follow-up JEP focused on non-Python codegen. +1. **Bidirectional tooling in Phase 1** - ship both the codegen CLI (Python → `.proto`) and a proto-first companion (`.proto` → Python interface + client + driver adapter). +2. **Python-first only** - ship only the codegen CLI and the interface check CLI. Proto-first is deferred to a follow-up JEP focused on non-Python codegen. -**Decision:** Option 2 — Python-first only. +**Decision:** Option 2 - Python-first only. -**Rationale:** For Python drivers, the proto-first adapter pattern adds an inheritance layer and an underscore-prefixed abstract-method indirection without materially reducing the code the author writes. Its main value is producing clients and servicers for **non-Python** languages — that design is orthogonal to the Python introspection work and benefits from a dedicated JEP. Shrinking scope unblocks Phase 1 and avoids committing to a Python adapter pattern before non-Python codegen design is complete. A reference prototype for non-Python codegen already exists and will be the basis for the follow-up JEP. +**Rationale:** For Python drivers, the proto-first adapter pattern adds an inheritance layer and an underscore-prefixed abstract-method indirection without materially reducing the code the author writes. Its main value is producing clients and servicers for **non-Python** languages - that design is orthogonal to the Python introspection work and benefits from a dedicated JEP. Shrinking scope unblocks Phase 1 and avoids committing to a Python adapter pattern before non-Python codegen design is complete. A reference prototype for non-Python codegen already exists and will be the basis for the follow-up JEP. ### DD-4: Dual inheritance for generated and migrated clients **Alternatives considered:** -1. **Keep single inheritance** — `class PowerClient(DriverClient)` — clients implement the interface by convention, not by declaration. -2. **Adopt dual inheritance** — `class PowerClient(PowerInterface, DriverClient)` — clients explicitly implement the interface; type checkers verify method coverage. +1. **Keep single inheritance** - `class PowerClient(DriverClient)` - clients implement the interface by convention, not by declaration. +2. **Adopt dual inheritance** - `class PowerClient(PowerInterface, DriverClient)` - clients explicitly implement the interface; type checkers verify method coverage. -**Decision:** Option 2 — dual inheritance. +**Decision:** Option 2 - dual inheritance. -**Rationale:** Dual inheritance makes the client-to-interface relationship structural, not nominal. Type checkers flag missing interface methods on the client at analysis time; new clients inherit a typed contract by construction. This also firms up the semantics across languages — for languages without multiple inheritance, the equivalent is single-inherit-from-interface with a `DriverClient` helper. +**Rationale:** Dual inheritance makes the client-to-interface relationship structural, not nominal. Type checkers flag missing interface methods on the client at analysis time; new clients inherit a typed contract by construction. This also firms up the semantics across languages - for languages without multiple inheritance, the equivalent is single-inherit-from-interface with a `DriverClient` helper. ### DD-5: Reflection is advisory in this JEP **Alternatives considered:** -1. **Reflect and invoke** — register native gRPC handlers alongside reflection so that reflected services are directly invocable (e.g., via `grpcurl`). -2. **Reflect only** — register services for schema discovery, leave invocation on the native gRPC path as `UNIMPLEMENTED` until a follow-up JEP designs the native transport. +1. **Reflect and invoke** - register native gRPC handlers alongside reflection so that reflected services are directly invocable (e.g., via `grpcurl`). +2. **Reflect only** - register services for schema discovery, leave invocation on the native gRPC path as `UNIMPLEMENTED` until a follow-up JEP designs the native transport. -**Decision:** Option 2 — reflect only. +**Decision:** Option 2 - reflect only. -**Rationale:** Native gRPC handlers require a substantial design for UUID routing, dual-path dispatch during transition, and backward compatibility with legacy `DriverCall` clients. That design exists as a sketch (see "Native gRPC Transport — Design Sketch") but belongs in its own JEP. In the meantime, reflection is still valuable for codegen, documentation, and typed-stub generation — clients use reflected schemas to drive the existing `DriverCall` transport. The `UNIMPLEMENTED` behavior is documented explicitly in the Proposal and integration test suite. +**Rationale:** Native gRPC handlers require a substantial design for UUID routing, dual-path dispatch during transition, and backward compatibility with legacy `DriverCall` clients. That design exists as a sketch (see "Native gRPC Transport - Design Sketch") but belongs in its own JEP. In the meantime, reflection is still valuable for codegen, documentation, and typed-stub generation - clients use reflected schemas to drive the existing `DriverCall` transport. The `UNIMPLEMENTED` behavior is documented explicitly in the Proposal and integration test suite. ### DD-6: Commit `.proto` source only; descriptor sets are build artifacts **Alternatives considered:** -1. **Commit both `.proto` source and the compiled descriptor set** (`protoc --descriptor_set_out` output) — the exporter loads the committed `.bin` directly; no build step required. -2. **Commit `.proto` source only; compile the descriptor set at package build time** — `hatchling` / `setuptools` (and equivalent backends in other languages) invoke `protoc` during `uv build` / `pip install`, bundling the compiled descriptor as part of the wheel payload. -3. **Commit `.proto` source only; compile the descriptor set at exporter startup** — the exporter invokes `protoc` (or an in-process equivalent) on every startup. +1. **Commit both `.proto` source and the compiled descriptor set** (`protoc --descriptor_set_out` output) - the exporter loads the committed `.bin` directly; no build step required. +2. **Commit `.proto` source only; compile the descriptor set at package build time** - `hatchling` / `setuptools` (and equivalent backends in other languages) invoke `protoc` during `uv build` / `pip install`, bundling the compiled descriptor as part of the wheel payload. +3. **Commit `.proto` source only; compile the descriptor set at exporter startup** - the exporter invokes `protoc` (or an in-process equivalent) on every startup. -**Decision:** Option 2 — commit source, build artifacts at package build time. +**Decision:** Option 2 - commit source, build artifacts at package build time. **Rationale:** This matches the project's existing convention for the wire protocol (`protocol/proto/*.proto` is committed; no `.bin` artifacts are checked in) and standard practice across the protobuf ecosystem (gRPC, Buf, `tonic-build`, Bazel). Committing binary descriptors (Option 1) creates source-tree bloat, generates noisy diffs on every regeneration, and risks drift when a `.proto` change is committed without recompiling the descriptor. Compiling at startup (Option 3) adds `protoc` as a runtime dependency on the exporter, slows boot, and turns descriptor-generation failures into runtime errors instead of build-time errors. Option 2 keeps the source tree text-only and reviewable, ships compiled descriptors as part of the package distribution (the same way generated language bindings ship), and ensures the exporter only ever consumes already-validated artifacts. **Consequences:** The package build must invoke `protoc --descriptor_set_out`. JEP-0011's codegen story already proposes this as a build step; the project's `.gitignore` should exclude `*.bin` / descriptor output paths from the `interfaces/` tree to prevent accidental commits. -**Same hook can also handle `.proto` generation.** Once the build is invoking `protoc` to produce the descriptor set, it can also invoke the codegen CLI immediately upstream — extracting `.proto` source from `@export`-decorated `DriverInterface` classes — so the entire pipeline (Python interface → `.proto` → descriptor set) runs as a single build step. Out-of-tree authors who set up the build plugin then never have to run the codegen CLI by hand: their normal `uv build` / `pip install` produces a wheel containing the `.proto` (committed in the source tree if the author chooses, or bundled only inside the wheel if not) and the compiled descriptor set. The `.proto` itself remains a normal source artifact: authors are encouraged to commit it for review, `buf breaking`, and polyglot consumption, but the *generation* of it is automated end-to-end. In-tree drivers use the same plugin against the in-repo `interfaces/` tree. +**Same hook can also handle `.proto` generation.** Once the build is invoking `protoc` to produce the descriptor set, it can also invoke the codegen CLI immediately upstream - extracting `.proto` source from `@export`-decorated `DriverInterface` classes - so the entire pipeline (Python interface → `.proto` → descriptor set) runs as a single build step. Out-of-tree authors who set up the build plugin then never have to run the codegen CLI by hand: their normal `uv build` / `pip install` produces a wheel containing the `.proto` (committed in the source tree if the author chooses, or bundled only inside the wheel if not) and the compiled descriptor set. The `.proto` itself remains a normal source artifact: authors are encouraged to commit it for review, `buf breaking`, and polyglot consumption, but the *generation* of it is automated end-to-end. In-tree drivers use the same plugin against the in-repo `interfaces/` tree. ## Design Details @@ -714,7 +718,7 @@ This JEP is a purely software-layer change. No hardware is required or affected. 4. **At `GetReport` time:** Each `DriverInstanceReport` carries the `file_descriptor_proto` bytes for its interface. Clients parse them with their language's protobuf library to discover the full schema. -### `DriverInterfaceMeta` and `DriverInterface` — Type-Safe Interface Definitions +### `DriverInterfaceMeta` and `DriverInterface` - Type-Safe Interface Definitions This JEP introduces a new metaclass + base class pair that provides type-safe, validated interface definitions, replacing the current convention of bare `ABCMeta`: @@ -846,17 +850,17 @@ class StorageMuxFlasherInterface(StorageMuxInterface): - Missing `client()` → `TypeError` at class definition time - Type checkers (mypy, pyright) see `client()` as required abstract classmethod -**Empty interfaces** (like `CompositeInterface`) work naturally — they inherit `DriverInterface`, define `client()`, and have no abstract methods. The builder produces an empty `ServiceDescriptorProto`. Note that `CompositeInterface` currently has no metaclass at all (it's a plain class, not even `ABCMeta`), so migration adds both the metaclass and `DriverInterface` base in one step. +**Empty interfaces** (like `CompositeInterface`) work naturally - they inherit `DriverInterface`, define `client()`, and have no abstract methods. The builder produces an empty `ServiceDescriptorProto`. Note that `CompositeInterface` currently has no metaclass at all (it's a plain class, not even `ABCMeta`), so migration adds both the metaclass and `DriverInterface` base in one step. -**Deferred: `UdsInterface` concrete mixin.** The `UdsInterface` pattern — where `@export` is placed directly on the interface class without `ABCMeta` — is an anti-pattern that conflates the interface contract with the dispatch implementation. `UdsInterface` should eventually be refactored to use `DriverInterface` with `@abstractmethod`, with the shared `@export` implementations moved to a separate mixin class (e.g., `UdsDriverMixin`). However, this refactoring involves ~18 methods shared between `UdsCan` and `UdsDoip` via multiple inheritance, making it a non-trivial migration with code duplication risk. **This refactoring is deferred to a follow-up task** and is not a prerequisite for Phase 1b. The `build_file_descriptor()` builder can detect `@export` on non-`DriverInterface` classes and handle them via a legacy fallback path during the transition period. +**Deferred: `UdsInterface` concrete mixin.** The `UdsInterface` pattern - where `@export` is placed directly on the interface class without `ABCMeta` - is an anti-pattern that conflates the interface contract with the dispatch implementation. `UdsInterface` should eventually be refactored to use `DriverInterface` with `@abstractmethod`, with the shared `@export` implementations moved to a separate mixin class (e.g., `UdsDriverMixin`). However, this refactoring involves ~18 methods shared between `UdsCan` and `UdsDoip` via multiple inheritance, making it a non-trivial migration with code duplication risk. **This refactoring is deferred to a follow-up task** and is not a prerequisite for Phase 1b. The `build_file_descriptor()` builder can detect `@export` on non-`DriverInterface` classes and handle them via a legacy fallback path during the transition period. **Discovery and registry:** - `DriverInterfaceMeta._registry` automatically tracks all defined interfaces - `build_file_descriptor()` checks `isinstance(cls.__class__, DriverInterfaceMeta)` for unambiguous discovery -- The codegen CLI's batch mode iterates the registry — no package entry-point scanning needed +- The codegen CLI's batch mode iterates the registry - no package entry-point scanning needed -**Migration:** Each interface changes from `metaclass=ABCMeta` to inheriting `DriverInterface`. Drivers that inherit from both the interface and `Driver` continue to work since `DriverInterfaceMeta` extends `ABCMeta`. The migration also requires adding full type annotations to all abstract methods — this is the forcing function for making the entire interface ecosystem type-safe. +**Migration:** Each interface changes from `metaclass=ABCMeta` to inheriting `DriverInterface`. Drivers that inherit from both the interface and `Driver` continue to work since `DriverInterfaceMeta` extends `ABCMeta`. The migration also requires adding full type annotations to all abstract methods - this is the forcing function for making the entire interface ecosystem type-safe. ### Opt-in type annotation enforcement for `@export` @@ -874,7 +878,7 @@ def export(func=None, *, strict=False): Otherwise, missing annotations emit a DeprecationWarning but do not block import. The codegen and interface check CLIs will still refuse - to produce a proto for an incompletely-typed interface — that is + to produce a proto for an incompletely-typed interface - that is where the contract is enforced for polyglot consumption. """ ... @@ -884,11 +888,11 @@ Three enforcement tiers exist: - **Permissive (default):** `@export` logs a `DeprecationWarning` for missing annotations. Existing drivers continue to import unchanged. - **Strict (`@export(strict=True)` or `JMP_EXPORT_STRICT=1`):** `TypeError` at decoration time. Opt in per package when the team is ready. -- **Tool-level (non-negotiable):** The codegen CLI fails with a clear error if the interface has incompletely annotated methods — there is no way to emit a proto with unknown types. The interface check CLI inherits the same requirement. +- **Tool-level (non-negotiable):** The codegen CLI fails with a clear error if the interface has incompletely annotated methods - there is no way to emit a proto with unknown types. The interface check CLI inherits the same requirement. Type enforcement is opt-in so it doesn't affect drivers that aren't yet consumed by polyglot clients. Teams that want the tighter contract enable strict mode package by package as they publish proto schemas. -**Annotation coverage in the current codebase.** An audit identified ~111 `@export` / `@exportstream` methods across 25 packages missing one or more annotations (mostly `-> None` return types on void methods, plus a handful of resource-handle `source` / `target` parameters). These fixes remain good practice and are recommended alongside Phase 1b, but they are **not blocking** for this JEP — packages migrate to fully-typed `@export` and emit proto schemas on their own schedule. +**Annotation coverage in the current codebase.** An audit identified ~111 `@export` / `@exportstream` methods across 25 packages missing one or more annotations (mostly `-> None` return types on void methods, plus a handful of resource-handle `source` / `target` parameters). These fixes remain good practice and are recommended alongside Phase 1b, but they are **not blocking** for this JEP - packages migrate to fully-typed `@export` and emit proto schemas on their own schedule. ### Driver Patterns and Introspection Scope @@ -908,7 +912,7 @@ PowerInterface (abstract) → PowerClient (DriverClient) └── SNMPPower (Driver) ``` -Every standard in-tree interface follows this pattern: `PowerInterface`, `NetworkInterface`, `FlasherInterface`, `StorageMuxInterface`, `StorageMuxFlasherInterface`, `CompositeInterface`. The interface class is the introspection target — `build_file_descriptor()` reads its abstract methods and type annotations to produce the `FileDescriptorProto`. This is the path the JEP is primarily designed for. +Every standard in-tree interface follows this pattern: `PowerInterface`, `NetworkInterface`, `FlasherInterface`, `StorageMuxInterface`, `StorageMuxFlasherInterface`, `CompositeInterface`. The interface class is the introspection target - `build_file_descriptor()` reads its abstract methods and type annotations to produce the `FileDescriptorProto`. This is the path the JEP is primarily designed for. When a driver implements an explicit interface, the `@export`-decorated methods on the driver class must match the abstract methods on the interface (same names, compatible signatures). The introspection reads from the interface, not the driver, so the proto describes the *contract*, not the *implementation*. Multiple driver implementations (MockPower, DutlinkPower, TasmotaPower) all produce the same proto because they implement the same interface. @@ -916,10 +920,10 @@ Interface inheritance also works naturally. `StorageMuxFlasherInterface` extends #### Pattern 2: `@exportstream` methods (raw byte channels) -Some drivers use the `@exportstream` decorator instead of (or in addition to) `@export`. This creates a fundamentally different kind of interaction — a raw bidirectional byte stream tunneled through the `RouterService`, not a structured `DriverCall` RPC: +Some drivers use the `@exportstream` decorator instead of (or in addition to) `@export`. This creates a fundamentally different kind of interaction - a raw bidirectional byte stream tunneled through the `RouterService`, not a structured `DriverCall` RPC: ```python -# TcpNetwork driver — @exportstream for the byte channel +# TcpNetwork driver - @exportstream for the byte channel class TcpNetwork(NetworkInterface, Driver): @exportstream @asynccontextmanager @@ -933,7 +937,7 @@ class TcpNetwork(NetworkInterface, Driver): ``` ```python -# PySerial driver — @exportstream for the serial connection +# PySerial driver - @exportstream for the serial connection class PySerial(Driver): @exportstream @asynccontextmanager @@ -945,7 +949,7 @@ class PySerial(Driver): The `@exportstream` methods are async context managers that yield raw byte streams. They are represented as native gRPC bidirectional streaming RPCs using a `StreamData { bytes payload }` message type that carries raw bytes. On the exporter, the generated servicer bridges between the gRPC bidi stream and the driver's byte stream. On the client side, non-Python clients call the native gRPC bidi endpoint directly and bridge it to local TCP/UDP sockets for port forwarding. -**Proto mapping for `@exportstream`:** The descriptor builder detects the `MARKER_STREAMCALL` attribute set by `@exportstream` and emits a bidi streaming RPC with `StreamData` — a simple message containing a `bytes payload` field. The `StreamData` message is auto-generated into the proto package: +**Proto mapping for `@exportstream`:** The descriptor builder detects the `MARKER_STREAMCALL` attribute set by `@exportstream` and emits a bidi streaming RPC with `StreamData` - a simple message containing a `bytes payload` field. The `StreamData` message is auto-generated into the proto package: ```protobuf service NetworkInterface { @@ -961,7 +965,7 @@ message StreamData { Note that the `NetworkInterface` in the current codebase only defines `connect()` as an abstract method. The `address()` method that exists on some implementations (e.g., `TcpNetwork`, `WebsocketNetwork`) is a driver-level extension, not part of the interface contract, and is therefore not included in the proto. -Codegen tools (including the deferred non-Python codegen) infer the dispatch mechanism from the proto structure: a bidirectional streaming RPC with `StreamData` request and response is a raw byte stream constructor (`@exportstream`). The `StreamData` pattern is unambiguous — no custom annotation is needed. +Codegen tools (including the deferred non-Python codegen) infer the dispatch mechanism from the proto structure: a bidirectional streaming RPC with `StreamData` request and response is a raw byte stream constructor (`@exportstream`). The `StreamData` pattern is unambiguous - no custom annotation is needed. For Python clients, the hand-written pattern under this JEP is: @@ -978,7 +982,7 @@ The `resource_handle` field option is defined in `jumpstarter/annotations/annota #### Pattern 3: Composite and nested drivers -Jumpstarter drivers form trees. A `Dutlink` board exposes a composite root with named children — `power` (PowerInterface), `storage` (StorageMuxFlasherInterface), `console` (serial) — each with its own UUID, interface, and client. The `GetReport` RPC returns this tree as a flat list of `DriverInstanceReport` entries linked by `parent_uuid`: +Jumpstarter drivers form trees. A `Dutlink` board exposes a composite root with named children - `power` (PowerInterface), `storage` (StorageMuxFlasherInterface), `console` (serial) - each with its own UUID, interface, and client. The `GetReport` RPC returns this tree as a flat list of `DriverInstanceReport` entries linked by `parent_uuid`: ``` Dutlink (CompositeInterface, uuid=root) @@ -998,7 +1002,7 @@ Each driver in the tree produces its own `FileDescriptorProto` based on its inte The tree structure is already encoded in the existing `uuid` / `parent_uuid` fields. The `file_descriptor_proto` field adds *what each node can do* alongside *where it sits in the tree*. -**CompositeInterface** defines no abstract methods — it's a pure container: +**CompositeInterface** defines no abstract methods - it's a pure container: ```python class CompositeInterface(DriverInterface): @@ -1016,11 +1020,11 @@ class CompositeClient(CompositeInterface, DriverClient): return self.children[name] ``` -**Proxy drivers** (`Proxy` class) are transparent to introspection — they delegate `report()` and `enumerate()` to their target, so the proto describes the target driver's interface, not the proxy itself. +**Proxy drivers** (`Proxy` class) are transparent to introspection - they delegate `report()` and `enumerate()` to their target, so the proto describes the target driver's interface, not the proxy itself. **Client tree reconstruction** works the same as today: `client_from_channel()` calls `GetReport()`, topologically sorts by `parent_uuid`, and instantiates client classes in dependency order. The `file_descriptor_proto` on each report is available for polyglot clients to discover the full typed API of every node in the tree. -**For native gRPC (future):** Each child driver registers its own native gRPC service on the exporter's server. The UUID routing interceptor dispatches to the correct instance. A Kotlin client leasing a Dutlink board would get three typed stubs — one for `PowerInterface`, one for `StorageMuxFlasherInterface`, one for `NetworkInterface` — each bound to the correct child UUID: +**For native gRPC (future):** Each child driver registers its own native gRPC service on the exporter's server. The UUID routing interceptor dispatches to the correct instance. A Kotlin client leasing a Dutlink board would get three typed stubs - one for `PowerInterface`, one for `StorageMuxFlasherInterface`, one for `NetworkInterface` - each bound to the correct child UUID: ```kotlin val report = stub.getReport(Empty.getDefaultInstance()) @@ -1039,13 +1043,13 @@ storage.host() Historically, some client classes added methods that aren't in the interface contract. The canonical example is `PowerClient.cycle()`: ```python -# Legacy pattern — client-side composition (avoid going forward) +# Legacy pattern - client-side composition (avoid going forward) class PowerClient(DriverClient): def on(self) -> None: # in PowerInterface self.call("on") def off(self) -> None: # in PowerInterface self.call("off") - def cycle(self, wait=2): # NOT in PowerInterface — pure client-side logic + def cycle(self, wait=2): # NOT in PowerInterface - pure client-side logic self.off() time.sleep(wait) self.on() @@ -1058,7 +1062,7 @@ class PowerClient(DriverClient): **Move convenience methods to the driver side.** Going forward, simple convenience methods like `cycle()` should be promoted to first-class `@export` methods on the driver and declared on the interface. The recommended shape: ```python -# Recommended pattern — convenience method on the driver +# Recommended pattern - convenience method on the driver class PowerInterface(DriverInterface): @abstractmethod def on(self) -> None: ... @@ -1083,11 +1087,11 @@ class PowerClient(PowerInterface, DriverClient): Putting `cycle()` on the wire gives it a proto entry, makes it reachable from every generated client, lets the driver implement it atomically (guarding against torn power transitions if the client crashes mid-cycle), and removes a class of subtle behavioral drift between Python and polyglot consumers. Reducing client-side logic is an explicit goal: the client should be a thin typed transport over the proto contract, not a layer with its own undeclared behavior. As part of the Phase 1b interface migration, simple composites like `cycle()` are migrated server-side. -**Keep on the client only when orchestration genuinely requires it.** A small set of drivers — primarily `NetworkInterface` and `FlasherInterface` / `StorageMuxFlasherInterface` — need real client-side orchestration that cannot be expressed across the wire: file hashing, compression negotiation, `OpendalAdapter` resource handle setup, byte-stream tunneling. Those clients keep their hand-written orchestration methods (`FlasherClient.flash()`, `StorageMuxFlasherClient.flash()`/`dump()`, console connect helpers, etc.). They are the exception, not the rule. When in doubt, push the composite to the driver. +**Keep on the client only when orchestration genuinely requires it.** A small set of drivers - primarily `NetworkInterface` and `FlasherInterface` / `StorageMuxFlasherInterface` - need real client-side orchestration that cannot be expressed across the wire: file hashing, compression negotiation, `OpendalAdapter` resource handle setup, byte-stream tunneling. Those clients keep their hand-written orchestration methods (`FlasherClient.flash()`, `StorageMuxFlasherClient.flash()`/`dump()`, console connect helpers, etc.). They are the exception, not the rule. When in doubt, push the composite to the driver. #### Pattern 5: Resource handle methods -Some interfaces use resource handles — opaque identifiers representing client-side streams negotiated through the Jumpstarter resource system. The `FlasherInterface` and `StorageMuxInterface` are the primary examples: +Some interfaces use resource handles - opaque identifiers representing client-side streams negotiated through the Jumpstarter resource system. The `FlasherInterface` and `StorageMuxInterface` are the primary examples: ```python class FlasherInterface(DriverInterface): @@ -1095,9 +1099,9 @@ class FlasherInterface(DriverInterface): def flash(self, source: str, target: str | None = None) -> None: ... ``` -On the driver side, `source` is a resource UUID received via `DriverCall`. On the client side, the actual `flash()` method creates an `OpendalAdapter` context manager, negotiates a stream handle, and passes it to `self.call("flash", handle, target)`. This orchestration involves file hashing, compression negotiation, and operator selection — none of which can be expressed in protobuf. +On the driver side, `source` is a resource UUID received via `DriverCall`. On the client side, the actual `flash()` method creates an `OpendalAdapter` context manager, negotiates a stream handle, and passes it to `self.call("flash", handle, target)`. This orchestration involves file hashing, compression negotiation, and operator selection - none of which can be expressed in protobuf. -On the wire, resource handles are UUIDs (strings) — they are passed as `string` parameters through `DriverCall`. The generated `.proto` represents these as `string` with a custom annotation `jumpstarter.annotations.resource_handle = true` on the field, signaling to codegen tools that this parameter is a resource reference, not a plain string. +On the wire, resource handles are UUIDs (strings) - they are passed as `string` parameters through `DriverCall`. The generated `.proto` represents these as `string` with a custom annotation `jumpstarter.annotations.resource_handle = true` on the field, signaling to codegen tools that this parameter is a resource reference, not a plain string. The hand-written `FlasherClient` with its `OpendalAdapter` orchestration (file hashing, compression negotiation, stream setup) remains the supported Python client pattern. The proto-level `resource_handle` annotation is a hint for future non-Python codegen; the polyglot resource handle protocol (how Java / Kotlin clients negotiate a stream and obtain a UUID to pass) will be specified in a follow-up JEP alongside non-Python codegen. @@ -1105,7 +1109,7 @@ This pattern affects: `FlasherInterface`, `StorageMuxInterface`, `StorageMuxFlas ### Error Handling and Failure Modes -- **Missing type annotations:** In the default `@export` mode, a missing annotation emits a `DeprecationWarning` but does not block import. In strict mode (`@export(strict=True)` or `JMP_EXPORT_STRICT=1`), a missing annotation raises `TypeError` at decoration time. The codegen and interface check CLIs refuse to produce a proto for an incompletely annotated interface regardless of mode — see "Opt-in type annotation enforcement for `@export`" above. +- **Missing type annotations:** In the default `@export` mode, a missing annotation emits a `DeprecationWarning` but does not block import. In strict mode (`@export(strict=True)` or `JMP_EXPORT_STRICT=1`), a missing annotation raises `TypeError` at decoration time. The codegen and interface check CLIs refuse to produce a proto for an incompletely annotated interface regardless of mode - see "Opt-in type annotation enforcement for `@export`" above. - **Unsupported types:** Complex Python types that don't have a clean protobuf mapping (e.g., `Union[str, int]`, custom metaclasses) cause the codegen CLI to warn and fall back to `google.protobuf.Value`. A future JEP may introduce `oneof` support for `Union` types. @@ -1113,17 +1117,17 @@ This pattern affects: `FlasherInterface`, `StorageMuxInterface`, `StorageMuxFlas - **Reflection registration failure:** If `grpcio-reflection` is not installed (it is an optional dependency), the exporter logs a warning and continues without reflection. The `file_descriptor_proto` field in the report is still populated. -- **Missing descriptor set at startup:** If the exporter cannot find the pre-compiled descriptor set bundled with the driver package, it logs a warning, skips reflection registration for that driver, and leaves `file_descriptor_proto` empty in the report. The driver still loads and serves `DriverCall` traffic normally — descriptor exposure is best-effort. +- **Missing descriptor set at startup:** If the exporter cannot find the pre-compiled descriptor set bundled with the driver package, it logs a warning, skips reflection registration for that driver, and leaves `file_descriptor_proto` empty in the report. The driver still loads and serves `DriverCall` traffic normally - descriptor exposure is best-effort. - **Proto parse failure in the interface check CLI:** If the committed `.proto` file is malformed, `protoc` (invoked as a subprocess) produces a standard error message. The check CLI surfaces this with context about which file failed, and CI fails the build. ### Concurrency and Thread-Safety -`build_file_descriptor()` is a pure function (no side effects, no mutation of inputs) and safe to call from any thread — but it is only called at codegen CLI invocation time, so concurrency is not relevant at runtime. The exporter's descriptor-set load is a single file read during startup before the gRPC server begins accepting connections. The gRPC reflection service is thread-safe by design (`grpcio-reflection` handles concurrent requests internally). +`build_file_descriptor()` is a pure function (no side effects, no mutation of inputs) and safe to call from any thread - but it is only called at codegen CLI invocation time, so concurrency is not relevant at runtime. The exporter's descriptor-set load is a single file read during startup before the gRPC server begins accepting connections. The gRPC reflection service is thread-safe by design (`grpcio-reflection` handles concurrent requests internally). ### Security Implications -gRPC Server Reflection exposes the full interface schema to any client that can reach the exporter's gRPC port. In Jumpstarter's architecture, the exporter is already behind the controller's authentication and lease system — only clients with a valid lease can dial the exporter. Reflection does not bypass this; it's registered on the same `grpc.Server` that serves `ExporterService` and inherits its transport security (mTLS via cert-manager). +gRPC Server Reflection exposes the full interface schema to any client that can reach the exporter's gRPC port. In Jumpstarter's architecture, the exporter is already behind the controller's authentication and lease system - only clients with a valid lease can dial the exporter. Reflection does not bypass this; it's registered on the same `grpc.Server` that serves `ExporterService` and inherits its transport security (mTLS via cert-manager). The `file_descriptor_proto` bytes in the report are served through the authenticated `GetReport` RPC and carry no additional security concern. @@ -1170,7 +1174,7 @@ No HiL tests are required for this JEP. The introspection layer operates entirel - [ ] Exporter loads the bundled descriptor set at startup, registers reflection, and populates `DriverInstanceReport.file_descriptor_proto`. - [ ] `grpcurl list` and `grpcurl describe` return the expected service names and method signatures against a running exporter; invoking a reflected method returns `UNIMPLEMENTED` as documented. - [ ] `jumpstarter/annotations/annotations.proto` is published and importable by external `.proto` files. -- [ ] `DriverCall` / `StreamingDriverCall` wire protocol is byte-for-byte unchanged — a client from before this JEP connects to an exporter that includes this JEP without modification. +- [ ] `DriverCall` / `StreamingDriverCall` wire protocol is byte-for-byte unchanged - a client from before this JEP connects to an exporter that includes this JEP without modification. ## Graduation Criteria @@ -1187,7 +1191,7 @@ No HiL tests are required for this JEP. The introspection layer operates entirel ### Stable - The type mapping table is finalized and documented. -- The interface check CLI runs in CI for all in-tree drivers, catching any drift between `.proto` files and Python interfaces — including doc comment and version drift. +- The interface check CLI runs in CI for all in-tree drivers, catching any drift between `.proto` files and Python interfaces - including doc comment and version drift. - At least one downstream JEP (DeviceClass, non-Python codegen, or Registry) has been implemented using the `.proto` artifacts from this JEP. - No breaking changes to `jumpstarter/annotations.proto` for at least one release cycle. @@ -1195,9 +1199,9 @@ No HiL tests are required for this JEP. The introspection layer operates entirel This JEP is **fully backward compatible.** All changes are additive: -- The `file_descriptor_proto` field (field number 6) is added to `DriverInstanceReport` as `optional bytes`. Old clients using generated stubs from the current `.proto` definition will simply ignore the unknown field — this is standard protobuf behavior. Old exporters will not populate the field, and clients must handle its absence. +- The `file_descriptor_proto` field (field number 6) is added to `DriverInstanceReport` as `optional bytes`. Old clients using generated stubs from the current `.proto` definition will simply ignore the unknown field - this is standard protobuf behavior. Old exporters will not populate the field, and clients must handle its absence. -- gRPC Server Reflection is a separate service (`grpc.reflection.v1.ServerReflection`) registered alongside `ExporterService`. It is invisible to clients that don't query it. No existing RPCs are modified. Reflected services return `UNIMPLEMENTED` when invoked directly — a known limitation scheduled for removal in the native-gRPC follow-up JEP. +- gRPC Server Reflection is a separate service (`grpc.reflection.v1.ServerReflection`) registered alongside `ExporterService`. It is invisible to clients that don't query it. No existing RPCs are modified. Reflected services return `UNIMPLEMENTED` when invoked directly - a known limitation scheduled for removal in the native-gRPC follow-up JEP. - The `@export` decorator is unchanged in its dispatch behavior. Existing markers, dispatch logic, and call semantics are untouched. The only addition is opt-in annotation validation (`strict=True` or `JMP_EXPORT_STRICT=1`), which is off by default. @@ -1227,7 +1231,7 @@ This JEP is **fully backward compatible.** All changes are additive: ### Risks -- **Scope creep.** "Proto-first for Python" is a tempting extension — a contributor might add a small code generator later that re-enters the territory this JEP explicitly left out. The follow-up non-Python codegen JEP needs to land first and set the pattern. +- **Scope creep.** "Proto-first for Python" is a tempting extension - a contributor might add a small code generator later that re-enters the territory this JEP explicitly left out. The follow-up non-Python codegen JEP needs to land first and set the pattern. - **Annotation migration stalls.** Opt-in enforcement is safer but means a package can live indefinitely in a half-annotated state. Mitigation: the codegen CLI refuses incomplete interfaces, so publishing a proto forces completion. - **Native-gRPC follow-up slips.** If the follow-up JEP takes longer than expected, the `UNIMPLEMENTED` reflection footgun persists. Mitigation: include a clear note in the exporter logs and in any `grpcurl` documentation. @@ -1268,34 +1272,34 @@ Encoding type information into the existing `methods_description` map (e.g., as ### Runtime dynamic `FileDescriptorProto` generation at exporter startup -An earlier revision of this JEP (seen in the initial PR discussion) had the exporter construct `FileDescriptorProto` objects dynamically at startup by introspecting `@export` method signatures — with type metadata captured on each function at import time (`MARKER_TYPE_INFO`, `ExportedMethodInfo`). This was rejected in favor of committed `.proto` files produced by the codegen CLI because: +An earlier revision of this JEP (seen in the initial PR discussion) had the exporter construct `FileDescriptorProto` objects dynamically at startup by introspecting `@export` method signatures - with type metadata captured on each function at import time (`MARKER_TYPE_INFO`, `ExportedMethodInfo`). This was rejected in favor of committed `.proto` files produced by the codegen CLI because: - **No reviewable artifact.** Dynamic generation produces no diff at review time. A signature change silently alters the wire schema; polyglot consumers get no CI signal until something breaks. - **Import-time cost and coupling.** Storing `ExportedMethodInfo` on every `@export` function couples dispatch to schema, lengthens import, and bloats memory for drivers that don't need polyglot exposure. -- **Drift detection is simpler without it.** The interface check CLI diffs the live Python class against the committed `.proto`, catching drift directly and deterministically. A dynamic approach would have to diff against a previous run — requiring a lockfile that is effectively the committed `.proto` by another name. +- **Drift detection is simpler without it.** The interface check CLI diffs the live Python class against the committed `.proto`, catching drift directly and deterministically. A dynamic approach would have to diff against a previous run - requiring a lockfile that is effectively the committed `.proto` by another name. - **Committed `.proto` files are the standard protobuf workflow.** `protoc`, `buf`, `grpcurl`, `buf breaking`, and every language's polyglot codegen pipeline expect a committed `.proto` source. Taking the standard path keeps the exporter free of schema-construction work and lets every existing tool participate. Runtime introspection remains available for development-time tooling (the codegen CLI), but it is no longer part of the exporter's runtime path. ## Prior Art -- **gRPC Server Reflection** ([grpc.io/docs/guides/reflection](https://grpc.io/docs/guides/reflection/)) — the standard mechanism for runtime service discovery in gRPC. This JEP uses the exact same `FileDescriptorProto` format and `ServerReflection` service definition. +- **gRPC Server Reflection** ([grpc.io/docs/guides/reflection](https://grpc.io/docs/guides/reflection/)) - the standard mechanism for runtime service discovery in gRPC. This JEP uses the exact same `FileDescriptorProto` format and `ServerReflection` service definition. -- **Buf Schema Registry** ([buf.build](https://buf.build/)) — a hosted registry for protobuf schemas. Jumpstarter's codegen CLI produces `.proto` files that are compatible with Buf's lint, breaking-change detection, and registry tooling. +- **Buf Schema Registry** ([buf.build](https://buf.build/)) - a hosted registry for protobuf schemas. Jumpstarter's codegen CLI produces `.proto` files that are compatible with Buf's lint, breaking-change detection, and registry tooling. -- **Kubernetes Custom Resource Definitions (CRDs)** — Kubernetes uses OpenAPI v3 schemas embedded in CRDs for the same purpose: making API resources self-describing. Jumpstarter's approach is analogous but uses protobuf's native self-description mechanism instead of OpenAPI. +- **Kubernetes Custom Resource Definitions (CRDs)** - Kubernetes uses OpenAPI v3 schemas embedded in CRDs for the same purpose: making API resources self-describing. Jumpstarter's approach is analogous but uses protobuf's native self-description mechanism instead of OpenAPI. -- **LAVA (Linaro Automated Validation Architecture)** — LAVA uses device type definitions and Jinja2 templates to describe hardware capabilities. Jumpstarter's approach is more strongly typed (protobuf vs. YAML templates) but serves the same goal of making device capabilities machine-discoverable. +- **LAVA (Linaro Automated Validation Architecture)** - LAVA uses device type definitions and Jinja2 templates to describe hardware capabilities. Jumpstarter's approach is more strongly typed (protobuf vs. YAML templates) but serves the same goal of making device capabilities machine-discoverable. -- **Robot Framework Remote Library Interface** — Robot Framework's remote library protocol uses XML-RPC with `get_keyword_names` and `get_keyword_arguments` introspection. This JEP serves a similar purpose but uses a modern, strongly-typed, multi-language format. +- **Robot Framework Remote Library Interface** - Robot Framework's remote library protocol uses XML-RPC with `get_keyword_names` and `get_keyword_arguments` introspection. This JEP serves a similar purpose but uses a modern, strongly-typed, multi-language format. ## Unresolved Questions -The following questions can be deferred until implementation. They do not block acceptance of this JEP — each has a reasonable default that can be refined as the codegen and check CLIs are built out. +The following questions can be deferred until implementation. They do not block acceptance of this JEP - each has a reasonable default that can be refined as the codegen and check CLIs are built out. 1. **`Union` type mapping:** How should `Union[str, int]` map to protobuf? `oneof` is the natural choice but adds complexity. Deferring to a future JEP is acceptable since `Union` is rarely used in current driver interfaces. -2. **Bidirectional streaming mapping:** The `@export` decorator supports `STREAM` (bidirectional) in addition to `UNARY` and `SERVER_STREAMING` — the TCP driver already uses bidirectional streaming. The proto mapping for bidirectional streaming (`stream → stream`) needs finalizing in `build_file_descriptor()`. This is required for completeness but can be added after unary and server-streaming support is stable. +2. **Bidirectional streaming mapping:** The `@export` decorator supports `STREAM` (bidirectional) in addition to `UNARY` and `SERVER_STREAMING` - the TCP driver already uses bidirectional streaming. The proto mapping for bidirectional streaming (`stream → stream`) needs finalizing in `build_file_descriptor()`. This is required for completeness but can be added after unary and server-streaming support is stable. 3. **Proto style guide:** Should generated `.proto` files follow Google's style guide, Buf's style guide, or a Jumpstarter-specific convention? This affects field naming (snake_case vs. camelCase) and file organization. @@ -1303,19 +1307,19 @@ The following questions can be deferred until implementation. They do not block 5. **Resource handle annotation in Phase 1:** The `jumpstarter.annotations.resource_handle = true` field option is specified by this JEP, but its consumer (non-Python codegen that understands how to negotiate resource streams) lands in a follow-up. Should the annotation ship in Phase 5 anyway so committed `.proto` files already carry it, or wait until the polyglot resource protocol is designed? -6. **Pydantic model features beyond simple fields:** Pydantic models can have validators, computed properties (`apparent_power` on `PowerReading`), model config, and custom serialization. The builder introspects `model_fields` only — validators and computed properties are not represented in the proto. Is this acceptable, or should computed properties be surfaced as read-only fields? +6. **Pydantic model features beyond simple fields:** Pydantic models can have validators, computed properties (`apparent_power` on `PowerReading`), model config, and custom serialization. The builder introspects `model_fields` only - validators and computed properties are not represented in the proto. Is this acceptable, or should computed properties be surfaced as read-only fields? ## Future Possibilities The following are **not** part of this JEP but are natural extensions enabled by it: -- **DeviceClass contracts and structural enforcement:** With machine-readable interface schemas, a `DeviceClass` CRD can reference specific interfaces and the controller can validate exporters against the contract — not just by checking labels, but by comparing actual `FileDescriptorProto` descriptors. Today, a driver declares that it implements `PowerInterface` by inheriting from the class, but there is no runtime or registration-time verification that the driver's `@export` methods actually match the interface contract. A typo in a method name, a missing parameter, or a wrong return type silently breaks clients at call time. The `FileDescriptorProto` from this JEP enables structural enforcement at every level of the DeviceClass mechanism: +- **DeviceClass contracts and structural enforcement:** With machine-readable interface schemas, a `DeviceClass` CRD can reference specific interfaces and the controller can validate exporters against the contract - not just by checking labels, but by comparing actual `FileDescriptorProto` descriptors. Today, a driver declares that it implements `PowerInterface` by inheriting from the class, but there is no runtime or registration-time verification that the driver's `@export` methods actually match the interface contract. A typo in a method name, a missing parameter, or a wrong return type silently breaks clients at call time. The `FileDescriptorProto` from this JEP enables structural enforcement at every level of the DeviceClass mechanism: - *At build time:* The interface check CLI verifies that a Python interface matches its `.proto` definition. This extends to verifying that a driver implementation's `@export` methods match the interface proto — catching signature mismatches before code is shipped. + *At build time:* The interface check CLI verifies that a Python interface matches its `.proto` definition. This extends to verifying that a driver implementation's `@export` methods match the interface proto - catching signature mismatches before code is shipped. - *At exporter registration time:* The controller receives `FileDescriptorProto` descriptors in each driver's `DriverInstanceReport`. It compares these against the canonical `FileDescriptorProto` stored in a DeviceClass or InterfaceClass CRD to perform structural validation — comparing actual method signatures, parameter types, return types, and streaming semantics. A driver that claims to implement `power-v1` but is missing the `read()` streaming method would be flagged at registration, not discovered at test time. + *At exporter registration time:* The controller receives `FileDescriptorProto` descriptors in each driver's `DriverInstanceReport`. It compares these against the canonical `FileDescriptorProto` stored in a DeviceClass or InterfaceClass CRD to perform structural validation - comparing actual method signatures, parameter types, return types, and streaming semantics. A driver that claims to implement `power-v1` but is missing the `read()` streaming method would be flagged at registration, not discovered at test time. - *At lease time:* A lease requesting a specific DeviceClass resolves to a set of required interface references, each with a canonical proto. The controller validates that every matched exporter's drivers produce compatible descriptors — ensuring that the leased device actually satisfies the contract the test code was generated against. + *At lease time:* A lease requesting a specific DeviceClass resolves to a set of required interface references, each with a canonical proto. The controller validates that every matched exporter's drivers produce compatible descriptors - ensuring that the leased device actually satisfies the contract the test code was generated against. *For driver certification:* A DeviceClass could declare compliance requirements: "this device provides `power-v1` at version `1.0.0` with these exact method signatures." A future registry could track which driver packages are certified against which interface versions, and `jmp validate` could verify local exporter configurations against the published DeviceClass contract before deployment. @@ -1323,21 +1327,21 @@ The following are **not** part of this JEP but are natural extensions enabled by - **Polyglot client code generation:** The `.proto` files produced by the codegen CLI feed directly into `protoc` for Kotlin, TypeScript, Rust, and other language stubs. A `jmp codegen` tool could wrap this pipeline. -- **Typed composite children:** Composite drivers today wire children dynamically (`self.children["power"] = DutlinkPower(...)`) with no enforceable contract — consumers cast manually (e.g., `tcp_driver: TcpNetwork = self.children["tcp"]`), and there is no static handle on a composite's shape on either the driver or client side. A follow-up JEP can introduce a `child()` field-style sentinel on `DriverInterface` subclasses (e.g., `power: PowerInterface = child()`), with `DriverInterfaceMeta` collecting the declarations once and the `Driver` and `CompositeClient` base classes enforcing them symmetrically — types validated at exporter startup against `self.children`, and at client construction against the `DriverInstanceReport` tree. The mechanism is purely Python-side (no `.proto` changes) and opt-in: composites that don't declare `child()` fields keep today's untyped behavior. Composition is already discoverable polyglot-side via the report tree plus each child's `file_descriptor_proto` (this JEP), so no proto annotation is needed. +- **Typed composite children:** Composite drivers today wire children dynamically (`self.children["power"] = DutlinkPower(...)`) with no enforceable contract - consumers cast manually (e.g., `tcp_driver: TcpNetwork = self.children["tcp"]`), and there is no static handle on a composite's shape on either the driver or client side. A follow-up JEP can introduce a `child()` field-style sentinel on `DriverInterface` subclasses (e.g., `power: PowerInterface = child()`), with `DriverInterfaceMeta` collecting the declarations once and the `Driver` and `CompositeClient` base classes enforcing them symmetrically - types validated at exporter startup against `self.children`, and at client construction against the `DriverInstanceReport` tree. The mechanism is purely Python-side (no `.proto` changes) and opt-in: composites that don't declare `child()` fields keep today's untyped behavior. Composition is already discoverable polyglot-side via the report tree plus each child's `file_descriptor_proto` (this JEP), so no proto annotation is needed. -- **Driver registry:** A controller-level registry that catalogs available drivers, interfaces, and DeviceClasses — serving `FileDescriptorProto` artifacts for codegen and reflection. +- **Driver registry:** A controller-level registry that catalogs available drivers, interfaces, and DeviceClasses - serving `FileDescriptorProto` artifacts for codegen and reflection. - **Interface versioning and compatibility checking:** Using `buf breaking` against committed `.proto` files to enforce backward-compatible interface evolution across releases. -- **Dynamic client construction:** A "generic driver client" that uses `FileDescriptorProto` and `DynamicMessage` to invoke any driver method without pre-generated stubs — useful for debugging, REPL exploration, and ad-hoc tooling. +- **Dynamic client construction:** A "generic driver client" that uses `FileDescriptorProto` and `DynamicMessage` to invoke any driver method without pre-generated stubs - useful for debugging, REPL exploration, and ad-hoc tooling. - **Additional custom options:** If the community identifies metadata that genuinely needs to be machine-readable beyond what proto comments provide (e.g., units of measurement, timing constraints, safety classifications), new options can be added to `jumpstarter/annotations.proto` via a follow-up JEP without changing the core introspection mechanism. -- **Interactive API documentation:** A web UI (served by the controller or Buf Schema Registry) that renders the `.proto` files as browsable, searchable API docs — similar to Swagger/OpenAPI but for gRPC driver interfaces, with proto comments displayed inline. +- **Interactive API documentation:** A web UI (served by the controller or Buf Schema Registry) that renders the `.proto` files as browsable, searchable API docs - similar to Swagger/OpenAPI but for gRPC driver interfaces, with proto comments displayed inline. - **Native protobuf wire protocol (future JEP):** The `.proto` files produced by this JEP are the foundation for migrating from string-based `DriverCall` dispatch to native gRPC services. A detailed design sketch follows. -### Native gRPC Transport — Design Sketch +### Native gRPC Transport - Design Sketch #### What changes @@ -1433,11 +1437,11 @@ class PowerServicer(power_pb2_grpc.PowerInterfaceServicer): ) ``` -The servicer is a thin adapter — it deserializes the compiled protobuf request, calls the driver method, and serializes the response. No `encode_value` / `decode_value`, no string lookup. +The servicer is a thin adapter - it deserializes the compiled protobuf request, calls the driver method, and serializes the response. No `encode_value` / `decode_value`, no string lookup. #### Duplicate instances: UUID routing interceptor -A single exporter can host multiple drivers implementing the same interface (e.g., `main_power` and `aux_power` both implementing `PowerInterface`). gRPC services are singletons — you can't register two `PowerInterfaceServicer` instances. +A single exporter can host multiple drivers implementing the same interface (e.g., `main_power` and `aux_power` both implementing `PowerInterface`). gRPC services are singletons - you can't register two `PowerInterfaceServicer` instances. The solution is a server interceptor that reads the driver UUID from gRPC metadata and dispatches to the correct instance: @@ -1458,7 +1462,7 @@ class DriverRoutingInterceptor(grpc.aio.ServerInterceptor): metadata = dict(handler_call_details.invocation_metadata) uuid_str = metadata.get("x-jumpstarter-driver-uuid") if uuid_str is None: - # No UUID header — fall through to legacy DriverCall + # No UUID header - fall through to legacy DriverCall return await continuation(handler_call_details) # Route to the correct driver's servicer @@ -1492,7 +1496,7 @@ async def serve_async(self, server): #### Server side: `@export` during transition -During the dual-path transition period, driver methods retain their `@export` decorators. The legacy `DriverCall` path still needs them for string-based dispatch. The native `PowerServicer` adapter calls the same underlying methods — both paths converge on the same driver implementation: +During the dual-path transition period, driver methods retain their `@export` decorators. The legacy `DriverCall` path still needs them for string-based dispatch. The native `PowerServicer` adapter calls the same underlying methods - both paths converge on the same driver implementation: ```python class MockPower(PowerInterface, Driver): @@ -1505,15 +1509,15 @@ class MockPower(PowerInterface, Driver): self.logger.info("power off") ``` -If `DriverCall` is eventually retired (see Migration phases below), the `@export` decorators would become unnecessary for dispatch — but they would continue to serve as the type introspection mechanism for `build_file_descriptor()` and `ExportedMethodInfo` capture. +If `DriverCall` is eventually retired (see Migration phases below), the `@export` decorators would become unnecessary for dispatch - but they would continue to serve as the type introspection mechanism for `build_file_descriptor()` and `ExportedMethodInfo` capture. #### Client side: `DriverClient` auto-generates native stubs -The `DriverClient` base class handles native stub creation automatically. When a driver's `DriverInstanceReport` includes a `file_descriptor_proto` and the exporter supports native gRPC, `DriverClient` creates the compiled stub internally — individual client classes don't need manual wiring: +The `DriverClient` base class handles native stub creation automatically. When a driver's `DriverInstanceReport` includes a `file_descriptor_proto` and the exporter supports native gRPC, `DriverClient` creates the compiled stub internally - individual client classes don't need manual wiring: ```python class AsyncDriverClient(Metadata): - """Base class — auto-creates native stub when available.""" + """Base class - auto-creates native stub when available.""" async def _init_native_stub(self): """Called during client setup if FileDescriptorProto is present.""" @@ -1540,10 +1544,10 @@ class AsyncDriverClient(Metadata): return decode_value(response.result) ``` -The generated client code stays clean — it calls `self.call("on")` as before, and the base class routes to the native stub transparently: +The generated client code stays clean - it calls `self.call("on")` as before, and the base class routes to the native stub transparently: ```python -# Generated client — unchanged from DriverCall era +# Generated client - unchanged from DriverCall era class PowerClient(PowerInterface, DriverClient): def on(self) -> None: self.call("on") # DriverClient routes to native stub if available @@ -1559,7 +1563,7 @@ class PowerClient(PowerInterface, DriverClient): For non-Python clients, the compiled stubs are used directly with standard gRPC patterns: ```kotlin -// Kotlin — standard gRPC stub with metadata +// Kotlin - standard gRPC stub with metadata val channel = ManagedChannelBuilder.forAddress(host, port).build() val interceptor = UuidMetadataInterceptor("abc-123") val stub = PowerInterfaceGrpcKt.PowerInterfaceCoroutineStub(channel) @@ -1575,10 +1579,10 @@ stub.read(Empty.getDefaultInstance()).collect { reading -> During the transition, the exporter serves both protocols simultaneously: -- **Legacy path:** `ExporterService.DriverCall(uuid, "on", [])` — string dispatch with `Value` serialization. Existing Python clients continue to work. -- **Native path:** `PowerInterface.On(Empty)` + `x-jumpstarter-driver-uuid` metadata — compiled protobuf. New and polyglot clients use this. +- **Legacy path:** `ExporterService.DriverCall(uuid, "on", [])` - string dispatch with `Value` serialization. Existing Python clients continue to work. +- **Native path:** `PowerInterface.On(Empty)` + `x-jumpstarter-driver-uuid` metadata - compiled protobuf. New and polyglot clients use this. -Both paths call the same underlying driver methods. The driver implementation is unchanged — it's the dispatch and serialization layers that differ. +Both paths call the same underlying driver methods. The driver implementation is unchanged - it's the dispatch and serialization layers that differ. ``` ┌─────────────────────────────┐ @@ -1598,14 +1602,14 @@ The first two phases are concrete proposals; what follows them is intentionally left open until the dual-path implementation has been validated in the field. 1. **This JEP:** Generate `FileDescriptorProto` and `.proto` files. Wire protocol unchanged. Polyglot clients can use `DynamicMessage` with `DriverCall` and the descriptor. -2. **Future JEP — dual path:** Exporter registers native gRPC services alongside `DriverCall`. Compile `.proto` files to stubs. New clients can opt into the native path; existing clients are unchanged. +2. **Future JEP - dual path:** Exporter registers native gRPC services alongside `DriverCall`. Compile `.proto` files to stubs. New clients can opt into the native path; existing clients are unchanged. **Possible future outcomes (not committed by this JEP):** After the dual-path implementation has been built, exercised in real deployments, and shown to be a complete substitute for `DriverCall`, the community may choose to take additional steps. Whether any of these steps -are taken — and on what timeline — is intentionally deferred. They are +are taken - and on what timeline - is intentionally deferred. They are listed here only to make the design space explicit: - **Deprecation (possible):** Mark `DriverCall` as deprecated and publish a migration guide, once the native path is known to cover every use case currently served by `DriverCall` (including resource-handle streaming, bidirectional drivers, and out-of-tree drivers). @@ -1615,28 +1619,28 @@ listed here only to make the design space explicit: | Phase | Deliverable | Depends On | | ----- | --------------------------------------------------------------------------------------------------------------------- | ------------- | -| 1a | `DriverInterfaceMeta` + `DriverInterface` base class — type-safe interface marking with registry and validation | — | +| 1a | `DriverInterfaceMeta` + `DriverInterface` base class - type-safe interface marking with registry and validation | - | | 1b | Migrate standard in-tree interfaces to `DriverInterface` and dual-inheritance clients (type annotations recommended) | Phase 1a | -| 2 | Opt-in `@export` annotation validation — warn by default, `@export(strict=True)` / `JMP_EXPORT_STRICT=1` | — | -| 3 | Type mapping module — Python types to protobuf field types, handling BaseModel, list, enum, UUID | — | +| 2 | Opt-in `@export` annotation validation - warn by default, `@export(strict=True)` / `JMP_EXPORT_STRICT=1` | - | +| 3 | Type mapping module - Python types to protobuf field types, handling BaseModel, list, enum, UUID | - | | 4 | `build_file_descriptor()` library function for build-time use | Phase 1a, 3 | -| 5 | `jumpstarter/annotations/annotations.proto` — `resource_handle` field option | — | -| 6 | Doc comment extraction — docstrings to proto comments in builder | Phase 4 | -| 7 | Codegen CLI — Python → `.proto` source files | Phase 4, 5, 6 | +| 5 | `jumpstarter/annotations/annotations.proto` - `resource_handle` field option | - | +| 6 | Doc comment extraction - docstrings to proto comments in builder | Phase 4 | +| 7 | Codegen CLI - Python → `.proto` source files | Phase 4, 5, 6 | | 8 | Commit `.proto` files and `protoc --descriptor_set_out` artifacts for standard in-tree interfaces | Phase 7 | | 9 | `DriverInstanceReport.file_descriptor_proto` populated from bundled descriptor set at exporter startup | Phase 8 | | 10 | gRPC Server Reflection registration from bundled descriptor set (advisory; services return `UNIMPLEMENTED` if called) | Phase 8 | -| 11 | Interface check CLI — CI drift detection between committed `.proto` and live Python interface | Phase 7 | +| 11 | Interface check CLI - CI drift detection between committed `.proto` and live Python interface | Phase 7 | -Phases 1a–1b establish the type-safe interface foundation and the dual-inheritance client convention. Phase 2 delivers opt-in annotation validation. Phases 3–4 build the build-time introspection core. Phases 5–7 deliver the developer-facing tooling. Phases 8–10 deliver runtime schema exposure from the committed artifacts. Phase 11 closes the loop with CI drift detection. +Phases 1a-1b establish the type-safe interface foundation and the dual-inheritance client convention. Phase 2 delivers opt-in annotation validation. Phases 3-4 build the build-time introspection core. Phases 5-7 deliver the developer-facing tooling. Phases 8-10 deliver runtime schema exposure from the committed artifacts. Phase 11 closes the loop with CI drift detection. Proto-first codegen and native gRPC transport are **out of scope** for this JEP and are planned as follow-up JEPs. ## Implementation History - 2026-04-06: JEP drafted -- 2026-04-07: JEP refined — added `DriverInterface` metaclass, type enforcement on `@export`, resource handle pattern, native gRPC migration sketch; fixed Pydantic BaseModel usage, NetworkInterface proto, driver adapter scope; expanded type mapping table and unresolved questions -- 2026-04-30: Simplified — pivoted to build-time generation of committed `.proto` files, dropped proto-first adapter and dynamic runtime introspection, made type enforcement opt-in, added grpcurl `UNIMPLEMENTED` note +- 2026-04-07: JEP refined - added `DriverInterface` metaclass, type enforcement on `@export`, resource handle pattern, native gRPC migration sketch; fixed Pydantic BaseModel usage, NetworkInterface proto, driver adapter scope; expanded type mapping table and unresolved questions +- 2026-04-30: Simplified - pivoted to build-time generation of committed `.proto` files, dropped proto-first adapter and dynamic runtime introspection, made type enforcement opt-in, added grpcurl `UNIMPLEMENTED` note - 2026-05-09: Deferred concrete CLI command names (now referred to as the codegen CLI and the interface check CLI); fixed spelling typos flagged by `typos`; added out-of-tree drivers section with no-proto fallback behavior; clarified interface check CLI discovery via `DriverInterfaceMeta._registry`; expanded Pattern 4 to recommend promoting most client-side composites to server-side `@export` methods, keeping client-side orchestration only for complex drivers like network and flasher ## References @@ -1646,8 +1650,8 @@ Proto-first codegen and native gRPC transport are **out of scope** for this JEP - [google.protobuf.FileDescriptorProto](https://github.com/protocolbuffers/protobuf/blob/main/src/google/protobuf/descriptor.proto) - [Buf Schema Registry](https://buf.build/docs/bsr/introduction) - [grpcurl](https://github.com/fullstorydev/grpcurl) -- [Jumpstarter Driver Architecture](https://docs.jumpstarter.dev/introduction/key-concepts.html) -- [Jumpstarter `@export` Decorator Source](https://github.com/jumpstarter-dev/jumpstarter/blob/main/packages/jumpstarter/jumpstarter/driver/decorators.py) +- [Jumpstarter Driver Architecture](https://jumpstarter.dev/main/introduction/) +- [Jumpstarter `@export` Decorator Source](https://github.com/jumpstarter-dev/jumpstarter/blob/main/python/packages/jumpstarter/jumpstarter/driver/decorators.py) - [Python `inspect.signature()`](https://docs.python.org/3/library/inspect.html#inspect.signature) --- diff --git a/python/docs/source/internal/jeps/JEP-0013-observability-telemetry-logs.md b/python/docs/source/contributing/jeps/JEP-0013-observability-telemetry-logs.md similarity index 88% rename from python/docs/source/internal/jeps/JEP-0013-observability-telemetry-logs.md rename to python/docs/source/contributing/jeps/JEP-0013-observability-telemetry-logs.md index 645d0d973..5d5debbec 100644 --- a/python/docs/source/internal/jeps/JEP-0013-observability-telemetry-logs.md +++ b/python/docs/source/contributing/jeps/JEP-0013-observability-telemetry-logs.md @@ -1,3 +1,7 @@ +--- +orphan: true +--- + # JEP-0013: Metrics, Tracing, and Log Observability | Field | Value | @@ -10,9 +14,9 @@ | **Created** | 2026-04-23 | | **Updated** | 2026-05-04 | | **Discussion** | https://github.com/jumpstarter-dev/jumpstarter/pull/631 | -| **Requires** | — | -| **Supersedes** | — | -| **Superseded-By** | — | +| **Requires** | - | +| **Supersedes** | - | +| **Superseded-By** | - | --- @@ -21,8 +25,8 @@ This JEP defines an optional, cross-component observability model for Jumpstarter covering lease context metadata, structured operational events, exporter/driver metrics, and standardized logging. It targets direct integration -with Prometheus (scrape), Loki (log aggregation), and Perses (dashboards) — -without mandating OpenTelemetry — and introduces an optional in-cluster +with Prometheus (scrape), Loki (log aggregation), and Perses (dashboards) - +without mandating OpenTelemetry - and introduces an optional in-cluster Jumpstarter Telemetry service that aggregates data from exporters and clients so that edge processes never need Loki or cluster-scrape credentials. Implementation is expected to land in phases; this JEP describes the end state @@ -79,14 +83,14 @@ exporter-level metrics that a monitoring stack can scrape or receive. ### Concepts -- **Lease context** — Identifiers and labels supplied by a client or CI and +- **Lease context** - Identifiers and labels supplied by a client or CI and associated for the life of a lease, propagated where safe so metrics, logs, and traces can be filtered and joined. -- **Lease events** (or *operations*) — Annotated, structured log entries +- **Lease events** (or *operations*) - Annotated, structured log entries recording significant actions (for example *flash started*, *flash failed*, *image reference*) with typed fields, queryable in **Loki** alongside regular logs and distinct from higher-frequency debug output (see **DD-2**). -- **Exporter metrics** — Counters (operations, bytes), histograms (operation +- **Exporter metrics** - Counters (operations, bytes), histograms (operation duration), and gauges (active sessions) exposed from the exporter and enriched by individual drivers via the `driver_type` label. Each driver selects a category from a predefined set in jumpstarter core (e.g. @@ -95,11 +99,11 @@ exporter-level metrics that a monitoring stack can scrape or receive. Composite drivers (e.g. Renode, QEMU) that bundle multiple sub-drivers do not emit a single top-level category for delegated work. Instead, each sub-driver emits its own `driver_type` when it performs an - operation — a Renode storage sub-driver emits `driver_type="storage"`, + operation - a Renode storage sub-driver emits `driver_type="storage"`, its power sub-driver emits `driver_type="power"`, and so on. Any top-level methods on the composite driver itself (e.g. VM lifecycle) emit `driver_type="composite"`. -- **Jumpstarter Telemetry** (optional) — a dedicated component that +- **Jumpstarter Telemetry** (optional) - a dedicated component that reverse-scrapes connected exporters for metrics via `MetricsStream` and receives structured logs via `PushLogs`, using the same trust model (mTLS, ServiceAccount) as Controller/Router. It isolates @@ -120,9 +124,9 @@ exporter-level metrics that a monitoring stack can scrape or receive. existing exporter↔control-plane trust boundary. On each Prometheus scrape, the Telemetry service fans out to connected exporters and serves the merged `/metrics` output (see **DD-3**, **DD-7**), with - cluster credentials — avoiding per-exporter Loki and metrics secrets. + cluster credentials - avoiding per-exporter Loki and metrics secrets. Exporters and clients also push structured log entries via `PushLogs` - (not unbounded default chatter — see *Control-plane aggregation* + (not unbounded default chatter - see *Control-plane aggregation* below). - The `jmp` CLI output remains human-readable, but when a Telemetry endpoint is available, `jmp` also pushes structured JSON logs to the @@ -161,11 +165,11 @@ message TelemetryEndpoint { Exporters call `GetServiceEndpoints` after `Register`; clients call it after authentication. An empty `telemetry_endpoints` list means telemetry -is not deployed — callers skip all telemetry RPCs. Older controllers +is not deployed - callers skip all telemetry RPCs. Older controllers that do not implement the method return `UNIMPLEMENTED`, which callers treat identically to an empty list. -#### gRPC: Telemetry service (`telemetry.proto` — new file) +#### gRPC: Telemetry service (`telemetry.proto` - new file) A new `protocol/proto/jumpstarter/v1/telemetry.proto` defines the `TelemetryService` implemented by `jumpstarter-telemetry`. It has two @@ -295,7 +299,7 @@ codes do not require a proto regeneration), and validation of allowed values is enforced at the application layer using the operator's configuration (e.g. `driverTypeEnum` allowlist). The same reasoning applies to `extra_fields` and `structured_fields` in -`LogStreamResponse` — they carry driver-specific key-value data +`LogStreamResponse` - they carry driver-specific key-value data destined for log bodies, not typed metrics. #### gRPC: `AuditStream` removal (`jumpstarter.proto`) @@ -304,7 +308,7 @@ The existing `AuditStream` RPC on `ControllerService` and its `AuditStreamRequest` message are removed. Analysis of the codebase shows this is dead code: -- The Go controller has no implementation — calls fall through to +- The Go controller has no implementation - calls fall through to `UnimplementedControllerServiceServer` which returns `codes.Unimplemented`. - No Python code (exporter or client) calls the RPC. @@ -316,7 +320,7 @@ message format. #### gRPC: `LogStreamResponse` enrichment (`jumpstarter.proto`) -The existing `LogStream` RPC on `ExporterService` is kept — it serves +The existing `LogStream` RPC on `ExporterService` is kept - it serves a fundamentally different purpose (real-time session logs from exporter to connected client) from the Telemetry log push. However, the `LogStreamResponse` message is enriched with optional additive @@ -337,14 +341,14 @@ message LogStreamResponse { } ``` -These fields are optional and backward compatible — older clients +These fields are optional and backward compatible - older clients ignore unknown fields; older exporters simply do not set them. The same size limits as `LogEntry.extra_fields` apply to `structured_fields` (16 entries, 64-char keys, 256-char values). #### Tracing scope -This JEP covers *correlation only* — `lease_id`, `trace_id`, +This JEP covers *correlation only* - `lease_id`, `trace_id`, and `span_id` are propagated as log fields and Prometheus exemplar keys so that metrics, logs, and (future) traces can be joined. Full distributed tracing (span creation, sampling policies, trace storage and visualization) is deferred @@ -361,7 +365,7 @@ metadata ignored by older servers). ### DD-1: How lease-scoped *context* metadata is stored **Scope:** This decision is about where to store generic metadata on a -`Lease` that describes *why* a run exists or *where* it came from — for example +`Lease` that describes *why* a run exists or *where* it came from - for example an external build id, pipeline id, VCS revision, or other operator-defined keys (team, environment), within the cardinality and size limits defined in *Cardinality guidelines*. The same stored context @@ -374,15 +378,15 @@ identity without re-typing it on every line. **Alternatives considered:** -1. **Annotation and label only** on the `Lease` object — Kube-native, no spec +1. **Annotation and label only** on the `Lease` object - Kube-native, no spec change; limited size for annotations; labels for select queries only. 2. **Typed subfields under `spec`** (for example `observability` or `context`) - — easier validation, clearer API, migration path in CRD. -3. **Only client-side** (environment / local config) — no cluster visibility; + - easier validation, clearer API, migration path in CRD. +3. **Only client-side** (environment / local config) - no cluster visibility; hard for operators to audit; no stable object-level link to per-lease metrics and server logs in the cluster. -**Decision:** **(2)** — a typed `spec.context` map under the Lease CRD for +**Decision:** **(2)** - a typed `spec.context` map under the Lease CRD for first-class, validated context. **(1)** (labels/annotations) remains allowed for integration with generic tooling that only understands Kubernetes metadata or benefits from lease label filtering. @@ -394,11 +398,11 @@ are still useful for selection and for tools that only understand metadata. **Alternatives considered:** -1. **Kubernetes `Event` objects** — built-in, TTL-limited, good for +1. **Kubernetes `Event` objects** - built-in, TTL-limited, good for "what happened" in `kubectl get events` but not long-term history by default. -2. **`Lease.status.conditions` only** — compact but poor for a sequence of +2. **`Lease.status.conditions` only** - compact but poor for a sequence of operations with payloads (image id, size). -3. **Dedicated CRD** (for example per-event or a single stream object) — more +3. **Dedicated CRD** (for example per-event or a single stream object) - more design and RBAC, better long-term retention and querying if backed properly. 4. **Annotated log events** Provides a lightweight alternative that can be traced and filtered along logs. @@ -417,7 +421,7 @@ are still useful for selection and for tools that only understand metadata. `status.conditions` **(2)** is a poor fit for a sequence of operations with variable payloads (image digest, byte count, duration); a dedicated CRD **(3)** adds schema versioning, RBAC surface, and per-event etcd writes - that scale with flash volume — all pressure the cluster does not need + that scale with flash volume - all pressure the cluster does not need for data whose primary consumers are dashboards and post-mortem queries, not reconciliation loops. Structured log events carry arbitrary fields without CRD migration, support configurable retention in Loki, @@ -428,21 +432,21 @@ are still useful for selection and for tools that only understand metadata. **Alternatives considered:** -1. **HTTP `GET /metrics` in Prometheus text format** (pull) — the default +1. **HTTP `GET /metrics` in Prometheus text format** (pull) - the default for in-cluster Prometheus in scrape mode; works with the Prometheus Operator (`ServiceMonitor`), `kube-prometheus`, and self-hosted jobs. The optional Jumpstarter Telemetry service exposes this for aggregated counters it holds after receiving +1 / +N from exporters. 2. **Prometheus remote write** (or a Mimir / Cortex receiver) - from a Jumpstarter component — useful in advanced topologies; not + from a Jumpstarter component - useful in advanced topologies; not part of the reference implementation in this JEP; operators can add a federation or `remote_write` from Prometheus to long-term storage without the application pushing to Prometheus. -3. **Both** — **(1)** is required for the documented path; **(2)** is +3. **Both** - **(1)** is required for the documented path; **(2)** is optional infrastructure behind Prometheus, not a second required app protocol. -4. **Reverse scrape via gRPC** — exporters maintain a local +4. **Reverse scrape via gRPC** - exporters maintain a local `prometheus_client.CollectorRegistry` and connect to the Telemetry service via a persistent bidirectional gRPC stream (`MetricsStream`). When Prometheus scrapes the Telemetry service's `/metrics` endpoint, @@ -452,7 +456,7 @@ are still useful for selection and for tools that only understand metadata. scrape (no change). This avoids push-increment complexity on the wire and keeps full counter state on the exporter at all times. -**Decision:** **(4)** — exporter-originated metrics are reverse-scraped +**Decision:** **(4)** - exporter-originated metrics are reverse-scraped through the Telemetry service via `MetricsStream`. **Rationale:** Exporters are often behind NAT or firewalls and cannot @@ -460,7 +464,7 @@ are still useful for selection and for tools that only understand metadata. solves this: the exporter initiates an outbound gRPC stream (NAT-friendly, same direction as the existing controller connection), the Telemetry service requests metric snapshots on demand, and full - counter state remains on the exporter at all times — eliminating + counter state remains on the exporter at all times - eliminating lost-increment concerns (see **DD-9**). The exporter uses standard `prometheus_client` primitives locally, so driver authors instrument with familiar counters and histograms. The OpenMetrics exposition @@ -521,11 +525,11 @@ are still useful for selection and for tools that only understand metadata. **Alternatives considered:** -1. **JSON always** for every process — best for machines; hard for humans. +1. **JSON always** for every process - best for machines; hard for humans. 2. **Human text default for `jmp`**, **JSON for long-running services** and a CLI push via the Telemetry ingest endpoint in JSON format (in addition to the human-friendly output) -3. **Single format** with a pretty-printer in front of developers — more moving +3. **Single format** with a pretty-printer in front of developers - more moving parts. **Decision:** **(2)**. Long-running services (`jumpstarter-controller`, @@ -541,13 +545,13 @@ are still useful for selection and for tools that only understand metadata. the same time all services get parseable, joinable log lines. Writing JSON to stdout and relying on the cluster log shipper for Loki delivery decouples the Controller reconciler and Router session handling from - Loki availability — a Loki outage does not affect lease operations. + Loki availability - a Loki outage does not affect lease operations. The Telemetry service retains a direct Loki-push because it is an isolated workload (**DD-7**) whose core job is Loki ingest. **Format:** JSONL (one JSON object per line), produced by setting `--zap-encoder=json` on the existing `controller-runtime` / Zap logger - (no changes to log call sites — existing `logr` structured fields become + (no changes to log call sites - existing `logr` structured fields become JSON keys automatically). The `ts`, `level`, and `msg` fields follow Zap's default JSON encoder output; application code adds domain fields via the standard `logr` `WithValues` / `Info` / `Error` API. @@ -578,7 +582,7 @@ are still useful for selection and for tools that only understand metadata. and used as indexed stream selectors. They must be low-cardinality to keep the active stream count manageable (Grafana recommends < 100 k active streams per tenant). With the labels above, a deployment with - 200 exporters across 5 namespaces produces roughly 1 000 streams — + 200 exporters across 5 namespaces produces roughly 1 000 streams - well within budget. High-cardinality fields like `client` or `lease_id` must stay in the JSON body: promoting `client` to a stream label in a 1 000-client, 200-exporter cluster would create @@ -596,22 +600,22 @@ are still useful for selection and for tools that only understand metadata. **Alternatives considered:** 1. **Each exporter and edge host** holds credentials (or a sidecar) to push - directly to Loki and to Prometheus (or a metrics gateway) — maximum + directly to Loki and to Prometheus (or a metrics gateway) - maximum flexibility; maximum secret distribution and rotation burden on lab and remote sites. 2. **Jumpstarter Controller and/or Router** receive metrics and structured events from exporters and (optionally) from client traffic they already handle, and forward to the Loki push API and to Prometheus-compatible sinks (scrape registration) - with in-cluster auth — one + with in-cluster auth - one credential surface; enriched with lease, exporter, and client context in one place; must be non-blocking, bounded, and optional so the control path does not depend on Loki or Prometheus availability. -3. **Hybrid** — generic in-cluster collectors for raw pod logs and scrape; +3. **Hybrid** - generic in-cluster collectors for raw pod logs and scrape; (2) for lease-scoped events and aggregated exporter metrics the platform understands. 4. **Dedicated Jumpstarter Telemetry Deployment** (see **DD-7**) - instead of folding everything into the Controller — only + instead of folding everything into the Controller - only Telemetry holds Loki-push credentials; isolated failure domain and scaling for reverse-scrape and log ingest. Router and Controller write structured JSON to stdout (see **DD-4**) and expose `/metrics` @@ -624,12 +628,12 @@ are still useful for selection and for tools that only understand metadata. cluster-ingest authentication to every exporter process while still attaching Jumpstarter-specific context. Among Jumpstarter components, only `jumpstarter-telemetry` - holds Loki-push credentials — the Controller and Router have no Loki + holds Loki-push credentials - the Controller and Router have no Loki client dependency (see **DD-4**); their pod logs reach Loki via the cluster's existing log shipping infrastructure. Generic in-cluster collectors solve *credentials* but not *semantic* correlation unless - integrated; alternative (2)'s trust-model advantage — which (4) - inherits — reuses the existing exporter→controller relationship and + integrated; alternative (2)'s trust-model advantage - which (4) + inherits - reuses the existing exporter→controller relationship and can inject labels and tenant context in one place. A separate Deployment (**4** / **DD-7**) is preferable to overloading the main reconciler when load or residency of counters matters. @@ -638,7 +642,7 @@ are still useful for selection and for tools that only understand metadata. **Alternatives considered:** -1. **Adopt OpenTelemetry** — instrument Controller, Router, Exporter, and +1. **Adopt OpenTelemetry** - instrument Controller, Router, Exporter, and clients with the OTel SDK, export OTLP to a cluster-local OpenTelemetry Collector, and let the Collector fan out to Loki, Prometheus (remote write), and Tempo. @@ -647,9 +651,9 @@ are still useful for selection and for tools that only understand metadata. (or logfmt) logs to stdout for shippers; optional W3C `traceparent` in gRPC metadata for correlation *without* shipping full distributed traces in the first iteration. If traces are ever needed, use Tempo - ingest where practical, *or* a thin sender — still + ingest where practical, *or* a thin sender - still without a project-wide requirement on the OTel SDK in every binary. -3. **Hybrid (OTel in one language, direct in another)** — lowest common +3. **Hybrid (OTel in one language, direct in another)** - lowest common implementation cost but inconsistent contributor experience and two operational models. @@ -657,30 +661,30 @@ are still useful for selection and for tools that only understand metadata. Collector) part of the required reference architecture. Vendors and operators who already run an OpenTelemetry Collector may scrape the same `/metrics`, receive logs shipped by existing agents, or - receive the Loki body the hub would have sent — compatibility + receive the Loki body the hub would have sent - compatibility is welcome; dependency is not mandatory. **Rationale:** The proposed Jumpstarter Telemetry service (**DD-7**) admittedly -reimplements a subset of OTel Collector functionality — metric +reimplements a subset of OTel Collector functionality - metric aggregation, log forwarding, backpressure, and multi-replica HA. The decision to build a purpose-built component rather than adopt the OTel Collector rests on three arguments, ordered by importance: -1. **Identity enforcement (primary)** — The Telemetry service operates +1. **Identity enforcement (primary)** - The Telemetry service operates inside Jumpstarter's existing authentication and trust domain (mTLS, registered client and exporter identities). It validates that every incoming `MetricsStream` or `PushLogs` call originates from the - claimed exporter — preventing impersonation or label - injection — using identities the platform already manages. A generic + claimed exporter - preventing impersonation or label + injection - using identities the platform already manages. A generic OTel Collector has no awareness of Jumpstarter identities; achieving the same guarantee would require an external auth policy layer (e.g. custom processors, mTLS-to-attribute mapping, and a sidecar or admission webhook to enforce label provenance), adding complexity that offsets the Collector's generality. -2. **Operational simplicity** — The Telemetry service is a single Go +2. **Operational simplicity** - The Telemetry service is a single Go binary with a single config surface (the operator CR), no separate version matrix, and no generic pipeline DSL. An OTel Collector requires operator familiarity with its configuration model @@ -690,7 +694,7 @@ Collector rests on three arguments, ordered by importance: monitor. This overhead is not justified when the data paths are known in advance. -3. **Narrow scope** — Jumpstarter metrics and lease events map directly +3. **Narrow scope** - Jumpstarter metrics and lease events map directly to Prometheus and Loki wire protocols that operators already use. Full three-pillar OTel (unified logs and metrics via OTLP) is *optional product territory*; this JEP optimizes for low ceremony @@ -710,7 +714,7 @@ project dependency. **Alternatives considered:** -1. **In-process** in the Controller (and Router) reconciler — few +1. **In-process** in the Controller (and Router) reconciler - few moving parts; risk of CPU / GC pressure and stronger coupling between leases and high-volume increments or Loki writes. 2. A **dedicated** in-cluster Service and Deployment (working name @@ -718,10 +722,10 @@ project dependency. exporters and clients, applies them to counters in memory, POSTs to Loki, exposes `/metrics`, and uses the same K8s ServiceAccount / mTLS as other control-plane binaries. -3. **Split** into separate sidecars (Loki-only, metrics-only) — more images to +3. **Split** into separate sidecars (Loki-only, metrics-only) - more images to build and version. 4. **Dedicated Deployment with reverse-scrape for metrics and push for - logs** — same dedicated `jumpstarter-telemetry` Deployment as **(2)**, + logs** - same dedicated `jumpstarter-telemetry` Deployment as **(2)**, but instead of receiving increment RPCs the service reverse-scrapes connected exporters via `MetricsStream` (see *API / Protocol Changes*). Exporters maintain local `prometheus_client` registries; @@ -729,7 +733,7 @@ project dependency. demand when its `/metrics` endpoint is hit, merges the results, and serves them to Prometheus. Logs and events are still pushed by exporters and clients via `PushLogs`. Client-side metrics are not - collected — all metrically-interesting operations are observable + collected - all metrically-interesting operations are observable from the exporter side. **Decision:** Prefer **(4)** for the optional aggregated-metrics + Loki @@ -742,7 +746,7 @@ project dependency. Loki spikes and ingest load cannot starve lease reconciliation in the controller. The reverse-scrape model **(4)** is preferred over the increment-push model **(2)** because full counter state stays on the - exporter — no metrics are lost when the Telemetry service restarts or + exporter - no metrics are lost when the Telemetry service restarts or is temporarily unavailable, and idempotency concerns are eliminated (see **DD-9**). @@ -757,7 +761,7 @@ supporting detail, not an independent responsibility. identity of every `MetricsStream` connection and `PushLogs` RPC from the mTLS certificate or ServiceAccount token. The `exporter` and `client` labels on incoming data are enforced server-side to match the - authenticated identity — a compromised or misconfigured exporter + authenticated identity - a compromised or misconfigured exporter cannot submit metrics under another exporter's name or inject arbitrary labels. @@ -766,9 +770,9 @@ supporting detail, not an independent responsibility. | Scenario | Behavior | | ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Telemetry service unavailable | Exporters keep counting locally; no metrics are lost. When the exporter reconnects, the next scrape returns the full current counter state. Log push RPCs are fire-and-forget with bounded retry; log entries may be lost but device operations are unaffected. | -| Telemetry pod restart | Metric state is rebuilt on the next scrape from each connected exporter — no permanent data loss. Prometheus `rate()` and `increase()` handle the apparent counter reset transparently. | +| Telemetry pod restart | Metric state is rebuilt on the next scrape from each connected exporter - no permanent data loss. Prometheus `rate()` and `increase()` handle the apparent counter reset transparently. | | Loki unreachable | The Telemetry service buffers log entries in a bounded queue (see *Backpressure* in the control-plane section). On overflow, entries are dropped and `jumpstarter_telemetry_dropped_total` incremented. | -| Prometheus scrape fails | No data loss — the next successful scrape triggers a fresh fan-out to connected exporters and returns current values. | +| Prometheus scrape fails | No data loss - the next successful scrape triggers a fresh fan-out to connected exporters and returns current values. | The Telemetry service exposes `/healthz` (liveness) and `/readyz` (readiness, gated on Loki reachability and at least one connected @@ -779,7 +783,7 @@ supporting detail, not an independent responsibility. parallel** and waits up to `spec.telemetry.metrics.scrapeTimeout` (default: 7 s) for responses. **Only metrics received during the current fan-out are included in the response.** Exporters that do not - respond in time are omitted entirely — no cached or stale data is + respond in time are omitted entirely - no cached or stale data is ever served. This eliminates any risk of double-counting from stale connections where the exporter may have already migrated to another replica (see **DD-8**). @@ -788,23 +792,23 @@ supporting detail, not an independent responsibility. temporarily holds metric snapshots from responding exporters until the merged response is written to Prometheus. With 200 exporters each producing ~50 series (bounded by `{operation, result, driver_type}` - label combinations), the peak is ~10 000 series at ~200–300 bytes - each, costing ~2–3 MB. Snapshots are discarded as soon as the - `/metrics` response is flushed — no metric data is retained between + label combinations), the peak is ~10 000 series at ~200-300 bytes + each, costing ~2-3 MB. Snapshots are discarded as soon as the + `/metrics` response is flushed - no metric data is retained between scrapes. ### DD-8: Multiple Telemetry replicas (HA) and persistent exporter connections **Context:** With the reverse-scrape model (see **DD-3** alternative 4 and *API / Protocol Changes*), the Telemetry service does not hold -authoritative counter state — exporters maintain their own local +authoritative counter state - exporters maintain their own local `prometheus_client` registries. The Telemetry service only caches the latest metric snapshot per exporter. Each exporter opens a single long-lived `MetricsStream` to one Telemetry replica. **Alternatives considered:** -1. **Single replica** for Telemetry — no cross-pod `sum` issue; SPOF for +1. **Single replica** for Telemetry - no cross-pod `sum` issue; SPOF for ingest and scrape of that `Service`. 2. **Multiple replicas** behind a load balancer; each RPC updates one pod, which only advances its partial counters for the label @@ -815,8 +819,8 @@ long-lived `MetricsStream` to one Telemetry replica. event is applied at most once in the system (counters are additive; increments are partitioned by traffic). 3. **Strong consistency** (Raft, Redis as source of truth for - counters) — higher operating cost than this JEP’s v1 scope. -4. **Multiple replicas with persistent exporter connections** — each exporter + counters) - higher operating cost than this JEP’s v1 scope. +4. **Multiple replicas with persistent exporter connections** - each exporter opens a single long-lived `MetricsStream` to one replica (persistent by stream). Each replica only caches metric snapshots for its connected exporters. Prometheus scrapes all replicas (via `PodMonitor`); @@ -825,7 +829,7 @@ long-lived `MetricsStream` to one Telemetry replica. double-counting, because each exporter’s metrics appear on exactly one replica’s `/metrics` output. On replica failure the exporter reconnects to a survivor and the next scrape returns its full - current counter state — no data is lost. + current counter state - no data is lost. **Decision:** **(4)** @@ -841,19 +845,19 @@ long-lived `MetricsStream` to one Telemetry replica. ### DD-9: Idempotency vs. best-effort **Context:** With the reverse-scrape model, metrics idempotency is a -non-issue — each scrape returns the full current counter state from the +non-issue - each scrape returns the full current counter state from the exporter, so there are no increments to deduplicate or double-count. The only remaining idempotency concern is for `PushLogs` RPCs, where a retry could result in duplicate log entries in Loki. **Alternatives considered:** -1. **Idempotent** log pushes (deduplication keys per `LogEntry`) — +1. **Idempotent** log pushes (deduplication keys per `LogEntry`) - appropriate for billing- or SLO-sensitive log pipelines; requires a dedup store or Loki-side dedup. 2. **Best effort** (at-least-once) for `PushLogs` without global - deduplication — simpler; rare duplicate log entries on retries. -3. **Metrics idempotency** (dedup keys on metric increments) — no + deduplication - simpler; rare duplicate log entries on retries. +3. **Metrics idempotency** (dedup keys on metric increments) - no longer applicable; the reverse-scrape model returns full state, making increment deduplication moot. @@ -868,35 +872,35 @@ a retry could result in duplicate log entries in Loki. **Alternatives considered:** -1. **Grafana** — mature, widely deployed, massive plugin and datasource +1. **Grafana** - mature, widely deployed, massive plugin and datasource ecosystem; governed by Grafana Labs (commercial); AGPL v3 license; custom JSON dashboard format; external to Kubernetes architecture. -2. **Perses** — CNCF project (vendor-neutral governance); Apache 2.0 +2. **Perses** - CNCF project (vendor-neutral governance); Apache 2.0 license; standardized dashboard spec (CUE/JSON) with built-in static validation and SDKs for GitOps; Kubernetes-native (CRD support for dashboards-as-code); data-source focus on Prometheus, Loki, and - Tempo — exactly the backends this JEP targets. + Tempo - exactly the backends this JEP targets. **Decision:** **(2)** **Rationale:** -- **License alignment** — Jumpstarter is Apache 2.0; recommending an +- **License alignment** - Jumpstarter is Apache 2.0; recommending an AGPL-licensed dashboard layer introduces license friction for downstream distributors and embedders. -- **CNCF governance** — vendor-neutral stewardship matches the project's +- **CNCF governance** - vendor-neutral stewardship matches the project's open-source posture; no single-vendor control over the dashboard layer. -- **Kubernetes-native CRDs** — dashboards can be managed as K8s resources, +- **Kubernetes-native CRDs** - dashboards can be managed as K8s resources, fitting the same declarative, reconciler-driven model Jumpstarter already uses for Leases, Exporters, and the optional Telemetry Deployment. -- **GitOps and validation** — CUE-based specs with static validation and SDKs +- **GitOps and validation** - CUE-based specs with static validation and SDKs enable dashboard-as-code in CI pipelines, consistent with the JEP's emphasis on automation and CI integration. -- **Backend focus** — Perses targets Prometheus, Loki, and Tempo — exactly the - three backends this JEP standardizes on — without carrying the cost of a +- **Backend focus** - Perses targets Prometheus, Loki, and Tempo - exactly the + three backends this JEP standardizes on - without carrying the cost of a broad plugin ecosystem the project does not need. -**Perses vs Grafana — practical comparison:** +**Perses vs Grafana - practical comparison:** | Aspect | Perses | Grafana | | -------------------- | --------------------------------------- | ------------------------------------------ | @@ -910,8 +914,8 @@ a retry could result in duplicate log entries in Loki. The main Perses gap today is exemplar visualization. Operators who need exemplar overlays on dashboards should use Grafana alongside Perses or -wait for upstream support. Grafana remains fully compatible — all -`/metrics` and Loki endpoints are standard — so the choice is +wait for upstream support. Grafana remains fully compatible - all +`/metrics` and Loki endpoints are standard - so the choice is non-exclusive. Operators who prefer Grafana can still point it at the same `/metrics` and Loki @@ -921,19 +925,19 @@ endpoints; this DD only governs the *recommended* dashboard experience. ### Correlation and fields -*Subject to review — names and cardinality rules should be fixed before +*Subject to review - names and cardinality rules should be fixed before "Implemented".* | Field / label | Prom label | Prom exemplar | Loki stream | Log line | Notes | | -------------------------------- | :--------: | :-----------: | :---------: | :------: | --------------------------------------------------- | -| `exporter` | yes | — | yes | yes | CRD name; bounded by cluster size. | -| `operation` | yes | — | no | yes | Small fixed enum (flash, power, …). | -| `result` | yes | — | no | yes | Small fixed enum (success, failure, …). | -| `driver_type` | yes | — | no | yes | Category from a predefined set in core (storage, power, …). | -| `error_type` | yes | — | no | yes | Failure class (timeout, device_error, …); on errors. | -| `direction` | yes | — | no | yes | tx / rx; for byte-counter and stream metrics only. | -| `component` | no | — | yes | yes | Fixed set (cli, controller, router, telemetry, exporter).| -| `namespace` | no | — | yes | yes | K8s namespace; bounded. | +| `exporter` | yes | - | yes | yes | CRD name; bounded by cluster size. | +| `operation` | yes | - | no | yes | Small fixed enum (flash, power, …). | +| `result` | yes | - | no | yes | Small fixed enum (success, failure, …). | +| `driver_type` | yes | - | no | yes | Category from a predefined set in core (storage, power, …). | +| `error_type` | yes | - | no | yes | Failure class (timeout, device_error, …); on errors. | +| `direction` | yes | - | no | yes | tx / rx; for byte-counter and stream metrics only. | +| `component` | no | - | yes | yes | Fixed set (cli, controller, router, telemetry, exporter).| +| `namespace` | no | - | yes | yes | K8s namespace; bounded. | | `lease_id` | **no** | yes | **no** | yes | Unbounded; exemplar for drill-down. | | `client` | **no** | yes | **no** | yes | CRD name; exemplar for client identity. | | `image_digest`, `build_id`, etc. | **no** | yes | **no** | yes | From `spec.context`; included when listed in `exemplarKeys`. | @@ -958,7 +962,7 @@ Rules of thumb for this JEP: - **Prometheus labels**: each metric label dimension should have < 100 distinct values per scrape target. The label set for Jumpstarter metrics is - `{exporter, operation, result, driver_type}` — all bounded enums. + `{exporter, operation, result, driver_type}` - all bounded enums. `error_type` is added on failure-path metrics and `direction` on byte-counter metrics. High-cardinality context is carried via exemplars, not labels. @@ -986,7 +990,7 @@ Default exemplar keys emitted on every counter/histogram observation: | `lease_id` | Lease UID | Correlate a metric sample with lease logs. | | `trace_id` | W3C `traceparent` | Included **only when present** in gRPC metadata.| -`trace_id` is not synthesized by Jumpstarter — it is included only when +`trace_id` is not synthesized by Jumpstarter - it is included only when an external caller (CI pipeline, user code) propagates a `traceparent`. Full distributed tracing (spans, storage, visualization) is deferred to a future JEP; when it lands, `trace_id` becomes a default key. Until @@ -994,14 +998,14 @@ then, omitting it saves ~45 characters of exemplar budget. `spec.context` keys (e.g. `build_id`, `image_digest`) are included as exemplar keys when listed in the operator's `exemplarKeys` allowlist (see -*Operator configuration*). Because exemplars are per-observation metadata — -not label dimensions — they have zero impact on series cardinality regardless +*Operator configuration*). Because exemplars are per-observation metadata - +not label dimensions - they have zero impact on series cardinality regardless of how many distinct values appear. **Exemplar size budget:** The OpenMetrics 1.0 limit is 128 UTF-8 characters for the combined key-value pairs in a single exemplar. -The two default keys (`client`, `lease_id`) consume roughly 30–50 -characters, leaving ~80–100 characters for `spec.context` entries +The two default keys (`client`, `lease_id`) consume roughly 30-50 +characters, leaving ~80-100 characters for `spec.context` entries (or more when `trace_id` is absent). To stay within budget: 1. Keys are added in the order specified by the operator's @@ -1057,7 +1061,7 @@ on every observation. | `jumpstarter_operations_total` | Dashboard | yes | Failure rate > 20 % over 15 min per exporter. | | `jumpstarter_operation_duration_seconds` | Dashboard | yes | p95 > 60 s per operation type. | | `jumpstarter_operation_errors_total` | Dashboard | yes | Error rate rising; group by `error_type`. | -| `jumpstarter_stream_bytes_total` | Dashboard | no | — | +| `jumpstarter_stream_bytes_total` | Dashboard | no | - | | `jumpstarter_active_sessions` | Dashboard | yes | 0 sessions for > 30 min (possible exporter issue). | | `jumpstarter_lease_acquisitions_total` | Dashboard | yes | Failure rate > 10 % over 15 min. | | `jumpstarter_telemetry_dropped_total` | Alerting | yes | Any increment (telemetry pipeline saturated). | @@ -1072,7 +1076,7 @@ environments with different baselines. **High-frequency byte counters:** `jumpstarter_stream_bytes_total` can be incremented at very high rates on serial and video streams. Because metrics live in the exporter's local `prometheus_client` registry, high -update rates do not generate any RPC traffic — the counter is updated +update rates do not generate any RPC traffic - the counter is updated in-process and only serialized when the Telemetry service sends a `MetricsScrapeRequest`. @@ -1194,7 +1198,7 @@ When this mode is enabled in a deployment: for the Loki log push path with a configurable depth (default: 10 000 entries, see `spec.telemetry.backpressure.queueDepth`). On overflow, dropped entries are replaced by a single **drop marker** - — a standard `LogEntry` with `severity="warning"`, + - a standard `LogEntry` with `severity="warning"`, `component="telemetry"`, `operation="backpressure"`, and the drop count and time window placed in `extra_fields` (`{"count":"142","window_seconds":"12"}`). Subsequent drops while the @@ -1204,7 +1208,7 @@ When this mode is enabled in a deployment: consumers do not need special-case parsing to detect or exclude it. A `jumpstarter_telemetry_dropped_total` counter (partitioned by `destination={loki}`) is also incremented on `/metrics` for alerting. - Metrics do not need backpressure — the reverse-scrape model is + Metrics do not need backpressure - the reverse-scrape model is pull-based and transient (no buffering between scrapes). Because the Controller and Router do not push to Loki, their lease/session operations are inherently isolated from Loki slowdowns. @@ -1300,7 +1304,7 @@ run one *alongside* and scrape the same targets if they choose. This JEP’s target wire protocols and components are Prometheus and Loki (and, if trace export is ever added, Tempo or Jaeger with -native ingest or HTTP — not OTLP as a *Jumpstarter* requirement; see +native ingest or HTTP - not OTLP as a *Jumpstarter* requirement; see **DD-6**). OpenTelemetry is a parallel ecosystem: teams can run a Collector next to Jumpstarter and still scrape `/metrics` and ship logs with Promtail-class agents; the reference design does not depend @@ -1315,14 +1319,14 @@ on the OTel SDK in application code. Perses (see **DD-10**) for search and with Promtail, Grafana Agent, or Grafana Alloy to ship logs, or with application push to Loki’s HTTP API as already discussed in the control-plane path. -- Traces (optional, future work) — if adopted, Grafana Tempo and Jaeger +- Traces (optional, future work) - if adopted, Grafana Tempo and Jaeger are typical stores; use W3C Trace Context in RPC metadata for correlation even when full trace export is off. OTLP may be *only* a convenience for operators; it is not a JEP-0011 core dependency. - A typical Kubernetes integration path: `ServiceMonitor` + Prometheus (or a compatible remote-write consumer), a Loki endpoint for logs - — any EKS, GKE, AKS, self-managed + - any EKS, GKE, AKS, self-managed Kubernetes, or bare-metal install that runs these same projects can be the target; the implementation plan should name tested combinations (Prometheus and Loki version @@ -1339,9 +1343,9 @@ can tune metrics, logging, and exemplar behavior without editing code. | Field | Type | Default | Description | | ----------------------------------------- | ---------- | ------------------------------------------------ | ---------------------------------------------------------------------------------------------- | | `spec.telemetry.enabled` | `bool` | `false` | Deploy the optional Telemetry service. | -| `spec.telemetry.loki.url` | `string` | — | Loki push endpoint; optional — Telemetry can run metrics-only without Loki. | -| `spec.telemetry.loki.secretRef` | `string` | — | Secret with Loki credentials (see **DD-5**). | -| `spec.telemetry.loki.tls.caSecretRef` | `string` | — | Secret containing a CA bundle (`ca.crt` key) to trust for the Loki endpoint. | +| `spec.telemetry.loki.url` | `string` | - | Loki push endpoint; optional - Telemetry can run metrics-only without Loki. | +| `spec.telemetry.loki.secretRef` | `string` | - | Secret with Loki credentials (see **DD-5**). | +| `spec.telemetry.loki.tls.caSecretRef` | `string` | - | Secret containing a CA bundle (`ca.crt` key) to trust for the Loki endpoint. | | `spec.telemetry.loki.tls.insecureSkipVerify` | `bool` | `false` | Disable TLS certificate verification (development/testing only). | | `spec.telemetry.exporterLabels` | `[]string` | `[]` | Exporter-level label keys (e.g. `board-type`) copied from Exporter CRD labels into log JSON fields and exemplar candidates. | | `spec.telemetry.metrics.exemplarKeys` | `[]string` | `["client", "lease_id"]` | Allowlist of keys to include in exemplars (including `spec.context` and `exporterLabels` keys). Only listed keys are emitted; unlisted keys are omitted even if present. | @@ -1404,11 +1408,11 @@ candidates for operations involving that exporter. For example, setting `exporterLabels: ["board-type"]` means an Exporter with the label `board-type: rpi4` will include `"board-type": "rpi4"` in its structured log lines and in the exemplar candidate pool. The list is -empty by default — no exporter labels are propagated unless the +empty by default - no exporter labels are propagated unless the administrator opts in. The `exemplarKeys` list is an **allowlist** that controls which keys are -included in Prometheus exemplars. This filters *everything* — built-in +included in Prometheus exemplars. This filters *everything* - built-in keys (`client`, `lease_id`), `spec.context` keys, and `exporterLabels` keys alike. Only keys present in `exemplarKeys` are emitted; unlisted keys are omitted even if available. This gives administrators full @@ -1467,14 +1471,14 @@ Each component must be testable in isolation without deploying the full stack: - **Structured logging**: unit tests validate JSON output format, base - fields, and `spec.context` propagation using an in-memory logger — no + fields, and `spec.context` propagation using an in-memory logger - no Loki required. - **Exporter metrics**: unit tests verify counter/histogram registration, label correctness, and exemplar attachment using a local Prometheus - registry — no Telemetry service required. + registry - no Telemetry service required. - **Telemetry service**: integration tests use mock gRPC clients and a mock Loki endpoint to verify ingest, counter aggregation, backpressure - behavior, and drop markers — no real exporters required. + behavior, and drop markers - no real exporters required. - **Operator configuration**: unit tests validate CRD admission (e.g. `spec.context` size limits) and `ServiceMonitor` generation. @@ -1492,7 +1496,7 @@ constraints make this impractical, at minimum: Loki path. - **Prometheus scrape**: the existing Go/Ginkgo E2E test suite performs direct HTTP scrapes of the `/metrics` endpoints on Controller, Router, - and Telemetry services — no separate Prometheus instance required. The + and Telemetry services - no separate Prometheus instance required. The test parses the OpenMetrics response and asserts that documented series, labels, and exemplars appear after a known operation sequence. - **Correlation round-trip**: an E2E test runs a lease lifecycle (create → @@ -1547,13 +1551,13 @@ all subsequent phases have E2E coverage from the start. applicable. - **`AuditStream` removal:** The `AuditStream` RPC and `AuditStreamRequest` message on `ControllerService` are removed. This RPC was never implemented - or called by any client — `Grep` across the codebase confirms zero usage + or called by any client - `Grep` across the codebase confirms zero usage outside its protobuf definition. Removing it is a no-op for all existing deployments. The new `PushLogs` RPC on `TelemetryService` supersedes the intended use case. - `LogStreamResponse` enrichment (new optional fields `driver_type`, `operation`, `timestamp`, `structured_fields`) is purely additive and - backward-compatible — existing clients ignore unknown fields. + backward-compatible - existing clients ignore unknown fields. - No removal of current default CLI behavior; JSON logging only when selected. ## Consequences @@ -1594,30 +1598,30 @@ all subsequent phases have E2E coverage from the start. Operators on older Prometheus versions still get full metrics and logs; exemplar-based drill-down is unavailable until they upgrade. - Prometheus / Loki / Perses-stack version drift in the field - — document tested pairs; W3C Trace Context in gRPC remains + - document tested pairs; W3C Trace Context in gRPC remains best-effort across Python and Go (no OTel SDK requirement to propagate `traceparent` where needed). ## Rejected Alternatives -- **"All metrics and facts are *generated* only in the controller"** — would +- **"All metrics and facts are *generated* only in the controller"** - would miss per-exporter and per-driver truth; rejected. *Forwarding* exporter-originated series and events *through* the control-plane (with stable labels) is not the same and remains in scope (see DD-5). - *Requiring Loki- and Prometheus-ingest credentials on every exporter - and edge* as the only supported model — rejected in favor of + and edge* as the only supported model - rejected in favor of optional hub forwarding and of cluster-native collectors that also avoid per-host secrets, even though those collectors are not Jumpstarter-specific. - **"Mandatory OpenTelemetry SDK and Collector"** for all metrics, - logs, and traces — rejected for the reference architecture; + logs, and traces - rejected for the reference architecture; rationale in **DD-6** (optional parallel deployment by operators is still fine). -- **"Unstructured logs everywhere; parse with regex"** — rejected as +- **"Unstructured logs everywhere; parse with regex"** - rejected as unscalable for joins with traces and multi-service incidents. -- **"Mandatory full tracing for every command"** — high overhead; rejected; prefer +- **"Mandatory full tracing for every command"** - high overhead; rejected; prefer sampling and opt-in for heavy paths. -- **"Push metric increments from exporters to telemetry"** — exporters +- **"Push metric increments from exporters to telemetry"** - exporters would send `+1`/`+N` counter increments and histogram observations to the Telemetry service, which would maintain in-memory counters and expose them on `/metrics`. Rejected because: (a) counter state would @@ -1626,7 +1630,7 @@ all subsequent phases have E2E coverage from the start. stream bytes) generate excessive RPC traffic. The reverse-scrape model keeps full counter state on the exporter and generates zero RPC traffic between scrapes (see **DD-3** alternative 4, **DD-7**). -- **"Reuse `AuditStream` for telemetry log push"** — `AuditStream` was an +- **"Reuse `AuditStream` for telemetry log push"** - `AuditStream` was an unimplemented stub on `ControllerService` with no message schema for structured telemetry data. Rather than retrofitting it, a purpose-built `PushLogs` RPC on the new `TelemetryService` provides a cleaner contract @@ -1635,25 +1639,25 @@ all subsequent phases have E2E coverage from the start. ## Prior Art - [Prometheus](https://prometheus.io/) and [Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/) - — time-series metrics and alerting; [Prometheus naming and labels](https://prometheus.io/docs/practices/naming/) + - time-series metrics and alerting; [Prometheus naming and labels](https://prometheus.io/docs/practices/naming/) on cardinality and naming; remote write for non-scrape topologies; - [Exemplars](https://prometheus.io/docs/instrumenting/exposition_formats/#exemplars) + [Exemplars](https://prometheus.io/docs/instrumenting/exposition_formats/#exemplars-experimental) for attaching high-cardinality context to individual samples. - [Grafana exemplar support](https://grafana.com/docs/grafana/latest/fundamentals/exemplars/) - — visualizing exemplars in metric panels and linking to traces or logs. -- [Loki](https://grafana.com/oss/loki/) — log aggregation, label model, and push + - visualizing exemplars in metric panels and linking to traces or logs. +- [Loki](https://grafana.com/oss/loki/) - log aggregation, label model, and push and query APIs; often combined with [Perses](https://perses.dev/) (see **DD-10**) and Grafana Agent / Alloy or [Promtail](https://grafana.com/docs/loki/latest/send-data/promtail/) for log shipping. -- [Grafana Tempo](https://grafana.com/oss/tempo/) or [Jaeger](https://www.jaegertracing.io/) — common trace backends - (native or HTTP ingest; OTLP where the operator uses it — not a +- [Grafana Tempo](https://grafana.com/oss/tempo/) or [Jaeger](https://www.jaegertracing.io/) - common trace backends + (native or HTTP ingest; OTLP where the operator uses it - not a Jumpstarter code dependency; see **DD-6**). -- [Perses](https://perses.dev/) — CNCF dashboard project; Apache 2.0; +- [Perses](https://perses.dev/) - CNCF dashboard project; Apache 2.0; Kubernetes-native CRDs; CUE/JSON spec with GitOps SDKs; focused on Prometheus, Loki, and Tempo data sources (see **DD-10**). - [OpenTelemetry](https://opentelemetry.io/) and the - [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) — + [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) - relevant as ecosystem and operator-side *optional* plumbing; this JEP intentionally does not adopt them in-process by default (**DD-6**). - Other HiL / test systems often separate "run metadata" (like Jenkins build @@ -1680,7 +1684,7 @@ all subsequent phases have E2E coverage from the start. ## References -- [JEP-0000 — JEP Process](JEP-0000-jep-process.md) +- [JEP-0000 - JEP Process](JEP-0000-jep-process.md) - [Kubernetes Events](https://kubernetes.io/docs/reference/kubernetes-api/cluster-resources/event-v1/) - [W3C Trace Context](https://www.w3.org/TR/trace-context/) (`traceparent`) - Upstream project docs for the Prometheus, Loki, and diff --git a/python/docs/source/internal/jeps/JEP-NNNN-template.md b/python/docs/source/contributing/jeps/JEP-NNNN-template.md similarity index 96% rename from python/docs/source/internal/jeps/JEP-NNNN-template.md rename to python/docs/source/contributing/jeps/JEP-NNNN-template.md index f3b9b5a93..c14c02532 100644 --- a/python/docs/source/internal/jeps/JEP-NNNN-template.md +++ b/python/docs/source/contributing/jeps/JEP-NNNN-template.md @@ -7,7 +7,7 @@ orphan: true ### User Stories *(optional)* @@ -116,7 +116,7 @@ orphan: true For each decision, state what was decided, what alternatives were considered, and why the chosen approach was preferred. This section - is the most important part of the JEP for long-term project memory — + is the most important part of the JEP for long-term project memory - future contributors will refer to it to understand *why* things are the way they are. --> @@ -125,8 +125,8 @@ orphan: true **Alternatives considered:** -1. **Option A** — Brief description. -2. **Option B** — Brief description. +1. **Option A** - Brief description. +2. **Option B** - Brief description. **Decision:** Option A. @@ -178,7 +178,7 @@ Reference specific project constraints, prior art, or technical tradeoffs. JmpMCP - JmpMCP -- "Lease & connect" --> DUTs -``` - -## Prerequisites - -- Jumpstarter CLI (`jmp`) installed and configured with a client identity -- An MCP-compatible AI tool (Cursor, Claude Code, Claude Desktop, or any - MCP client) - -The MCP server package, which is normally provided when you perform a full install -through the `jumpstarter-mcp` package which provides the `jmp mcp serve` subcommand on the CLI. - -## Setup - -### Cursor - -Add to your Cursor MCP configuration (`~/.cursor/mcp.json`): - -```json -{ - "mcpServers": { - "jumpstarter": { - "command": "jmp", - "args": ["mcp", "serve"] - } - } -} -``` - -Restart Cursor, then verify the server appears in **Settings > MCP**. The -Jumpstarter tools will be available to the AI agent in Composer. - -### Claude Code - -Register the MCP server with a single command: - -```bash -claude mcp add jumpstarter -- jmp mcp serve -``` - -This writes the configuration to `~/.claude.json`. Verify with: - -```bash -claude mcp list -``` - -Alternatively, you can add it manually to `~/.claude.json`: - -```json -{ - "mcpServers": { - "jumpstarter": { - "command": "jmp", - "args": ["mcp", "serve"] - } - } -} -``` - -### Claude Desktop - -Add to your Claude Desktop configuration: - -- **macOS**: `~/Library/Application Support/Claude/claude_desktop_config.json` -- **Windows**: `%APPDATA%\Claude\claude_desktop_config.json` - -```json -{ - "mcpServers": { - "jumpstarter": { - "command": "jmp", - "args": ["mcp", "serve"] - } - } -} -``` - -Restart Claude Desktop and the Jumpstarter tools will appear in the tools menu. - -### Other MCP Clients - -Any MCP-compatible client can use the Jumpstarter server. The server -communicates over stdio using the standard MCP protocol. Launch it with: - -```bash -jmp mcp serve -``` - -## Available Tools - -The MCP server exposes the following tools: - -### Lease & Exporter Management - -| Tool | Description | -|---|---| -| `jmp_list_exporters` | List exporters with online status and lease info | -| `jmp_list_leases` | List active leases | -| `jmp_create_lease` | Create a new lease by selector or exporter name | -| `jmp_delete_lease` | Release a lease | - -### Connection Management - -| Tool | Description | -|---|---| -| `jmp_connect` | Connect to a device (by lease, selector, or exporter) | -| `jmp_disconnect` | Disconnect from a device | -| `jmp_list_connections` | List active connections | - -### Device Interaction - -| Tool | Description | -|---|---| -| `jmp_run` | Execute CLI commands on a connected device | -| `jmp_get_env` | Get shell/Python environment for direct device access | - -### Discovery & Introspection - -| Tool | Description | -|---|---| -| `jmp_explore` | Discover available CLI commands and their arguments | -| `jmp_drivers` | List Python driver objects and their methods | -| `jmp_driver_methods` | Inspect method signatures, docstrings, and parameters | - -## Usage Examples - -### Example: Interactive Hardware Exploration - -Once the MCP server is configured, you can interact with hardware using natural -language from your AI assistant: - -> **You**: What devices are available on the cluster? -> -> *Agent calls `jmp_list_exporters` and shows a summary of available hardware.* -> -> **You**: Get me a QEMU target and power it on. -> -> *Agent calls `jmp_create_lease`, `jmp_connect`, then `jmp_run` with -> `["power", "on"]`.* -> -> **You**: Check what OS is running via SSH. -> -> *Agent calls `jmp_run` with `["ssh", "--", "cat", "/etc/os-release"]` and -> interprets the output.* -> -> **You**: Give me a Python example to automate this. -> -> *Agent calls `jmp_get_env` and generates a script using the `env()` helper.* - -### Example: Claude Code Session - -``` -$ claude - -> /mcp - -Connected MCP servers: - - jumpstarter (jmp mcp serve) - -> Can you list the hardware available on the jumpstarter cluster? - -I'll check what devices are available... - -[Uses jmp_list_exporters] - -Here's what's available: - - qemu-test-01 (online, no active lease) - - arm-board-01 (online, leased by alice) - - arm-board-02 (online, no active lease) - -> Lease arm-board-02 and check if it boots to Linux - -[Uses jmp_create_lease, jmp_connect, jmp_run to power on and SSH] - -The board is running Fedora 41 (aarch64). Here's the full `uname -a` output... -``` - -### Example: Cursor Agent Mode - -In Cursor's Composer (Agent mode), the Jumpstarter tools are available -alongside your code. This enables workflows like: - -1. Ask the agent to flash a new firmware image to a board -2. Have it verify the board boots successfully via serial console -3. Run your test suite against the live hardware -4. Iterate on code fixes with the agent retesting on real hardware - -## Typical Workflow - -```{mermaid} -:config: {"theme":"base","themeVariables":{"primaryColor":"#f8f8f8","primaryTextColor":"#000","primaryBorderColor":"#e5e5e5","lineColor":"#3d94ff","secondaryColor":"#f8f8f8","tertiaryColor":"#fff"}} -sequenceDiagram - participant User - participant Agent as AI Agent - participant MCP as MCP Server - participant Ctrl as Controller - - User->>Agent: "Get me an ARM board" - Agent->>MCP: jmp_create_lease(selector="arch=arm64") - MCP->>Ctrl: Request lease - Ctrl-->>MCP: Lease ID - MCP-->>Agent: Lease created - - Agent->>MCP: jmp_connect(lease_id) - MCP-->>Agent: Connected - - Agent->>MCP: jmp_explore() - MCP-->>Agent: Available commands: power, ssh, serial, storage - - User->>Agent: "Power it on and check the OS" - Agent->>MCP: jmp_run(["power", "on"]) - Agent->>MCP: jmp_run(["ssh", "--", "cat", "/etc/os-release"]) - MCP-->>Agent: OS info - - User->>Agent: "Done, release it" - Agent->>MCP: jmp_disconnect() - Agent->>MCP: jmp_delete_lease() -``` - -## Tips - -- **Use `jmp_explore` first**: Each device type exposes different commands. - Always explore before assuming what's available. -- **Set `timeout_seconds` for streaming commands**: Commands like `serial pipe` - block indefinitely. Use a short `timeout_seconds` (e.g., 10-15) so the - command is killed after capturing available output. -- **Use `jmp_drivers` for Python access**: When you need programmatic control - beyond CLI commands, inspect the Python driver tree to discover available - methods and their signatures. -- **Connections are persistent**: Create once, run many commands. No need to - reconnect between commands. - -## Logging and Debugging - -The MCP server logs to `~/.jumpstarter/logs/mcp-server.log`. Monitor it with: - -```bash -tail -f ~/.jumpstarter/logs/mcp-server.log -``` - -## Writing Python with AI Assistance - -The MCP server is especially useful when writing Python code that interacts with -hardware. While connected to a device, the agent can introspect the live -connection to discover available drivers, methods, and their signatures -- then -use that knowledge to help you write correct code. - -**Ask the agent to explore what's available on your target:** - -> "I'm connected to an ARM board. What drivers and methods are available?" -> -> *The agent calls `jmp_drivers` and `jmp_driver_methods` to inspect the live -> connection and gives you a summary of power, ssh, serial, storage, etc.* - -**Ask for help writing automation scripts:** - -> "Write me a Python script that power-cycles the board, waits for it to boot, -> and grabs the kernel version over SSH." -> -> *The agent inspects the driver methods to discover exact signatures and -> generates a working script using the `env()` helper.* - -**Debug a failing interaction:** - -> "My serial expect is timing out. Can you read the serial output and tell me -> what the board is printing?" -> -> *The agent calls `jmp_run` with `["serial", "pipe"]` and a short timeout -> to capture what the console is outputting right now.* - -**Discover capabilities you didn't know about:** - -> "What can I do with the storage driver on this device?" -> -> *The agent calls `jmp_driver_methods` for the storage driver and shows you -> methods like `flash`, `write_local_file`, `read_to_local_file`, etc. with -> their full signatures and docstrings.* - -**Iterate on code with live hardware feedback:** - -> "Run my test script and tell me if the board boots successfully." -> -> *The agent uses `jmp_get_env` to get the shell environment, executes your -> script, and reports back with the actual device output.* - -See the [jumpstarter-mcp package reference](../../reference/package-apis/mcp.md) -for the full list of tools and their parameters. diff --git a/python/docs/source/getting-started/guides/examples.md b/python/docs/source/getting-started/guides/examples.md deleted file mode 100644 index 80ce7cecf..000000000 --- a/python/docs/source/getting-started/guides/examples.md +++ /dev/null @@ -1,113 +0,0 @@ -# Examples - -This guide provides practical examples for using Jumpstarter in both local and -distributed modes. Each example demonstrates how to accomplish common tasks. - -## Starting and Exiting a Session - -Start a local exporter session: -```console -$ jmp shell --exporter example-local -``` - -Start a distributed exporter session: -```console -$ jmp shell --client hello --selector example.com/board=foo -``` - -When finished, simply exit the shell: -```console -$ exit -``` - -## Interact with the Exporter Shell - -The exporter shell provides access to driver CLI interfaces through the magic -`j` command: - -```console -$ jmp shell # Use appropriate --exporter or --client parameters -$ j -Usage: j [OPTIONS] COMMAND [ARGS]... - - Generic composite device - -Options: - --help Show this message and exit. - -Commands: - power Generic power - storage Generic storage mux -$ j power on -ok -$ j power off -ok -$ exit -``` - -When you run the `j` command in the exporter shell, you're accessing the CLI -interfaces exposed by the drivers configured in your exporter. In this example: - -- `j power` - Would access the power interface from the MockPower driver -- `j storage` - Would access the storage interface from the MockStorageMux - driver - -Each driver can expose different commands through this interface, making it easy -to interact with the mock hardware. The command structure follows `j - `, where available actions depend on the specific driver. - -## Use the Python API in a Shell - -The exporter shell exposes the local exporter via environment variables, -enabling you to run any Python code that interacts with the client/exporter. -This approach works especially well for complex operations or when a driver -doesn't provide a CLI. - -### Using Python with Jumpstarter - -Create a Python file for interacting with your exporter. This example -(`example.py`) demonstrates a complete power cycle workflow: - -```python -import time -from jumpstarter.common.utils import env - -with env() as client: - client.power.on() - client.power.off() -``` - -```console -$ jmp shell # Use appropriate --exporter or --client parameters -$ python ./example.py -$ exit -``` - -This example demonstrates how Python interacts with the exporter: - -1. The `env()` function from `jumpstarter.common.utils` automatically connects - to the exporter configured in your shell environment. - -2. The `with env() as client:` statement creates a client connected to your - local exporter and handles connection setup and cleanup. - -3. `client.power.on()` directly calls the power driver's "on" method—the same - action that `j power on` performs in the CLI. - -4. `client.power.off()` directly calls the power driver's "off" method—the same - action that `j power off` performs in the CLI. - -Using a Python with Jumpstarter allows you to: - - - Create sequences of operations (power on → wait → power off) - - Save and reuse complex workflows - - Add logic, error handling, and conditional operations - - Import other Python libraries (like `time` in this example) - - Build sophisticated automation scripts - -### Running `pytest` in the Shell - -For structured test suites, Jumpstarter provides a `JumpstarterTest` base class -that handles connection management automatically. See the -[Testing with pytest](pytest-usage.md) guide for full details on writing tests, -custom fixtures, markers, and CI integration. diff --git a/python/docs/source/getting-started/guides/examples/index.md b/python/docs/source/getting-started/guides/examples/index.md new file mode 100644 index 000000000..39298be04 --- /dev/null +++ b/python/docs/source/getting-started/guides/examples/index.md @@ -0,0 +1,16 @@ +# Examples + +Practical examples for using Jumpstarter in {term}`local mode`, {term}`direct mode`, and {term}`distributed mode`. + +- [Shell](shell.md): Interacting with {term}`device`s through the + {term}`exporter shell` +- [Scripting](scripting.md): Writing Python scripts that interact with hardware +- [Testing](testing.md): Writing and running hardware tests using pytest + +```{toctree} +:maxdepth: 1 +:hidden: +shell.md +scripting.md +testing.md +``` diff --git a/python/docs/source/getting-started/guides/examples/scripting.md b/python/docs/source/getting-started/guides/examples/scripting.md new file mode 100644 index 000000000..809f61ec0 --- /dev/null +++ b/python/docs/source/getting-started/guides/examples/scripting.md @@ -0,0 +1,57 @@ +# Scripting + +## Use the Python API in a Shell + +The {term}`exporter shell` exposes the local {term}`exporter` via environment variables, +enabling you to run any Python code that interacts with the client/{term}`exporter`. +This approach works especially well for complex operations or when a driver +doesn't provide a CLI. + +### Using Python with Jumpstarter + +Create a Python file for interacting with your {term}`exporter`. This example +(`example.py`) demonstrates a complete power cycle workflow: + +```python +import time +from jumpstarter.common.utils import env + +with env() as client: + client.power.on() + client.power.off() +``` + +```console +$ jmp shell # Use appropriate --exporter or --client parameters +$ python ./example.py +$ exit +``` + +This example demonstrates how Python interacts with the {term}`exporter`: + +1. The `env()` function from `jumpstarter.common.utils` automatically connects + to the {term}`exporter` configured in your shell environment. + +2. The `with env() as client:` statement creates a client connected to your + local {term}`exporter` and handles connection setup and cleanup. + +3. `client.power.on()` directly calls the power driver's "on" method--the same + action that `j power on` performs in the CLI. + +4. `client.power.off()` directly calls the power driver's "off" method--the same + action that `j power off` performs in the CLI. + +Using a Python with Jumpstarter allows you to: + + - Create sequences of operations (power on -> wait -> power off) + - Save and reuse complex workflows + - Add logic, error handling, and conditional operations + - Import other Python libraries (like `time` in this example) + - Build sophisticated automation scripts + +### Running `pytest` in the Shell + +For structured test suites, Jumpstarter provides a `JumpstarterTest` base class +that handles connection management automatically. See the +[Testing](testing.md) guide for full details on writing tests, +custom fixtures, markers, and CI integration. diff --git a/python/docs/source/getting-started/guides/examples/shell.md b/python/docs/source/getting-started/guides/examples/shell.md new file mode 100644 index 000000000..3cd07c8e3 --- /dev/null +++ b/python/docs/source/getting-started/guides/examples/shell.md @@ -0,0 +1,54 @@ +# Shell + +## Starting and Exiting a Session + +Start a {term}`local mode` {term}`exporter` {term}`session`: +```console +$ jmp shell --exporter example-local +``` + +Start a {term}`distributed mode` {term}`exporter` {term}`session`: +```console +$ jmp shell --client hello --selector example.com/board=foo +``` + +When finished, simply exit the shell: +```console +$ exit +``` + +## Interact with the Exporter Shell + +The {term}`exporter shell` provides access to driver CLI interfaces through the magic +{term}`j` command: + +```console +$ jmp shell # Use appropriate --exporter or --client parameters +$ j +Usage: j [OPTIONS] COMMAND [ARGS]... + + Generic composite device + +Options: + --help Show this message and exit. + +Commands: + power Generic power + storage Generic storage mux +$ j power on +ok +$ j power off +ok +$ exit +``` + +When you run the `j` command in the {term}`exporter shell`, you're accessing the CLI +interfaces exposed by the drivers configured in your {term}`exporter`. In this example: + +- `j power` - Would access the power interface from the MockPower driver +- `j storage` - Would access the storage interface from the MockStorageMux + driver + +Each driver can expose different commands through this interface, making it easy +to interact with the mock hardware. The command structure follows `j + `, where available actions depend on the specific driver. diff --git a/python/docs/source/getting-started/guides/pytest-usage.md b/python/docs/source/getting-started/guides/examples/testing.md similarity index 79% rename from python/docs/source/getting-started/guides/pytest-usage.md rename to python/docs/source/getting-started/guides/examples/testing.md index 1451280aa..ff663a9bf 100644 --- a/python/docs/source/getting-started/guides/pytest-usage.md +++ b/python/docs/source/getting-started/guides/examples/testing.md @@ -1,4 +1,4 @@ -# Testing with pytest +# Testing This guide explains how to write and run hardware tests using [pytest](https://docs.pytest.org/) with Jumpstarter. The `jumpstarter-testing` @@ -9,7 +9,7 @@ focus on test logic. Install the following packages in your Python environment: -- `jumpstarter-testing` - pytest integration for Jumpstarter +- `jumpstarter-testing` - `pytest` integration for Jumpstarter - `pytest` - the test framework Install any driver packages your tests require (for example, @@ -19,14 +19,14 @@ guide that use console interaction with `PexpectAdapter` require ## The JumpstarterTest base class -`JumpstarterTest` is a pytest class that provides a `client` fixture scoped to -the test class. It connects to a Jumpstarter exporter in one of two ways: +`JumpstarterTest` is a `pytest` class that provides a `client` fixture scoped to +the test class. It connects to a Jumpstarter {term}`exporter` in one of two ways: 1. **Shell mode**: when the `JUMPSTARTER_HOST` environment variable is set (for - example, inside a `jmp shell` session), it connects to the exporter from that + example, inside a `jmp shell` session), it connects to the {term}`exporter` from that environment. -2. **Lease mode**: when `JUMPSTARTER_HOST` is not set, it loads the default - client configuration and acquires a lease using the `selector` class variable. +2. **{term}`Lease` mode**: when `JUMPSTARTER_HOST` is not set, it loads the default + client config and acquires a {term}`lease` using the `selector` class variable. ```python from jumpstarter_testing.pytest import JumpstarterTest @@ -42,20 +42,19 @@ class TestPowerCycle(JumpstarterTest): client.dutlink.power.off() ``` -The `selector` class variable is a comma-separated list of label selectors that -identify which exporter to lease. It is only used when running outside a shell -session. +The `selector` class variable is a comma-separated list of {term}`label selector`s that +identify which {term}`exporter` to {term}`lease`. It is only used when running outside a shell +{term}`session`. The `client` object exposes driver interfaces as nested attributes. In the example above, `dutlink` is a composite driver that provides child drivers like -`power` and `storage`. The exact attribute names depend on your exporter -configuration. +`power` and `storage`. The exact attribute names depend on your exporter config. ## Running tests ### Inside a shell session -Start an exporter shell first, then run pytest inside it: +Start an {term}`exporter shell` first, then run `pytest` inside it: ```console $ jmp shell --exporter my-exporter @@ -64,24 +63,24 @@ $ exit ``` In this mode, `JumpstarterTest` detects `JUMPSTARTER_HOST` and connects to the -active exporter. The `selector` class variable is ignored. +active {term}`exporter`. The `selector` class variable is ignored. ### With automatic lease acquisition -Run pytest directly without a shell session. `JumpstarterTest` loads the default -client configuration and acquires a lease matching your `selector`: +Run `pytest` directly without a shell {term}`session`. `JumpstarterTest` loads the default +client configuration and acquires a {term}`lease` matching your `selector`: ```console $ pytest test_my_device.py ``` This requires a configured client (see -[Setup Distributed Mode](setup-distributed-mode.md)). +[Setup Distributed Mode](../setup/distributed-mode.md)). ## Writing custom fixtures -Create additional pytest fixtures that build on the `client` fixture provided by -`JumpstarterTest`. This is useful for setting up device state or wrapping driver +Create additional `pytest` fixtures that build on the `client` fixture provided by +`JumpstarterTest`. This is useful for setting up {term}`device` state or wrapping driver interfaces. ```python @@ -115,14 +114,14 @@ The `client` fixture has class scope, so it is shared across all test methods in a class. Custom fixtures can have any scope up to `class`. Serial console interaction uses `PexpectAdapter` from `jumpstarter-driver-network`, -which wraps a driver client into a [pexpect](https://pexpect.readthedocs.io/) +which wraps a driver client class into a [pexpect](https://pexpect.readthedocs.io/) `fdspawn` object. Use `expect()` and `sendline()` instead of `read_until()`. ## Combining with pytest features ### Logging -Use Python's `logging` module to add diagnostic output to tests. Pytest captures +Use Python's `logging` module to add diagnostic output to tests. `pytest` captures log output by default and displays it for failing tests. ```python @@ -152,7 +151,7 @@ class TestDiagnostics(JumpstarterTest): ### Skipping and marking tests -Use standard pytest markers to control test execution: +Use standard `pytest` markers to control test execution: ```python import pytest @@ -207,7 +206,7 @@ class TestWithFirmware(JumpstarterTest): ## CI integration -`JumpstarterTest` works in CI pipelines. Use either shell mode or lease mode +`JumpstarterTest` works in CI pipelines. Use either shell mode or {term}`lease` mode depending on your setup. ### Shell mode in CI @@ -240,7 +239,7 @@ hardware-test: ### Lease mode in CI When tests use `selector` and run outside a shell, configure the client before -running pytest: +running `pytest`: ````{tab} GitHub ```yaml @@ -276,11 +275,11 @@ hardware-test: client configured with `jmp config client use `. **Lease acquisition times out** -: Verify that an exporter matching your `selector` labels is running and - registered with the controller. Check available exporters with +: Verify that an {term}`exporter` matching your `selector` labels is running and + registered with the {term}`controller`. Check available {term}`exporter`s with `jmp get exporters`. **`client` fixture setup fails** : Confirm that `jumpstarter-testing` is installed, and either: `JUMPSTARTER_HOST` is set correctly in shell mode, or a valid default client is configured for - lease mode. + {term}`lease` mode. diff --git a/python/docs/source/getting-started/guides/index.md b/python/docs/source/getting-started/guides/index.md index cc427873a..374e2da63 100644 --- a/python/docs/source/getting-started/guides/index.md +++ b/python/docs/source/getting-started/guides/index.md @@ -1,32 +1,17 @@ # Guides This section provides guidance on how to use Jumpstarter effectively in your -development workflow. The guides cover: - -- [Setup Local Mode](setup-local-mode.md): Running Jumpstarter in local mode for - individual development -- [Setup Direct Mode](setup-direct-mode.md): Connecting a client directly to an - exporter over TCP, without a controller -- [Setup Distributed Mode](setup-distributed-mode.md): Configuring Jumpstarter - for team environments with shared resources -- [Examples](examples.md): Practical examples of Jumpstarter usage in common - scenarios -- [Integration Patterns](integration-patterns.md): Integrate Jumpstarter into - your existing workflows and systems -- [AI Agent Integration](ai-agent-integration.md): Use AI coding agents - (Cursor, Claude Code, Claude Desktop) to interact with hardware via MCP -- [Testing with pytest](pytest-usage.md): Write and run hardware tests using - pytest with Jumpstarter +development workflow. +- [Setup](setup/index.md): Step-by-step instructions for each operation mode +- [Examples](examples/index.md): Practical examples for common tasks +- [Integration Patterns](integration-patterns/index.md): Incorporate Jumpstarter + into your existing workflows and systems ```{toctree} :maxdepth: 1 :hidden: -setup-local-mode.md -setup-direct-mode.md -setup-distributed-mode.md -examples.md -integration-patterns.md -ai-agent-integration.md -pytest-usage.md -``` \ No newline at end of file +setup/index.md +examples/index.md +integration-patterns/index.md +``` diff --git a/python/docs/source/getting-started/guides/integration-patterns.md b/python/docs/source/getting-started/guides/integration-patterns.md deleted file mode 100644 index bd83d6d6d..000000000 --- a/python/docs/source/getting-started/guides/integration-patterns.md +++ /dev/null @@ -1,417 +0,0 @@ -# Integration Patterns - -This document outlines common integration patterns for Jumpstarter, helping you -incorporate it into your development and testing workflows. - -Jumpstarter integrates with various tools and platforms across the hardware -development lifecycle: - -- **Infrastructure**: Kubernetes, Prometheus, Grafana -- **Developer Environments**: IDE, scripts, GitHub Actions, GitLab CI, Tekton -- **Testing Frameworks**: pytest, unittest, Robot Framework - -## Infrastructure - -### Continuous Integration with System Testing - -```{mermaid} -:config: {"theme":"base","themeVariables":{"primaryColor":"#f8f8f8","primaryTextColor":"#000","primaryBorderColor":"#e5e5e5","lineColor":"#3d94ff","secondaryColor":"#f8f8f8","tertiaryColor":"#fff"}} -flowchart TB - subgraph "Version Control" - GitRepo["Git Repository"] - Actions["GitHub/GitLab CI"] - end - - subgraph "Jumpstarter Infrastructure" - Controller["Controller"] - Exporters["Exporter"] - DUTs["Device Under Test"] - end - - GitRepo -- "Code changes" --> Actions - Actions -- "Request access" --> Controller - Controller -- "Assign lease" --> Actions - Controller -- "Connect to" --> Exporters - Exporters -- "Control" --> DUTs - Actions -- "Update status" --> GitRepo -``` - -This architecture integrates Jumpstarter with CI/CD pipelines to enable -automated testing on real systems: - -1. Code changes trigger the CI pipeline -2. The pipeline runs tests that use Jumpstarter to access systems -3. Jumpstarter's controller manages device access and leases -4. Test results are reported back to the CI system - -**CI Configuration Examples:** - -````{tab} GitHub -```yaml -# .github/workflows/hardware-test.yml -jobs: - hardware-test: - runs-on: self-hosted - steps: - - uses: actions/checkout@v3 - - name: Request hardware lease - run: | - jmp config client use ci-client - jmp create lease --selector project=myproject --wait 300 - - name: Run tests - run: pytest tests/hardware_tests/ - - name: Release hardware lease - if: always() - run: jmp delete lease -``` -```` - -````{tab} GitLab -```yaml -# .gitlab-ci.yml -hardware-test: - tags: - - self-hosted - script: - - jmp config client use ci-client - - jmp create lease --selector project=myproject --wait 300 - - pytest tests/hardware_tests/ - after_script: - - jmp delete lease -``` -```` - -### Self-Hosted CI Runner with Attached System - -```{mermaid} -:config: {"theme":"base","themeVariables":{"primaryColor":"#f8f8f8","primaryTextColor":"#000","primaryBorderColor":"#e5e5e5","lineColor":"#3d94ff","secondaryColor":"#f8f8f8","tertiaryColor":"#fff"}} -flowchart TB - subgraph "Version Control" - GitRepo["Git Repository"] - Actions["GitHub/GitLab CI"] - end - - subgraph "Runner" - Runner1["Self-Hosted Runner"] - JmpLocal["Local Mode"] - Devices["Device Under Test"] - end - - GitRepo -- "Code changes" --> Actions - Actions -- "Dispatch job" --> Runner1 - - Runner1 -- "Execute tests" --> JmpLocal - JmpLocal -- "Control" --> Devices - - Runner1 -- "Report results" --> Actions - Actions -- "Update status" --> GitRepo -``` - -This architecture leverages a self-hosted runner with directly attached system: - -1. The self-hosted runner has physical devices connected directly to it -2. Jumpstarter runs in local mode on the runner, controlling the attached system -3. Code changes trigger CI jobs which are dispatched to the runner -4. Tests execute on the runner using Jumpstarter to interface with the system -5. Results are reported back to the CI system - -This approach works best when: - -- You need to permanently connect systems to a specific test machine -- You want to integrate system testing into existing CI/CD workflows without - additional infrastructure -- You need a simple setup for initial system-in-the-loop testing - -**CI Configuration Examples:** - -````{tab} GitHub -```yaml -# .github/workflows/self-hosted-hw-test.yml -jobs: - hardware-test: - runs-on: self-hosted-hw-attached - steps: - - uses: actions/checkout@v3 - - name: Run Jumpstarter in local mode - run: jmp local start --config=./.jumpstarter/local-config.yaml - - name: Run tests - run: pytest tests/hardware_tests/ - - name: Cleanup - if: always() - run: jmp local stop -``` -```` - -````{tab} GitLab -```yaml -# .gitlab-ci.yml -hardware-test: - tags: - - hw-attached - script: - - jmp local start --config=./.jumpstarter/local-config.yaml - - pytest tests/hardware_tests/ - after_script: - - jmp local stop -``` -```` - -### Cost Management and Chargeback - -Organizations can implement usage-based billing for teams through a cost -management layer. - -```{mermaid} -:config: {"theme":"base","themeVariables":{"primaryColor":"#f8f8f8","primaryTextColor":"#000","primaryBorderColor":"#e5e5e5","lineColor":"#3d94ff","secondaryColor":"#f8f8f8","tertiaryColor":"#fff"}} -flowchart LR - subgraph "Kubernetes" - Controller["Controller"] - - subgraph "Telemetry" - Prometheus["Prometheus"] - Grafana["Grafana"] - AlertManager["AlertManager"] - end - - subgraph "Cost Management" - UsageTracker["Usage Tracker"] - OpenCost["OpenCost"] - Accounting["Chargeback System"] - end - end - - subgraph "Lab" - Rack1["Exporter 1"] - Rack2["Exporter 2"] - end - - subgraph "Users" - Team["Team"] - end - - Team -- "Request access" --> Controller - Controller -- "Assign lease" --> Team - Controller -- "Record lease\nmetadata" --> Prometheus - - Controller -- "Connect to" --> Rack1 - Controller -- "Connect to" --> Rack2 - - Rack1 -- "Report usage\nmetrics" --> Prometheus - Rack2 -- "Report usage\nmetrics" --> Prometheus - - Prometheus -- "Store\nmetrics" --> Grafana - Prometheus -- "Threshold\nalerts" --> AlertManager - Prometheus -- "Usage\nmetrics" --> UsageTracker - - UsageTracker -- "Monthly billing\nreport" --> Team - - UsageTracker -- "Team resource\nusage" --> OpenCost - OpenCost -- "Cost\nallocation" --> Accounting -``` - -This architecture implements a cost chargeback model for infrastructure -resources: - -1. Prometheus collects and stores all resource utilization metrics -2. Teams request resources through the controller, which records team - identifiers with each lease -3. System resources export detailed utilization metrics to Prometheus: - - Resource uptime and availability - - Utilization metrics (CPU, memory, I/O) - - Team attribution via metadata - -## AI Agent Integration - -Jumpstarter provides an MCP (Model Context Protocol) server that enables AI -coding agents to interact with hardware using natural language. This works with -Cursor, Claude Code, Claude Desktop, and any MCP-compatible client. - -See the [AI Agent Integration](ai-agent-integration.md) guide for full setup -instructions and examples. - -## Developer Environments - -### Traditional Developer Workflow - -```{mermaid} -:config: {"theme":"base","themeVariables":{"primaryColor":"#f8f8f8","primaryTextColor":"#000","primaryBorderColor":"#e5e5e5","lineColor":"#3d94ff","secondaryColor":"#f8f8f8","tertiaryColor":"#fff"}} -flowchart TB - subgraph "Workstation" - TestCode["Test Code"] - end - - subgraph "Local Environment" - LocalExporter["Local Exporter"] - DeviceOnDesk["Device Under Test"] - end - - subgraph "Lab" - Controller["Controller"] - RemoteExporters["Exporter"] - LabDevices["Device Under Test"] - end - - TestCode --> LocalExporter - LocalExporter --> DeviceOnDesk - - TestCode -- "Request access" --> Controller - Controller -- "Assign lease" --> TestCode - Controller -- "Connect to" --> RemoteExporters - RemoteExporters --> LabDevices -``` - -This architecture supports developers working with both local systems and shared -lab resources: - -1. Developers write and test code in their IDE -2. For quick tests, they use the test code to access a system on their desk -3. For more complex tests, they connect to remote lab systems through the - controller -4. The same test code works in both environments - -See [Setup Local Mode](setup-local-mode.md) for more information on configuring -your local environment. - -### Cloud Native Developer Workflow - -```{mermaid} -:config: {"theme":"base","themeVariables":{"primaryColor":"#f8f8f8","primaryTextColor":"#000","primaryBorderColor":"#e5e5e5","lineColor":"#3d94ff","secondaryColor":"#f8f8f8","tertiaryColor":"#fff"}} -flowchart TB - subgraph "Web Browser" - Dev["Developer"] - end - - subgraph "Kubernetes Cluster" - subgraph "Eclipse Che" - Workspace["Developer Workspace"] - TestCode["Test Code"] - PortFwd["Port Forwarding"] - end - - Controller["Controller"] - end - - subgraph "Local Environment" - LocalExporter["Local Exporter"] - DeviceOnDesk["Device Under Test"] - end - - subgraph "Lab" - RemoteExporters["Exporter"] - LabDevices["Device Under Test"] - end - - Dev -- "Access via browser" --> Workspace - Workspace -- "Contains" --> TestCode - - TestCode -- "Local system access" --> PortFwd - PortFwd -- "Forward connection" --> LocalExporter - LocalExporter -- "Control" --> DeviceOnDesk - - TestCode -- "Request access" --> Controller - Controller -- "Assign lease" --> TestCode - Controller -- "Connect to" --> RemoteExporters - RemoteExporters -- "Control" --> LabDevices -``` - -This architecture provides a cloud-native development experience while -maintaining flexibility to work with both local and remote systems: - -1. Developers access a containerized development environment through a web - browser using Eclipse Che -2. The development workspace contains all necessary tools, dependencies, and - test code -3. For quick iterations with locally connected systems: - - Port forwarding enables the cloud workspace to communicate with systems - connected to the developer's machine - - The local Jumpstarter exporter manages the device directly -4. For access to shared lab resources: - - The same test code can request access to remote devices through the - controller - - The controller manages leases and routes connections through the standard - infrastructure - -Key benefits of this approach: - -- **Consistent Development Environment**: Standardized, reproducible workspaces - for all team members -- **Flexibility**: Seamless transition between local and remote system testing -- **Collaboration**: Web-based IDE enables real-time collaboration and knowledge - sharing -- **Scalability**: Easy onboarding of new team members with zero local - configuration -- **System Flexibility**: Enables a hybrid approach where developers can test - locally first, then validate on shared lab systems - -This workflow eliminates the distinction between local and cloud development -while providing the best of both worlds for system testing. - -See [Setup Distributed Mode](setup-distributed-mode.md) for more details on -configuring your distributed environment. - -## Testing Frameworks - -### pytest Integration - -Jumpstarter integrates with pytest through the `jumpstarter-testing` package: - -```python -from jumpstarter_testing.pytest import JumpstarterTest - -class TestMyDevice(JumpstarterTest): - # Optional: specify which exporter to use based on labels - exporter_selector = "vendor=acme,model=widget-v2" - - def test_power_cycle(self): - # Access the device driver through the provided client - self.client.power.on() - assert self.client.serial.read_until("boot complete") is not None - self.client.power.off() -``` - -### Robot Framework Integration - -For teams using Robot Framework, Jumpstarter drivers can be exposed as keywords: - -```robotframework -*** Settings *** -Library JumpstarterLibrary - -*** Test Cases *** -Device Boot Test - Connect To Exporter selector=vendor=acme,model=widget-v2 - Power On - ${output}= Read Serial Until boot complete - Should Not Be Empty ${output} - Power Off -``` - -## Recommended Practices - -### Labeling Strategy - -Develop a consistent labeling strategy for your exporters to make device -selection straightforward: - -- **System Properties**: `arch=arm64`, `cpu=cortex-a53` -- **Organization**: `team=platform`, `project=widget` -- **Capabilities**: `has-video=true`, `has-can=true` -- **Environment**: `env=dev`, `env=production` - -### Resource Management - -Implement these practices to ensure efficient use of shared systems: - -- Set appropriate lease timeouts to prevent orphaned resources -- Use CI systems' concurrency controls to manage test parallelism -- Implement monitoring and alerting for device availability -- Create "pools" of identical devices to improve scalability - -### Security Considerations - -When deploying Jumpstarter in a multi-user environment: - -- Use role-based access control to limit which users can access which devices -- Restrict driver access to prevent untrusted code execution -- Isolate the Jumpstarter network from production systems -- Rotate JWT tokens regularly for enhanced security \ No newline at end of file diff --git a/python/docs/source/getting-started/guides/integration-patterns/agentic.md b/python/docs/source/getting-started/guides/integration-patterns/agentic.md new file mode 100644 index 000000000..becd09911 --- /dev/null +++ b/python/docs/source/getting-started/guides/integration-patterns/agentic.md @@ -0,0 +1,187 @@ +# Agentic + +Jumpstarter exposes hardware control as structured {term}`MCP` tools, enabling +AI coding agents to interact with {term}`device`s using natural language from +IDEs and AI assistants. + +```{mermaid} +flowchart TB + subgraph "Developer" + IDE["IDE / AI Assistant"] + end + + subgraph "MCP Server" + JmpMCP["jmp mcp serve"] + end + + subgraph "Jumpstarter Infrastructure" + DUTs["Device Under Test"] + end + + IDE -- "MCP Protocol" --> JmpMCP + JmpMCP -- "Lease & connect" --> DUTs +``` + +## Prerequisites + +- Jumpstarter CLI ({term}`jmp`) installed and configured with a client identity +- An {term}`MCP`-compatible AI tool (Cursor, Claude Code, Claude Desktop, or any + {term}`MCP` client) +- The `jumpstarter-mcp` package (included in a full install) + +## Setup + +### Cursor + +Add to your Cursor {term}`MCP` configuration (`~/.cursor/mcp.json`): + +```json +{ + "mcpServers": { + "jumpstarter": { + "command": "jmp", + "args": ["mcp", "serve"] + } + } +} +``` + +### Claude Code + +```console +claude mcp add jumpstarter -- jmp mcp serve +``` + +### Claude Desktop + +Add to your Claude Desktop configuration: + +- **macOS**: `~/Library/Application Support/Claude/claude_desktop_config.json` +- **Windows**: `%APPDATA%\Claude\claude_desktop_config.json` + +```json +{ + "mcpServers": { + "jumpstarter": { + "command": "jmp", + "args": ["mcp", "serve"] + } + } +} +``` + +### Other Clients + +Any {term}`MCP`-compatible client can use the Jumpstarter server. It +communicates over stdio: + +```console +jmp mcp serve +``` + +For the full list of available tools and their parameters, see the +[MCP package reference](../../../reference/package-apis/mcp.md). + +## Usage Examples + +### Interactive Hardware Exploration + +> **You**: What devices are available on the cluster? +> +> *Agent calls `jmp_list_exporters` and shows a summary of available hardware.* +> +> **You**: Get me a QEMU target and power it on. +> +> *Agent calls `jmp_create_lease`, `jmp_connect`, then `jmp_run` with +> `["power", "on"]`.* +> +> **You**: Check what OS is running via SSH. +> +> *Agent calls `jmp_run` with `["ssh", "--", "cat", "/etc/os-release"]`.* + +### Claude Code Session + +``` +$ claude + +> /mcp + +Connected MCP servers: + - jumpstarter (jmp mcp serve) + +> Can you list the hardware available on the jumpstarter cluster? + +I'll check what devices are available... + +[Uses jmp_list_exporters] + +Here's what's available: + - qemu-test-01 (online, no active lease) + - arm-board-01 (online, leased by alice) + - arm-board-02 (online, no active lease) + +> Lease arm-board-02 and check if it boots to Linux + +[Uses jmp_create_lease, jmp_connect, jmp_run to power on and SSH] + +The board is running Fedora 41 (aarch64). Here's the full `uname -a` output... +``` + +### Cursor Agent Mode + +In Cursor's Composer (Agent mode), the Jumpstarter tools are available +alongside your code: + +1. Ask the agent to flash a new firmware image to a board +2. Have it verify the board boots successfully via serial console +3. Run your test suite against the live hardware +4. Iterate on code fixes with the agent retesting on real hardware + +## Typical Workflow + +```{mermaid} +sequenceDiagram + participant User + participant Agent as AI Agent + participant MCP as MCP Server + participant Ctrl as Controller + + User->>Agent: "Get me an ARM board" + Agent->>MCP: jmp_create_lease(selector="arch=arm64") + MCP->>Ctrl: Request lease + Ctrl-->>MCP: Lease ID + MCP-->>Agent: Lease created + + Agent->>MCP: jmp_connect(lease_id) + MCP-->>Agent: Connected + + Agent->>MCP: jmp_explore() + MCP-->>Agent: Available commands: power, ssh, serial, storage + + User->>Agent: "Power it on and check the OS" + Agent->>MCP: jmp_run(["power", "on"]) + Agent->>MCP: jmp_run(["ssh", "--", "cat", "/etc/os-release"]) + MCP-->>Agent: OS info + + User->>Agent: "Done, release it" + Agent->>MCP: jmp_disconnect() + Agent->>MCP: jmp_delete_lease() +``` + +## Tips + +- **Use `jmp_explore` first** - each {term}`device` type exposes different + commands +- **Set `timeout_seconds` for streaming commands** - commands like `serial pipe` + block indefinitely +- **Use `jmp_drivers` for Python access** - inspect the driver tree to discover + methods and signatures +- **Connections are persistent** - create once, run many commands + +## Logging + +The {term}`MCP` server logs to `~/.jumpstarter/logs/mcp-server.log`: + +```console +tail -f ~/.jumpstarter/logs/mcp-server.log +``` diff --git a/python/docs/source/getting-started/guides/integration-patterns/cicd.md b/python/docs/source/getting-started/guides/integration-patterns/cicd.md new file mode 100644 index 000000000..1638e4347 --- /dev/null +++ b/python/docs/source/getting-started/guides/integration-patterns/cicd.md @@ -0,0 +1,135 @@ +# CI/CD + +## Continuous Integration with System Testing + +```{mermaid} +flowchart TB + subgraph "Version Control" + GitRepo["Git Repository"] + Actions["GitHub/GitLab CI"] + end + + subgraph "Jumpstarter Infrastructure" + Controller["Controller"] + Exporters["Exporter"] + DUTs["Device Under Test"] + end + + GitRepo -- "Code changes" --> Actions + Actions -- "Request access" --> Controller + Controller -- "Assign lease" --> Actions + Controller -- "Connect to" --> Exporters + Exporters -- "Control" --> DUTs + Actions -- "Update status" --> GitRepo +``` + +This architecture integrates Jumpstarter with CI/CD pipelines to enable +automated testing on real systems: + +1. Code changes trigger the CI pipeline +2. The pipeline runs tests that use Jumpstarter to access systems +3. Jumpstarter's {term}`controller` manages {term}`device` access and {term}`lease`s +4. Test results are reported back to the CI system + +**CI Configuration Examples:** + +````{tab} GitHub +```yaml +# .github/workflows/hardware-test.yml +jobs: + hardware-test: + runs-on: self-hosted + steps: + - uses: actions/checkout@v3 + - name: Request hardware lease + run: | + jmp config client use ci-client + LEASE_ID=$(jmp create lease --selector project=myproject --wait 300 -o name) + - name: Run tests + run: jmp shell --lease ${LEASE_ID} pytest tests/hardware_tests/ + - name: Release hardware lease + if: always() + run: jmp delete lease ${LEASE_ID} +``` +```` + +````{tab} GitLab +```yaml +# .gitlab-ci.yml +hardware-test: + tags: + - self-hosted + script: + - jmp config client use ci-client + - LEASE_ID=$(jmp create lease --selector project=myproject --wait 300 -o name) + - jmp shell --lease ${LEASE_ID} pytest tests/hardware_tests/ + after_script: + - jmp delete lease ${LEASE_ID} +``` +```` + +## Self-Hosted CI Runner with Attached System + +```{mermaid} +flowchart TB + subgraph "Version Control" + GitRepo["Git Repository"] + Actions["GitHub/GitLab CI"] + end + + subgraph "Runner" + Runner1["Self-Hosted Runner"] + JmpLocal["Local Mode"] + Devices["Device Under Test"] + end + + GitRepo -- "Code changes" --> Actions + Actions -- "Dispatch job" --> Runner1 + + Runner1 -- "Execute tests" --> JmpLocal + JmpLocal -- "Control" --> Devices + + Runner1 -- "Report results" --> Actions + Actions -- "Update status" --> GitRepo +``` + +This architecture leverages a self-hosted runner with directly attached system: + +1. The self-hosted runner has physical {term}`device`s connected directly to it +2. Jumpstarter runs in {term}`local mode` on the runner, controlling the attached system +3. Code changes trigger CI jobs which are dispatched to the runner +4. Tests execute on the runner using Jumpstarter to interface with the system +5. Results are reported back to the CI system + +This approach works best when: + +- You need to permanently connect systems to a specific test machine +- You want to integrate system testing into existing CI/CD workflows without + additional infrastructure +- You need a simple setup for initial system-in-the-loop testing + +**CI Configuration Examples:** + +````{tab} GitHub +```yaml +# .github/workflows/self-hosted-hw-test.yml +jobs: + hardware-test: + runs-on: self-hosted-hw-attached + steps: + - uses: actions/checkout@v3 + - name: Run Jumpstarter in local mode + run: jmp shell --exporter-config=./.jumpstarter/local-config.yaml pytest test/hardware/tests/ +``` +```` + +````{tab} GitLab +```yaml +# .gitlab-ci.yml +hardware-test: + tags: + - hw-attached + script: + - jmp shell --exporter-config=./.jumpstarter/local-config.yaml pytest tests/hardware_tests/ +``` +```` diff --git a/python/docs/source/getting-started/guides/integration-patterns/development.md b/python/docs/source/getting-started/guides/integration-patterns/development.md new file mode 100644 index 000000000..4b0dd80ce --- /dev/null +++ b/python/docs/source/getting-started/guides/integration-patterns/development.md @@ -0,0 +1,117 @@ +# Development + +## Traditional Developer Workflow + +```{mermaid} +flowchart TB + subgraph "Workstation" + TestCode["Test Code"] + end + + subgraph "Local Environment" + LocalExporter["Local Exporter"] + DeviceOnDesk["Device Under Test"] + end + + subgraph "Lab" + Controller["Controller"] + RemoteExporters["Exporter"] + LabDevices["Device Under Test"] + end + + TestCode --> LocalExporter + LocalExporter --> DeviceOnDesk + + TestCode -- "Request access" --> Controller + Controller -- "Assign lease" --> TestCode + Controller -- "Connect to" --> RemoteExporters + RemoteExporters --> LabDevices +``` + +This architecture supports developers working with both local systems and shared +lab resources: + +1. Developers write and test code in their IDE +2. For quick tests, they use the test code to access a system on their desk +3. For more complex tests, they connect to remote lab systems through the + {term}`controller` +4. The same test code works in both environments + +See [Setup Local Mode](../setup/local-mode.md) for more information on configuring +your local environment. + +## Cloud Native Developer Workflow + +```{mermaid} +flowchart TB + subgraph "Web Browser" + Dev["Developer"] + end + + subgraph "Kubernetes Cluster" + subgraph "Eclipse Che" + Workspace["Developer Workspace"] + TestCode["Test Code"] + PortFwd["Port Forwarding"] + end + + Controller["Controller"] + end + + subgraph "Local Environment" + LocalExporter["Local Exporter"] + DeviceOnDesk["Device Under Test"] + end + + subgraph "Lab" + RemoteExporters["Exporter"] + LabDevices["Device Under Test"] + end + + Dev -- "Access via browser" --> Workspace + Workspace -- "Contains" --> TestCode + + TestCode -- "Local system access" --> PortFwd + PortFwd -- "Forward connection" --> LocalExporter + LocalExporter -- "Control" --> DeviceOnDesk + + TestCode -- "Request access" --> Controller + Controller -- "Assign lease" --> TestCode + Controller -- "Connect to" --> RemoteExporters + RemoteExporters -- "Control" --> LabDevices +``` + +This architecture provides a cloud-native development experience while +maintaining flexibility to work with both local and remote systems: + +1. Developers access a containerized development environment through a web + browser using Eclipse Che +2. The development workspace contains all necessary tools, dependencies, and + test code +3. For quick iterations with locally connected systems: + - Port forwarding enables the cloud workspace to communicate with systems + connected to the developer's machine + - The local Jumpstarter {term}`exporter` manages the {term}`device` directly +4. For access to shared lab resources: + - The same test code can request access to remote {term}`device`s through the + {term}`controller` + - The {term}`controller` manages {term}`lease`s and routes connections through the standard + infrastructure + +Key benefits of this approach: + +- **Consistent Development Environment**: Standardized, reproducible workspaces + for all team members +- **Flexibility**: Seamless transition between local and remote system testing +- **Collaboration**: Web-based IDE enables real-time collaboration and knowledge + sharing +- **Scalability**: Easy onboarding of new team members with zero local + configuration +- **System Flexibility**: Enables a hybrid approach where developers can test + locally first, then validate on shared lab systems + +This workflow eliminates the distinction between local and cloud development +while providing the best of both worlds for system testing. + +See [Setup Distributed Mode](../setup/distributed-mode.md) for more details on +configuring your distributed environment. diff --git a/python/docs/source/getting-started/guides/integration-patterns/index.md b/python/docs/source/getting-started/guides/integration-patterns/index.md new file mode 100644 index 000000000..c7ca8ed92 --- /dev/null +++ b/python/docs/source/getting-started/guides/integration-patterns/index.md @@ -0,0 +1,19 @@ +# Integration Patterns + +Common patterns for incorporating Jumpstarter into your development and testing +workflows. + +- [CI/CD](cicd.md): Pipeline configs for GitHub Actions, GitLab CI, and other + CI/CD systems +- [Development](development.md): Local and cloud-native development setups +- [Agentic](agentic.md): AI agent interaction with hardware via {term}`MCP` +- [Best Practices](practices.md): Labeling, security, and resource management + +```{toctree} +:maxdepth: 1 +:hidden: +cicd.md +development.md +agentic.md +practices.md +``` diff --git a/python/docs/source/getting-started/guides/integration-patterns/practices.md b/python/docs/source/getting-started/guides/integration-patterns/practices.md new file mode 100644 index 000000000..39ae563b3 --- /dev/null +++ b/python/docs/source/getting-started/guides/integration-patterns/practices.md @@ -0,0 +1,29 @@ +# Best Practices + +## Labeling Strategy + +Develop a consistent labeling strategy for your exporters to make device +selection straightforward: + +- **System Properties**: `arch=arm64`, `cpu=cortex-a53` +- **Organization**: `team=platform`, `project=widget` +- **Capabilities**: `has-video=true`, `has-can=true` +- **Environment**: `env=dev`, `env=production` + +## Resource Management + +Implement these practices to ensure efficient use of shared systems: + +- Set appropriate {term}`lease` timeouts to prevent orphaned resources +- Use CI systems' concurrency controls to manage test parallelism +- Implement monitoring and alerting for device availability +- Create "pools" of identical devices to improve scalability + +## Security Considerations + +When deploying Jumpstarter in a multi-user environment: + +- Use role-based access control to limit which users can access which devices +- Restrict driver access to prevent untrusted code execution +- Isolate the Jumpstarter network from production systems +- Rotate JWT tokens regularly for enhanced security diff --git a/python/docs/source/getting-started/guides/setup-direct-mode.md b/python/docs/source/getting-started/guides/setup/direct-mode.md similarity index 55% rename from python/docs/source/getting-started/guides/setup-direct-mode.md rename to python/docs/source/getting-started/guides/setup/direct-mode.md index 7d5fcabbc..1b6e2edd2 100644 --- a/python/docs/source/getting-started/guides/setup-direct-mode.md +++ b/python/docs/source/getting-started/guides/setup/direct-mode.md @@ -1,23 +1,23 @@ -# Setup Direct Mode +# Direct Mode -This guide shows you how to run a Jumpstarter exporter that clients connect to -directly over TCP — no controller or Kubernetes cluster required. +This guide shows you how to run a Jumpstarter {term}`exporter` that clients connect to +directly over TCP - no {term}`controller` or Kubernetes cluster required. -Direct mode is useful when you want to expose hardware on one machine to clients -on another, without setting up a controller. +{term}`Direct mode ` is useful when you want to expose hardware on one machine to clients +on another, without setting up a {term}`controller`. ```{note} -Direct mode skips the controller's lease management. Only one client should +{term}`Direct mode` skips the {term}`controller`'s {term}`lease` management. Only one client should connect at a time. For shared, multi-user environments use -[distributed mode](setup-distributed-mode.md) instead. +[distributed mode](distributed-mode.md) instead. ``` ## Instructions ### Create an Exporter Configuration -Unlike distributed mode, you don't need `endpoint` or `token` fields — there -is no controller to register with. +Unlike {term}`distributed mode`, you don't need `endpoint` or `token` fields - there +is no {term}`controller` to register with. Create `example-direct.yaml`: @@ -42,13 +42,13 @@ hooks: timeout: 30 ``` -The `hooks` section is optional. `beforeLease` runs once when the exporter -starts (before any client connects), and `afterLease` runs on shutdown. Hook -scripts can use `j` commands to interact with the drivers. +The {term}`hook`s section is optional. `beforeLease` hook runs once when the {term}`exporter` +starts (before any client connects), and `afterLease` hook runs on shutdown. {term}`Hook` +scripts can use {term}`j` commands to interact with the drivers. ### Start the Exporter -Run the exporter and tell it to listen on a TCP port with `--tls-grpc-listener`: +Run the {term}`exporter` and tell it to listen on a TCP port with `--tls-grpc-listener`: ```console $ jmp run --exporter-config example-direct.yaml \ @@ -74,14 +74,14 @@ $ jmp run --exporter-config example-direct.yaml \ $ jmp shell --tls-grpc :19090 --tls-grpc-insecure ``` -If the exporter requires a passphrase: +If the {term}`exporter` requires a passphrase: ```console $ jmp shell --tls-grpc :19090 --tls-grpc-insecure --passphrase my-secret ``` -Replace `` with the exporter machine's IP address or hostname. Once -connected, interact with the exporter using `j` commands: +Replace `` with the {term}`exporter` machine's IP address or hostname. Once +connected, interact with the {term}`exporter` using `j` commands: ```console $ j power on diff --git a/python/docs/source/getting-started/guides/setup-distributed-mode.md b/python/docs/source/getting-started/guides/setup/distributed-mode.md similarity index 59% rename from python/docs/source/getting-started/guides/setup-distributed-mode.md rename to python/docs/source/getting-started/guides/setup/distributed-mode.md index 037e30466..d2e3c1287 100644 --- a/python/docs/source/getting-started/guides/setup-distributed-mode.md +++ b/python/docs/source/getting-started/guides/setup/distributed-mode.md @@ -1,19 +1,19 @@ -# Setup Distributed Mode +# Distributed Mode -This guide walks you through the process of creating an exporter using the -controller service, configuring drivers, and running the exporter. +This guide walks you through the process of creating an {term}`exporter` using the +{term}`controller` {term}`service`, configuring drivers, and running the exporter. ```{warning} The jumpstarter-controller endpoints are secured by TLS. However, in release 0.7.x, the certificates are self-signed and rotated on every restart. This means the client will not be able to verify the server certificate. To bypass this, you should use the -`--insecure-tls` flag when creating clients and exporters. +`--insecure-tls` flag when creating clients and {term}`exporter`s. Alternatively, you can configure the ingress/route in reencrypt mode with your own key and certificate. ``` ## Prerequisites -Install [the following packages](../installation/packages.md) in your Python +Install [the following packages](../../installation/packages.md) in your Python environment: - `jumpstarter-cli` - The core Jumpstarter CLI @@ -21,27 +21,27 @@ environment: - `jumpstarter-driver-power` - The base power driver These driver packages include mock implementations, enabling you to test the -connection between an exporter and client without physical hardware. +connection between an {term}`exporter` and client without physical hardware. -You need the [service](../../introduction/service.md) running in a Kubernetes +You need the [service](../../../introduction/service.md) running in a Kubernetes cluster with admin access. For installation instructions, refer to the -[installation guide](../installation/service/index.md). +[installation guide](../../installation/service/index.md). ## Instructions ### Create an Exporter Configuration Create an exporter using the controller service API. The `jmp admin` CLI -provides commands to interact with the controller directly. +provides commands to interact with the {term}`controller` directly. -Run this command to create an exporter named `example-distributed` and save the +Run this command to create an {term}`exporter` named `example-distributed` and save the configuration locally: ```console $ jmp admin create exporter example-distributed --label foo=bar --save --insecure-tls ``` -After creating the exporter, find the new configuration file at +After creating the exporter, find the new exporter config file at `/etc/jumpstarter/exporters/example-distributed.yaml`. Edit the configuration using your default text editor with: @@ -69,17 +69,17 @@ export: ### Run an Exporter -Start the exporter locally using the `jmp` CLI tool: +Start the {term}`exporter` locally using the {term}`jmp` CLI tool: ```console $ jmp run --exporter example-distributed ``` -The exporter runs until you terminate the process with or close the shell. +The {term}`exporter` runs until you terminate the process with or close the shell. ### Create a Client -Create a client to connect to your new exporter using the `jmp admin` CLI: +Create a client to connect to your new {term}`exporter` using the `jmp admin` CLI: The following command creates a client named "hello", enables unsafe drivers for development purposes, and saves the configuration locally in @@ -91,10 +91,10 @@ $ jmp admin create client hello --save --unsafe --insecure-tls ### Spawn an Exporter Shell -Interact with your distributed exporter using the "client shell" functionality -in the `jmp` CLI. When you spawn a shell, the client attempts to acquire a lease -on an exporter. Once the lease is acquired, you can interact with the exporter -through your shell session. +Interact with your distributed {term}`exporter` using the {term}`exporter shell` functionality +in the {term}`jmp` CLI. When you spawn a shell, the client attempts to acquire a {term}`lease` +on an {term}`exporter`. Once the {term}`lease` is acquired, you can interact with the {term}`exporter` +through your shell {term}`session`. ```console $ jmp shell --client hello --selector example.com/board=foo @@ -102,7 +102,7 @@ $ jmp shell --client hello --selector example.com/board=foo ### Exiting the Exporter Shell -To terminate the local exporter, simply exit the shell: +To terminate the local {term}`exporter`, simply exit the shell: ```console $ exit @@ -110,6 +110,6 @@ $ exit ## Next Steps -Once you have your exporter shell running, you can start using Jumpstarter +Once you have your {term}`exporter shell` running, you can start using Jumpstarter commands to interact with your hardware. To learn more about common workflow -patterns and implementation examples, see [Examples](./examples.md). +patterns and implementation examples, see [Examples](../examples/index.md). diff --git a/python/docs/source/getting-started/guides/setup/index.md b/python/docs/source/getting-started/guides/setup/index.md new file mode 100644 index 000000000..aaf19be90 --- /dev/null +++ b/python/docs/source/getting-started/guides/setup/index.md @@ -0,0 +1,18 @@ +# Setup + +Step-by-step instructions for each operation mode. + +- [Local Mode](local-mode.md): Running Jumpstarter with devices connected + directly to your machine +- [Direct Mode](direct-mode.md): Connecting a client directly to an {term}`exporter` + over TCP, without a {term}`controller` +- [Distributed Mode](distributed-mode.md): Configuring Jumpstarter for team + environments with shared resources + +```{toctree} +:maxdepth: 1 +:hidden: +local-mode.md +direct-mode.md +distributed-mode.md +``` diff --git a/python/docs/source/getting-started/guides/setup-local-mode.md b/python/docs/source/getting-started/guides/setup/local-mode.md similarity index 59% rename from python/docs/source/getting-started/guides/setup-local-mode.md rename to python/docs/source/getting-started/guides/setup/local-mode.md index d1b5131da..f261dfd5f 100644 --- a/python/docs/source/getting-started/guides/setup-local-mode.md +++ b/python/docs/source/getting-started/guides/setup/local-mode.md @@ -1,11 +1,11 @@ -# Setup Local Mode +# Local Mode This guide shows you how to use Jumpstarter with a client and exporter running -on the same host. +on the same {term}`host`. ## Prerequisites -Install [the following packages](../installation/packages.md) in your Python +Install [the following packages](../../installation/packages.md) in your Python environment: - `jumpstarter-cli` - The Jumpstarter CLI for interacting with exporters @@ -19,10 +19,10 @@ connection between an exporter and client without physical hardware. ### Create an Exporter Configuration -Create an exporter configuration named `example-local` to define the -capabilities of your local test exporter. This configuration mirrors a regular +Create an exporter config named `example-local` to define the +capabilities of your local test {term}`exporter`. This configuration mirrors a regular exporter config but without the `endpoint` and `token` fields since you -don't need to connect to the controller service. +don't need to connect to the {term}`controller` {term}`service`. Create `example-local.yaml` in `/etc/jumpstarter/exporters` with this content: @@ -41,9 +41,9 @@ export: ### Spawn an Exporter Shell -Interact with your local exporter using the "exporter shell" functionality in -the `jmp` CLI. When you spawn a shell, Jumpstarter runs a local exporter -instance in the background for the duration of your shell session. +Interact with your local exporter using the {term}`exporter shell` functionality in +the {term}`jmp` CLI. When you spawn a shell, Jumpstarter runs a local {term}`exporter` +instance in the background for the duration of your shell {term}`session`. ```console $ jmp shell --exporter example-local @@ -51,7 +51,7 @@ $ jmp shell --exporter example-local ### Exiting the Exporter Shell -To terminate the local exporter, simply exit the shell: +To terminate the local {term}`exporter`, simply exit the shell: ```console $ exit @@ -59,6 +59,6 @@ $ exit ## Next Steps -Once you have your exporter shell running, you can start using Jumpstarter +Once you have your {term}`exporter shell` running, you can start using Jumpstarter commands to interact with your hardware. To learn more about common workflow -patterns and implementation examples, see [Examples](./examples.md). +patterns and implementation examples, see [Examples](../examples/index.md). diff --git a/python/docs/source/getting-started/index.md b/python/docs/source/getting-started/index.md index 6b08c26f6..9d3ac922a 100644 --- a/python/docs/source/getting-started/index.md +++ b/python/docs/source/getting-started/index.md @@ -10,8 +10,9 @@ environment. The guides cover: - [Guides](guides/index.md): Running your first tests and integrating with your development workflow -These guides support both local-mode for individual development and -distributed-mode for team environments with shared hardware resources. +These guides support all three operation modes: {term}`local mode` for individual +development, {term}`direct mode` for single-user remote access, and +{term}`distributed mode` for team environments with shared hardware resources. ```{toctree} :maxdepth: 1 diff --git a/python/docs/source/getting-started/installation/index.md b/python/docs/source/getting-started/installation/index.md index 2a7756e50..19126b964 100644 --- a/python/docs/source/getting-started/installation/index.md +++ b/python/docs/source/getting-started/installation/index.md @@ -4,7 +4,7 @@ This section provides guidance on installing Jumpstarter components in your environment. The guides cover: - [Packages](packages.md): Installing Jumpstarter software packages -- [Service](service/index.md): Setting up Jumpstarter as a Kubernetes service +- [Service](service/index.md): Setting up Jumpstarter as a Kubernetes {term}`service` ```{toctree} :maxdepth: 1 diff --git a/python/docs/source/getting-started/installation/packages.md b/python/docs/source/getting-started/installation/packages.md index b222a85f0..5a14aa3e3 100644 --- a/python/docs/source/getting-started/installation/packages.md +++ b/python/docs/source/getting-started/installation/packages.md @@ -4,10 +4,10 @@ Jumpstarter includes the following installable Python packages: -- `jumpstarter`: Core package for exporter interaction and service hosting +- `jumpstarter`: Core package for {term}`exporter` interaction and {term}`service` hosting - `jumpstarter-cli`: CLI components metapackage including admin and user interfaces -- `jumpstarter-cli-admin`: Admin CLI for controller management and lease control +- `jumpstarter-cli-admin`: Admin CLI for {term}`controller` management and {term}`lease` control - `jumpstarter-driver-*`: Drivers for device connectivity - `jumpstarter-imagehash`: Image checking library for video inputs - `jumpstarter-testing`: Tools for Jumpstarter-powered pytest integration @@ -29,7 +29,7 @@ curl -fsSL https://raw.githubusercontent.com/jumpstarter-dev/jumpstarter/main/py Or with explicit source specification (main branch example) ```{code-block} console -curl -fsSL https://raw.githubusercontent.com/jumpstarter-dev/jumpstarter/main/python/install.sh | bash -s -- -s main +curl -fsSL https://raw.githubusercontent.com/jumpstarter-dev/jumpstarter/main/python/install.sh | bash -s - -s main ``` ##### Local Installation @@ -59,12 +59,12 @@ After installation, the following structure is created: ##### Activating the Environment -Activate for current session, adjust ~/.local/jumpstarter if you picked a custom install directory. +Activate for current session, adjust `~/.local/jumpstarter` if you picked a custom install directory. ```{code-block} console source ~/.local/jumpstarter/set ``` -Or add to your shell profile for permanent activation, adjust ~/.local/jumpstarter if you picked a custom install directory. +Or add to your shell profile for permanent activation, adjust `~/.local/jumpstarter` if you picked a custom install directory. ```{code-block} console echo 'source ~/.local/jumpstarter/set' >> ~/.bashrc ``` @@ -185,11 +185,11 @@ Run the following commands to clone the repository and create a virtual environm # Clone the git repository $ git clone https://github.com/jumpstarter-dev/jumpstarter.git -# Open Jumpstarter -jumpstarter$ cd jumpstarter +# Open the Python workspace +$ cd jumpstarter/python # Install Python venv and sync packages with uv -jumpstarter$ make sync +$ make sync # Create local config directories for Jumpstarter $ mkdir -p "${HOME}/.config/jumpstarter/" @@ -237,12 +237,12 @@ $ docker run --rm -it \ ```` ```{tip} -If you need Kubernetes access (e.g. for `jmp admin` commands), also mount your kubeconfig: +If you need Kubernetes access (e.g. for jmp admin commands), also mount your kubeconfig: `-v "${HOME}/.kube/config:/root/.kube/config":z` ``` To interact with Jumpstarter without local Python package installation, -create an alias to run the `jmp` client in a container. +create an alias to run the {term}`jmp` client in a container. We recommend adding this alias to your shell profile (`~/.bashrc` or `~/.zshrc`) for persistent use: @@ -266,13 +266,13 @@ $ alias jmp='docker run --rm -it -w /home \ ``` ```` -If you've configured a `jmp` alias you can undefine it with: +If you've configured a {term}`jmp` alias you can undefine it with: ```console $ unalias jmp ``` -When you need hardware access for running the `jmp` command or following the +When you need hardware access for running the {term}`jmp` command or following the [local-only workflow](../../introduction/index.md#local-mode), configure the container with device access, host networking, and privileged mode. This typically requires `root` privileges: diff --git a/python/docs/source/getting-started/installation/service/development.md b/python/docs/source/getting-started/installation/service/development.md new file mode 100644 index 000000000..034168b23 --- /dev/null +++ b/python/docs/source/getting-started/installation/service/development.md @@ -0,0 +1,151 @@ +# Development + +Install Jumpstarter on local Kubernetes clusters using kind or minikube. Ideal +for learning about the {term}`service` quickly or for validating Jumpstarter +drivers in CI/CD pipelines. For production deployments, see +[Production](production.md). + +## Prerequisites + +- Docker or Podman installed +- `kubectl` installed and configured +- Administrator access to your cluster (required for CRD installation) + +## Install + +The `jmp admin` CLI can create a local cluster and install Jumpstarter in +a single command: + +```{code-block} console +$ jmp admin create cluster +``` + +```{warning} +If automatic IP detection fails, check with `jmp admin ip` and use `--ip` to +set your address manually. +``` + +```{tip} +By default, Jumpstarter uses kind if available. Use `--minikube` to force +minikube instead. +``` + +````{tab} kind +```{code-block} console +$ jmp admin create cluster --kind +``` + +Options: `--force-recreate`, `--skip-install`, `--extra-certs `, +`--kind-extra-args`, custom cluster name as first argument. +```` + +````{tab} minikube +```{code-block} console +$ jmp admin create cluster --minikube +``` + +Options: `--force-recreate`, `--skip-install`, `--extra-certs `, +`--minikube-extra-args`, custom cluster name as first argument. +```` + +### Install on an Existing Cluster + +```{warning} +Jumpstarter requires specific NodePort configurations. It is recommended to +create a new cluster or use the automatic creation above. +``` + +````{tab} kind +```{code-block} console +$ jmp admin install --kind +``` +```` + +````{tab} minikube +```{code-block} console +$ jmp admin install --minikube +``` +```` + +## Verify + +```{code-block} console +$ kubectl get pods -n jumpstarter-lab --watch +``` + +## Configuration + +### Manual Cluster Setup + +For more control, create the cluster yourself before installing: + +````{tab} kind +Create a kind cluster config that enables NodePorts. Save as `kind_config.yaml`: + +```{code-block} yaml +kind: Cluster +apiVersion: kind.x-k8s.io/v1alpha4 +kubeadmConfigPatches: +- | + kind: ClusterConfiguration + apiServer: + extraArgs: + "service-node-port-range": "3000-32767" +- | + kind: InitConfiguration + nodeRegistration: + kubeletExtraArgs: + node-labels: "ingress-ready=true" +nodes: +- role: control-plane + extraPortMappings: + - containerPort: 80 + hostPort: 5080 + protocol: TCP + - containerPort: 30010 + hostPort: 8082 + protocol: TCP + - containerPort: 30011 + hostPort: 8083 + protocol: TCP + - containerPort: 443 + hostPort: 5443 + protocol: TCP +``` + +```{code-block} console +$ kind create cluster --config kind_config.yaml +``` +```` + +````{tab} minikube +```{code-block} console +$ minikube start --extra-config=apiserver.service-node-port-range=8000-9000 +``` +```` + +Then follow the [Production](production.md) guide using a `baseDomain` +appropriate for your local environment (for example, `nip.io` based hostnames). + +## Uninstall + +```{code-block} console +$ jmp admin uninstall +``` + +To delete the local cluster completely: + +````{tab} kind +```{code-block} console +$ jmp admin delete cluster --kind +``` +```` + +````{tab} minikube +```{code-block} console +$ jmp admin delete cluster --minikube +``` +```` + +For complete documentation of all `jmp admin` options, see the +[MAN pages](../../../reference/man-pages/jmp.md). diff --git a/python/docs/source/getting-started/installation/service/index.md b/python/docs/source/getting-started/installation/service/index.md index f07dcc2d6..4cb3bc6bb 100644 --- a/python/docs/source/getting-started/installation/service/index.md +++ b/python/docs/source/getting-started/installation/service/index.md @@ -1,23 +1,19 @@ # Service -This section explains how to install and configure the Jumpstarter service in your Kubernetes cluster. The service enables centralized management of your Jumpstarter environment. +This section explains how to install the Jumpstarter {term}`service`. -## Getting Started - -For most users, we recommend starting with a **local installation** to get familiar with Jumpstarter before moving to production deployments. +- [Development](development.md): Set up a local cluster with + `jmp admin` using kind or minikube +- [Standalone](standalone.md): Lightweight deployment with MicroShift and a + bootable container image +- [Production](production.md): Deploy on a Kubernetes or OpenShift + cluster with the Jumpstarter {term}`operator` ```{toctree} :maxdepth: 2 +:hidden: -service-local.md -service-production.md -service-operator.md +development.md +standalone.md +production.md ``` - -## Quick Start - -**New to Jumpstarter?** Start with the [Local Installation](service-local.md) guide to get up and running quickly on your development machine. - -**Ready for production?** See the [Production Deployment](service-production.md) guide for Kubernetes and OpenShift clusters with proper security, monitoring, and ingress configurations. - -**Installing with the Kubernetes operator?** Use [Install with Operator](service-operator.md) for `Jumpstarter` CR examples using Kubernetes Ingress or OpenShift Routes. diff --git a/python/docs/source/getting-started/installation/service/production.md b/python/docs/source/getting-started/installation/service/production.md new file mode 100644 index 000000000..4a1bae718 --- /dev/null +++ b/python/docs/source/getting-started/installation/service/production.md @@ -0,0 +1,257 @@ +# Production + +For production deployments, install Jumpstarter on a Kubernetes or OpenShift +cluster using the Jumpstarter {term}`operator`. + +## Prerequisites + +- A Kubernetes or OpenShift cluster +- `kubectl` (or `oc`) configured for your cluster +- Cluster-admin permissions (required to install CRDs and {term}`operator` RBAC) +- A DNS domain for Jumpstarter {term}`service` endpoints (for example, + `jumpstarter.example.com`) +- An ingress controller on Kubernetes, or Routes on OpenShift + +```{note} +`spec.baseDomain` creates these {term}`service` hostnames with +`jumpstarter.example.com`: +- `grpc.jumpstarter.example.com` +- `router.jumpstarter.example.com` +- `login.jumpstarter.example.com` +``` + +## Install + +### Install the Operator + +Apply the {term}`operator` installer from a release asset: + +```{code-block} console +$ kubectl apply -f https://github.com/jumpstarter-dev/jumpstarter/releases/download/v0.8.1/operator-installer.yaml +$ kubectl wait --namespace jumpstarter-operator-system \ + --for=condition=available deployment/jumpstarter-operator-controller-manager \ + --timeout=120s +``` + +Alternatively, install via OLM or OperatorHub: + +```{tab} Kubernetes +Install from [OperatorHub](https://operatorhub.io/operator/jumpstarter-operator). +Requires OLM to be installed in your cluster. +``` + +```{tab} OpenShift +1. Go to **Operators -> OperatorHub** in the web console. +2. Search for **Jumpstarter Operator** and install it. +3. Verify: `oc get csv -n openshift-operators | grep jumpstarter` + +Or via CLI subscription: + + apiVersion: operators.coreos.com/v1alpha1 + kind: Subscription + metadata: + name: jumpstarter-operator + namespace: openshift-operators + spec: + channel: alpha + name: jumpstarter-operator + source: community-operators + sourceNamespace: openshift-marketplace + installPlanApproval: Automatic +``` + +### Create a Namespace + +```{code-block} console +$ kubectl create namespace jumpstarter-lab +``` + +### Create a Jumpstarter Custom Resource + +The {term}`operator` reconciles the `Jumpstarter` CR and creates Deployments, +Services, and networking resources for {term}`controller`/{term}`router`/login +endpoints. + +````{tab} Kubernetes +```{code-block} yaml +apiVersion: operator.jumpstarter.dev/v1alpha1 +kind: Jumpstarter +metadata: + name: jumpstarter + namespace: jumpstarter-lab +spec: + baseDomain: jumpstarter.example.com + certManager: + enabled: true + controller: + image: quay.io/jumpstarter-dev/jumpstarter-controller:0.8.1-rc.2 + imagePullPolicy: IfNotPresent + replicas: 1 + grpc: + endpoints: + - address: grpc.jumpstarter.example.com:443 + ingress: + enabled: true + class: nginx + login: + endpoints: + - address: login.jumpstarter.example.com:443 + ingress: + enabled: true + class: nginx + routers: + image: quay.io/jumpstarter-dev/jumpstarter-controller:0.8.1-rc.2 + imagePullPolicy: IfNotPresent + replicas: 1 + grpc: + endpoints: + - address: router.jumpstarter.example.com:443 + ingress: + enabled: true + class: nginx +``` +```` + +````{tab} OpenShift +```{code-block} yaml +apiVersion: operator.jumpstarter.dev/v1alpha1 +kind: Jumpstarter +metadata: + name: jumpstarter + namespace: jumpstarter-lab +spec: + baseDomain: jumpstarter.example.com + certManager: + enabled: true + controller: + image: quay.io/jumpstarter-dev/jumpstarter-controller:0.8.1-rc.2 + imagePullPolicy: IfNotPresent + replicas: 1 + grpc: + endpoints: + - address: grpc.jumpstarter.example.com:443 + route: + enabled: true + login: + endpoints: + - address: login.jumpstarter.example.com:443 + route: + enabled: true + routers: + image: quay.io/jumpstarter-dev/jumpstarter-controller:0.8.1-rc.2 + imagePullPolicy: IfNotPresent + replicas: 1 + grpc: + endpoints: + - address: router.jumpstarter.example.com:443 + route: + enabled: true +``` +```` + +```{code-block} console +$ kubectl apply -f jumpstarter.yaml +``` + +## Verify + +````{tab} Kubernetes +```{code-block} console +$ kubectl get jumpstarter -n jumpstarter-lab +$ kubectl get deploy,svc,ingress -n jumpstarter-lab +``` +```` + +````{tab} OpenShift +```{code-block} console +$ kubectl get jumpstarter -n jumpstarter-lab +$ kubectl get deploy,svc,route -n jumpstarter-lab +``` + +Ensure DNS is configured so route hostnames resolve correctly. +```` + +## Configuration + +### TLS and gRPC + +Jumpstarter uses {term}`gRPC` for communication, which requires HTTP/2 support. +The {term}`operator` configures TLS passthrough at the ingress or route for +{term}`gRPC` endpoints and edge TLS termination for login endpoints. + +```{note} +When using ingress-nginx, enable +[`--enable-ssl-passthrough`](https://kubernetes.github.io/ingress-nginx/user-guide/cli-arguments/) +on the ingress controller. +``` + +### OAuth and OIDC + +Configure through `spec.authentication.jwt` in the `Jumpstarter` CR. The +{term}`operator` applies this to {term}`controller` runtime settings but does +not install your identity provider. See +[Authentication](../../configuration/authentication.md) for examples. + +### cert-manager + +Set `spec.certManager.enabled: true` for {term}`operator`-managed certificates. + +````{tab} Self-signed +```{code-block} yaml +spec: + certManager: + enabled: true + server: + selfSigned: + enabled: true +``` + +Creates: `-selfsigned-issuer`, `-ca`, `-ca-issuer`, +`-controller-tls`, `-router--tls`. +```` + +````{tab} External issuer +```{code-block} yaml +spec: + certManager: + enabled: true + server: + issuerRef: + name: my-cluster-issuer + kind: ClusterIssuer +``` +```` + +````{tab} ACME +```{code-block} yaml +spec: + controller: + login: + endpoints: + - address: login.jumpstarter.example.com:443 + ingress: + enabled: true + class: nginx + annotations: + cert-manager.io/cluster-issuer: letsencrypt-prod +``` +```` + +### GitOps + +Use the {term}`operator` installer and manage your `Jumpstarter` CR +declaratively in GitOps flows. + +### Operator Behavior + +- If `spec.baseDomain` is empty on OpenShift, the {term}`operator` auto-detects + the cluster domain. +- If no endpoint service type is enabled, the {term}`operator` auto-selects: + route, then ingress, then clusterIP. +- {term}`Controller` and {term}`router` auth secrets persist across CR + deletion/recreation. +- {term}`Router` replicas are one Deployment per replica; `$(replica)` + placeholders are substituted per replica. + +For the full `Jumpstarter` CRD field reference, see the +[CRDs](../../../reference/crds/index.md). diff --git a/python/docs/source/getting-started/installation/service/service-local.md b/python/docs/source/getting-started/installation/service/service-local.md deleted file mode 100644 index 1da85e2bd..000000000 --- a/python/docs/source/getting-started/installation/service/service-local.md +++ /dev/null @@ -1,219 +0,0 @@ -# Local Installation - -For local development and testing, you can install Jumpstarter on local Kubernetes clusters using tools like kind or minikube. This is ideal for learning about the distributed service quickly or for creating CI/CD pipelines to validate your own Jumpstarter drivers. - -## Prerequisites - -Before installing locally, ensure you have: - -- Docker or Podman installed (for kind) -- `kubectl` installed and configured to access your cluster -- Administrator access to your cluster (required for CRD installation) - -## Install with Jumpstarter CLI - -The Jumpstarter CLI provides convenient commands for local demo/test cluster management and Jumpstarter installation: - -- `jmp admin create cluster` - Creates a local cluster and installs Jumpstarter (recommended for getting started quickly) -- `jmp admin delete cluster` - Deletes a local cluster completely -- `jmp admin get clusters` - Get local clusters from a Kubeconfig -- `jmp admin install` - Installs Jumpstarter on an existing cluster -- `jmp admin uninstall` - Removes Jumpstarter from a cluster (but keeps the cluster) - -```{warning} -Sometimes the automatic IP address detection for will not work correctly, to check if Jumpstarter can determine your IP address, run `jmp admin ip`. If the IP address cannot be determined, use the `--ip` argument to manually set your IP address. -``` - -### Create a Local Cluster and Install Jumpstarter - -If you want to test Jumpstarter locally with more control over the setup, you can create a local cluster using tools such as [minikube](https://minikube.sigs.k8s.io/docs/start/) and [kind](https://kind.sigs.k8s.io/docs/user/quick-start/). - -[**kind**](https://kind.sigs.k8s.io/docs/user/quick-start/) (Kubernetes in Docker) is a tool for running local Kubernetes clusters using Docker or Podman containerized "nodes". It's lightweight and fast to start, making it excellent for CI/CD pipelines and quick local testing. - -[**minikube**](https://minikube.sigs.k8s.io/docs/start/) runs local Kubernetes clusters using VMs or container "nodes". It works across several platforms and supports different hypervisors, making it ideal for local development and testing. Minikube works better if you don't have a local Docker/Podman installation. - -The admin CLI can automatically create a local cluster and install Jumpstarter with a single command: - -By default, Jumpstarter will try to detect which local cluster tools are installed: - -```{tip} -By default, Jumpstarter will use `kind` if available, use the `--minikube` argument to force Jumpstarter to use minikube instead. -``` - -```{code-block} console -$ jmp admin create cluster -``` - -However, you can also explicitly specify a local cluster tool: - -````{tab} kind -```{code-block} console -$ jmp admin create cluster --kind -``` - -Additional options for cluster creation: - -- Custom cluster name: Specify as the first argument (default: `jumpstarter-lab`) -- `--kind `: Path to the kind binary to use for cluster management -- `--force-recreate`: Force recreate the cluster if it already exists (destroys all data) -- `--kind-extra-args`: Pass additional arguments to kind cluster creation -- `--skip-install`: Create the cluster without installing Jumpstarter -- `--extra-certs `: Path to custom CA certificate bundle file to inject into the cluster -```` - -````{tab} minikube -```{code-block} console -$ jmp admin create cluster --minikube -``` - -Additional options for cluster creation: - -- Custom cluster name: Specify as the first argument (default: `jumpstarter-lab`) -- `--minikube `: Path to the minikube binary to use for cluster management -- `--force-recreate`: Force recreate the cluster if it already exists (destroys all data) -- `--minikube-extra-args`: Pass additional arguments to minikube cluster creation -- `--skip-install`: Create the cluster without installing Jumpstarter -- `--extra-certs `: Path to custom CA certificate bundle file to inject into the cluster -```` - -To set a custom cluster name: - -````{tab} kind -```{code-block} console -$ jmp admin create cluster my-jumpstarter-cluster --kind -``` -```` - -````{tab} minikube -```{code-block} console -$ jmp admin create cluster my-jumpstarter-cluster --minikube -``` -```` - -### Install Jumpstarter in an Existing Local Cluster - -```{warning} -Jumpstarter requires specific `NodePort` configurations, it is recommended to create a new cluster for Jumpstarter or use the automatic creation above. -``` - -If you already have a local cluster, install Jumpstarter with default options for your local cluster tool: - -````{tab} kind -```{code-block} console -$ jmp admin install --kind -``` -```` - -````{tab} minikube -```{code-block} console -$ jmp admin install --minikube -``` -```` - -### Uninstall Jumpstarter - -Uninstall Jumpstarter from the cluster with the CLI: - -```{code-block} console -$ jmp admin uninstall -``` - -To delete the local cluster completely, use the cluster delete command: - -````{tab} kind -```{code-block} console -$ jmp admin delete cluster --kind -``` -```` - -````{tab} minikube -```{code-block} console -$ jmp admin delete cluster --minikube -``` -```` - -To delete a cluster with a custom name: - -````{tab} kind -```{code-block} console -$ jmp admin delete cluster my-jumpstarter-cluster --kind -``` -```` - -````{tab} minikube -```{code-block} console -$ jmp admin delete cluster my-jumpstarter-cluster --minikube -``` -```` - -For complete documentation of the `jmp admin create cluster`, `jmp admin delete cluster`, `jmp admin get clusters`, and `jmp admin install` commands and all available options, see the [MAN pages](../../../reference/man-pages/jmp.md). - -## Manual Local Cluster Install - -If you want to customize the local cluster further, you can create the cluster yourself. - -### Create a Local Cluster - -````{tab} kind -#### Create a kind cluster - -First, create a kind cluster config that enables nodeports to host the Services. -Save this as `kind_config.yaml`: - -```{code-block} yaml -kind: Cluster -apiVersion: kind.x-k8s.io/v1alpha4 -kubeadmConfigPatches: -- | - kind: ClusterConfiguration - apiServer: - extraArgs: - "service-node-port-range": "3000-32767" -- | - kind: InitConfiguration - nodeRegistration: - kubeletExtraArgs: - node-labels: "ingress-ready=true" -nodes: -- role: control-plane - extraPortMappings: - - containerPort: 80 - hostPort: 5080 - protocol: TCP - - containerPort: 30010 - hostPort: 8082 - protocol: TCP - - containerPort: 30011 - hostPort: 8083 - protocol: TCP - - containerPort: 443 - hostPort: 5443 - protocol: TCP -``` - -Next, create a kind cluster using the config you created: - -```{code-block} console -$ kind create cluster --config kind_config.yaml -``` -```` - -````{tab} minikube -#### Create a minikube cluster - -Expand the default NodePort range to include the Jumpstarter ports: - -```{code-block} console -$ minikube start --extra-config=apiserver.service-node-port-range=8000-9000 -``` -```` - -### Install Local Jumpstarter with Operator - -For manual installation after creating the local cluster, follow [Install with Operator](service-operator.md). Use a `baseDomain` and endpoint addresses appropriate for your local environment (for example, `nip.io` based hostnames), then apply your `Jumpstarter` CR. - -To check the status of the installation, run: - -```{code-block} console -$ kubectl get pods -n jumpstarter-lab --watch -``` \ No newline at end of file diff --git a/python/docs/source/getting-started/installation/service/service-operator.md b/python/docs/source/getting-started/installation/service/service-operator.md deleted file mode 100644 index f96d76544..000000000 --- a/python/docs/source/getting-started/installation/service/service-operator.md +++ /dev/null @@ -1,452 +0,0 @@ -# Install with Operator - -This guide covers installing Jumpstarter with the Kubernetes operator, using: - -- **Ingress** on vanilla Kubernetes -- **Route** on OpenShift - -It mirrors how `make deploy METHOD=operator` deploys the operator and creates a `Jumpstarter` custom resource (CR), but uses production-friendly manifests and release artifacts. - -## Prerequisites - -- A Kubernetes, OpenShift, or OKD cluster -- `kubectl` (or `oc`) configured for your cluster -- Cluster-admin permissions (required to install CRDs and operator RBAC) -- A DNS domain for Jumpstarter service endpoints (for example, `jumpstarter.example.com`) -- An ingress controller on Kubernetes, or Routes on OpenShift/OKD - -```{note} -This page focuses on operator installation and core CR configuration. It does not cover full setup of external components such as Dex/other OIDC providers or cert-manager installation. -``` - -## Install the operator - -````{tab} Kubernetes (OLM installed) -If your Kubernetes cluster already has OLM, install the operator from OperatorHub and then continue with the `Jumpstarter` custom resource in this guide. - -OperatorHub package page: - -- [Jumpstarter Operator on OperatorHub](https://operatorhub.io/operator/jumpstarter-operator) - -```{note} -On vanilla Kubernetes, this OperatorHub path assumes OLM is already installed and configured in your cluster. -``` -```` - -````{tab} OpenShift / OKD (OperatorHub recommended) -1. Log in to the OpenShift/OKD web console with cluster-admin permissions. -2. Go to **Operators -> OperatorHub**. -3. Search for **Jumpstarter Operator** and install it. -4. Wait until the installed operator status is `Succeeded`. - -Verify from CLI: - -```{code-block} console -$ oc get csv -n openshift-operators | grep jumpstarter -``` -```` - -````{tab} OpenShift / OKD (CLI OLM subscription) -Create a `Subscription` (example: `subscription.yaml`): - -```yaml -apiVersion: operators.coreos.com/v1alpha1 -kind: Subscription -metadata: - name: jumpstarter-operator - namespace: openshift-operators -spec: - channel: alpha - name: jumpstarter-operator - source: community-operators - sourceNamespace: openshift-marketplace - installPlanApproval: Automatic -``` - -Apply and verify: - -```{code-block} console -$ oc apply -f subscription.yaml -$ oc get csv -n openshift-operators | grep jumpstarter -``` -```` - -````{tab} Manual installer YAML (any cluster) -Apply the operator installer from a release asset: - -```{code-block} console -$ kubectl apply -f https://github.com/jumpstarter-dev/jumpstarter/releases/download//operator-installer.yaml -``` - -For example: - -```{code-block} console -$ kubectl apply -f https://github.com/jumpstarter-dev/jumpstarter/releases/download/v0.8.1/operator-installer.yaml -``` - -Wait for the operator deployment: - -```{code-block} console -$ kubectl wait --namespace jumpstarter-operator-system \ - --for=condition=available deployment/jumpstarter-operator-controller-manager \ - --timeout=120s -``` -```` - -## Create a namespace for Jumpstarter - -```{code-block} console -$ kubectl create namespace jumpstarter-lab -``` - -## Create a `Jumpstarter` custom resource - -The operator reconciles the `Jumpstarter` CR and creates Deployments, Services, and networking resources (Ingresses or Routes) for controller/router/login endpoints. - -````{tab} Kubernetes (Ingress) -```yaml -apiVersion: operator.jumpstarter.dev/v1alpha1 -kind: Jumpstarter -metadata: - name: jumpstarter - namespace: jumpstarter-lab -spec: - baseDomain: jumpstarter.example.com - certManager: - enabled: true - controller: - image: quay.io/jumpstarter-dev/jumpstarter-controller:0.8.1-rc.2 - imagePullPolicy: IfNotPresent - replicas: 1 - grpc: - endpoints: - - address: grpc.jumpstarter.example.com:443 - ingress: - enabled: true - class: nginx - login: - endpoints: - - address: login.jumpstarter.example.com:443 - ingress: - enabled: true - class: nginx - routers: - image: quay.io/jumpstarter-dev/jumpstarter-controller:0.8.1-rc.2 - imagePullPolicy: IfNotPresent - replicas: 1 - grpc: - endpoints: - - address: router.jumpstarter.example.com:443 - ingress: - enabled: true - class: nginx -``` -```` - -````{tab} OpenShift / OKD (Route) -```yaml -apiVersion: operator.jumpstarter.dev/v1alpha1 -kind: Jumpstarter -metadata: - name: jumpstarter - namespace: jumpstarter-lab -spec: - baseDomain: jumpstarter.example.com - certManager: - enabled: true - controller: - image: quay.io/jumpstarter-dev/jumpstarter-controller:0.8.1-rc.2 - imagePullPolicy: IfNotPresent - replicas: 1 - grpc: - endpoints: - - address: grpc.jumpstarter.example.com:443 - route: - enabled: true - login: - endpoints: - - address: login.jumpstarter.example.com:443 - route: - enabled: true - routers: - image: quay.io/jumpstarter-dev/jumpstarter-controller:0.8.1-rc.2 - imagePullPolicy: IfNotPresent - replicas: 1 - grpc: - endpoints: - - address: router.jumpstarter.example.com:443 - route: - enabled: true -``` -```` - -Save as `jumpstarter.yaml`, then apply: - -```{code-block} console -$ kubectl apply -f jumpstarter.yaml -``` - -## Verify deployment - -Check CR status and workloads: - -```{code-block} console -$ kubectl get jumpstarter -n jumpstarter-lab -$ kubectl get deploy,svc,ingress -n jumpstarter-lab # Kubernetes -$ kubectl get deploy,svc,route -n jumpstarter-lab # OpenShift/OKD -``` - -```{note} -`route` is only available on OpenShift/OKD. On vanilla Kubernetes, use `ingress`. -``` - -```{note} -For OpenShift/OKD, set `spec.baseDomain` to a domain that resolves to your route hosts (for example, `jumpstarter.example.com`). Ensure DNS is configured so these route hostnames resolve correctly. -``` - -## OAuth and cert-manager integration notes - -- **OAuth / OIDC integration**: Configure this through `spec.authentication.jwt` in the `Jumpstarter` CR (issuer URL, audiences, and claim mappings). The operator applies this configuration to controller runtime settings, but does not install or configure your identity provider. -- **cert-manager integration**: Set `spec.certManager.enabled: true` to let the operator manage server certificates. You can use operator-managed self-signed certificates or reference an existing `Issuer`/`ClusterIssuer` with `spec.certManager.server.issuerRef`. Installing and configuring cert-manager itself remains an external prerequisite. - -For detailed authentication examples and field-level reference, see [Authentication](../../configuration/authentication.md). - -## cert-manager configuration examples - -These examples are aligned with scenarios covered by the operator e2e tests in `controller/deploy/operator/test/e2e/e2e_test.go`. - -### Self-signed cert-manager mode - -```yaml -apiVersion: operator.jumpstarter.dev/v1alpha1 -kind: Jumpstarter -metadata: - name: jumpstarter - namespace: jumpstarter-lab -spec: - baseDomain: jumpstarter.example.com - certManager: - enabled: true - server: - selfSigned: - enabled: true - controller: - grpc: - endpoints: - - address: grpc.jumpstarter.example.com:443 - ingress: - enabled: true - class: nginx - routers: - grpc: - endpoints: - - address: router.jumpstarter.example.com:443 - ingress: - enabled: true - class: nginx -``` - -The operator creates and uses: - -- `-selfsigned-issuer` -- `-ca` -- `-ca-issuer` -- `-controller-tls` -- `-router--tls` - -### External issuer reference (ClusterIssuer) - -```yaml -apiVersion: operator.jumpstarter.dev/v1alpha1 -kind: Jumpstarter -metadata: - name: jumpstarter - namespace: jumpstarter-lab -spec: - baseDomain: jumpstarter.example.com - certManager: - enabled: true - server: - issuerRef: - name: my-cluster-issuer - kind: ClusterIssuer - controller: - grpc: - endpoints: - - address: grpc.jumpstarter.example.com:443 - route: - enabled: true - routers: - grpc: - endpoints: - - address: router.jumpstarter.example.com:443 - route: - enabled: true -``` - -In this mode, the operator issues certificates with your referenced issuer and does not create the self-signed issuer chain. - -### Login endpoint with cert-manager default TLS secret naming - -When cert-manager is enabled and `controller.login.tls.secretName` is not set, the generated login Ingress uses the default TLS secret name `login-tls`. - -For Ingress-based login endpoints, you can use `controller.login.endpoints[].ingress.annotations` to integrate with ACME issuers (for example Let's Encrypt) managed by cert-manager. - -```yaml -apiVersion: operator.jumpstarter.dev/v1alpha1 -kind: Jumpstarter -metadata: - name: jumpstarter - namespace: jumpstarter-lab -spec: - baseDomain: jumpstarter.example.com - certManager: - enabled: true - server: - selfSigned: - enabled: true - controller: - login: - endpoints: - - address: login.jumpstarter.example.com:443 - ingress: - enabled: true - class: nginx - annotations: - cert-manager.io/cluster-issuer: letsencrypt-prod -``` - -### Login endpoint with explicit TLS secret - -If you want a specific login certificate secret, set `controller.login.tls.secretName`: - -```yaml -apiVersion: operator.jumpstarter.dev/v1alpha1 -kind: Jumpstarter -metadata: - name: jumpstarter - namespace: jumpstarter-lab -spec: - baseDomain: jumpstarter.example.com - certManager: - enabled: true - server: - selfSigned: - enabled: true - controller: - login: - tls: - secretName: login-custom-tls - endpoints: - - address: login.jumpstarter.example.com:443 - ingress: - enabled: true - class: nginx -``` - -## Operator behavior insights - -From the current operator implementation in `controller/deploy/operator`, these behaviors are useful to know when authoring manifests: - -- If `spec.baseDomain` is empty and the cluster exposes OpenShift Route APIs, the operator auto-detects the cluster domain and sets `spec.baseDomain` to `jumpstarter..`. -- If an endpoint has no enabled service type, the operator auto-selects one in this order: `route` (if available), then `ingress`, then `clusterIP`. -- gRPC endpoints (`controller.grpc`, `routers.grpc`) use TLS passthrough semantics in generated Ingress/Route resources; login endpoints use edge TLS termination. -- Controller and router auth secrets are created once with fixed names (`jumpstarter-controller-secret`, `jumpstarter-router-secret`) and are intentionally not owner-referenced, so they persist across CR deletion/recreation. -- Router replicas are implemented as one Deployment per replica, and `$(replica)` placeholders in endpoint addresses are substituted per replica. -- When router NodePort is enabled for multiple replicas, the operator offsets NodePort by replica index for router services. -- Even when cert-manager is disabled, the operator still creates `jumpstarter-service-ca-cert` (with empty `ca.crt`) for CLI discoverability. -- Status conditions are populated on the `Jumpstarter` resource and include deployment readiness plus cert-manager/certificate readiness when cert-manager is enabled. - -## Jumpstarter API field reference - -The `Jumpstarter` CRD is `operator.jumpstarter.dev/v1alpha1`. - -### Top-level spec fields - -| Field | Type | Description | -| --- | --- | --- | -| `spec.baseDomain` | `string` | Base DNS domain for generated endpoint hostnames (for example `grpc.`). | -| `spec.certManager` | `object` | Certificate management settings for controller/router/login TLS integration. | -| `spec.controller` | `object` | Controller deployment, endpoint, and runtime settings. | -| `spec.routers` | `object` | Router deployment scale, resources, topology, and endpoint settings. | -| `spec.authentication` | `object` | Internal, Kubernetes token, JWT, and auto-provisioning authentication settings. | - -### Controller and router fields - -| Field | Type | Description | -| --- | --- | --- | -| `spec.controller.image` | `string` | Controller container image. | -| `spec.controller.imagePullPolicy` | `string` | Pull policy (`Always`, `IfNotPresent`, `Never`). | -| `spec.controller.resources` | `object` | Controller resource requests/limits. | -| `spec.controller.replicas` | `integer` | Number of controller pods. | -| `spec.controller.exporterOptions.offlineTimeout` | `duration` | Timeout before exporter is considered offline. | -| `spec.controller.grpc.tls.certSecret` | `string` | Manual TLS secret name for controller gRPC when cert-manager is disabled. | -| `spec.controller.grpc.endpoints[]` | `array` | Controller gRPC endpoint definitions (address + exposure method). | -| `spec.controller.grpc.keepalive.*` | `object` | gRPC keepalive tuning options. | -| `spec.controller.login.tls.secretName` | `string` | Optional TLS secret used by login edge-termination ingress/route. | -| `spec.controller.login.endpoints[]` | `array` | Login endpoint definitions (address + exposure method). | -| `spec.routers.image` | `string` | Router container image. | -| `spec.routers.imagePullPolicy` | `string` | Pull policy (`Always`, `IfNotPresent`, `Never`). | -| `spec.routers.resources` | `object` | Router resource requests/limits. | -| `spec.routers.replicas` | `integer` | Router replica count (operator creates one deployment per replica). | -| `spec.routers.topologySpreadConstraints[]` | `array` | Pod spread constraints for router deployments. | -| `spec.routers.grpc.tls.certSecret` | `string` | Manual TLS secret name for router gRPC when cert-manager is disabled. | -| `spec.routers.grpc.endpoints[]` | `array` | Router endpoint definitions; supports `$(replica)` placeholder in address. | -| `spec.routers.grpc.keepalive.*` | `object` | Router gRPC keepalive tuning options. | - -### Authentication fields - -| Field | Type | Description | -| --- | --- | --- | -| `spec.authentication.internal.enabled` | `boolean` | Enables internal token-based auth. | -| `spec.authentication.internal.prefix` | `string` | Username/subject prefix for internal auth. | -| `spec.authentication.internal.tokenLifetime` | `duration` | Internal token validity period. | -| `spec.authentication.k8s.enabled` | `boolean` | Enables Kubernetes service account token auth. | -| `spec.authentication.jwt[]` | `array` | JWT authenticators (issuer, audiences, claim mappings). | -| `spec.authentication.autoProvisioning.enabled` | `boolean` | Auto-create users authenticated by external providers. | - -### cert-manager fields - -| Field | Type | Description | -| --- | --- | --- | -| `spec.certManager.enabled` | `boolean` | Enables operator cert-manager integration. | -| `spec.certManager.server.selfSigned.enabled` | `boolean` | Enables operator-managed self-signed CA mode. | -| `spec.certManager.server.selfSigned.caDuration` | `duration` | Self-signed CA certificate duration. | -| `spec.certManager.server.selfSigned.certDuration` | `duration` | Issued server certificate duration. | -| `spec.certManager.server.selfSigned.renewBefore` | `duration` | Renewal lead time before expiration. | -| `spec.certManager.server.issuerRef.name` | `string` | Existing Issuer/ClusterIssuer name. | -| `spec.certManager.server.issuerRef.kind` | `string` | `Issuer` or `ClusterIssuer`. | -| `spec.certManager.server.issuerRef.group` | `string` | Issuer API group (default `cert-manager.io`). | -| `spec.certManager.server.issuerRef.caBundle` | `bytes` | Optional PEM CA bundle published for clients. | - -### Endpoint schema (used in gRPC/login endpoint arrays) - -| Field | Type | Description | -| --- | --- | --- | -| `address` | `string` | Host/address, optional port, supports `$(replica)` for router endpoints. | -| `route.enabled` | `boolean` | Create OpenShift Route for endpoint. | -| `route.annotations` / `route.labels` | `map` | Route metadata overrides. | -| `ingress.enabled` | `boolean` | Create Kubernetes Ingress for endpoint. | -| `ingress.class` | `string` | Ingress class name. | -| `ingress.annotations` / `ingress.labels` | `map` | Ingress metadata overrides. | -| `nodeport.enabled` | `boolean` | Create NodePort service for endpoint. | -| `nodeport.port` | `integer` | Requested NodePort value. | -| `nodeport.annotations` / `nodeport.labels` | `map` | NodePort service metadata overrides. | -| `loadBalancer.enabled` | `boolean` | Create LoadBalancer service for endpoint. | -| `loadBalancer.port` | `integer` | Service port for LoadBalancer exposure. | -| `loadBalancer.annotations` / `loadBalancer.labels` | `map` | LoadBalancer service metadata overrides. | -| `clusterIP.enabled` | `boolean` | Create ClusterIP service for endpoint. | -| `clusterIP.annotations` / `clusterIP.labels` | `map` | ClusterIP service metadata overrides. | - -### Status conditions - -| Condition type | Meaning | -| --- | --- | -| `Ready` | Overall operator-managed deployment readiness. | -| `ControllerDeploymentReady` | Controller deployment is available. | -| `RouterDeploymentsReady` | All router deployments are available. | -| `CertManagerAvailable` | cert-manager CRDs are present (when enabled). | -| `IssuerReady` | Configured issuer is ready (when enabled). | -| `ControllerCertificateReady` | Controller TLS secret is ready (when enabled). | -| `RouterCertificatesReady` | Router TLS secrets are ready for all replicas (when enabled). | - diff --git a/python/docs/source/getting-started/installation/service/service-production.md b/python/docs/source/getting-started/installation/service/service-production.md deleted file mode 100644 index b411904fa..000000000 --- a/python/docs/source/getting-started/installation/service/service-production.md +++ /dev/null @@ -1,48 +0,0 @@ -# Production Deployment - -For production deployments, you can install Jumpstarter on Kubernetes or OpenShift clusters with proper ingress, monitoring, and security configurations. - -## Prerequisites - -Before installing in production, ensure you have: - -- A production Kubernetes cluster available -- `kubectl` installed and configured to access your cluster -- Administrator access to your cluster (required for CRD installation) -- Domain name for service endpoints -- Ingress controller installed (for Kubernetes) or Routes configured (for OpenShift) -```{note} -`global.baseDomain` creates these service hostnames with `jumpstarter.example.com`: -- `grpc.jumpstarter.example.com` -- `router.jumpstarter.example.com` (for router endpoints) -``` - -## TLS and gRPC Configuration - -Jumpstarter uses gRPC for communication, which has specific requirements for production deployments: - -### gRPC Requirements - -- **HTTP/2 Support**: gRPC requires HTTP/2; ensure the path from clients to the service supports it -- **Keep-Alive Settings**: The Jumpstarter service and client configure gRPC keep-alive by default; you usually do not need to tune these separately. - -### TLS for gRPC - -The [Jumpstarter operator](service-operator.md) installs gRPC with **TLS passthrough** at the ingress or route: encrypted traffic is forwarded to the controller and router pods, which terminate TLS. HTTP login endpoints use edge TLS termination instead. - -```{note} -When using ingress-nginx, you must enable the [`--enable-ssl-passthrough`](https://kubernetes.github.io/ingress-nginx/user-guide/cli-arguments/) flag on the ingress controller, as SSL passthrough is disabled by default. See the [ingress-nginx TLS documentation](https://kubernetes.github.io/ingress-nginx/user-guide/tls/#ssl-passthrough) for more details. -``` - -## Installation - -To install Jumpstarter, see [Install with Operator](service-operator.md). That guide includes: - -- Installing the operator from the release asset (`operator-installer.yaml`), OperatorHub and OLM. -- Creating a `Jumpstarter` custom resource for vanilla Kubernetes with Ingress -- Creating a `Jumpstarter` custom resource for OpenShift with Routes -- Notes on integrating external OAuth/OIDC and cert-manager setups - -## GitOps and ArgoCD - -Use the operator installer and manage your `Jumpstarter` custom resource declaratively in GitOps flows. See [Install with Operator](service-operator.md) for the manifests and endpoint patterns to use on Kubernetes (Ingress) and OpenShift (Route). \ No newline at end of file diff --git a/python/docs/source/getting-started/installation/service/standalone.md b/python/docs/source/getting-started/installation/service/standalone.md new file mode 100644 index 000000000..2b79cdc79 --- /dev/null +++ b/python/docs/source/getting-started/installation/service/standalone.md @@ -0,0 +1,155 @@ +# Standalone + +Lightweight deployment using MicroShift and a bootable container (bootc) image +with the Jumpstarter {term}`operator` pre-installed. Ideal for edge devices, +development environments, and small labs. For production deployments, see +[Production](production.md). + +## Prerequisites + +- Fedora/RHEL-based system (tested on Fedora 42) +- Podman installed and configured +- Root/sudo access required for privileged operations +- At least 4GB RAM and 20GB disk space recommended + +## Install + +### Build the Image + +```console +make bootc-build +``` + +### Run as Container + +```console +make bootc-run +``` + +This creates a 1GB LVM disk image, starts MicroShift in a privileged container, +sets up LVM volume groups for TopoLVM, and waits for MicroShift to be ready. + +### Create a Bootable QCOW2 Image + +For bare-metal or VM deployments: + +```console +make build-image +``` + +```{note} +If the container is running, stop it first with `make bootc-rm` to avoid LVM +conflicts. +``` + +## Verify + +Access the services: + +- **Configuration Web UI**: `http://localhost:8880` (login: `root` / `jumpstarter`, + password change required on first use) +- **MicroShift API**: `https://jumpstarter..nip.io:6443` +- **Pod Monitoring**: `http://localhost:8880/pods` + +Check running pods: + +```console +sudo podman exec -it jumpstarter-microshift-okd oc get pods -A +``` + +## Configuration + +### Customization + +```console +BOOTC_IMG=quay.io/your-org/microshift-bootc:v1.0 make bootc-build +``` + +Add Kubernetes manifests to `/etc/microshift/manifests.d/002-jumpstarter/` by +editing `kustomization.yaml`. For live config service changes without rebuild: + +```console +make bootc-reload-app +``` + +### QCOW2 Image + +The QCOW2 image is configured via `config.toml` (LVM partitioning with 20GB +minimum, XFS root filesystem, default password `root:jumpstarter`). + +```console +qemu-system-x86_64 \ + -m 4096 \ + -smp 2 \ + -drive file=output/qcow2/disk.qcow2,format=qcow2 \ + -net nic -net user,hostfwd=tcp::8880-:8880,hostfwd=tcp::443-:443 +``` + +### Network + +The system uses `nip.io` for automatic DNS resolution (e.g. +`jumpstarter.10.0.2.2.nip.io`). + +| Port | Service | Description | +|------|---------|-------------| +| 80 | HTTP | MicroShift ingress | +| 443 | HTTPS | MicroShift API and ingress | +| 8880 | Config UI | Web configuration interface | +| 6443 | API Server | Kubernetes API (internal) | + +### Security + +1. **Default Password:** `root:jumpstarter`. Console login forces a change. Web + UI requires a change before access. +2. **TLS Certificates:** MicroShift uses self-signed certs by default. +3. **Privileged Container:** Required for systemd, LVM, and networking. +4. **Authentication:** Web UI uses PAM authentication with root credentials. + +## Troubleshooting + +### LVM/TopoLVM Issues + +```console +sudo podman exec jumpstarter-microshift-okd vgs +sudo podman exec jumpstarter-microshift-okd pvs +make bootc-rm && make clean && make bootc-run +``` + +### MicroShift Not Starting + +```console +sudo podman logs jumpstarter-microshift-okd +sudo podman exec jumpstarter-microshift-okd journalctl -u microshift -f +``` + +### Configuration Service Issues + +```console +sudo podman exec jumpstarter-microshift-okd systemctl status config-svc +sudo podman exec jumpstarter-microshift-okd journalctl -u config-svc -f +``` + +## Uninstall + +```console +make bootc-stop +make bootc-rm +make clean +``` + +`make bootc-rm` stops the container, cleans up LVM volume groups, and detaches +loop devices. `make clean` removes the LVM disk image. + +## Makefile Targets + +| Target | Description | +|--------|-------------| +| `make bootc-build` | Build the bootc container image | +| `make bootc-run` | Run MicroShift in a container | +| `make bootc-stop` | Stop the running container | +| `make bootc-rm` | Remove container and clean up LVM resources | +| `make bootc-sh` | Open shell in container | +| `make bootc-reload-app` | Reload config service without rebuild | +| `make build-image` | Create bootable QCOW2 image | +| `make bootc-push` | Push image to registry | +| `make clean` | Clean up images, artifacts, and LVM disk | diff --git a/python/docs/source/glossary.md b/python/docs/source/glossary.md index 1549a2090..e3b679899 100644 --- a/python/docs/source/glossary.md +++ b/python/docs/source/glossary.md @@ -2,103 +2,105 @@ ## Acronyms -* `DUT`: Device Under Test -* `CRD`: Custom Resource Definition -* `CI/CD`: Continuous Integration/Continuous Deployment -* `gRPC`: Google Remote Procedure Call -* `JWT`: JSON Web Token -* `KVM`: Keyboard, Video, Mouse +```{glossary} +:sorted: + +CRD + Custom Resource Definition - Kubernetes extension for Jumpstarter resources. + +DUT + Device Under Test. + +gRPC + Google Remote Procedure Call - Jumpstarter's communication framework. + +HiL + Hardware-in-the-Loop - testing with real hardware in the loop. + +MAN + Manual - reference documentation for command-line tools. + +JEP + Jumpstarter Enhancement Proposal - design document for significant changes. + +MCP + Model Context Protocol - enables AI agents to interact with hardware. +``` ## Entities -* `exporter`: A Linux service that exports the interfaces to the DUTs. An - exporter connects directly to a Jumpstarter server or directly to a client. +```{glossary} +:sorted: -* `client`: A developer or a CI/CD pipeline that connects to the Jumpstarter - server and leases exporters. The client can run tests on the leased resources. +client + A user or CI pipeline that connects to the service and leases exporters. -* `controller`: The central service that authenticates and connects the - exporters and clients, manages leases, and provides an inventory of available - exporters and clients. +controller + Central service for authentication, lease management, and inventory. -* `router`: A service used by the controller to route messages between clients - and exporters through a gRPC tunnel, enabling remote access to exported - interfaces. +exporter + Service that exposes hardware interfaces to clients over gRPC. -* `host`: A system running the exporter service, typically a low-cost test - system such as a single board computer with sufficient interfaces to connect - to hardware. +host + Machine running the exporter, typically a single board computer. + +operator + Kubernetes operator that deploys the controller, router, and CRDs. + +router + Routes traffic between clients and exporters through a gRPC tunnel. + +service + Kubernetes backend providing controller, router, and authentication. +``` ## Concepts -* `device`: A device that is exposed on an exporter. The exporter enumerates - these devices and makes them available for use in tests. Examples of resources - include: - * Network interface - * Serial port - * GPIO pin - * Storage device (USB Muxer, SD-Wire, etc.) - * CAN bus interface - -* `hook`: A shell script configured on an exporter that runs automatically at - lease boundaries. A `beforeLease` hook runs after a lease is assigned but - before drivers are available to the client, and an `afterLease` hook runs - after the session ends but before the lease is released. Hooks use the `j` - CLI to interact with exported devices. - -* `lease`: A time-limited reservation of an exporter. A lease is created by a - client and allows the client to use the exporter resources for a limited time. - Leases ensure exclusive access to specific devices/exporters. - -* `adapter`: A component that transforms connections exposed by drivers into - different forms or interfaces. Adapters take a driver client as input and - provide alternative ways to interact with the underlying connection, such as - port forwarding, VNC access, or terminal emulation. - -* `interface class`: An abstract base class that defines the contract for driver - implementations. It specifies the required methods that must be implemented by - driver classes and provides the client class path through the `client()` class - method. - -* `driver class`: A class that implements an interface and inherits from the - `Driver` base class. It uses the `@export` decorator to expose methods that - can be called remotely by clients. - -* `driver client class`: The driver client class that is used directly by end - users. It interacts with the `driver class` remotely via remote procedure call - to invoke exported methods, which in turn interact with the exporter - resources. - -* `driver`: The term for both the `driver class` and the corresponding `driver - client class`, not to be confused with `Driver`, the base class of all `driver - classes`. Drivers in the main `jumpstarter` repository are called `in-tree - drivers`, otherwise they are called `out-of-tree drivers`. Drivers - implementing predefined interfaces are called `standard drivers`, otherwise - they are called `custom drivers`. - -* `composite driver`: A driver that combines multiple lower-level drivers to - create higher-level abstractions or specialized workflows, organized in a tree - structure to represent complex device configurations. - -* `local mode`: An operation mode where clients communicate directly with - exporters running on the same machine or through direct network connections, - ideal for individual developers working directly with accessible hardware or - virtual devices. - -* `distributed mode`: An operation mode that enables multiple teams to securely - share hardware resources across a network using a Kubernetes-based controller - to coordinate access to exporters and manage leases. - -* `stream`: A continuous data exchange channel established by drivers for - communications like serial connections or video streaming, enabling real-time - interaction with both physical and virtual interfaces across the network. - -* `message`: Commands sent from driver clients to driver implementations, - allowing the client to trigger actions or retrieve information from the - device. - -* `exporter status`: The current state of an exporter in its lifecycle. States - include `AVAILABLE`, `BEFORE_LEASE_HOOK`, `LEASE_READY`, - `AFTER_LEASE_HOOK`, `BEFORE_LEASE_HOOK_FAILED`, `AFTER_LEASE_HOOK_FAILED`, - and `OFFLINE`. The status tracks transitions through the lease and hook - lifecycle. \ No newline at end of file +```{glossary} +:sorted: + +adapter + Transforms driver connections into other forms (port forwarding, VNC, etc). + +device + Hardware or virtual resource exposed on an exporter. + +direct mode + Client connects to an exporter over TCP without a controller. + +distributed mode + Shared hardware access across teams via a jumpstarter-controller. + +driver + Modular component providing a standardized interface to a device type. + +exporter shell + Interactive shell spawned by `jmp shell` for driver CLI access. + +hook + Shell script that runs automatically at lease boundaries. + +label selector + Key-value metadata for selecting exporters when leasing. + +lease + Time-limited reservation of an exporter with exclusive access. + +local mode + Client and exporter on the same machine, no Kubernetes required. + +session + Connection context between client and exporter during testing. +``` + +## Tools + +```{glossary} +:sorted: + +j + Shorthand CLI for driver access within the exporter shell. + +jmp + Primary Jumpstarter CLI for managing clients, exporters, and leases. +``` diff --git a/python/docs/source/index.md b/python/docs/source/index.md index 97d831dbb..7f18178ce 100644 --- a/python/docs/source/index.md +++ b/python/docs/source/index.md @@ -38,9 +38,10 @@ using cloud native principles. See Jumpstarter in action: One tool, any target. Jumpstarter decouples devices from test runners, letting you use identical automation scripts everywhere - your *Makefile* for device -testing. +testing. Every interface is programmatic, so human developers, test scripts, CI +pipelines, and AI agents all interact with hardware through the same APIs. -```{include} ../../README.md +```{include} ../../../README.md :start-after: "## Highlights" :end-before: "##" ``` @@ -59,5 +60,4 @@ contributing.md glossary.md reference/index.md -internal/index.md ``` diff --git a/python/docs/source/internal/index.md b/python/docs/source/internal/index.md deleted file mode 100644 index 964e6d9fb..000000000 --- a/python/docs/source/internal/index.md +++ /dev/null @@ -1,11 +0,0 @@ -# Project Governance - -This section contains Jumpstarter Enhancement Proposals (JEPs) that document -significant design decisions and process changes. - -```{toctree} -:maxdepth: 2 -:hidden: - -jeps/README.md -``` diff --git a/python/docs/source/introduction/adapters.md b/python/docs/source/introduction/adapters.md index 139f5151b..d83bc22e2 100644 --- a/python/docs/source/introduction/adapters.md +++ b/python/docs/source/introduction/adapters.md @@ -1,40 +1,40 @@ # Adapters -Jumpstarter uses adapters to transform network connections established by +Jumpstarter uses {term}`adapter`s to transform network connections established by drivers into different forms or interfaces that are more appropriate for specific use cases. ## Architecture -Adapters in Jumpstarter follow a transformation pattern where: +{term}`Adapter`s in Jumpstarter follow a transformation pattern where: -- Adapters take a driver client as input +- {term}`Adapter`s take a driver client class as input - They transform the connection into a different interface format - The transformed interface is exposed to the user in a way that's tailored for specific scenarios The architecture consists of these key components: -- **Adapter Base** - Adapters typically follow a context manager pattern using - Python's `with` statement for resource management. Each adapter takes a driver +- **{term}`Adapter` Base** - {term}`Adapter`s typically follow a context manager pattern using + Python's `with` statement for resource management. Each {term}`adapter` takes a driver client as input and transforms its connection. -- **Connection Transformation** - Adapters create a new interface on top of an +- **Connection Transformation** - {term}`Adapter`s create a new interface on top of an existing driver connection, such as forwarding ports, providing web interfaces, or offering terminal-like access. -- **Resource Lifecycle** - Adapters handle proper setup and teardown of +- **Resource Lifecycle** - {term}`Adapter`s handle proper setup and teardown of resources, ensuring connections are properly established and cleaned up. Unlike [Drivers](drivers.md), which establish the foundational connections to -hardware or virtual interfaces, adapters focus on providing alternative ways to +hardware or virtual interfaces, {term}`adapter`s focus on providing alternative ways to interact with those connections without modifying the underlying drivers. -Adapters operate entirely on the client side and transform existing connections +{term}`Adapter`s operate entirely on the client side and transform existing connections rather than establishing new ones directly with hardware or virtual devices. ## Types -Different types of adapters serve different needs: +Different types of {term}`adapter`s serve different needs: - **Port Forwarding Adapters** - Convert network connections to local ports or sockets @@ -45,17 +45,17 @@ Different types of adapters serve different needs: - **UI Adapters** - Create user interfaces for interacting with devices (e.g., web-based VNC) -Adapters can be composed and extended for more complex scenarios: +{term}`Adapter`s can be composed and extended for more complex scenarios: -- **Chaining adapters**: Use the output of one adapter as the input to another -- **Custom adapters**: Create specialized adapters for specific hardware or +- **Chaining {term}`adapter`s**: Use the output of one {term}`adapter` as the input to another +- **Custom {term}`adapter`s**: Create specialized {term}`adapter`s for specific hardware or software interfaces - **Extended functionality**: Add logging, monitoring, or security features on - top of base adapters + top of base {term}`adapter`s ## Implementation Patterns -Adapters typically implement the context manager protocol (`__enter__` and +{term}`Adapter`s typically implement the context manager protocol (`__enter__` and `__exit__`) to ensure proper resource management. The general pattern is: 1. Initialize with a driver client reference @@ -63,10 +63,10 @@ Adapters typically implement the context manager protocol (`__enter__` and 3. Return the appropriate interface (URL, address, interactive object) 4. Clean up resources in `__exit__` -This allows adapters to be used in `with` statements for clean, deterministic +This allows {term}`adapter`s to be used in `with` statements for clean, deterministic resource handling. -When working with adapters, follow these recommended practices: +When working with {term}`adapter`s, follow these recommended practices: 1. **Always use context managers** (`with` statements) to ensure proper resource cleanup and prevent resource leaks diff --git a/python/docs/source/introduction/clients.md b/python/docs/source/introduction/clients.md index d419c0214..b8b6e3d37 100644 --- a/python/docs/source/introduction/clients.md +++ b/python/docs/source/introduction/clients.md @@ -10,23 +10,23 @@ Jumpstarter supports two types of client configurations: *local* and *remote*. ### Local Clients -When using Jumpstarter in *local-only* mode, you can use the local client +When using Jumpstarter in {term}`local mode`, you can use the local client functionality to directly access your hardware. The local client will automatically use any drivers that are registered without the need for an -exporter instance running in the background. +{term}`exporter` instance running in the background. ### Remote Clients -When using Jumpstarter in *distributed* mode, the client must be configured to -connect to an instance of the Service that can authenticate and route requests -to the appropriate exporter instance. +When using Jumpstarter in {term}`distributed mode`, the client must be configured to +connect to an instance of the {term}`service` that can authenticate and route requests +to the appropriate {term}`exporter` instance. The following parameters are required to set up a remote client: -- The URL of a Service endpoint to connect to -- An authentication token generated by the Service +- The URL of a {term}`service` endpoint to connect to +- An authentication token generated by the {term}`service` ```{note} The endpoint must be accessible from your client machine to communicate -with the Service. +with the {term}`service`. ``` \ No newline at end of file diff --git a/python/docs/source/introduction/drivers.md b/python/docs/source/introduction/drivers.md index 45d98b1fc..c8d58cc0c 100644 --- a/python/docs/source/introduction/drivers.md +++ b/python/docs/source/introduction/drivers.md @@ -4,43 +4,43 @@ Jumpstarter uses a modular driver model to build abstractions around the interfaces used to interact with target devices, both physical hardware and virtual systems. -An [Exporter](exporters.md) uses Drivers to "export" these interfaces from a -host machine to the clients via [gRPC](https://grpc.io/). Drivers can be thought +An [{term}`Exporter`](exporters.md) uses Drivers to "export" these interfaces from a +{term}`host` machine to the clients via {term}`gRPC`. Drivers can be thought of as a simplified API for an interface or device type. ## Architecture Drivers in Jumpstarter follow a client/server architecture where: -- Driver implementations run on the exporter side and interact directly with - hardware or virtual devices -- Driver clients run on the client side and communicate with drivers via gRPC +- Driver implementations run on the {term}`exporter` side and interact directly with + hardware or virtual {term}`device`s +- Driver clients run on the client side and communicate with drivers via {term}`gRPC` - Interface classes define the contract between implementations and clients The architecture follows a pattern with these key components: -- **Interface Class** - An abstract base class using Python's ABCMeta to define +- **Interface class** - An abstract base class using Python's ABCMeta to define the contract (methods and their signatures) that driver implementations must fulfill. The interface also specifies the client class through the `client()` class method. -- **Driver Class** - Inherits from both the Interface and the base `Driver` +- **Driver class** - Inherits from both the Interface and the base `Driver` class, implementing the logic to configure and use hardware interfaces. Driver methods are marked with the `@export` decorator to expose them over the network. -- **Driver Client** - Provides a user-friendly interface that can be used by +- **Driver client class** - Provides a user-friendly interface that can be used by clients to interact with the driver either locally or remotely over the network. -When a client requests a lease and connects to an exporter, a session is created -for all tests the client needs to execute. Within this session, the specified +When a client requests a {term}`lease` and connects to an {term}`exporter`, a {term}`session` is created +for all tests the client needs to execute. Within this {term}`session`, the specified `Driver` subclass is instantiated for each configured interface. These driver -instances live throughout the session's duration, maintaining state and +instances live throughout the {term}`session`'s duration, maintaining state and executing setup/teardown logic. On the client side, a `DriverClient` subclass is instantiated for each exported -interface. Since clients may run on different machines than exporters, +interface. Since clients may run on different machines than {term}`exporter`s, `DriverClient` classes are loaded dynamically when specified in the allowed packages list. @@ -49,34 +49,34 @@ methods when needed but preserve existing signatures. If breaking changes are required, create new interface, client, and driver versions within the same module. -Drivers are often used with [Adapters](adapters.md), which transform driver +Drivers are often used with [{term}`adapter`s](adapters.md), which transform driver connections into different forms or interfaces for specific use cases. ## Types -The API reference of the documentation provides a complete list of all standard -drivers, you can find it here: [Driver API +The API reference of the documentation provides a complete list of all +standard drivers, you can find it here: [Driver API Reference](../reference/package-apis/drivers/index.md). Some categories of drivers include: -- [System - Control](../reference/package-apis/drivers/index.md#system-control-drivers): +- [System Control](../reference/package-apis/drivers/index.md#system-control): Control power to devices, or general control. -- [Communication](../reference/package-apis/drivers/index.md#communication-drivers): +- [Communication](../reference/package-apis/drivers/index.md#communication): Provide protocols for network communication, such as TCP/IP, Serial, CAN bus, etc. -- [Data and - Storage](../reference/package-apis/drivers/index.md#storage-and-data-drivers): +- [Storage and Data](../reference/package-apis/drivers/index.md#storage-and-data): Control storage devices, such as SD cards or USB drives, and data. -- [Media](../reference/package-apis/drivers/index.md#media-drivers): Provide +- [Media](../reference/package-apis/drivers/index.md#media): Provide interfaces for media capture and playback, such as video or audio. -- [Debug and - Programming](../reference/package-apis/drivers/index.md#debug-and-programming-drivers): - Provide interfaces for debugging and programming devices, such as JTAG or SWD, - remote flashing, emulation, etc. -- [Utility](../reference/package-apis/drivers/index.md#utility-drivers): Provide - utility functions, such as shell driver commands on a exporter. +- [Automotive Diagnostics](../reference/package-apis/drivers/index.md#automotive-diagnostics): + Provide automotive diagnostic protocol interfaces. +- [Flashing and Programming](../reference/package-apis/drivers/index.md#flashing-and-programming): + Provide interfaces for flashing firmware and programming devices. +- [Emulation](../reference/package-apis/drivers/index.md#emulation): + Manage virtual and emulated targets. +- [Utility](../reference/package-apis/drivers/index.md#utility): Provide + utility functions, such as shell driver commands on an {term}`exporter`. ### Composite Drivers @@ -85,7 +85,7 @@ abstractions or specialized workflows. For example, a composite driver might coordinate power cycling, storage re-flashing, and serial communication to automate a device initialization process. -In Jumpstarter, drivers are organized in a tree structure which allows for the +In Jumpstarter, drivers are organized in a driver tree structure which allows for the representation of complex device configurations that may be found in your environment. @@ -103,7 +103,7 @@ MyHarness # Custom composite driver for the entire target device harness ## Configuration -Drivers are configured using a YAML Exporter config file, which specifies the +Drivers are configured using a YAML exporter config file, which specifies the drivers to load and the parameters for each. Drivers are distributed as Python packages making it easy to develop and install your own drivers. @@ -143,20 +143,41 @@ export: ## Communication -Drivers use two primary methods to communicate between client and exporter: - -### Messages - -Commands are sent as messages from driver clients to driver implementations, -allowing the client to trigger actions or retrieve information from the device. -Methods marked with the `@export` decorator are made available over the network. - -### Streams +Drivers expose their methods over {term}`gRPC` using three RPC styles (see +[RPC life cycle](https://grpc.io/docs/what-is-grpc/core-concepts/#rpc-life-cycle) +for details on gRPC counterparts): + +```{mermaid} +flowchart LR + subgraph "Unary RPC" + direction TB + C1["Client"] -- "DriverCall\n(desired state)" --> D1["Driver"] + D1 -- "Result" --> C1 + E1["Example: power on/off"] + end + + subgraph "Server Streaming RPC" + direction TB + C2["Client"] -- "StreamingDriverCall\n(interval)" --> D2["Driver"] + D2 -- "Result Stream" --> C2 + E2["Example: power readings"] + end + + subgraph "Bidirectional Streaming RPC" + direction TB + C3["Client"] <-- "DriverStream\n(Byte Stream)" --> D3["Driver"] + E3["Example: video capture"] + end +``` -Drivers can establish streams for continuous data exchange, such as for serial -communication or video streaming. This enables real-time interaction with both -physical and virtual interfaces across the network. Methods marked with the -`@exportstream` decorator create streams for bidirectional communication. +- **Unary** - Methods marked with `@export` send a single request and receive a + single response. Used for commands like power on/off or querying device state. +- **Server Streaming** - Methods marked with `@export` that return a generator + produce a stream of responses from a single request. Used for continuous data + like sensor readings. +- **Bidirectional Streaming** - Methods marked with the `@exportstream` decorator open a + full-duplex byte stream. Used for serial communication, video capture, or + tunneling existing protocols (such as SSH) over Jumpstarter. ## Authentication and Security @@ -165,33 +186,33 @@ Driver access is controlled through Jumpstarter's authentication mechanisms: ### Local Mode Authentication -In local mode, drivers are accessible to any process that can connect to the +In {term}`local mode`, drivers are accessible to any process that can connect to the local Unix socket. This is typically restricted by file system permissions. When running tests locally, authentication is simplified since everything runs in the same user context. ### Distributed Mode Authentication -In distributed mode, authentication is handled through JWT tokens: +In {term}`distributed mode`, authentication is handled through JWT tokens: -- **Client Authentication**: Clients authenticate to the controller using JWT +- **Client Authentication**: Clients authenticate to the {term}`controller` using JWT tokens, which establishes their identity and permissions -- **Exporter Authentication**: Similarly, exporters authenticate to the - controller with their own tokens -- **Driver Access Control**: The controller enforces access control by only - allowing authorized clients to acquire leases on exporters and their drivers -- **Driver Allowlist**: Client configurations can specify which driver packages +- **Exporter Authentication**: Similarly, {term}`exporter`s authenticate to the + {term}`controller` with their own tokens +- **Driver Access Control**: The {term}`controller` enforces access control by only + allowing authorized clients to acquire {term}`lease`s on {term}`exporter`s and their drivers +- **Driver allowlist**: Client configurations can specify which driver packages are allowed to be loaded, preventing unintended execution of untrusted code ### Driver Package Security -When using distributed mode, driver security considerations include: +When using {term}`distributed mode`, driver security considerations include: - **Package Verification**: Clients can verify that only trusted driver packages are loaded by configuring allowlists - **Capability Restrictions**: Access to specific driver functionality can be restricted based on client permissions -- **Session Isolation**: Each client session operates with its own driver +- **{term}`Session` Isolation**: Each client {term}`session` operates with its own driver instances to prevent interference between users ## Custom Drivers diff --git a/python/docs/source/introduction/exporters.md b/python/docs/source/introduction/exporters.md index fa5da10f2..d927f6317 100644 --- a/python/docs/source/introduction/exporters.md +++ b/python/docs/source/introduction/exporters.md @@ -1,25 +1,25 @@ # Exporters -Jumpstarter uses a program called an Exporter to enable remote access to your -hardware. The Exporter typically runs on a "host" system directly connected to -your hardware. It is called an Exporter because it "exports" the interfaces +Jumpstarter uses a program called an {term}`exporter` to enable remote access to your +hardware. The {term}`exporter` typically runs on a {term}`host` system directly connected to +your hardware. It is called an {term}`exporter` because it "exports" the interfaces connected to the target device for client access. ## Hosts -Typically, the host will be a low-cost test system such as a single board +Typically, the {term}`host` will be a low-cost test system such as a single board computer with sufficient interfaces to connect to your hardware. It is also -possible to use a local high-power server (or CI runner) as the host device. +possible to use a local high-power server (or CI runner) as the {term}`host` device. -A host can run multiple Exporter instances simultaneously if it needs to +A {term}`host` can run multiple Exporter instances simultaneously if it needs to interact with several different devices at the same time. ## Exporter Configuration -Exporters use a YAML configuration file to define which Drivers must be loaded +Exporters use a YAML configuration file (exporter config) to define which Drivers must be loaded and the configuration required. -Here is an example Exporter config file which would typically be saved at +Here is an example exporter config file which would typically be saved at `/etc/jumpstarter/exporters/demo.yaml`: ```yaml @@ -62,27 +62,27 @@ documentation](https://grpc.github.io/grpc/core/group__grpc__arg__keys.html). ## Running an Exporter -To run an Exporter on a host system, you must have Python {{requires_python}} +To run an Exporter on a {term}`host` system, you must have Python {{requires_python}} installed and the driver packages specified in the config installed in your current Python environment. -You can run the exporter in your local terminal with: +You can run the {term}`exporter` in your local terminal with: ```console $ jmp run --exporter myexporter ``` -Exporters can also be run in a privileged container or as a systemd daemon. It -is recommended to run the Exporter service in the background with auto-restart +{term}`Exporter`s can also be run in a privileged container or as a `systemd` daemon. It +is recommended to run the {term}`exporter` service in the background with auto-restart capabilities in case something goes wrong and it needs to be restarted. ## Lifecycle Hooks -Exporters support lifecycle hooks that execute shell scripts at lease -boundaries. A `beforeLease` hook runs after a lease is assigned but before +{term}`Exporter`s support lifecycle {term}`hook`s that execute shell scripts at {term}`lease` +boundaries. A `beforeLease` hook runs after a {term}`lease` is assigned but before the client can access drivers, and an `afterLease` hook runs after the -session ends but before the lease is released. +{term}`session` ends but before the {term}`lease` is released. -Hooks are configured in the `hooks` section of the exporter config file and -use the `j` CLI to interact with exported devices. For full details, see +{term}`Hook`s are configured in the `hooks` section of the exporter config file and +use the {term}`j` CLI to interact with exported devices. For full details, see [Hooks](hooks.md). diff --git a/python/docs/source/introduction/hooks.md b/python/docs/source/introduction/hooks.md index f02ec6b88..d728445b6 100644 --- a/python/docs/source/introduction/hooks.md +++ b/python/docs/source/introduction/hooks.md @@ -1,15 +1,15 @@ # Hooks -Jumpstarter supports lifecycle hooks that execute shell scripts automatically before or after a lease. +Jumpstarter supports lifecycle hooks that execute shell scripts automatically before or after a {term}`lease`. A `beforeLease` hook runs after a lease is assigned but before drivers are available to the client, and an `afterLease` hook runs after -the session ends but before the lease is released. Hooks are optional and -configured in the [Exporter](exporters.md) YAML configuration file. +the {term}`session` ends but before the lease is released. Hooks are optional and +configured in the [Exporter](exporters.md) YAML configuration file (exporter config). -Hooks execute on the exporter host using a configurable interpreter (defaulting -to `/bin/sh`) and can use the `j` CLI to interact with drivers locally on the -exporter. The `exec` field lets you choose a different interpreter such as +Hooks execute on the exporter {term}`host` using a configurable interpreter (defaulting +to `/bin/sh`) and can use the {term}`j` CLI to interact with drivers locally on the +{term}`exporter`. The `exec` field lets you choose a different interpreter such as `/bin/bash` or `python3`. The `script` field accepts either an inline script or a path to a script file on disk. Common use cases include powering on devices, validating hardware state, flashing firmware, and cleaning up after @@ -17,11 +17,10 @@ tests. ## How Hooks Work -The following diagram shows the full lifecycle of a lease with both hooks +The following diagram shows the full lifecycle of a {term}`lease` with both {term}`hook`s configured: ```{mermaid} -:config: {"theme":"base","themeVariables":{"primaryColor":"#f8f8f8","primaryTextColor":"#000","primaryBorderColor":"#e5e5e5","lineColor":"#3d94ff","secondaryColor":"#f8f8f8","tertiaryColor":"#fff"}} sequenceDiagram participant Controller participant Exporter @@ -46,28 +45,28 @@ sequenceDiagram Exporter->>Controller: Release lease ``` -The exporter transitions through these states during a lease: +The {term}`exporter` transitions through these states during a {term}`lease`: -1. **Lease assigned** -- The controller assigns a lease to the exporter. -2. **`BEFORE_LEASE_HOOK`** -- The `beforeLease` script runs. Driver access is - blocked until the hook completes successfully. -3. **`LEASE_READY`** -- The hook succeeded and the client can now access +1. **{term}`Lease` assigned** - The {term}`controller` assigns a {term}`lease` to the {term}`exporter`. +2. **`BEFORE_LEASE_HOOK`** - The `beforeLease` script runs. Driver access is + blocked until the {term}`hook` completes successfully. +3. **`LEASE_READY`** - The {term}`hook` succeeded and the client can now access drivers. -4. **Client session** -- The client uses drivers normally. -5. **Session ends** -- The client disconnects or the lease is released. -6. **`AFTER_LEASE_HOOK`** -- The `afterLease` script runs. The session remains +4. **Client {term}`session`** - The client uses drivers normally. +5. **{term}`Session` ends** - The client disconnects or the {term}`lease` is released. +6. **`AFTER_LEASE_HOOK`** - The `afterLease` script runs. The {term}`session` remains open so `j` commands can still interact with drivers. -7. **`AVAILABLE`** -- The hook completed and the lease is released. The - exporter is ready for the next lease. +7. **`AVAILABLE`** - The {term}`hook` completed and the {term}`lease` is released. The + {term}`exporter` is ready for the next {term}`lease`. ```{note} -If no hooks are configured, the exporter transitions directly from lease -assignment to `LEASE_READY` and from session end to `AVAILABLE`. +If no {term}`hook`s are configured, the {term}`exporter` transitions directly from {term}`lease` +assignment to `LEASE_READY` and from {term}`session` end to `AVAILABLE`. ``` ## Configuration -Hooks are configured in the `hooks` section of the exporter config file: +{term}`Hook`s are configured in the `hooks` section of the exporter config file: ```yaml apiVersion: jumpstarter.dev/v1alpha1 @@ -101,8 +100,8 @@ hooks: | Field | Type | Default | Description | | ------------------------ | ------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `hooks.beforeLease` | object | *(none)* | Hook that runs after lease assignment, before drivers are available | -| `hooks.afterLease` | object | *(none)* | Hook that runs after the session ends, before the lease is released | +| `hooks.beforeLease` | object | *(none)* | {term}`Hook` that runs after {term}`lease` assignment, before drivers are available | +| `hooks.afterLease` | object | *(none)* | {term}`Hook` that runs after the {term}`session` ends, before the {term}`lease` is released | | `hooks..exec` | string | *(auto)* | Interpreter used to execute the script. Auto-detected from file extension when not set (`.py` uses the exporter's Python, `.sh` uses `/bin/sh`). Defaults to `/bin/sh` for inline scripts. | | `hooks..script` | string | *(required)* | Inline script or path to a script file (auto-detected) | | `hooks..timeout` | integer | `120` | Maximum execution time in seconds | @@ -123,7 +122,7 @@ auto-detected from the file extension: | Extension | Interpreter | Notes | | --------- | ------------------------------------ | ----------------------------------------------------------------------------- | -| `.py` | Exporter's Python (`sys.executable`) | Has access to all installed packages including the Jumpstarter client library | +| `.py` | {term}`Exporter`'s Python (`sys.executable`) | Has access to all installed packages including the Jumpstarter client library | | `.sh` | `/bin/sh` | POSIX shell | | *(other)* | `/bin/sh` | Fallback for unrecognized extensions | @@ -132,22 +131,22 @@ a `.sh` file that needs bash features). ## Environment Variables -Hook scripts receive a pre-configured environment that enables the `j` CLI to -communicate with the exporter session: +{term}`Hook` scripts receive a pre-configured environment that enables the `j` CLI to +communicate with the {term}`exporter` {term}`session`: | Variable | Description | | ------------------- | ----------------------------------------------------------------------------------- | -| `JUMPSTARTER_HOST` | Unix socket path for `j` CLI access to the exporter session | -| `LEASE_NAME` | Name of the current lease assigned by the controller | -| `CLIENT_NAME` | Name of the client holding the lease | -| `JMP_DRIVERS_ALLOW` | Set to `UNSAFE` to enable access to all drivers (hooks run locally on the exporter) | +| `JUMPSTARTER_HOST` | Unix socket path for `j` CLI access to the {term}`exporter` {term}`session` | +| `LEASE_NAME` | Name of the current {term}`lease` assigned by the {term}`controller` | +| `CLIENT_NAME` | Name of the client holding the {term}`lease` | +| `JMP_DRIVERS_ALLOW` | Set to `UNSAFE` to enable access to all drivers ({term}`hook`s run locally on the {term}`exporter`) | -These variables are set automatically. The hook uses a dedicated Unix socket +These variables are set automatically. The {term}`hook` uses a dedicated Unix socket separate from the client connection to avoid protocol interference. -The hook environment is also configured to signal noninteractive mode. Even -though hooks run in a PTY (for line-buffered output), they are not interactive -sessions. The following variables are set to prevent programs from displaying +The {term}`hook` environment is also configured to signal noninteractive mode. Even +though {term}`hook`s run in a PTY (for line-buffered output), they are not interactive +{term}`session`s. The following variables are set to prevent programs from displaying prompts or interactive UI: | Variable | Value | Purpose | @@ -161,83 +160,83 @@ emit a prompt. ## Logging -Hook output is streamed to the client in real time. Every line written to -stdout or stderr by the hook script is captured and forwarded to the client -through the exporter's log stream. The `beforeLease` hook output is tagged +{term}`Hook` output is streamed to the client in real time. Every line written to +stdout or stderr by the {term}`hook` script is captured and forwarded to the client +through the {term}`exporter`'s log stream. The `beforeLease` {term}`hook` output is tagged with the `BEFORE_LEASE_HOOK` log source, and `afterLease` output is tagged with `AFTER_LEASE_HOOK`. -Hooks run inside a pseudo-terminal (PTY) to force line buffering, so output +{term}`Hook`s run inside a pseudo-terminal (PTY) to force line buffering, so output appears on the client as each line is written rather than being held in a block buffer. This means `echo` statements, `j` CLI output, and any other text written to the terminal will be visible immediately. ```{note} -Because hooks use a PTY, programs that detect terminal mode (such as +Because {term}`hook`s use a PTY, programs that detect terminal mode (such as `grep --color=auto`) will behave as though running interactively. ``` ## Failure Handling The `onFailure` field controls what happens when a hook script exits with a -non-zero exit code or exceeds its timeout. A hook is considered failed when the +non-zero exit code or exceeds its timeout. A {term}`hook` is considered failed when the shell process returns a non-zero exit code or when execution exceeds the configured `timeout`. ### `warn` -The default mode. The failure is logged as a warning and the lease lifecycle -continues as if the hook succeeded: +The default mode. The failure is logged as a warning and the {term}`lease` lifecycle +continues as if the {term}`hook` succeeded: - **`beforeLease`**: Drivers are unblocked and the client can connect normally. The exporter status transitions to `LEASE_READY`. -- **`afterLease`**: The exporter returns to `AVAILABLE` and the lease is +- **`afterLease`**: The {term}`exporter` returns to `AVAILABLE` and the {term}`lease` is released normally. -This is useful for hooks that perform best-effort actions where failure should +This is useful for {term}`hook`s that perform best-effort actions where failure should not disrupt the workflow. ### `endLease` -The lease is ended and the client is notified of the failure: +The {term}`lease` is ended and the client is notified of the failure: - **`beforeLease`**: The exporter status transitions to `BEFORE_LEASE_HOOK_FAILED`. The client discovers the failure through status - polling and the lease is released. The interactive shell is skipped. + polling and the {term}`lease` is released. The interactive shell is skipped. - **`afterLease`**: The exporter status transitions to - `AFTER_LEASE_HOOK_FAILED`. Since the session has already ended, this + `AFTER_LEASE_HOOK_FAILED`. Since the {term}`session` has already ended, this primarily serves as a signal to the client that cleanup did not complete - successfully. The exporter remains available for new leases. + successfully. The {term}`exporter` remains available for new {term}`lease`s. -This is the recommended mode for `beforeLease` validation hooks where you want -the client to know immediately that the device is not ready. +This is the recommended mode for `beforeLease` validation {term}`hook`s where you want +the client to know immediately that the {term}`device` is not ready. ### `exit` -The exporter shuts down entirely with exit code `1` (Failure): +The {term}`exporter` shuts down entirely with exit code `1` (Failure): - **`beforeLease`**: The exporter status transitions to - `BEFORE_LEASE_HOOK_FAILED`. The exporter then shuts down, going offline. The + `BEFORE_LEASE_HOOK_FAILED`. The {term}`exporter` then shuts down, going offline. The shutdown is deferred until the client has had a chance to observe the failure status. - **`afterLease`**: The exporter status transitions to - `AFTER_LEASE_HOOK_FAILED` and the exporter shuts down immediately. + `AFTER_LEASE_HOOK_FAILED` and the {term}`exporter` shuts down immediately. The exit code `1` signals to service managers such as `systemd` that the shutdown -was intentional. If your systemd unit uses `Restart=always`, you should +was intentional. If your `systemd` unit uses `Restart=always`, you should configure `RestartPreventExitStatus=1` to prevent automatic restarts after an `exit` failure. ```{warning} The `exit` failure mode is a drastic action intended for critical failures -where the device may be in an unusable state. It takes the exporter offline +where the {term}`device` may be in an unusable state. It takes the {term}`exporter` offline until manually restarted. Use `endLease` for most validation scenarios and reserve `exit` for critical failures. ``` ### Timeout Behavior -When a hook exceeds its `timeout`, the process is terminated with `SIGTERM` +When a {term}`hook` exceeds its `timeout`, the process is terminated with `SIGTERM` followed by `SIGKILL` if the process does not exit within a few seconds. The resulting failure is then handled according to the `onFailure` setting, exactly as if the script had exited with a non-zero exit code. @@ -246,7 +245,7 @@ as if the script had exited with a non-zero exit code. ### Device Initialization -Power on the device and wait until it is reachable over SSH before the client +Power on the {term}`device` and wait until it is reachable over SSH before the client connects: ```yaml @@ -257,7 +256,7 @@ hooks: j power on echo "Waiting for SSH to become available..." for i in $(seq 1 30); do - if j ssh -o ConnectTimeout=2 -- echo "Device ready"; then + if j ssh -o ConnectTimeout=2 - echo "Device ready"; then exit 0 fi sleep 1 @@ -271,11 +270,11 @@ hooks: Note that the `j ssh` command does not have a built-in connection timeout, so each attempt uses the system SSH default (typically ~30 seconds). Passing `-o ConnectTimeout=2` keeps each retry attempt short so the loop can complete -within the hook's `timeout`. +within the {term}`hook`'s `timeout`. ### Device Cleanup -Power off the device after each lease to ensure a clean environment for the +Power off the {term}`device` after each {term}`lease` to ensure a clean environment for the next user: ```yaml @@ -290,7 +289,7 @@ hooks: ### Firmware Flashing -Flash known-good firmware before each test session to guarantee a consistent +Flash known-good firmware before each test {term}`session` to guarantee a consistent starting state: ```yaml @@ -324,11 +323,11 @@ hooks: ### Using Python -Point `script` to a `.py` file. The exporter auto-detects the `.py` -extension and runs it with its own Python interpreter, so the hook has +Point `script` to a `.py` file. The {term}`exporter` auto-detects the `.py` +extension and runs it with its own Python interpreter, so the {term}`hook` has access to all installed packages including the Jumpstarter client library. -Python hooks can use the driver client APIs directly by importing -`jumpstarter.utils.env.env`, which connects to the local exporter session +Python {term}`hook`s can use the driver client APIs directly by importing +`jumpstarter.utils.env.env`, which connects to the local {term}`exporter` {term}`session` via the `JUMPSTARTER_HOST` socket automatically. Exporter config: @@ -358,7 +357,7 @@ with env() as client: The `env()` context manager returns a `DriverClient` whose attributes correspond to the exported drivers (e.g. `client.power`, `client.storage`). This is the same API used by the `j` CLI and by test scripts connecting to -an exporter. +an {term}`exporter`. ### Using a Script File @@ -376,20 +375,20 @@ hooks: ## Best Practices -- Keep hook scripts short and focused on a single concern (initialization or +- Keep {term}`hook` scripts short and focused on a single concern (initialization or cleanup). -- Set an appropriate `timeout` for each hook. The default of 120 seconds may be +- Set an appropriate `timeout` for each {term}`hook`. The default of 120 seconds may be too generous for simple scripts and too short for firmware flashing. - Use `onFailure: endLease` for `beforeLease` validation so clients get - immediate feedback when a device is not ready. -- Use `onFailure: warn` for `afterLease` cleanup unless leaving the device in a + immediate feedback when a {term}`device` is not ready. +- Use `onFailure: warn` for `afterLease` cleanup unless leaving the {term}`device` in a bad state poses a safety risk. - Reserve `onFailure: exit` for critical failures that require manual intervention. -- Hook output is streamed to the client in real time. Include informative +- {term}`Hook` output is streamed to the client in real time. Include informative `echo` statements for observability. - The interpreter is auto-detected from the file extension (`.py` uses the - exporter's Python, `.sh` uses `/bin/sh`). Set `exec` explicitly to + {term}`exporter`'s Python, `.sh` uses `/bin/sh`). Set `exec` explicitly to override (e.g. `exec: /bin/bash` for bash-specific syntax). -- The `j` CLI is available in hook scripts because `JUMPSTARTER_HOST` is set +- The `j` CLI is available in {term}`hook` scripts because `JUMPSTARTER_HOST` is set automatically. No additional configuration is needed. diff --git a/python/docs/source/introduction/index.md b/python/docs/source/introduction/index.md index 7521bd1a7..60b1a75c8 100644 --- a/python/docs/source/introduction/index.md +++ b/python/docs/source/introduction/index.md @@ -2,66 +2,112 @@ Jumpstarter is an open source framework that brings enterprise-grade testing capabilities to everyone. While established industries like automotive and -manufacturing have long used HiL testing, these tools have typically been +manufacturing have long used {term}`HiL` testing, these tools have typically been expensive proprietary systems. Jumpstarter democratizes this technology through a free, cloud native approach that works with both physical hardware and virtual devices. At its core, Jumpstarter uses a client/server architecture where a single client can control multiple devices under test. Its modular design supports both local -development (devices connected directly to your machine) and distributed testing -environments (devices accessed remotely through a central controller). All -communication happens over gRPC, providing a consistent interface regardless of -deployment model. +development (devices connected directly to your machine) and {term}`distributed mode` +testing environments (devices accessed remotely through a central {term}`controller`). All +communication happens over {term}`gRPC`, providing a consistent interface regardless of +deployment model. Every interface is programmatic - there is no GUI-only +workflow that a script or agent cannot replicate. A human developer running +`jmp shell`, a [pytest](https://docs.pytest.org/en/stable/) script, a CI +pipeline, and an +[AI agent](../getting-started/guides/integration-patterns/agentic.md) +all use the exact same APIs, authentication, and access controls. Built on Python, Jumpstarter integrates easily with existing development workflows and runs almost anywhere. It works with common testing tools like -[pytest](https://docs.pytest.org/en/stable/), shell scripts, Makefiles, and -typical CI/CD systems. Beyond testing, it can function as a virtual KVM -(Keyboard, Video, Mouse) switch, enabling remote access to physical devices for -development. +`pytest`, shell scripts, Makefiles, and typical CI/CD systems. Beyond testing, it +can function as a virtual KVM (Keyboard, Video, Mouse) switch, enabling remote +access to physical devices for development. ## Core Components Jumpstarter architecture is based on the following key components: -- Device Under Test (DUT) - Hardware or virtual device being tested -- [Drivers](drivers.md) - Interfaces for DUT communication -- [Adapters](adapters.md) - Convert driver connections into various formats -- [Exporters](exporters.md) - Expose device interfaces over network via gRPC -- [Hooks](hooks.md) - Lifecycle scripts that run at lease boundaries -- [Clients](clients.md) - Libraries and CLI tools for device interaction -- [Service](service.md) - Kubernetes controller for resource management +- {term}`DUT` - Hardware or virtual {term}`device` being tested +- [Drivers](drivers.md) - Interfaces for {term}`DUT` communication +- [Adapters](adapters.md) - Convert {term}`driver` connections into various formats +- [Exporters](exporters.md) - Expose {term}`device` interfaces over network via {term}`gRPC` +- [Hooks](hooks.md) - Lifecycle scripts that run at {term}`lease` boundaries +- [Clients](clients.md) - Libraries and CLI tools for {term}`device` interaction +- [Service](service.md) - Kubernetes {term}`controller` for resource management Component interactions include: -- **DUT and Drivers** - Drivers provide standardized interfaces to DUT's - hardware connections -- **Drivers and Adapters** - Adapters transform driver connections for - specialized use cases -- **Drivers/Adapters and Exporters** - Exporters manage drivers/adapters and - expose them via gRPC -- **Hooks and Exporters** - Hooks execute shell scripts at lease boundaries, - running before drivers are available and after the session ends -- **Exporters and Clients** - Clients connect to exporters to control devices -- **Clients/Exporters and Service** - Service manages access control and - resource allocation in distributed mode +- **{term}`DUT` and {term}`Driver`s** - {term}`Driver`s provide standardized + interfaces to {term}`DUT`'s hardware connections +- **{term}`Driver`s and {term}`Adapter`s** - {term}`Adapter`s transform + {term}`driver` connections for specialized use cases +- **{term}`Driver`s/{term}`Adapter`s and {term}`Exporter`s** - + {term}`Exporter`s manage {term}`driver`s/{term}`adapter`s and expose them + via {term}`gRPC` +- **{term}`Hook`s and {term}`Exporter`s** - {term}`Hook`s execute shell + scripts at {term}`lease` boundaries, running before {term}`driver`s are + available and after the {term}`session` ends +- **{term}`Exporter`s and {term}`Client`s** - {term}`Client`s connect to + {term}`exporter`s to control {term}`device`s +- **{term}`Client`s/{term}`Exporter`s and {term}`Service`** - + {term}`Service` manages access control and resource allocation in + {term}`distributed mode` Together, these components form a comprehensive testing framework that bridges the gap between development and deployment environments. +```{mermaid} +flowchart TB + subgraph "Kubernetes Cluster" + Controller["Controller\nInventory / Lease / Access Control"] + Router["Router\nNAT Traversal"] + CRDs["CRDs\nExporter, Client, Lease"] + Controller --- CRDs + Controller <--> Router + end + + subgraph "Exporter Host" + Exporter["Exporter"] + subgraph "Drivers" + GPIO["GPIO"] + HDMI["HDMI"] + Serial["Serial"] + Storage["Storage"] + end + Exporter --- GPIO + Exporter --- HDMI + Exporter --- Serial + Exporter --- Storage + end + + DUT["Device Under Test"] + GPIO --> DUT + HDMI --> DUT + Serial --> DUT + Storage --> DUT + + Client["Client\n(CLI / Python API)"] + + Client -- "Distributed\n(gRPC via Router)" --> Router + Router <--> Exporter + Client -. "Direct\n(gRPC via TCP)" .-> Exporter + Client -. "Local\n(gRPC via Socket)" .-> Exporter +``` + ## Operation Modes -Building on these components, Jumpstarter implements two operation modes that -provide flexibility for different scenarios: *local* and *distributed* modes. +Building on these components, Jumpstarter implements three operation modes that +provide flexibility for different scenarios: {term}`local mode`, +{term}`direct mode`, and {term}`distributed mode`. ### Local Mode -In local mode, clients communicate directly with exporters running on the same +In {term}`local mode`, clients communicate directly with {term}`exporter`s running on the same machine or through direct network connections. ```{mermaid} -:config: {"theme":"base","themeVariables":{"primaryColor":"#f8f8f8","primaryTextColor":"#000","primaryBorderColor":"#e5e5e5","lineColor":"#3d94ff","secondaryColor":"#f8f8f8","tertiaryColor":"#fff"}} flowchart TB subgraph "Developer Machine" Client["Client\n(Python Library/CLI)"] @@ -86,8 +132,8 @@ flowchart TB This mode is ideal for individual developers working directly with accessible hardware or virtual devices. When no client configuration or environment -variables are present, Jumpstarter runs in local mode and communicates with a -built-in exporter service via a local socket connection, requiring no Kubernetes +variables are present, Jumpstarter runs in {term}`local mode` and communicates with a +built-in {term}`exporter` service via a local socket connection, requiring no Kubernetes or other infrastructure. Developers can work with devices on their desk, develop drivers, create automation scripts, and test with QEMU or other virtualization tools. @@ -97,22 +143,59 @@ $ jmp shell --exporter my-exporter $ pytest test_device.py ``` -The example above shows typical local mode usage: first connecting to an -exporter (which manages the device interfaces) using the `jmp shell` command, -and then running tests against the device with pytest. The `--exporter` flag +The example above shows typical {term}`local mode` usage: first connecting to an +{term}`exporter` (which manages the {term}`device` interfaces) using the `jmp shell` command, +and then running tests against the device with `pytest`. The `--exporter` flag specifies which exporter configuration to use, allowing you to easily switch -between different hardware or virtual device setups. +between different hardware or virtual {term}`device` setups. + +### Direct Mode + +{term}`Direct mode` connects a client to an {term}`exporter` over TCP without a +{term}`controller` or Kubernetes cluster. This is useful when hardware is on one +machine and the client is on another, but you don't need multi-user +{term}`lease` management. + +```{mermaid} +flowchart LR + subgraph "Client Machine" + Client["Client\n(Python Library/CLI)"] + end + + subgraph "Exporter Machine" + Exporter["Exporter\n(Remote Service)"] + Power["Power"] + Serial["Serial"] + Storage["Storage"] + end + + DUT["Device Under Test"] + + Client <--> |"gRPC via TCP"| Exporter + Exporter --> Power + Exporter --> Serial + Exporter --> Storage + Power --> DUT + Serial --> DUT + Storage --> DUT +``` + +```console +$ jmp shell --exporter example-direct +``` + +Only one client should connect at a time. For shared, multi-user environments +use {term}`distributed mode` instead. ### Distributed Mode -Distributed mode enables multiple teams to securely share hardware resources -across a network. It uses a Kubernetes-based controller to coordinate access to -exporters, managing leases that grant exclusive access to DUT resources, while +{term}`Distributed mode` enables multiple teams to securely share hardware resources +across a network. It uses a Kubernetes-based {term}`controller` to coordinate access to +{term}`exporter`s, managing {term}`lease`s that grant exclusive access to {term}`DUT` resources, while JWT token-based authentication secures all connections between clients and -exporters. +{term}`exporter`s. ```{mermaid} -:config: {"theme":"base","themeVariables":{"primaryColor":"#f8f8f8","primaryTextColor":"#000","primaryBorderColor":"#e5e5e5","lineColor":"#3d94ff","secondaryColor":"#f8f8f8","tertiaryColor":"#fff"}} flowchart TB subgraph "Kubernetes Cluster" Controller["Controller\nResource Management"] @@ -151,33 +234,33 @@ flowchart TB Exporter2 --> DUT3 ``` -Distributed mode is ideal for environments where teams need to share hardware +{term}`Distributed mode` is ideal for environments where teams need to share hardware resources, especially in CI/CD pipelines requiring scheduled device testing. It excels in geographically distributed test environments where devices are spread across multiple locations, and in any scenario requiring centralized management of testing resources. All these scenarios require a robust security model to manage access rights and prevent resource conflicts. -To address these security needs, the distributed mode implements a comprehensive +To address these security needs, the {term}`distributed mode` implements a comprehensive authentication system that secures access through: - **Client Registration** - Clients register in the Kubernetes cluster with unique identities -- **Token Issuance** - Controller issues JWT tokens to authenticated clients and - exporters -- **Secure Communication** - All gRPC communication between components uses +- **Token Issuance** - {term}`Controller` or an OIDC server issues JWT tokens to authenticated clients and + {term}`exporter`s +- **Secure Communication** - All {term}`gRPC` communication between components uses token authentication -- **Access Control** - Controller enforces permissions based on token identity: - - Which exporters a client can lease +- **Access Control** - {term}`Controller` enforces permissions based on token identity: + - Which {term}`exporter`s a client can {term}`lease` - What actions a client can perform - Which driver packages can be loaded -This security model enables dynamic registration of clients and exporters, +This security model enables dynamic registration of clients and {term}`exporter`s, allowing fine-grained access control in multi-user environments. For example, CI -pipelines can be granted access only to specific exporters based on their +pipelines can be granted access only to specific {term}`exporter`s based on their credentials, ensuring proper resource isolation in shared testing environments. -The following example shows how to run tests in distributed mode: +The following example shows how to run tests in {term}`distributed mode`: ```console $ jmp config client use my-client @@ -185,10 +268,10 @@ $ jmp create lease --selector vendor=acme,model=widget-v2 $ pytest test_device.py ``` -The example above demonstrates the distributed mode workflow: first configuring -the client with connection information for the central controller, then -requesting a lease on an exporter that matches specific criteria (using selector -labels), and finally running tests against the acquired DUT. The lease system +The example above demonstrates the {term}`distributed mode` workflow: first configuring +the client with connection information for the central {term}`controller`, then +requesting a {term}`lease` on an {term}`exporter` that matches specific criteria (using +{term}`label selector`s), and finally running tests against the acquired {term}`DUT`. The {term}`lease` system ensures exclusive access to the requested resources for the duration of testing, preventing conflicts with other users or pipelines in the shared environment. diff --git a/python/docs/source/introduction/service.md b/python/docs/source/introduction/service.md index f06588d17..a59f51480 100644 --- a/python/docs/source/introduction/service.md +++ b/python/docs/source/introduction/service.md @@ -1,12 +1,12 @@ # Service -When building a lab with many devices under test, it quickly becomes difficult -to keep track of devices, schedule access for automated tests, and perform +When building a lab with many {term}`DUT`s, it quickly becomes difficult +to keep track of {term}`device`s, schedule access for automated tests, and perform routine maintenance such as batch updates. -Jumpstarter provides a service that can be installed in any +Jumpstarter provides a {term}`service` that can be installed in any [Kubernetes](https://kubernetes.io/) cluster to manage connected clients and -exporters. +{term}`exporter`s. If you're already using a Kubernetes-native CI tool such as [Tekton](https://tekton.dev/), [Jenkins X](https://jenkins-x.io/), @@ -16,23 +16,22 @@ can integrate directly into your existing cloud or on-premises cluster. ## Controller -The core of the Service is the Controller, which manages access to devices, -authenticates clients/exporters, and maintains a set of labels to easily -identify specific devices. +The core of the {term}`service` is the {term}`controller`, which manages access to {term}`device`s, +authenticates clients/{term}`exporter`s, and maintains a set of {term}`label selector`s to easily +identify specific {term}`device`s. -The Controller is implemented as a Kubernetes -[controller](https://github.com/jumpstarter-dev/jumpstarter-controller) using -[Custom Resource Definitions -(CRDs)](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) -to store information about clients, exporters, leases, and other resources. +The {term}`Controller` is implemented as a Kubernetes controller using +{term}`CRD`s to store information about clients, {term}`exporter`s, +{term}`lease`s, and other resources. See the +[CRDs reference](../reference/crds/index.md) for the full field definitions. ### Leases -When a client requests access to an exporter and a matching instance is found, a -Lease is created. The lease ensures that each lessee (client) has exclusive -access to a specific device/exporter. +When a client requests access to an {term}`exporter` and a matching instance is found, a +{term}`lease` is created. The {term}`lease` ensures that each lessee (client) has exclusive +access to a specific {term}`device`/{term}`exporter`. -Clients can be scheduled to access a specific exporter or any exporter that +Clients can be scheduled to access a specific {term}`exporter` or any {term}`exporter` that matches a set of requested labels, similar to [node selection](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector) in Kubernetes. This enables flexible CI-driven testing even when physical @@ -40,11 +39,15 @@ resources are limited. ## Router -The Router service is used by the controller to route messages between clients -and exporters through a gRPC tunnel. This enables remote access to exported -interfaces via the client. +The {term}`router` routes traffic between clients and {term}`exporter`s through a {term}`gRPC` tunnel. +This allows clients to reach {term}`exporter`s without public IP addresses or behind +NATs/firewalls. Clients on the same network can also connect directly to an +{term}`exporter`, bypassing the {term}`router`. -Once a lease is established, all traffic between the client and the exporter -flows through a router instance. While there may only be one controller, the -router can be scaled with multiple instances to handle traffic between many -clients and exporters simultaneously. \ No newline at end of file +Once a {term}`lease` is established, all traffic flows through a {term}`router` instance. While +there may only be one {term}`controller`, the {term}`router` can be scaled with multiple +instances to handle many clients and {term}`exporter`s simultaneously. + +All communication between clients and drivers uses {term}`gRPC` with three RPC styles +(unary, server streaming, and bidirectional streaming). See +[Driver Communication](drivers.md#communication) for details. \ No newline at end of file diff --git a/python/docs/source/reference/crds/index.md b/python/docs/source/reference/crds/index.md new file mode 100644 index 000000000..7b26be867 --- /dev/null +++ b/python/docs/source/reference/crds/index.md @@ -0,0 +1,25 @@ +# CRDs + +This section provides the {term}`CRD` field reference for all Jumpstarter custom +resources. The documentation covers: + +- [Client](client.md): Client identity and credentials +- [Exporter](exporter.md): {term}`Exporter` registration and status +- [ExporterAccessPolicy](exporteraccesspolicy.md): Access control policies for + {term}`exporter`s +- [Lease](lease.md): {term}`Lease` reservations and lifecycle +- [Jumpstarter](jumpstarter.md): {term}`Operator` deployment configuration + +These references are useful for administrators deploying and managing the +Jumpstarter {term}`service`. + +```{toctree} +:maxdepth: 1 +:hidden: + +client.md +exporter.md +exporteraccesspolicy.md +lease.md +jumpstarter.md +``` diff --git a/python/docs/source/reference/generate-crd-docs.py b/python/docs/source/reference/generate-crd-docs.py new file mode 100644 index 000000000..2d6429a73 --- /dev/null +++ b/python/docs/source/reference/generate-crd-docs.py @@ -0,0 +1,130 @@ +#!/usr/bin/env python3 +"""Generate markdown API reference from Kubernetes CRD YAML files.""" + +import glob +import os + +import yaml + +CRD_DIR = os.path.join( + os.path.dirname(__file__), + "../../../../controller/deploy/operator/config/crd/bases", +) +OUTPUT_DIR = os.path.join(os.path.dirname(__file__), "crds") + +SKIP_EXPAND = { + "topologySpreadConstraints", + "resources", + "labelSelector", + "matchExpressions", + "claims", +} + + +def flatten_properties(properties, prefix="", depth=0): + rows = [] + for name, prop in sorted(properties.items()): + path = f"{prefix}{name}" if prefix else name + typ = prop.get("type", "object") + desc = prop.get("description", "").split("\n")[0].strip() + default = prop.get("default") + enum = prop.get("enum") + + type_str = typ + if enum: + type_str = " | ".join(f"`{e}`" for e in enum) + if len(desc) > 120: + desc = desc[:117] + "..." + + if default is not None: + desc += f" (default: `{default}`)" + + rows.append((f"`{path}`", type_str, desc)) + + if name in SKIP_EXPAND: + continue + + if typ == "object" and "properties" in prop and depth < 2: + rows.extend( + flatten_properties(prop["properties"], f"{path}.", depth + 1) + ) + elif typ == "array" and "items" in prop: + items = prop["items"] + if items.get("type") == "object" and "properties" in items and depth < 2: + rows.extend( + flatten_properties(items["properties"], f"{path}[].", depth + 1) + ) + + return rows + + +def render_table(rows): + if not rows: + return "*No fields defined.*\n" + lines = ["| Field | Type | Description |", "| --- | --- | --- |"] + for field, typ, desc in rows: + lines.append(f"| {field} | {typ} | {desc} |") + return "\n".join(lines) + "\n" + + +def process_crd(filepath): + with open(filepath) as f: + crd = yaml.safe_load(f) + + group = crd["spec"]["group"] + kind = crd["spec"]["names"]["kind"] + version = crd["spec"]["versions"][0] + ver = version["name"] + schema = version["schema"]["openAPIV3Schema"] + + sections = [] + sections.append(f"# {kind}\n") + sections.append(f"`{group}/{ver}`\n") + + desc = schema.get("description", "") + if desc: + sections.append(desc.split("\n")[0] + "\n") + + spec = schema.get("properties", {}).get("spec", {}) + if spec.get("properties"): + sections.append("## Spec\n") + rows = flatten_properties(spec["properties"], "spec.") + sections.append(render_table(rows)) + + status = schema.get("properties", {}).get("status", {}) + if status.get("properties"): + sections.append("## Status\n") + rows = flatten_properties(status["properties"], "status.") + sections.append(render_table(rows)) + + return kind, "\n".join(sections) + + +def main(): + crds = sorted(glob.glob(os.path.join(CRD_DIR, "*.yaml"))) + if not crds: + print(f"No CRD files found in {CRD_DIR}") + return + + os.makedirs(OUTPUT_DIR, exist_ok=True) + + toctree_entries = [] + index_entries = [] + + for crd_file in crds: + print(f"Processing {os.path.basename(crd_file)}") + kind, content = process_crd(crd_file) + slug = kind.lower() + filename = f"{slug}.md" + + with open(os.path.join(OUTPUT_DIR, filename), "w") as f: + f.write(content) + + toctree_entries.append(filename) + index_entries.append(f"- [{kind}]({filename})") + + print(f"Generated {len(toctree_entries)} CRD docs in {OUTPUT_DIR}/") + + +if __name__ == "__main__": + main() diff --git a/python/docs/source/reference/index.md b/python/docs/source/reference/index.md index 3832e27cc..b2fa24a88 100644 --- a/python/docs/source/reference/index.md +++ b/python/docs/source/reference/index.md @@ -3,12 +3,10 @@ This section provides reference documentation for Jumpstarter. The documentation covers: -- [API Pages](man-pages/index.md): Command-line tools and utilities - documentation -- [Packages](package-apis/index.md): API documentation for Jumpstarter packages - and components - -These references are useful for developers working with Jumpstarter. +- [MAN Pages](man-pages/index.md): Command-line tools and utilities +- [Package APIs](package-apis/index.md): API documentation for Jumpstarter + packages and components +- [CRDs](crds/index.md): Field reference for all Jumpstarter custom resources ```{toctree} :maxdepth: 1 @@ -16,4 +14,5 @@ These references are useful for developers working with Jumpstarter. man-pages/index.md package-apis/index.md +crds/index.md ``` \ No newline at end of file diff --git a/python/docs/source/reference/man-pages/index.md b/python/docs/source/reference/man-pages/index.md index b55c61e28..3a338e248 100644 --- a/python/docs/source/reference/man-pages/index.md +++ b/python/docs/source/reference/man-pages/index.md @@ -6,8 +6,8 @@ interfaces. The documentation covers: - [`jmp`](jmp.md): Main command-line interface for Jumpstarter - [`j`](j.md): Shorthand utility for quick interactions with Jumpstarter -These references support both local and distributed deployment modes of -Jumpstarter. +These references support all three operation modes of Jumpstarter: local, +direct, and distributed. ```{toctree} :maxdepth: 1 diff --git a/python/docs/source/reference/package-apis/drivers/index.md b/python/docs/source/reference/package-apis/drivers/index.md index 509331d8d..5e3e4fdf9 100644 --- a/python/docs/source/reference/package-apis/drivers/index.md +++ b/python/docs/source/reference/package-apis/drivers/index.md @@ -1,4 +1,4 @@ -# Driver Packages +# Drivers This section documents the drivers from the Jumpstarter packages directory. Each driver is contained in a separate package in the form of @@ -10,112 +10,89 @@ with different hardware components and systems. Jumpstarter includes several types of drivers organized by their primary function: -### System Control Drivers +### System Control Drivers that control the power state and basic operation of devices: -* **[Power](power.md)** (`jumpstarter-driver-power`) - Power control for devices -* **[gpiod](gpiod.md)** (`jumpstarter-driver-gpiod`) - - gpiod hardware control -* **[Yepkit](yepkit.md)** (`jumpstarter-driver-yepkit`) - Yepkit hardware - control -* **[DUT Link](dutlink.md)** (`jumpstarter-driver-dutlink`) - [DUT Link - Board](https://github.com/jumpstarter-dev/dutlink-board) hardware control -* **[Energenie PDU](energenie.md)** (`jumpstarter-driver-energenie`) - Energenie PDUs -* **[Tasmota](tasmota.md)** (`jumpstarter-driver-tasmota`) - Tasmota hardware control -* **[HTTP Power](http-power.md)** (`jumpstarter-driver-http-power`) - HTTP-based power - control, useful for smart sockets, like the Shelly Smart Plug or similar -* **[Noyito Relay](noyito-relay.md)** (`jumpstarter-driver-noyito-relay`) - NOYITO USB relay - board control (1/2-channel serial and 4/8-channel HID variants) - -### Communication Drivers +- **[Power](power.md)** (`jumpstarter-driver-power`) - Power control for devices +- **[gpiod](gpiod.md)** (`jumpstarter-driver-gpiod`) - GPIO hardware control via libgpiod +- **[Yepkit](yepkit.md)** (`jumpstarter-driver-yepkit`) - Yepkit USB hub hardware control +- **[DUT Link](dutlink.md)** (`jumpstarter-driver-dutlink`) - [DUT Link Board](https://github.com/jumpstarter-dev/dutlink-board) hardware control +- **[Energenie PDU](energenie.md)** (`jumpstarter-driver-energenie`) - Energenie PDU control +- **[Tasmota](tasmota.md)** (`jumpstarter-driver-tasmota`) - Tasmota device control +- **[HTTP Power](http-power.md)** (`jumpstarter-driver-http-power`) - HTTP-based power control for smart sockets +- **[Noyito Relay](noyito-relay.md)** (`jumpstarter-driver-noyito-relay`) - NOYITO USB relay board control + +### Communication Drivers that provide various communication interfaces: -* **[ADB](adb.md)** (`jumpstarter-driver-adb`) - Android Debug Bridge tunneling - for remote Android device access -* **[BLE](ble.md)** (`jumpstarter-driver-ble`) - Bluetooth Low Energy communication -* **[CAN](can.md)** (`jumpstarter-driver-can`) - Controller Area Network - communication -* **[HTTP](http.md)** (`jumpstarter-driver-http`) - HTTP communication -* **[Mitmproxy](mitmproxy.md)** (`jumpstarter-driver-mitmproxy`) - HTTP(S) interception, mocking, and traffic recording -* **[DUT Network](dut-network.md)** (`jumpstarter-driver-dut-network`) - DUT network - isolation with bridge, DHCP, DNS, and NAT -* **[Network](network.md)** (`jumpstarter-driver-network`) - Network interfaces - and configuration -* **[PySerial](pyserial.md)** (`jumpstarter-driver-pyserial`) - Serial port - communication -* **[SNMP](snmp.md)** (`jumpstarter-driver-snmp`) - Simple Network Management - Protocol -* **[TFTP](tftp.md)** (`jumpstarter-driver-tftp`) - Trivial File Transfer - Protocol -* **[VNC](vnc.md)** (`jumpstarter-driver-vnc`) - VNC (Virtual Network Computing) remote desktop protocol -* **[XCP](xcp.md)** (`jumpstarter-driver-xcp`) - Universal Measurement and - Calibration Protocol communication - -### Storage and Data Drivers +- **[ADB](adb.md)** (`jumpstarter-driver-adb`) - Android Debug Bridge tunneling +- **[BLE](ble.md)** (`jumpstarter-driver-ble`) - Bluetooth Low Energy communication +- **[CAN](can.md)** (`jumpstarter-driver-can`) - Controller Area Network communication +- **[HTTP](http.md)** (`jumpstarter-driver-http`) - HTTP communication +- **[mitmproxy](mitmproxy.md)** (`jumpstarter-driver-mitmproxy`) - HTTP/HTTPS interception, mocking, and traffic recording +- **[DUT Network](dut-network.md)** (`jumpstarter-driver-dut-network`) - DUT network isolation with bridge, DHCP, DNS, and NAT +- **[Network](network.md)** (`jumpstarter-driver-network`) - Network interfaces and configuration +- **[PySerial](pyserial.md)** (`jumpstarter-driver-pyserial`) - Serial port communication +- **[SNMP](snmp.md)** (`jumpstarter-driver-snmp`) - Simple Network Management Protocol +- **[SSH](ssh.md)** (`jumpstarter-driver-ssh`) - SSH wrapper driver +- **[SSH MITM](ssh-mitm.md)** (`jumpstarter-driver-ssh-mitm`) - SSH proxy with server-side private key storage +- **[TFTP](tftp.md)** (`jumpstarter-driver-tftp`) - Trivial File Transfer Protocol +- **[VNC](vnc.md)** (`jumpstarter-driver-vnc`) - Virtual Network Computing remote desktop +- **[XCP](xcp.md)** (`jumpstarter-driver-xcp`) - Universal Measurement and Calibration Protocol + +### Storage and Data Drivers that control storage devices and manage data: -* **[OpenDAL](opendal.md)** (`jumpstarter-driver-opendal`) - Open Data Access - Layer -* **[SD Wire](sdwire.md)** (`jumpstarter-driver-sdwire`) - SD card switching - utilities -* **[iSCSI](iscsi.md)** (`jumpstarter-driver-iscsi`) - iSCSI server to serve LUNs +- **[OpenDAL](opendal.md)** (`jumpstarter-driver-opendal`) - Open Data Access Layer +- **[SD Wire](sdwire.md)** (`jumpstarter-driver-sdwire`) - SD card switching +- **[iSCSI](iscsi.md)** (`jumpstarter-driver-iscsi`) - iSCSI target server for LUN export -### Media Drivers +### Media Drivers that handle media streams: -* **[UStreamer](ustreamer.md)** (`jumpstarter-driver-ustreamer`) - Video - streaming functionality +- **[uStreamer](ustreamer.md)** (`jumpstarter-driver-ustreamer`) - Video streaming -### Automotive Diagnostics Drivers +### Automotive Diagnostics Drivers for automotive diagnostic protocols: -* **[DoIP](doip.md)** (`jumpstarter-driver-doip`) - Raw Diagnostics over Internet - Protocol (ISO-13400) -* **[UDS](uds.md)** (`jumpstarter-driver-uds`) - Shared UDS interface and models - (ISO-14229) -* **[UDS over DoIP](uds-doip.md)** (`jumpstarter-driver-uds-doip`) - UDS - diagnostics over DoIP transport -* **[UDS over CAN](uds-can.md)** (`jumpstarter-driver-uds-can`) - UDS - diagnostics over CAN/ISO-TP transport -* **[SOME/IP](someip.md)** (`jumpstarter-driver-someip`) - SOME/IP protocol - operations (RPC, service discovery, events) via opensomeip - -### Debug and Programming Drivers - -Drivers for debugging and programming devices: - -* **[ESP32](esp32.md)** (`jumpstarter-driver-esp32`) - ESP32 flashing and - management via esptool -* **[Flashers](flashers.md)** (`jumpstarter-driver-flashers`) - Flash memory - programming tools -* **[Pi Pico](pi-pico.md)** (`jumpstarter-driver-pi-pico`) - Raspberry Pi Pico - UF2 flashing via BOOTSEL mass storage -* **[Probe-RS](probe-rs.md)** (`jumpstarter-driver-probe-rs`) - Debugging probe - support -* **[ST-LINK MSD](stlink-msd.md)** (`jumpstarter-driver-stlink-msd`) - ST-LINK - mass storage flasher for STM32 Nucleo and Discovery boards -* **[Android Emulator](androidemulator.md)** (`jumpstarter-driver-androidemulator`) - - Android emulator lifecycle management with ADB tunneling -* **[QEMU](qemu.md)** (`jumpstarter-driver-qemu`) - QEMU virtualization platform -* **[Renode](renode.md)** (`jumpstarter-driver-renode`) - Renode embedded systems emulation -* **[Corellium](corellium.md)** (`jumpstarter-driver-corellium`) - Corellium - virtualization platform -* **[U-Boot](uboot.md)** (`jumpstarter-driver-uboot`) - Universal Bootloader - interface -* **[RideSX](ridesx.md)** (`jumpstarter-driver-ridesx`) - Flashing and power management for Qualcomm RideSX devices - -### Utility Drivers +- **[DoIP](doip.md)** (`jumpstarter-driver-doip`) - Diagnostics over Internet Protocol (ISO 13400) +- **[UDS](uds.md)** (`jumpstarter-driver-uds`) - Unified Diagnostic Services (ISO 14229) +- **[UDS over DoIP](uds-doip.md)** (`jumpstarter-driver-uds-doip`) - UDS diagnostics over DoIP transport +- **[UDS over CAN](uds-can.md)** (`jumpstarter-driver-uds-can`) - UDS diagnostics over CAN/ISO-TP transport +- **[SOME/IP](someip.md)** (`jumpstarter-driver-someip`) - SOME/IP protocol operations via opensomeip + +### Flashing and Programming + +Drivers for flashing firmware and programming devices: + +- **[ESP32](esp32.md)** (`jumpstarter-driver-esp32`) - ESP32 flashing via esptool +- **[Flashers](flashers.md)** (`jumpstarter-driver-flashers`) - Flash memory programming tools +- **[Pi Pico](pi-pico.md)** (`jumpstarter-driver-pi-pico`) - Raspberry Pi Pico UF2 flashing via BOOTSEL +- **[Probe-RS](probe-rs.md)** (`jumpstarter-driver-probe-rs`) - Debug probe support +- **[ST-LINK MSD](stlink-msd.md)** (`jumpstarter-driver-stlink-msd`) - ST-LINK mass storage flasher for STM32 +- **[U-Boot](uboot.md)** (`jumpstarter-driver-uboot`) - Universal Bootloader interface +- **[RideSX](ridesx.md)** (`jumpstarter-driver-ridesx`) - Flashing and power management for Qualcomm RideSX + +### Emulation + +Drivers for virtual and emulated targets: + +- **[Android Emulator](androidemulator.md)** (`jumpstarter-driver-androidemulator`) - Android emulator lifecycle management with ADB tunneling +- **[QEMU](qemu.md)** (`jumpstarter-driver-qemu`) - QEMU virtual machine management +- **[Renode](renode.md)** (`jumpstarter-driver-renode`) - Renode embedded systems emulation +- **[Corellium](corellium.md)** (`jumpstarter-driver-corellium`) - Corellium virtualization platform + +### Utility General-purpose utility drivers: -* **[Shell](shell.md)** (`jumpstarter-driver-shell`) - Shell command execution -* **[TMT](tmt.md)** (`jumpstarter-driver-tmt`) - TMT (Test Management Tool) wrapper driver -* **[SSH](ssh.md)** (`jumpstarter-driver-ssh`) - SSH wrapper driver +- **[Shell](shell.md)** (`jumpstarter-driver-shell`) - Shell command execution +- **[TMT](tmt.md)** (`jumpstarter-driver-tmt`) - Test Management Tool wrapper ```{toctree} :hidden: @@ -130,6 +107,7 @@ dutlink.md energenie.md esp32.md flashers.md +gpiod.md http.md http-power.md iscsi.md @@ -143,17 +121,17 @@ probe-rs.md pyserial.md qemu.md renode.md -gpiod.md ridesx.md sdwire.md shell.md -ssh.md snmp.md someip.md +ssh.md +ssh-mitm.md stlink-msd.md tasmota.md -tmt.md tftp.md +tmt.md uboot.md uds.md uds-can.md diff --git a/python/docs/source/reference/package-apis/drivers/iscsi.md b/python/docs/source/reference/package-apis/drivers/iscsi.md deleted file mode 100644 index e40b1780f..000000000 --- a/python/docs/source/reference/package-apis/drivers/iscsi.md +++ /dev/null @@ -1,62 +0,0 @@ -# iSCSI server driver - -`jumpstarter-driver-iscsi` provides a lightweight iSCSI **target** implementation powered by the Linux -[RFC-tgt](https://github.com/open-iscsi/tcmu-runner/) framework via the -[`rtslib-fb`](https://github.com/open-iscsi/rtslib-fb) Python bindings. - -> ⚠️ The driver **creates and manages an iSCSI _target_** (server). To access the -> exported LUNs you still need a separate iSCSI **initiator** (client) on the -> machine running your test-code / DUT. - ---- - -## Installation - -`rtslib-fb` relies on the in-kernel LIO target framework which is packaged -differently by each distribution. **You should be able to run `sudo targetcli` -without errors before you start the Jumpstarter driver.** - -Fedora: - -```{code-block} console -$ sudo dnf install targetcli python3-rtslib -``` - -Finally, install the driver itself from the Jumpstarter package index: - -```{code-block} console -:substitutions: -$ pip3 install --extra-index-url {{index_url}} jumpstarter-driver-iscsi -``` - -## Configuration - -The driver is configured through the exporter YAML file. A minimal example -exports the local file `disk.img` as a 5 GiB LUN: - -```yaml -export: - iscsi: - type: jumpstarter_driver_iscsi.driver.ISCSI - config: - root_dir: "/var/lib/iscsi" - target_name: "demo" - # When size_mb is 0 a pre-existing file size is used. -``` - -### Config parameters - -| Parameter | Description | Type | Required | Default | -| ----------- | ---------------------------------------------------------------------------- | ---- | -------- | --------------------------------- | -| `root_dir` | Directory where image files will be stored. | str | no | `/var/lib/iscsi` | -| `iqn_prefix`| IQN prefix to use when building the target IQN. | str | no | `iqn.2024-06.dev.jumpstarter` | -| `target_name`| The target name appended to the IQN prefix. | str | no | `target1` | -| `host` | IP address to bind the target to. Empty string will auto-detect default IP. | str | no | *auto* | -| `port` | TCP port the target listens on. | int | no | `3260` | - -## API Reference - -```{eval-rst} -.. autoclass:: jumpstarter_driver_iscsi.client.ISCSIServerClient() - :members: start, stop, get_host, get_port, get_target_iqn, add_lun, remove_lun, list_luns, upload_image -``` diff --git a/python/docs/source/reference/package-apis/drivers/iscsi.md b/python/docs/source/reference/package-apis/drivers/iscsi.md new file mode 120000 index 000000000..15e55a4ff --- /dev/null +++ b/python/docs/source/reference/package-apis/drivers/iscsi.md @@ -0,0 +1 @@ +../../../../../packages/jumpstarter-driver-iscsi/README.md \ No newline at end of file diff --git a/python/docs/source/reference/package-apis/drivers/ssh-mitm.md b/python/docs/source/reference/package-apis/drivers/ssh-mitm.md new file mode 120000 index 000000000..5c33b2c34 --- /dev/null +++ b/python/docs/source/reference/package-apis/drivers/ssh-mitm.md @@ -0,0 +1 @@ +../../../../../packages/jumpstarter-driver-ssh-mitm/README.md \ No newline at end of file diff --git a/python/docs/source/reference/package-apis/index.md b/python/docs/source/reference/package-apis/index.md index e1096f1ae..26c946b33 100644 --- a/python/docs/source/reference/package-apis/index.md +++ b/python/docs/source/reference/package-apis/index.md @@ -1,15 +1,12 @@ # Package APIs -This section provides reference documentation for Jumpstarter's package APIs and -components. The documentation covers: +This section provides reference documentation for Jumpstarter's package APIs. +The documentation covers: - [Drivers](drivers/index.md): APIs for various driver categories -- [MCP Server](mcp.md): AI agent integration via Model Context Protocol +- [MCP](mcp.md): {term}`MCP` server package API - [Exceptions](exceptions.md): Exceptions raised by driver clients -These references are useful for developers extending Jumpstarter or integrating -with custom hardware. - ```{toctree} :maxdepth: 1 :hidden: diff --git a/python/examples/android-emulator/jumpstarter_example_android_emulator/test_android_emulator.py b/python/examples/android-emulator/jumpstarter_example_android_emulator/test_android_emulator.py index 95260881e..c4f076e53 100644 --- a/python/examples/android-emulator/jumpstarter_example_android_emulator/test_android_emulator.py +++ b/python/examples/android-emulator/jumpstarter_example_android_emulator/test_android_emulator.py @@ -2,7 +2,7 @@ These tests demonstrate interacting with an Android device through the Jumpstarter ADB tunnel using the adbutils Python API. No APK -is required — all tests use built-in Android capabilities. +is required - all tests use built-in Android capabilities. """ import os diff --git a/python/examples/automotive/README.md b/python/examples/automotive/README.md index 41c51f190..042e34740 100644 --- a/python/examples/automotive/README.md +++ b/python/examples/automotive/README.md @@ -8,15 +8,15 @@ Unit) diagnostics using UDS (Unified Diagnostic Services) over DoIP A **stateful mock ECU** simulates a realistic diagnostic target with: -- **Session management** -- default, extended, and programming sessions with +- **Session management** - default, extended, and programming sessions with enforced preconditions (e.g., DID writes require extended session) -- **Security access** -- seed/key challenge-response gating privileged +- **Security access** - seed/key challenge-response gating privileged operations -- **DID store** -- readable/writable Data Identifiers (VIN, part number, +- **DID store** - readable/writable Data Identifiers (VIN, part number, software version, supplier ID) -- **DTC memory** -- pre-populated Diagnostic Trouble Codes that can be read +- **DTC memory** - pre-populated Diagnostic Trouble Codes that can be read and cleared, restored on ECU reset -- **Negative responses** -- proper NRC codes when preconditions are violated +- **Negative responses** - proper NRC codes when preconditions are violated The test exercises a complete diagnostic workflow through the full Jumpstarter pipeline (driver -> gRPC -> client), validating the end-to-end use case. @@ -47,10 +47,10 @@ export: request_timeout: 5 ``` -The test code using Jumpstarter's client API would remain unchanged -- only the +The test code using Jumpstarter's client API would remain unchanged - only the exporter configuration changes between mock and real hardware. ## Drivers used -- **jumpstarter-driver-uds-doip** -- UDS over DoIP transport -- **jumpstarter-driver-uds** -- UDS service interface (base) +- **jumpstarter-driver-uds-doip** - UDS over DoIP transport +- **jumpstarter-driver-uds** - UDS service interface (base) diff --git a/python/examples/automotive/jumpstarter_example_automotive/mock_ecu.py b/python/examples/automotive/jumpstarter_example_automotive/mock_ecu.py index 6e711a3bd..c711c79a2 100644 --- a/python/examples/automotive/jumpstarter_example_automotive/mock_ecu.py +++ b/python/examples/automotive/jumpstarter_example_automotive/mock_ecu.py @@ -232,7 +232,7 @@ def _dispatch_doip( return [] - # -- UDS stateful engine -------------------------------------------------- + # - UDS stateful engine -------------------------------------------------- def _handle_uds(self, data: bytes) -> bytes: if not data: @@ -335,7 +335,7 @@ def _uds_read_dtc_info(self, data: bytes) -> bytes: result += struct.pack(">I", dtc_id)[1:] + bytes([status]) return result - # -- RoutineControl (0x31) ------------------------------------------------ + # - RoutineControl (0x31) ------------------------------------------------ def _uds_routine_control(self, data: bytes) -> bytes: if len(data) < 4: @@ -367,7 +367,7 @@ def _uds_routine_control(self, data: bytes) -> bytes: return bytes([data[0] + 0x40, control_type, data[2], data[3]]) + status_record - # -- Authentication (0x29) ------------------------------------------------ + # - Authentication (0x29) ------------------------------------------------ def _uds_authentication(self, data: bytes) -> bytes: if len(data) < 2: @@ -424,7 +424,7 @@ def _uds_authentication(self, data: bytes) -> bytes: return _nrc(data[0], NRC_REQUEST_OUT_OF_RANGE) - # -- RequestFileTransfer (0x38) ------------------------------------------- + # - RequestFileTransfer (0x38) ------------------------------------------- def _uds_file_transfer(self, data: bytes) -> bytes: if len(data) < 4: diff --git a/python/examples/automotive/jumpstarter_example_automotive/test_diagnostic_flow.py b/python/examples/automotive/jumpstarter_example_automotive/test_diagnostic_flow.py index dccbba08c..f26dc18a4 100644 --- a/python/examples/automotive/jumpstarter_example_automotive/test_diagnostic_flow.py +++ b/python/examples/automotive/jumpstarter_example_automotive/test_diagnostic_flow.py @@ -1,7 +1,7 @@ """Representative end-to-end diagnostic test using jumpstarter. Demonstrates a realistic ECU diagnostic workflow: session management, -DID read/write, DTC handling, security access, and ECU reset -- all +DID read/write, DTC handling, security access, and ECU reset - all running through the full jumpstarter driver/gRPC/client pipeline against a stateful mock ECU. """ @@ -37,7 +37,7 @@ def test_full_diagnostic_workflow(ecu_client): assert resp.success is True assert resp.service == "DiagnosticSessionControl" - # 4. Read DTCs -- mock ECU has pre-populated faults + # 4. Read DTCs - mock ECU has pre-populated faults dtcs = ecu_client.read_dtc_by_status_mask(0xFF) assert len(dtcs) == len(INITIAL_DTCS) initial_ids = {dtc_id for dtc_id, _ in INITIAL_DTCS} @@ -144,7 +144,7 @@ def test_security_access_in_default_session(ecu_client): assert seed_resp.nrc == 0x22 -# -- RoutineControl tests ---------------------------------------------------- +# - RoutineControl tests ---------------------------------------------------- def test_routine_start_stop_result(ecu_client): @@ -185,7 +185,7 @@ def test_routine_unknown_id_rejected(ecu_client): assert resp.nrc == 0x31 -# -- Authentication tests ---------------------------------------------------- +# - Authentication tests ---------------------------------------------------- ALGO_INDICATOR = bytes(16) @@ -245,7 +245,7 @@ def test_deauthenticate(ecu_client): assert resp.success is True -# -- RequestFileTransfer tests ----------------------------------------------- +# - RequestFileTransfer tests ----------------------------------------------- def test_file_transfer_read_file(ecu_client): diff --git a/python/examples/soc-pytest/README.md b/python/examples/soc-pytest/README.md index 1fca5b5d2..9b01d8b7f 100644 --- a/python/examples/soc-pytest/README.md +++ b/python/examples/soc-pytest/README.md @@ -18,7 +18,7 @@ This example requires the following hardware: 1) Setup an environment with the required hardware, and customize the exporter.yaml -2) Setup the exporter to be run from a container (TODO: link) +2) Setup the exporter to be run from a container (see [Exporter Installation](https://jumpstarter.dev/main/getting-started/installation/index.html)) 3) Label the exporter in k8s with the `board=rpi4` label 4) Prepare the images by running `make` in the `image` directory 5) Run the tests in this directory by running: @@ -26,7 +26,7 @@ This example requires the following hardware: $ cd jumpstarter_example_soc_pytest $ uv run pytest -s ================================================================== test session starts =================================================================== -platform linux -- Python 3.12.3, pytest-8.3.3, pluggy-1.5.0 +platform linux - Python 3.12.3, pytest-8.3.3, pluggy-1.5.0 rootdir: /home/majopela/jumpstarter/examples/soc-pytest configfile: pyproject.toml plugins: anyio-4.6.2.post1, cov-5.0.0 diff --git a/python/examples/xcp-ecu/jumpstarter_example_xcp_ecu/mock_ecu.py b/python/examples/xcp-ecu/jumpstarter_example_xcp_ecu/mock_ecu.py index a027eda70..c0e609510 100644 --- a/python/examples/xcp-ecu/jumpstarter_example_xcp_ecu/mock_ecu.py +++ b/python/examples/xcp-ecu/jumpstarter_example_xcp_ecu/mock_ecu.py @@ -16,18 +16,18 @@ # Pre-populated calibration parameters (address -> value) # All byte values kept in 0x00-0x7F range (UTF-8 safe for gRPC transport) CALIBRATION_MAP: dict[int, bytes] = { - 0x0010_0000: b"\x64\x00\x00\x00", # int32 100 – max engine RPM scale - 0x0010_0004: b"\x32\x00\x00\x00", # int32 50 – idle RPM target - 0x0010_0008: b"\x0A\x00\x00\x00", # int32 10 – fuel trim % - 0x0010_000C: b"\x01", # bool True – traction control enabled + 0x0010_0000: b"\x64\x00\x00\x00", # int32 100 - max engine RPM scale + 0x0010_0004: b"\x32\x00\x00\x00", # int32 50 - idle RPM target + 0x0010_0008: b"\x0A\x00\x00\x00", # int32 10 - fuel trim % + 0x0010_000C: b"\x01", # bool True - traction control enabled } # Pre-populated measurement signals (address -> initial value) MEASUREMENT_MAP: dict[int, bytes] = { - 0x0020_0000: b"\x5A\x00\x00\x00", # int32 90 – coolant temperature (C) - 0x0020_0004: b"\x00\x00\x00\x00", # int32 0 – vehicle speed (km/h) - 0x0020_0008: b"\x37\x00\x00\x00", # int32 55 – battery voltage (x10) - 0x0020_000C: b"\x03\x04\x00\x00", # int32 1027 – engine RPM + 0x0020_0000: b"\x5A\x00\x00\x00", # int32 90 - coolant temperature (C) + 0x0020_0004: b"\x00\x00\x00\x00", # int32 0 - vehicle speed (km/h) + 0x0020_0008: b"\x37\x00\x00\x00", # int32 55 - battery voltage (x10) + 0x0020_000C: b"\x03\x04\x00\x00", # int32 1027 - engine RPM } # Flash region: uses 0x00 as erased state (UTF-8 safe, unlike 0xFF) @@ -130,13 +130,13 @@ def __init__(self) -> None: def _require_connected(self): if not self._connected: - raise RuntimeError("Not connected – call connect() first") + raise RuntimeError("Not connected - call connect() first") def _require_unlocked(self): if not self._unlocked: - raise RuntimeError("Resource protected – unlock required") + raise RuntimeError("Resource protected - unlock required") - # -- Session Management -------------------------------------------------- + # - Session Management -------------------------------------------------- def connect(self, mode: int = 0): self._connected = True @@ -157,7 +157,7 @@ def getStatus(self): def getCurrentProtectionStatus(self) -> dict[str, bool]: return dict(self._protection) - # -- Security (Seed & Key) ----------------------------------------------- + # - Security (Seed & Key) ----------------------------------------------- def cond_unlock(self, resources=None): self._require_connected() @@ -165,7 +165,7 @@ def cond_unlock(self, resources=None): for key in self._protection: self._protection[key] = False - # -- Memory Access ------------------------------------------------------- + # - Memory Access ------------------------------------------------------- def setMta(self, address: int, ext: int = 0): self._require_connected() @@ -190,10 +190,10 @@ def shortUpload(self, length: int, address: int, ext: int = 0) -> bytes: def download(self, data: bytes): self._require_connected() if self._protection.get("calpag", False): - raise RuntimeError("CAL/PAG resource is protected – unlock first") + raise RuntimeError("CAL/PAG resource is protected - unlock first") self._memory[self._mta_address] = data - # -- Checksum ------------------------------------------------------------ + # - Checksum ------------------------------------------------------------ def buildChecksum(self, block_size: int): self._require_connected() @@ -201,7 +201,7 @@ def buildChecksum(self, block_size: int): csum = sum(raw) & 0xFFFFFFFF return _AttrDict(checksumType=0x01, checksum=csum) - # -- DAQ ----------------------------------------------------------------- + # - DAQ ----------------------------------------------------------------- def getDaqInfo(self): self._require_connected() @@ -265,12 +265,12 @@ def startStopSynch(self, mode: int): for dl in self._daq_lists: dl.running = (mode == 1) - # -- Programming (Flashing) ---------------------------------------------- + # - Programming (Flashing) ---------------------------------------------- def programStart(self): self._require_connected() if self._protection.get("pgm", False): - raise RuntimeError("PGM resource is protected – unlock first") + raise RuntimeError("PGM resource is protected - unlock first") self._programming = True self._program_cleared = False return _AttrDict( diff --git a/python/examples/xcp-ecu/jumpstarter_example_xcp_ecu/test_xcp_flow.py b/python/examples/xcp-ecu/jumpstarter_example_xcp_ecu/test_xcp_flow.py index 4bcba441b..92330a9d1 100644 --- a/python/examples/xcp-ecu/jumpstarter_example_xcp_ecu/test_xcp_flow.py +++ b/python/examples/xcp-ecu/jumpstarter_example_xcp_ecu/test_xcp_flow.py @@ -1,7 +1,7 @@ """Representative end-to-end XCP ECU tests using jumpstarter. Demonstrates realistic XCP workflows: connection, measurement, calibration, -DAQ configuration, and flash programming -- all running through the full +DAQ configuration, and flash programming - all running through the full jumpstarter driver/gRPC/client pipeline against a stateful mock ECU. """ @@ -25,7 +25,7 @@ def _to_bytes(data) -> bytes: return bytes(data, "latin-1") if isinstance(data, str) else data -# -- Full workflow tests ------------------------------------------------------- +# - Full workflow tests ------------------------------------------------------- def test_full_measurement_and_calibration_workflow(ecu_client, mock_ecu): @@ -56,7 +56,7 @@ def test_full_measurement_and_calibration_workflow(ecu_client, mock_ecu): coolant_temp = struct.unpack(" adb devices ``` -### Android Studio +#### Android Studio Android Studio automatically starts and maintains its own ADB server on port 5037. Because of this, the `tunnel` command uses an auto-assigned port @@ -142,7 +142,7 @@ j adb tunnel -P 5037 # causing a conflict. If this happens, use the auto-assigned port instead. ``` -### Trade Federation (tradefed) +#### Trade Federation (tradefed) tradefed discovers devices through the ADB server via the `ANDROID_ADB_SERVER_PORT` environment variable: @@ -158,7 +158,7 @@ tradefed.sh # > list devices <-- shows remote devices ``` -### Python API +#### Python API You can also perform interactions via ADB using the [`adbutils`](https://github.com/openatx/adbutils) Python package. @@ -173,9 +173,9 @@ with client.adb.forward_adb(port=0) as (host, port): print(device.serial, device.prop.model) ``` -## CLI Reference +### CLI -### Standard ADB commands (passed through) +#### Standard ADB commands (passed through) | Usage | Description | | ----------------------------- | ------------------------------------------------- | @@ -187,13 +187,13 @@ with client.adb.forward_adb(port=0) as (host, port): | `j adb pull ` | Pull a file from the device | | `j adb logcat` | View device logs | -### Jumpstarter-specific commands +#### Jumpstarter-specific commands | Usage | Description | | ------------------------ | ----------------------------------------------------------------------- | | `j adb tunnel [-P PORT]` | Create a persistent ADB tunnel (auto-assigned port, or specify with -P) | -### Options +#### Options | Option | Description | Default | | ------------ | ------------------------------------ | --------- | diff --git a/python/packages/jumpstarter-driver-adb/jumpstarter_driver_adb/client.py b/python/packages/jumpstarter-driver-adb/jumpstarter_driver_adb/client.py index 7775d3bf3..0b7a02da9 100644 --- a/python/packages/jumpstarter-driver-adb/jumpstarter_driver_adb/client.py +++ b/python/packages/jumpstarter-driver-adb/jumpstarter_driver_adb/client.py @@ -193,7 +193,7 @@ def adb(host: str, port: int, adb: str, args: tuple[str, ...]): ) return process.wait() - # No persistent tunnel — create an ephemeral one + # No persistent tunnel - create an ephemeral one with self.forward_adb(host, port) as addr: env = os.environ | { "ANDROID_ADB_SERVER_ADDRESS": addr[0], diff --git a/python/packages/jumpstarter-driver-androidemulator/README.md b/python/packages/jumpstarter-driver-androidemulator/README.md index 45898ba71..56a3b4215 100644 --- a/python/packages/jumpstarter-driver-androidemulator/README.md +++ b/python/packages/jumpstarter-driver-androidemulator/README.md @@ -16,6 +16,24 @@ For the optional Python ADB API: pip3 install --extra-index-url https://pkg.jumpstarter.dev/simple/ "jumpstarter-driver-androidemulator[python-api]" ``` +### Prerequisites + +- Android SDK with emulator and platform-tools installed +- `emulator` and `adb` available on PATH (or specify `emulator_path`) +- An AVD created via Android Studio or `avdmanager` + +#### Quick AVD Setup + +```bash +# Apple Silicon (arm64) +sdkmanager "system-images;android-35;google_apis;arm64-v8a" +avdmanager create avd -n Pixel_6 -k "system-images;android-35;google_apis;arm64-v8a" -d pixel_6 + +# Intel/AMD (x86_64) +sdkmanager "system-images;android-35;google_apis;x86_64" +avdmanager create avd -n Pixel_6 -k "system-images;android-35;google_apis;x86_64" -d pixel_6 +``` + ## Configuration Example exporter configuration: @@ -41,19 +59,6 @@ export: | console_port | Emulator console port | int | no | 5554 | | adb_server_port | Port for the custom ADB server | int | no | 15037 | -## Architecture - -This is a composite driver with two children: - -- **`adb`** (`AdbServer` from `jumpstarter-driver-adb`): Manages the ADB server - and provides TCP tunneling for remote ADB access -- **`power`** (`AndroidEmulatorPower`): Controls the emulator process lifecycle - via the standard `PowerInterface` (on/off/read) - -The emulator registers with the custom ADB server on port 15037 (via the -`ANDROID_ADB_SERVER_PORT` environment variable) to avoid conflicts with any -local ADB server on the standard port 5037. - ## Usage ### CLI @@ -93,23 +98,18 @@ with serve(driver) as client: client.power.off() ``` -## Prerequisites - -- Android SDK with emulator and platform-tools installed -- `emulator` and `adb` available on PATH (or specify `emulator_path`) -- An AVD created via Android Studio or `avdmanager` +## Architecture -### Quick AVD Setup +This is a composite driver with two children: -```bash -# Apple Silicon (arm64) -sdkmanager "system-images;android-35;google_apis;arm64-v8a" -avdmanager create avd -n Pixel_6 -k "system-images;android-35;google_apis;arm64-v8a" -d pixel_6 +- **`adb`** (`AdbServer` from `jumpstarter-driver-adb`): Manages the ADB server + and provides TCP tunneling for remote ADB access +- **`power`** (`AndroidEmulatorPower`): Controls the emulator process lifecycle + via the standard `PowerInterface` (on/off/read) -# Intel/AMD (x86_64) -sdkmanager "system-images;android-35;google_apis;x86_64" -avdmanager create avd -n Pixel_6 -k "system-images;android-35;google_apis;x86_64" -d pixel_6 -``` +The emulator registers with the custom ADB server on port 15037 (via the +`ANDROID_ADB_SERVER_PORT` environment variable) to avoid conflicts with any +local ADB server on the standard port 5037. ## API Reference diff --git a/python/packages/jumpstarter-driver-ble/README.md b/python/packages/jumpstarter-driver-ble/README.md index e18d43da9..05e12f21c 100644 --- a/python/packages/jumpstarter-driver-ble/README.md +++ b/python/packages/jumpstarter-driver-ble/README.md @@ -1,4 +1,4 @@ -# Bluetooth Low Energy (BLE) driver +# BLE Driver `jumpstarter-driver-ble` provides communication functionality via ble with the DUT. The driver expects a ble service with a write and notify characteristic to send and receive data respectively. diff --git a/python/packages/jumpstarter-driver-can/README.md b/python/packages/jumpstarter-driver-can/README.md index 055f6d749..067f5134e 100644 --- a/python/packages/jumpstarter-driver-can/README.md +++ b/python/packages/jumpstarter-driver-can/README.md @@ -1,4 +1,4 @@ -# CAN driver +# CAN Driver `jumpstarter-driver-can` provides functionality for interacting with CAN bus connections based on the [python-can](https://python-can.readthedocs.io/en/stable/index.html) @@ -11,14 +11,14 @@ library. $ pip3 install --extra-index-url {{index_url}} jumpstarter-driver-can ``` -## `jumpstarter_driver_can.Can` +## Configuration + +### Can A generic CAN bus driver. Available on any platform, supports many different CAN interfaces through the `python-can` library. -### Configuration - Example configuration: ```yaml @@ -35,22 +35,13 @@ export: | interface | Refer to the [python-can](https://python-can.readthedocs.io/en/stable/interfaces.html) list of interfaces | str | yes | | | channel | channel to be used, refer to the interface documentation | int or str | yes | | -### API Reference +### IsoTpPython -```{eval-rst} -.. autoclass:: jumpstarter_driver_can.client.CanClient() - :members: -``` - -## `jumpstarter_driver_can.IsoTpPython` - A Pure python ISO-TP socket driver Available on any platform (does not require Linux ISO-TP kernel module), moderate performance and reliability, wide support for non-standard hardware interfaces -### Configuration - Example configuration: ```yaml @@ -78,20 +69,12 @@ export: | params | IsoTp parameters, refer to the [IsoTpParams](#isotpparams) section table | `IsoTpParams` | no | see table | | read_timeout | Read timeout for the bus in seconds | `float` | no | 0.05 | -### API Reference -```{eval-rst} -.. autoclass:: jumpstarter_driver_can.client.IsoTpClient() - :members: -``` - -## `jumpstarter_driver_can.IsoTpSocket` +### IsoTpSocket -Pure python ISO-TP socket driver +Linux kernel ISO-TP socket driver Available on any platform, moderate performance and reliability, wide support for non-standard hardware interfaces -### Configuration - Example configuration: ```yaml @@ -116,14 +99,7 @@ export: | address | Refer to the [isotp.Address](https://can-isotp.readthedocs.io/en/latest/isotp/addressing.html#isotp.Address) documentation | isotp.Address | yes | | | params | IsoTp parameters, refer to the [IsoTpParams](#isotpparams) section table | `IsoTpParams` | no | see table | -### API Reference -```{eval-rst} -.. autoclass:: jumpstarter_driver_can.client.IsoTpClient() - :noindex: - :members: -``` - -## IsoTpParams +### IsoTpParams | Parameter | Description | Type | Required | Default | |-----------------------------|-------------------------------------------------------------------------------------------------------|------------------|----------|------------| | `stmin` | Minimum Separation Time minimum in milliseconds between consecutive frames. | `int` | No | `0` | @@ -143,4 +119,23 @@ export: | `rate_limit_max_bitrate` | Maximum bitrate in bits per second for rate limiting if enabled. | `int` | No | `10000000` | | `rate_limit_window_size` | Time window in seconds over which the rate limit is calculated. | `float` | No | `0.2` | | `listen_mode` | If `True`, the stack operates in listen-only mode (does not send any frames). | `bool` | No | `False` | -| `blocking_send` | If `True`, send operations will block until the message is fully transmitted or an error occurs. | `bool` | No | `False` | \ No newline at end of file +| `blocking_send` | If `True`, send operations will block until the message is fully transmitted or an error occurs. | `bool` | No | `False` | + +## API Reference + +```{eval-rst} +.. autoclass:: jumpstarter_driver_can.driver.Can() + :members: + +.. autoclass:: jumpstarter_driver_can.client.CanClient() + :members: + +.. autoclass:: jumpstarter_driver_can.driver.IsoTpPython() + :members: + +.. autoclass:: jumpstarter_driver_can.driver.IsoTpSocket() + :members: + +.. autoclass:: jumpstarter_driver_can.client.IsoTpClient() + :members: +``` diff --git a/python/packages/jumpstarter-driver-composite/README.md b/python/packages/jumpstarter-driver-composite/README.md index 203a88714..4063df4dd 100644 --- a/python/packages/jumpstarter-driver-composite/README.md +++ b/python/packages/jumpstarter-driver-composite/README.md @@ -22,4 +22,6 @@ export: ## API Reference -Add API documentation here. +```{eval-rst} +.. autoclass:: jumpstarter_driver_composite.driver.Composite() +``` diff --git a/python/packages/jumpstarter-driver-corellium/README.md b/python/packages/jumpstarter-driver-corellium/README.md index 0bfc5e63e..1ce7d9954 100644 --- a/python/packages/jumpstarter-driver-corellium/README.md +++ b/python/packages/jumpstarter-driver-corellium/README.md @@ -68,3 +68,9 @@ export: device_os: "1.0" device_build: "Critical Application Monitor (Baremetal)" ``` + +## API Reference + +```{eval-rst} +.. autoclass:: jumpstarter_driver_corellium.driver.Corellium() +``` diff --git a/python/packages/jumpstarter-driver-doip/README.md b/python/packages/jumpstarter-driver-doip/README.md index 140508ba7..f0e421ec3 100644 --- a/python/packages/jumpstarter-driver-doip/README.md +++ b/python/packages/jumpstarter-driver-doip/README.md @@ -36,16 +36,8 @@ export: ecu_logical_address: 224 # 0x00E0 ``` -## Client API - -| Method | Description | -|--------------------------------|--------------------------------------------------| -| `entity_status()` | Request DoIP entity status | -| `alive_check()` | Request alive check | -| `diagnostic_power_mode()` | Request diagnostic power mode | -| `request_vehicle_identification()` | Request vehicle identification (VIN, EID, etc.) | -| `routing_activation(type)` | Request routing activation | -| `send_diagnostic(payload)` | Send raw diagnostic payload bytes | -| `receive_diagnostic(timeout)` | Receive raw diagnostic response bytes | -| `reconnect(close_delay)` | Reconnect after ECU reset | -| `close_connection()` | Close the DoIP connection | +## API Reference + +```{eval-rst} +.. autoclass:: jumpstarter_driver_doip.driver.DoIP() +``` diff --git a/python/packages/jumpstarter-driver-dut-network/README.md b/python/packages/jumpstarter-driver-dut-network/README.md index 510e8d9f4..7cc06e20b 100644 --- a/python/packages/jumpstarter-driver-dut-network/README.md +++ b/python/packages/jumpstarter-driver-dut-network/README.md @@ -1,4 +1,4 @@ -# DutNetwork Driver +# DUT Network Driver `jumpstarter-driver-dut-network` provides network isolation for DUTs (Devices Under Test) by configuring a dedicated network interface with NAT, DHCP, and nftables-based firewall rules on the exporter host. @@ -21,17 +21,6 @@ The following must be available on the exporter host: Optional: - `nmcli` (NetworkManager) - only needed if NM is running; the driver marks its interfaces as unmanaged -## How It Works - -The driver configures an isolated network for the DUT: - -1. Takes over a dedicated Ethernet interface (e.g., USB NIC) and assigns a gateway IP directly to it -2. Runs dnsmasq to provide DHCP to DUTs connected to that interface -3. Configures nftables rules for NAT (masquerade or 1:1) -4. Enables IP forwarding so DUT traffic routes through the exporter - -When NetworkManager is detected, the driver marks managed interfaces as `unmanaged` to prevent interference. On cleanup, existing addresses are flushed and the interface is restored to NetworkManager control. - ## Configuration ### Masquerade NAT (recommended for most use cases) @@ -118,7 +107,7 @@ export: ip: "10.26.28.2" ``` -## Configuration Reference +### Reference | Parameter | Type | Default | Description | |-----------|------|---------|-------------| @@ -136,7 +125,7 @@ export: | `nat_mode` | str | `masquerade` | NAT mode: `masquerade`, `1to1`, `disabled`, or `none` | | `public_interface` | str | None | Interface for IP alias (defaults to upstream) | -### Address Entry Fields +#### Address Entry Fields | Field | Required | Description | |-------|----------|-------------| @@ -145,7 +134,9 @@ export: | `hostname` | no | Hostname for DHCP | | `public_ip` | no | Public IP for 1:1 NAT (per-entry). At least one entry must have `public_ip` when `nat_mode=1to1` | -## Client CLI +## Usage + +### CLI Inside a `jmp shell` session: @@ -181,7 +172,7 @@ j dut-network add-dns controller.lab.local 10.26.28.1 j dut-network remove-dns controller.lab.local ``` -## Python API +### Python ```python from jumpstarter.common.utils import env @@ -212,11 +203,16 @@ with env() as client: client.dut_network.remove_dns_entry("myhost.lab.local") ``` -## nftables Coexistence +## Architecture -The driver uses a dedicated nftables table (named after the interface, e.g. `table ip jumpstarter_enx00e04c683af1`) that does not conflict with firewalld or other nftables users. Firewalld manages its own `firewalld` table and does not touch other tables, even during reloads. +The driver configures an isolated network for the DUT: -## Architecture +1. Takes over a dedicated Ethernet interface (e.g., USB NIC) and assigns a gateway IP directly to it +2. Runs dnsmasq to provide DHCP to DUTs connected to that interface +3. Configures nftables rules for NAT (masquerade or 1:1) +4. Enables IP forwarding so DUT traffic routes through the exporter + +When NetworkManager is detected, the driver marks managed interfaces as `unmanaged` to prevent interference. On cleanup, existing addresses are flushed and the interface is restored to NetworkManager control. ```text Exporter Host @@ -259,6 +255,17 @@ The driver uses a dedicated nftables table (named after the interface, e.g. `tab isolation or when routing is handled externally. ``` +## API Reference + +```{eval-rst} +.. autoclass:: jumpstarter_driver_dut_network.driver.DutNetwork() +``` + +```{note} +The driver uses a dedicated nftables table (named after the interface) that +does not conflict with firewalld or other nftables users. +``` + ## Troubleshooting ### NAT traffic not forwarding (Docker hosts) @@ -267,7 +274,7 @@ On hosts running Docker, the default iptables policy is often set to `iptables -P FORWARD DROP` to isolate container networks. Since modern Linux translates iptables rules into nftables under the hood, this creates a `table ip filter { chain FORWARD { policy drop } }` base chain that -**all** forwarded packets must pass — including traffic routed through +**all** forwarded packets must pass - including traffic routed through the DUT interface. The driver **automatically** detects this situation using native nftables: @@ -290,12 +297,3 @@ sysctl net.ipv4.conf..forwarding sysctl net.ipv4.conf..forwarding ``` -## Running Tests - -Integration tests require root privileges through passwordless sudo, or direct root access: - -```shell -make pkg-test-dut-network -``` - -Tests use veth pairs and network namespaces to simulate the DUT without real hardware. diff --git a/python/packages/jumpstarter-driver-dut-network/jumpstarter_driver_dut_network/driver.py b/python/packages/jumpstarter-driver-dut-network/jumpstarter_driver_dut_network/driver.py index 96f140fd5..8f43b65d5 100644 --- a/python/packages/jumpstarter-driver-dut-network/jumpstarter_driver_dut_network/driver.py +++ b/python/packages/jumpstarter-driver-dut-network/jumpstarter_driver_dut_network/driver.py @@ -115,6 +115,8 @@ def from_dict(cls, data: dict) -> "FilterConfig": @dataclass(kw_only=True) class DutNetwork(Driver): + """DUT network isolation with bridge, DHCP, DNS, and NAT.""" + interface: str subnet: str = "192.168.100.0/24" gateway_ip: str = "192.168.100.1" diff --git a/python/packages/jumpstarter-driver-dut-network/jumpstarter_driver_dut_network/nftables.py b/python/packages/jumpstarter-driver-dut-network/jumpstarter_driver_dut_network/nftables.py index ca508d787..a1225ff7b 100644 --- a/python/packages/jumpstarter-driver-dut-network/jumpstarter_driver_dut_network/nftables.py +++ b/python/packages/jumpstarter-driver-dut-network/jumpstarter_driver_dut_network/nftables.py @@ -110,7 +110,7 @@ def _build_forward_chain( extras = extra_forward_rules or [] if filter_config is None: - # Legacy behaviour — no filtering. + # Legacy behaviour - no filtering. lines.append(f' iifname "{interface}" oifname "{upstream}" accept') lines.append( f' iifname "{upstream}" oifname "{interface}" ct state related,established accept' @@ -125,7 +125,7 @@ def _build_forward_chain( egress = filter_config.egress ingress = filter_config.ingress - # -- Egress (DUT -> upstream) ---------------------------------------- + # - Egress (DUT -> upstream) ---------------------------------------- if egress: for rule in egress.rules: lines.append(_render_filter_rule(rule, interface, upstream, "destination")) @@ -133,10 +133,10 @@ def _build_forward_chain( egress_policy = egress.policy if egress else "accept" lines.append(f' iifname "{interface}" oifname "{upstream}" {egress_policy}') - # -- Extra forward rules (e.g. 1:1 NAT per-mapping accepts) --------- + # - Extra forward rules (e.g. 1:1 NAT per-mapping accepts) --------- lines.extend(extras) - # -- Ingress (upstream -> DUT) — new connections only ---------------- + # - Ingress (upstream -> DUT) - new connections only ---------------- if ingress: for rule in ingress.rules: lines.append(_render_filter_rule(rule, upstream, interface, "source")) diff --git a/python/packages/jumpstarter-driver-dutlink/README.md b/python/packages/jumpstarter-driver-dutlink/README.md index 6f29f0ea9..ebd02da02 100644 --- a/python/packages/jumpstarter-driver-dutlink/README.md +++ b/python/packages/jumpstarter-driver-dutlink/README.md @@ -1,4 +1,4 @@ -# DUT Link driver +# DUT Link Driver `jumpstarter-driver-dutlink` provides functionality for interacting with DUT Link devices. @@ -24,4 +24,6 @@ export: ## API Reference -Add API documentation here. +```{eval-rst} +.. autoclass:: jumpstarter_driver_dutlink.driver.Dutlink() +``` diff --git a/python/packages/jumpstarter-driver-dutlink/jumpstarter_driver_dutlink/driver.py b/python/packages/jumpstarter-driver-dutlink/jumpstarter_driver_dutlink/driver.py index d7d0e45b3..86478dfb9 100644 --- a/python/packages/jumpstarter-driver-dutlink/jumpstarter_driver_dutlink/driver.py +++ b/python/packages/jumpstarter-driver-dutlink/jumpstarter_driver_dutlink/driver.py @@ -234,6 +234,8 @@ async def read(self, dst: str): @dataclass(kw_only=True) class Dutlink(DutlinkConfig, CompositeInterface, Driver): + """DUT Link Board composite driver for power, serial, and storage.""" + alternate_console: str | None = field(default=None) storage_device: str baudrate: int = field(default=115200) diff --git a/python/packages/jumpstarter-driver-energenie/README.md b/python/packages/jumpstarter-driver-energenie/README.md index 5a8da5ea5..75cd28cf2 100644 --- a/python/packages/jumpstarter-driver-energenie/README.md +++ b/python/packages/jumpstarter-driver-energenie/README.md @@ -1,13 +1,9 @@ -# EnerGenie +# Energenie PDU Driver Drivers for EnerGenie products. -## EnerGenie driver - This driver provides a client for the [EnerGenie Programmable power switch](https://energenie.com/products.aspx?sg=239). The driver was tested on EG-PMS2-LAN device only but should be easy to support other devices. -**driver**: `jumpstarter_driver_energenie.driver.EnerGenie` - ## Installation ```{code-block} console @@ -15,7 +11,9 @@ This driver provides a client for the [EnerGenie Programmable power switch](http $ pip3 install --extra-index-url {{index_url}} jumpstarter-driver-energenie ``` -### Configuration +## Configuration + +Example configuration: ```yaml export: @@ -24,25 +22,20 @@ export: config: host: "192.168.0.1" password: "password" - slot: "1" + slot: 1 ``` -### Config parameters - | Parameter | Description | Type | Required | Default | |-----------|-------------|------|----------|---------| -| host | The ip address of the EnerGenie system | string | yes | None | -| password | The password of the EnerGenie system | string | no | None | -| slot | The slot number to be managed, 1, 2, 3, 4 | int | yes | 1 | - -### PowerClient API +| host | The IP address of the EnerGenie system | `str` | yes | | +| password | The password of the EnerGenie system | `str` | no | `"1"` | +| slot | The slot number to be managed (1, 2, 3, or 4) | `int` | no | `1` | -The EnerGenie driver provides a `PowerClient` with the following API: +## API Reference ```{eval-rst} -.. autoclass:: jumpstarter_driver_power.client.PowerClient() - :no-index: - :members: on, off +.. autoclass:: jumpstarter_driver_energenie.driver.EnerGenie() + :members: ``` ### Examples diff --git a/python/packages/jumpstarter-driver-esp32/README.md b/python/packages/jumpstarter-driver-esp32/README.md index 06895e6b4..06e936939 100644 --- a/python/packages/jumpstarter-driver-esp32/README.md +++ b/python/packages/jumpstarter-driver-esp32/README.md @@ -1,4 +1,4 @@ -# ESP32 driver +# ESP32 Driver `jumpstarter-driver-esp32` provides functionality for flashing and managing ESP32 devices using [esptool](https://github.com/espressif/esptool) as a @@ -45,12 +45,7 @@ the child driver. Use a `ref` proxy to share the serial driver with the top-level composite, enabling both `j serial start-console` and `j storage flash` to work. -## API Reference - -```{eval-rst} -.. autoclass:: jumpstarter_driver_esp32.client.Esp32FlasherClient() - :members: flash, dump, get_chip_info, erase, hard_reset, enter_bootloader -``` +## Usage ### CLI @@ -74,8 +69,6 @@ Commands: pipe Pipe serial port data to stdout or file ``` -## Examples - ### CLI usage ```bash @@ -127,3 +120,10 @@ console = client.serial.open() console.sendline("import machine") console.expect(">>>") ``` + +## API Reference + +```{eval-rst} +.. autoclass:: jumpstarter_driver_esp32.client.Esp32FlasherClient() + :members: flash, dump, get_chip_info, erase, hard_reset, enter_bootloader +``` diff --git a/python/packages/jumpstarter-driver-flashers/README.md b/python/packages/jumpstarter-driver-flashers/README.md index b973eed4c..b6af078a9 100644 --- a/python/packages/jumpstarter-driver-flashers/README.md +++ b/python/packages/jumpstarter-driver-flashers/README.md @@ -1,4 +1,4 @@ -# Flashers +# Flashers Driver The flasher drivers are used to flash images to DUTs via network, typically using TFTP and HTTP. It is designed to interact with the target bootloader and @@ -16,14 +16,16 @@ See the [bundle](#oci-bundles) section for more details. $ pip3 install --extra-index-url {{index_url}} jumpstarter-driver-flashers ``` -## Available drivers and bundles +## Configuration + +### Available drivers and bundles | Driver | Bundle | | --------------- | ------------------------------------------------------------ | | TIJ784S4Flasher | quay.io/jumpstarter-dev/jumpstarter-flasher-ti-j784s4:latest | -## Driver configuration +### Driver configuration **driver**: `jumpstarter_driver_flashers.driver.${DRIVER}` ```yaml @@ -72,15 +74,60 @@ HTTP servers are used to serve images to the DUT bootloader and busybox shell. | manifest | The manifest to use from the bundle. Every bundle can have multiple manifests, this is the name of the manifest to use | str | no | manifest.yaml | | default_target | The default target to flash to if none specified | str | no | | -## BaseFlasher API +### oci-bundles -The `BaseFlasher` class provides a set of methods to flash the DUT, -```{eval-rst} -.. autoclass:: jumpstarter_driver_flashers.client.BaseFlasherClient() - :members: flash, busybox_shell, bootloader_shell, use_dtb, use_initram, use_kernel +The flasher drivers require some artifacts and basic information about the +target device to operate. To make this easy to distribute and use, we use OCI +bundles to package the artifacts and metadata. + +The bundle is a container that uses [oras](https://oras.land/) to transport the +artifacts and metadata. It is a container that contains the following: +- `manifest.yaml`: The manifest file that describes the bundle +- `data/*`: The artifacts, including kernel, initram, dtbs, etc. + +### The format of the manifest is as follows: + +```{literalinclude} ../../../../../packages/jumpstarter-driver-flashers/oci_bundles/test/manifest.yaml +:language: yaml +``` + +### Table with the spec fields of the manifest: + +| Field | Description | Default | +| -------------------- | -------------------------------------------------------------------------- | ------- | +| `manufacturer` | Name of the device manufacturer | | +| `link` | URL to device documentation or manufacturer website | | +| `bootcmd` | Command used to boot the device (e.g. booti, bootz) | | +| `default_target` | Default target device to flash to if none specified | | +| `targets` | Map of target names to device paths | | +| `login.type` | Type of login shell | busybox | +| `login.login_prompt` | Expected login prompt string | login: | +| `login.username` | Username to log in with, leave empty if not needed | | +| `login.password` | Password for login, leave empty if not needed | | +| `login.prompt` | Shell prompt after successful login | # | +| `preflash_commands` | List of commands to run before flashing, useful to clear boot entries, etc | | +| `kernel.file` | Path to kernel image within bundle | | +| `kernel.address` | Memory address to load kernel to | | +| `initram.file` | Path to initramfs within bundle (if any) | | +| `initram.address` | Memory address to load initramfs to (if any) | | +| `dtb.default` | Default DTB variant to use | | +| `dtb.address` | Memory address to load DTB to | | +| `dtb.variants` | Map of DTB variant names to files | | + +### Bundle Examples + +An example bundle for the TI J784S4XEVM looks like this: + +```{literalinclude} ../../../../../packages/jumpstarter-driver-flashers/oci_bundles/ti_j784s4xevm/manifest.yaml +:language: yaml ``` -## CLI +You can find a script to build and push a bundle to a registry here: +[oci_bundles](https://github.com/jumpstarter-dev/jumpstarter/tree/main/python/packages/jumpstarter-driver-flashers/oci_bundles) + +## Usage + +### CLI The flasher driver provides a CLI to perform flashing, access to busybox shell and uboot. @@ -111,7 +158,7 @@ Commands: ``` -### flash +#### flash ```shell Usage: j storage flash [OPTIONS] [FILE] @@ -206,7 +253,7 @@ Environment variables for OCI auth: - `OCI_USERNAME`: registry username - `OCI_PASSWORD`: registry password -### bootloader-shell +#### bootloader-shell ```shell Usage: j storage bootloader-shell [OPTIONS] @@ -231,7 +278,7 @@ U-Boot 2024.01-rc3 (Jan 09 2024 - 00:00:00 +0000) gcc (GCC) 11.4.1 20231218 (Red Hat 11.4.1-3) GNU ld version 2.35.2-42.el9 ``` -### busybox-shell +#### busybox-shell ```shell Usage: j storage busybox-shell [OPTIONS] @@ -263,7 +310,7 @@ Linux buildroot 6.1.46-dirty #2 SMP PREEMPT Thu Mar 14 14:37:01 UTC 2024 aarch64 # ``` -## Examples +### Python Examples Flash the device with a specific image ```python @@ -280,8 +327,7 @@ Flash into a specific partition flasherclient.flash("/path/to/image.raw.xz", partition="emmc") ``` - -## Examples of utility consoles +### Utility Consoles In addition to the flashing mechanisms, the flasher drivers also provide a way to access the DUT bootloader and busybox shell for convenience and debugging, @@ -304,53 +350,10 @@ with flasherclient.bootloader_shell() as serial: print(serial.before) ``` -## oci-bundles +## API Reference -The flasher drivers require some artifacts and basic information about the -target device to operate. To make this easy to distribute and use, we use OCI -bundles to package the artifacts and metadata. - -The bundle is a container that uses [oras](https://oras.land/) to transport the -artifacts and metadata. It is a container that contains the following: -- `manifest.yaml`: The manifest file that describes the bundle -- `data/*`: The artifacts, including kernel, initram, dtbs, etc. - -## The format of the manifest is as follows: - -```{literalinclude} ../../../../../packages/jumpstarter-driver-flashers/oci_bundles/test/manifest.yaml -:language: yaml -``` - -## Table with the spec fields of the manifest: - -| Field | Description | Default | -| -------------------- | -------------------------------------------------------------------------- | ------- | -| `manufacturer` | Name of the device manufacturer | | -| `link` | URL to device documentation or manufacturer website | | -| `bootcmd` | Command used to boot the device (e.g. booti, bootz) | | -| `default_target` | Default target device to flash to if none specified | | -| `targets` | Map of target names to device paths | | -| `login.type` | Type of login shell | busybox | -| `login.login_prompt` | Expected login prompt string | login: | -| `login.username` | Username to log in with, leave empty if not needed | | -| `login.password` | Password for login, leave empty if not needed | | -| `login.prompt` | Shell prompt after successful login | # | -| `preflash_commands` | List of commands to run before flashing, useful to clear boot entries, etc | | -| `kernel.file` | Path to kernel image within bundle | | -| `kernel.address` | Memory address to load kernel to | | -| `initram.file` | Path to initramfs within bundle (if any) | | -| `initram.address` | Memory address to load initramfs to (if any) | | -| `dtb.default` | Default DTB variant to use | | -| `dtb.address` | Memory address to load DTB to | | -| `dtb.variants` | Map of DTB variant names to files | | - -## Examples - -An example bundle for the TI J784S4XEVM looks like this: - -```{literalinclude} ../../../../../packages/jumpstarter-driver-flashers/oci_bundles/ti_j784s4xevm/manifest.yaml -:language: yaml +The `BaseFlasher` class provides a set of methods to flash the DUT, +```{eval-rst} +.. autoclass:: jumpstarter_driver_flashers.client.BaseFlasherClient() + :members: flash, busybox_shell, bootloader_shell, use_dtb, use_initram, use_kernel ``` - -You can find a script to build and push a bundle to a registry here: -[oci_bundles](https://github.com/jumpstarter-dev/jumpstarter/tree/main/python/packages/jumpstarter-driver-flashers/oci_bundles) diff --git a/python/packages/jumpstarter-driver-flashers/jumpstarter_driver_flashers/client_test.py b/python/packages/jumpstarter-driver-flashers/jumpstarter_driver_flashers/client_test.py index c9888654a..81023b68c 100644 --- a/python/packages/jumpstarter-driver-flashers/jumpstarter_driver_flashers/client_test.py +++ b/python/packages/jumpstarter-driver-flashers/jumpstarter_driver_flashers/client_test.py @@ -97,7 +97,7 @@ def test_resolve_oci_credentials_partial_env_falls_through_to_auth_file(monkeypa monkeypatch.setenv("OCI_USERNAME", "env-user") monkeypatch.delenv("OCI_PASSWORD", raising=False) - # When auth file has no match, result is (None, None) — no error + # When auth file has no match, result is (None, None) - no error with patch("jumpstarter.common.oci.read_auth_file_credentials", return_value=(None, None)): username, password = client._resolve_oci_credentials("oci://quay.io/org/image:tag", None, None) assert username is None diff --git a/python/packages/jumpstarter-driver-gpiod/README.md b/python/packages/jumpstarter-driver-gpiod/README.md index 2c5745dbf..f8685b304 100644 --- a/python/packages/jumpstarter-driver-gpiod/README.md +++ b/python/packages/jumpstarter-driver-gpiod/README.md @@ -1,4 +1,4 @@ -# gpiod driver +# gpiod Driver `jumpstarter-driver-gpiod` provides functionality for interacting with gpiod GPIO pins for digital input/output operations. @@ -13,6 +13,12 @@ This requires the /dev/gpiochip[0..N] device available on the system, and you ca $ pip3 install --extra-index-url {{index_url}} jumpstarter-driver-gpiod ``` +### Hardware Requirements + +- gpiod with GPIO access +- Python `gpiod` library installed +- Appropriate permissions to access `/dev/gpiochip0` + ## Configuration The gpiod driver provides three main driver types: @@ -76,23 +82,7 @@ export: | initial_value | The initial value for output pins. Options: "active", "inactive", "on", "off", True, False | str/bool | no | "inactive" | DigitalOutput, PowerSwitch | | mode | The mode for PowerSwitch (same as drive parameter) | str | no | "push_pull" | PowerSwitch | -## API Reference - -### DigitalOutputClient - -```{eval-rst} -.. autoclass:: jumpstarter_driver_gpiod.client.DigitalOutputClient() - :members: on, off, read -``` - -### DigitalInputClient - -```{eval-rst} -.. autoclass:: jumpstarter_driver_gpiod.client.DigitalInputClient() - :members: wait_for_active, wait_for_inactive, wait_for_edge, read -``` - -## Examples +## Usage ### Digital Output Examples @@ -146,42 +136,46 @@ state = power_switch.read() print(f"Power state: {state}") ``` -## Pin Configuration Details +### Pin Configuration Details -### Drive Modes +#### Drive Modes - **push_pull**: Standard push-pull output (default) - **open_drain**: Open-drain output (useful for I2C, etc.) - **open_source**: Open-source output -### Bias Configuration +#### Bias Configuration - **as_is**: No bias (default) - **pull_up**: Internal pull-up resistor - **pull_down**: Internal pull-down resistor - **disabled**: Disable bias -### Active Low vs Active High +#### Active Low vs Active High - **active_low: false** (default): Pin is active when HIGH - **active_low: true**: Pin is active when LOW -### Initial Values +#### Initial Values For output pins, you can set the initial state: - **"inactive"** or **"off"** or **False**: Start with pin LOW - **"active"** or **"on"** or **True**: Start with pin HIGH -## Hardware Requirements +## API Reference -- gpiod with GPIO access -- Python `gpiod` library installed -- Appropriate permissions to access `/dev/gpiochip0` +### DigitalOutputClient + +```{eval-rst} +.. autoclass:: jumpstarter_driver_gpiod.client.DigitalOutputClient() + :members: on, off, read +``` + +### DigitalInputClient -## Error Handling +```{eval-rst} +.. autoclass:: jumpstarter_driver_gpiod.client.DigitalInputClient() + :members: wait_for_active, wait_for_inactive, wait_for_edge, read +``` -The driver includes comprehensive error handling for: -- Invalid pin numbers -- Invalid drive/bias configurations -- Hardware access errors - Timeout conditions for input operations diff --git a/python/packages/jumpstarter-driver-http-power/README.md b/python/packages/jumpstarter-driver-http-power/README.md index a47b45718..4095096f0 100644 --- a/python/packages/jumpstarter-driver-http-power/README.md +++ b/python/packages/jumpstarter-driver-http-power/README.md @@ -106,8 +106,8 @@ http_power_client.off() ``` -## Notes - -- The power reading response parsing is not yet implemented. The driver currently returns dummy values (0.0V, 0.0A). -- Authentication is optional and currently supports HTTP Basic Auth only. -- All HTTP requests will raise exceptions on HTTP error status codes. +```{note} +Power reading response parsing is not yet implemented - the driver returns +dummy values (0.0V, 0.0A). Authentication is optional and supports HTTP +Basic Auth only. +``` diff --git a/python/packages/jumpstarter-driver-http/README.md b/python/packages/jumpstarter-driver-http/README.md index 956f7dae3..955cfd8d0 100644 --- a/python/packages/jumpstarter-driver-http/README.md +++ b/python/packages/jumpstarter-driver-http/README.md @@ -1,4 +1,4 @@ -# HTTP driver +# HTTP Driver `jumpstarter-driver-http` provides functionality for HTTP communication. @@ -41,4 +41,6 @@ The internal HTTP server driver automatically tracks files and directories creat ## API Reference -Add API documentation here. +```{eval-rst} +.. autoclass:: jumpstarter_driver_http.driver.HttpServer() +``` diff --git a/python/packages/jumpstarter-driver-iscsi/README.md b/python/packages/jumpstarter-driver-iscsi/README.md index 03c8335a5..9b0737c8d 100644 --- a/python/packages/jumpstarter-driver-iscsi/README.md +++ b/python/packages/jumpstarter-driver-iscsi/README.md @@ -1,4 +1,4 @@ -# iSCSI server driver +# iSCSI Driver `jumpstarter-driver-iscsi` provides a lightweight iSCSI **target** implementation powered by the Linux [RFC-tgt](https://github.com/open-iscsi/tcmu-runner/) framework via the @@ -58,7 +58,7 @@ export: | `host` | IP address to bind the target to. Empty string will auto-detect | str | no | _auto_ | | `port` | TCP port the target listens on | int | no | `3260` | | `remove_created_on_close`| Automatically remove created files/directories when driver closes| bool | no | false | -| `block_device_allowlist`| List of allowed block device paths for `is_block=True` LUNs. Symlinks are resolved before matching. Must be set to use block devices. | list[str] | no | `[]` (empty -- block devices disabled) | +| `block_device_allowlist`| List of allowed block device paths for `is_block=True` LUNs. Symlinks are resolved before matching. Must be set to use block devices. | list[str] | no | `[]` (empty - block devices disabled) | ### File Management diff --git a/python/packages/jumpstarter-driver-mitmproxy/README.md b/python/packages/jumpstarter-driver-mitmproxy/README.md index 7f5f3f82e..9043b83cb 100644 --- a/python/packages/jumpstarter-driver-mitmproxy/README.md +++ b/python/packages/jumpstarter-driver-mitmproxy/README.md @@ -1,18 +1,16 @@ -# jumpstarter-driver-mitmproxy +# mitmproxy Driver -A [Jumpstarter](https://jumpstarter.dev) driver for [mitmproxy](https://mitmproxy.org) — bringing HTTP(S) interception, backend mocking, and traffic recording to Hardware-in-the-Loop testing. - -## What it does +A [Jumpstarter](https://jumpstarter.dev) driver for [mitmproxy](https://mitmproxy.org) - bringing HTTP(S) interception, backend mocking, and traffic recording to Hardware-in-the-Loop testing. This driver manages a `mitmdump` or `mitmweb` process on the Jumpstarter exporter host, providing your pytest HiL tests with: -- **Backend mocking** — Return deterministic JSON responses for any API endpoint, with hot-reloadable definitions, wildcard path matching, conditional rules, sequences, templates, and custom addons -- **SSL/TLS interception** — Inspect and modify HTTPS traffic from your DUT, with easy CA certificate retrieval for DUT provisioning -- **Traffic recording & replay** — Capture a "golden" session against real servers, then replay it offline in CI -- **Request capture** — Record every request the DUT makes and assert on them in your tests -- **Browser-based UI** — Launch `mitmweb` for interactive traffic inspection, with TCP port forwarding through the Jumpstarter tunnel -- **Scenario files** — Load complete mock configurations from YAML or JSON, swap between test scenarios instantly -- **Full CLI** — Control the proxy interactively from `jmp shell` sessions +- **Backend mocking** - Return deterministic JSON responses for any API endpoint, with hot-reloadable definitions, wildcard path matching, conditional rules, sequences, templates, and custom addons +- **SSL/TLS interception** - Inspect and modify HTTPS traffic from your DUT, with easy CA certificate retrieval for DUT provisioning +- **Traffic recording & replay** - Capture a "golden" session against real servers, then replay it offline in CI +- **Request capture** - Record every request the DUT makes and assert on them in your tests +- **Browser-based UI** - Launch `mitmweb` for interactive traffic inspection, with TCP port forwarding through the Jumpstarter tunnel +- **Scenario files** - Load complete mock configurations from YAML or JSON, swap between test scenarios instantly +- **Full CLI** - Control the proxy interactively from `jmp shell` sessions ## Installation @@ -29,7 +27,9 @@ uv build pip install dist/jumpstarter_driver_mitmproxy-*.whl ``` -## Exporter Configuration +## Configuration + +### Exporter Configuration ```yaml # /etc/jumpstarter/exporters/my-bench.yaml @@ -77,7 +77,43 @@ export: See `examples/exporter.yaml` in the package source for a full exporter config with DUT Link, serial, and video drivers. -## Modes +### SSL/TLS Setup + +For HTTPS interception, the mitmproxy CA certificate must be installed on the DUT. The certificate is generated the first time the proxy starts. + +#### From the CLI + +```console +j proxy cert # writes ./mitmproxy-ca-cert.pem +j proxy cert /tmp/ca.pem # custom output path +``` + +#### From Python + +```python +# Get the PEM certificate contents +pem = proxy.get_ca_cert() + +# Write to a local file +from pathlib import Path +Path("/tmp/mitmproxy-ca.pem").write_text(pem) + +# Or push directly to the DUT via serial/ssh/adb +dut.write_file("/etc/ssl/certs/mitmproxy-ca.pem", pem) +``` + +#### Exporter-side path + +If you need the path on the exporter host itself (for provisioning scripts that run locally): + +```python +cert_path = proxy.get_ca_cert_path() +# -> /opt/jumpstarter/mitmproxy/conf/mitmproxy-ca-cert.pem +``` + +## Usage + +### Modes | Mode | Description | |---------------|--------------------------------------------------| @@ -88,11 +124,11 @@ See `examples/exporter.yaml` in the package source for a full exporter config wi Add `web_ui=True` (Python) or `--web-ui` (CLI) to any mode for the mitmweb browser interface. -## CLI Commands +### CLI Commands During a `jmp shell` session, control the proxy with `j proxy `: -### Lifecycle +#### Lifecycle ```console j proxy start # start in mock mode (default) @@ -106,7 +142,7 @@ j proxy restart -m passthrough # restart with new mode j proxy status # show proxy status ``` -### Mock Management +#### Mock Management ```console j proxy mock list # list configured mocks @@ -115,7 +151,7 @@ j proxy mock load happy-path.yaml # load a scenario file j proxy mock load my-capture/ # load a saved capture directory ``` -### Traffic Capture +#### Traffic Capture ```console j proxy capture list # show captured requests @@ -125,7 +161,7 @@ j proxy capture save -f '/api/v1/*' ./my-capture # with path filter j proxy capture save --exclude-mocked ./my-capture ``` -### Flow Files +#### Flow Files ```console j proxy flow list # list recorded flow files @@ -133,7 +169,7 @@ j proxy flow save capture_20260101.bin # download to current directory j proxy flow save capture_20260101.bin /tmp/my.bin # download to specific path ``` -### Web UI & Certificates +#### Web UI & Certificates ```console j proxy web # forward mitmweb UI to localhost:8081 @@ -142,9 +178,63 @@ j proxy cert # download CA cert to ./mitmproxy-ca-ce j proxy cert /tmp/ca.pem # download to a specific path ``` -## Python API +### Mock Scenarios + +Create YAML or JSON files with endpoint definitions: + +```yaml +# scenarios/happy-path.yaml +endpoints: + GET /api/v1/status: + status: 200 + body: + id: device-001 + status: active + firmware_version: "2.5.1" + + POST /api/v1/telemetry/upload: + status: 202 + body: + accepted: true + + GET /api/v1/search*: # wildcard prefix match + status: 200 + body: + results: [] +``` + +Load from CLI or Python: + +```console +j proxy mock load happy-path.yaml +j proxy mock load my-capture/ # directory from 'capture save' +``` + +```python +proxy.load_mock_scenario("happy-path.yaml") + +# Or with automatic cleanup: +with proxy.mock_scenario("happy-path.yaml"): + run_tests() +``` + +See `examples/scenarios/` in the package source for complete scenario examples including conditional rules, templates, and sequences. + +### Web UI Port Forwarding + +The mitmweb UI runs on the exporter host and is not directly reachable from the test client. The `web` command tunnels it through the Jumpstarter gRPC transport: + +```console +j proxy start -m mock -w # start with web UI on the exporter +j proxy web # tunnel to localhost:8081 +j proxy web --port 9090 # use a custom local port +``` + +Then open `http://localhost:8081` in your browser to inspect traffic in real time. + +### Python API -### Basic Usage +#### Basic Usage ```python def test_device_status(client): @@ -164,7 +254,7 @@ def test_device_status(client): proxy.stop() ``` -### Context Managers +#### Context Managers Context managers ensure clean teardown even if the test fails: @@ -195,7 +285,7 @@ Available context managers: | `proxy.recording()` | Record traffic to a flow file | | `proxy.capture()` | Capture and assert on requests | -### Request Capture +#### Request Capture Verify that the DUT is making the right API calls: @@ -212,9 +302,9 @@ def test_telemetry_sent(client): cap.assert_request_made("POST", "/api/v1/telemetry") ``` -### Advanced Mocking +#### Advanced Mocking -#### Conditional responses +##### Conditional responses Return different responses based on request headers, body, or query params: @@ -229,7 +319,7 @@ proxy.set_mock_conditional("POST", "/api/auth", [ ]) ``` -#### Response sequences +##### Response sequences Return different responses on successive calls: @@ -241,7 +331,7 @@ proxy.set_mock_sequence("GET", "/api/v1/auth/token", [ ]) ``` -#### Dynamic templates +##### Dynamic templates Responses with per-request dynamic values: @@ -254,7 +344,7 @@ proxy.set_mock_template("GET", "/api/v1/weather", { }) ``` -#### Simulated latency +##### Simulated latency ```python proxy.set_mock_with_latency( @@ -264,7 +354,7 @@ proxy.set_mock_with_latency( ) ``` -#### File serving +##### File serving ```python proxy.set_mock_file( @@ -274,7 +364,7 @@ proxy.set_mock_file( ) ``` -#### Custom addon scripts +##### Custom addon scripts ```python proxy.set_mock_addon( @@ -284,7 +374,7 @@ proxy.set_mock_addon( ) ``` -### State Store +#### State Store Share state between tests and conditional mock rules: @@ -298,95 +388,13 @@ all_state = proxy.get_all_state() # {"auth_token": "...", "retries": 3} proxy.clear_state() ``` -## SSL/TLS Setup - -For HTTPS interception, the mitmproxy CA certificate must be installed on the DUT. The certificate is generated the first time the proxy starts. - -### From the CLI - -```console -j proxy cert # writes ./mitmproxy-ca-cert.pem -j proxy cert /tmp/ca.pem # custom output path -``` - -### From Python - -```python -# Get the PEM certificate contents -pem = proxy.get_ca_cert() - -# Write to a local file -from pathlib import Path -Path("/tmp/mitmproxy-ca.pem").write_text(pem) - -# Or push directly to the DUT via serial/ssh/adb -dut.write_file("/etc/ssl/certs/mitmproxy-ca.pem", pem) -``` - -### Exporter-side path +## API Reference -If you need the path on the exporter host itself (for provisioning scripts that run locally): - -```python -cert_path = proxy.get_ca_cert_path() -# -> /opt/jumpstarter/mitmproxy/conf/mitmproxy-ca-cert.pem +```{eval-rst} +.. autoclass:: jumpstarter_driver_mitmproxy.driver.MitmproxyDriver() ``` -## Mock Scenarios - -Create YAML or JSON files with endpoint definitions: - -```yaml -# scenarios/happy-path.yaml -endpoints: - GET /api/v1/status: - status: 200 - body: - id: device-001 - status: active - firmware_version: "2.5.1" - - POST /api/v1/telemetry/upload: - status: 202 - body: - accepted: true - - GET /api/v1/search*: # wildcard prefix match - status: 200 - body: - results: [] -``` - -Load from CLI or Python: - -```console -j proxy mock load happy-path.yaml -j proxy mock load my-capture/ # directory from 'capture save' -``` - -```python -proxy.load_mock_scenario("happy-path.yaml") - -# Or with automatic cleanup: -with proxy.mock_scenario("happy-path.yaml"): - run_tests() -``` - -See `examples/scenarios/` in the package source for complete scenario examples including conditional rules, templates, and sequences. - -## Web UI Port Forwarding - -The mitmweb UI runs on the exporter host and is not directly reachable from the test client. The `web` command tunnels it through the Jumpstarter gRPC transport: - -```console -j proxy start -m mock -w # start with web UI on the exporter -j proxy web # tunnel to localhost:8081 -j proxy web --port 9090 # use a custom local port -``` - -Then open `http://localhost:8081` in your browser to inspect traffic in real time. - -## Container Deployment +### Container Deployment ```bash podman build -t jumpstarter-mitmproxy:latest . @@ -398,7 +406,3 @@ podman run --rm -it --privileged \ jumpstarter-mitmproxy:latest \ jmp exporter start my-bench ``` - -## License - -Apache-2.0 diff --git a/python/packages/jumpstarter-driver-mitmproxy/demo/scenarios/happy-path/scenario.yaml b/python/packages/jumpstarter-driver-mitmproxy/demo/scenarios/happy-path/scenario.yaml index 069efa4bc..1362fae24 100644 --- a/python/packages/jumpstarter-driver-mitmproxy/demo/scenarios/happy-path/scenario.yaml +++ b/python/packages/jumpstarter-driver-mitmproxy/demo/scenarios/happy-path/scenario.yaml @@ -1,12 +1,12 @@ # Happy-path scenario: all endpoints return mock success data. # -# Demonstrates per-entry match conditions — entries are evaluated in +# Demonstrates per-entry match conditions - entries are evaluated in # order and the first matching one wins. An entry with no "match" key # is an unconditional default. # -# match.query — require specific query parameters -# match.body_json — match on JSON fields in the request body -# match.headers — require specific request headers +# match.query - require specific query parameters +# match.body_json - match on JSON fields in the request body +# match.headers - require specific request headers endpoints: http://127.0.0.1:9000/api/v1/status: diff --git a/python/packages/jumpstarter-driver-mitmproxy/demo/test_demo.py b/python/packages/jumpstarter-driver-mitmproxy/demo/test_demo.py index 8e0cd56bd..45ba5da38 100644 --- a/python/packages/jumpstarter-driver-mitmproxy/demo/test_demo.py +++ b/python/packages/jumpstarter-driver-mitmproxy/demo/test_demo.py @@ -25,7 +25,7 @@ class TestPassthrough: - """No mocks configured — requests flow through the proxy to the real backend.""" + """No mocks configured - requests flow through the proxy to the real backend.""" def test_status_from_real_backend( self, backend_server, proxy_client, http_session, diff --git a/python/packages/jumpstarter-driver-mitmproxy/examples/addons/_template.py b/python/packages/jumpstarter-driver-mitmproxy/examples/addons/_template.py index a9e779a83..48f1455ef 100644 --- a/python/packages/jumpstarter-driver-mitmproxy/examples/addons/_template.py +++ b/python/packages/jumpstarter-driver-mitmproxy/examples/addons/_template.py @@ -29,7 +29,7 @@ cleanup() Called when the addon is unloaded (not currently triggered - automatically — reserved for future use). + automatically - reserved for future use). """ from __future__ import annotations @@ -40,7 +40,7 @@ class Handler: - """Template handler — replace with your implementation.""" + """Template handler - replace with your implementation.""" def __init__(self): # Initialize any state your handler needs. @@ -107,6 +107,6 @@ def websocket_message(self, flow: http.HTTPFlow, config: dict): def cleanup(self) -> None: """Called when the addon is unloaded. - Reserved for future use — not yet triggered automatically. + Reserved for future use - not yet triggered automatically. Add teardown logic here (close connections, flush buffers, etc.). """ diff --git a/python/packages/jumpstarter-driver-mitmproxy/examples/addons/data_stream_websocket.py b/python/packages/jumpstarter-driver-mitmproxy/examples/addons/data_stream_websocket.py index 289007ce0..4ce5f69b3 100644 --- a/python/packages/jumpstarter-driver-mitmproxy/examples/addons/data_stream_websocket.py +++ b/python/packages/jumpstarter-driver-mitmproxy/examples/addons/data_stream_websocket.py @@ -86,10 +86,10 @@ def handle(self, flow: http.HTTPFlow, config: dict) -> bool: injector take over. Returns True to indicate the request was handled (but we - don't set flow.response — we let the WebSocket handshake + don't set flow.response - we let the WebSocket handshake complete naturally by NOT intercepting it here). """ - # Don't block the handshake — return False to let it through + # Don't block the handshake - return False to let it through # to the server (or get intercepted later by websocket hooks) return False diff --git a/python/packages/jumpstarter-driver-mitmproxy/examples/conftest.py b/python/packages/jumpstarter-driver-mitmproxy/examples/conftest.py index a8d90dbd4..330c6b287 100644 --- a/python/packages/jumpstarter-driver-mitmproxy/examples/conftest.py +++ b/python/packages/jumpstarter-driver-mitmproxy/examples/conftest.py @@ -14,7 +14,7 @@ import pytest from jumpstarter_driver_mitmproxy.client import MitmproxyClient -# -- Proxy session fixtures -------------------------------------------------- +# - Proxy session fixtures -------------------------------------------------- @pytest.fixture(scope="session") @@ -48,7 +48,7 @@ def proxy(proxy_session): proxy_session.clear_mocks() -# -- Scenario fixtures ------------------------------------------------------- +# - Scenario fixtures ------------------------------------------------------- @pytest.fixture diff --git a/python/packages/jumpstarter-driver-mitmproxy/examples/exporter.yaml b/python/packages/jumpstarter-driver-mitmproxy/examples/exporter.yaml index 9e2870bc5..c7ec35177 100644 --- a/python/packages/jumpstarter-driver-mitmproxy/examples/exporter.yaml +++ b/python/packages/jumpstarter-driver-mitmproxy/examples/exporter.yaml @@ -27,7 +27,7 @@ # j proxy cert [output.pem] # download CA certificate export: - # -- Hardware interfaces --------------------------------------------------- + # - Hardware interfaces --------------------------------------------------- dutlink: type: jumpstarter_driver_dutlink.driver.Dutlink @@ -45,7 +45,7 @@ export: args: device: "/dev/video0" - # -- Network proxy / mocking ----------------------------------------------- + # - Network proxy / mocking ----------------------------------------------- proxy: type: jumpstarter_driver_mitmproxy.driver.MitmproxyDriver diff --git a/python/packages/jumpstarter-driver-mitmproxy/examples/scenarios/backend-degraded.yaml b/python/packages/jumpstarter-driver-mitmproxy/examples/scenarios/backend-degraded.yaml index 519595bd2..cc89dd81c 100644 --- a/python/packages/jumpstarter-driver-mitmproxy/examples/scenarios/backend-degraded.yaml +++ b/python/packages/jumpstarter-driver-mitmproxy/examples/scenarios/backend-degraded.yaml @@ -38,7 +38,7 @@ endpoints: body: {id: device-001, status: active} latency_ms: 200 - # All telemetry uploads time out -- tests retry logic + # All telemetry uploads time out - tests retry logic POST /api/v1/telemetry/upload: status: 503 body: {error: Backend overloaded} diff --git a/python/packages/jumpstarter-driver-mitmproxy/jumpstarter_driver_mitmproxy/bundled_addon.py b/python/packages/jumpstarter-driver-mitmproxy/jumpstarter_driver_mitmproxy/bundled_addon.py index c958c24ad..63d64c40e 100644 --- a/python/packages/jumpstarter-driver-mitmproxy/jumpstarter_driver_mitmproxy/bundled_addon.py +++ b/python/packages/jumpstarter-driver-mitmproxy/jumpstarter_driver_mitmproxy/bundled_addon.py @@ -530,7 +530,7 @@ class MitmproxyMockAddon: Also supports the v1 flat format (just endpoints, no wrapper). """ - # Default config directory — overridden by env var or config + # Default config directory - overridden by env var or config MOCK_DIR = os.environ.get( "MITMPROXY_MOCK_DIR", "/opt/jumpstarter/mitmproxy/mock-responses" ) @@ -988,7 +988,7 @@ async def _handle_rules( await self._send_response(flow, rule) return - # No rule matched — passthrough + # No rule matched - passthrough ctx.log.info( f"No conditional rule matched for {key}, passing through" ) diff --git a/python/packages/jumpstarter-driver-mitmproxy/jumpstarter_driver_mitmproxy/client.py b/python/packages/jumpstarter-driver-mitmproxy/jumpstarter_driver_mitmproxy/client.py index 9506bcea1..a33fda9f3 100644 --- a/python/packages/jumpstarter-driver-mitmproxy/jumpstarter_driver_mitmproxy/client.py +++ b/python/packages/jumpstarter-driver-mitmproxy/jumpstarter_driver_mitmproxy/client.py @@ -679,7 +679,7 @@ def set_mock_sequence(self, method: str, path: str, sequence: List of response steps. Each step has: - status (int) - body (dict) - - repeat (int, optional — last entry repeats forever) + - repeat (int, optional - last entry repeats forever) Example:: @@ -1380,7 +1380,7 @@ def _format_capture_entry(entry: dict) -> str: else: ts_str = click.style("--:--:--", fg="bright_black") - # Color-code HTTP method (padded to 7 chars — length of "OPTIONS") + # Color-code HTTP method (padded to 7 chars - length of "OPTIONS") styled_method = click.style( method.ljust(7), fg=_METHOD_COLORS.get(method, "white"), bold=True, ) @@ -1412,7 +1412,7 @@ def _format_capture_entry(entry: dict) -> str: # Format response size (fixed 8-char column) size_str = click.style(_human_size(response_size).rjust(8), fg="bright_black") - # Mock/patched/passthrough tag (fixed 13-char column — length of "[passthrough]") + # Mock/patched/passthrough tag (fixed 13-char column - length of "[passthrough]") was_patched = entry.get("was_patched", False) if was_patched: tag = click.style("[patched]".ljust(13), fg="yellow") diff --git a/python/packages/jumpstarter-driver-mitmproxy/jumpstarter_driver_mitmproxy/driver.py b/python/packages/jumpstarter-driver-mitmproxy/jumpstarter_driver_mitmproxy/driver.py index ad6f59075..71fc55354 100644 --- a/python/packages/jumpstarter-driver-mitmproxy/jumpstarter_driver_mitmproxy/driver.py +++ b/python/packages/jumpstarter-driver-mitmproxy/jumpstarter_driver_mitmproxy/driver.py @@ -160,7 +160,7 @@ def _convert_url_endpoints(endpoints: dict) -> dict: converted: dict[str, dict] = {} for key, ep in endpoints.items(): if not key.startswith(("http://", "https://")): - # Legacy format — keep as-is + # Legacy format - keep as-is converted[key] = ep continue @@ -622,12 +622,12 @@ def _check_startup_failure(self, web_ui: bool) -> str | None: if self._is_port_in_use(self.listen.host, self.listen.port): port_hint = ( f" (port {self.listen.port} is already in use" - " — is another mitmproxy instance running?)" + " - is another mitmproxy instance running?)" ) elif web_ui and self._is_port_in_use(self.web.host, self.web.port): port_hint = ( f" (web UI port {self.web.port} is already in use" - " — is another mitmproxy instance running?)" + " - is another mitmproxy instance running?)" ) logger.error( "mitmproxy exited during startup (exit code %s)%s: %s", @@ -722,7 +722,7 @@ def stop(self) -> str: self._web_ui_enabled = False self._current_flow_file = None - # Stop capture server (do NOT clear _captured_requests — tests may + # Stop capture server (do NOT clear _captured_requests - tests may # read captures after stop) self._stop_capture_server() @@ -1699,7 +1699,7 @@ def _export_body_to_file( ) -> str: """Write a response body to a file, formatting JSON if possible. - Always writes to a file — never inlines into the YAML. JSON + Always writes to a file - never inlines into the YAML. JSON bodies are pretty-printed for readability. Returns the relative file path for the client to download. @@ -1715,7 +1715,7 @@ def _export_body_to_file( except (UnicodeDecodeError, json.JSONDecodeError, TypeError, ValueError): pass - # Non-JSON — write as-is with an appropriate extension + # Non-JSON - write as-is with an appropriate extension ext = _content_type_to_ext(content_type) return _write_captured_file( method, file_key, ext, raw, endpoint, files_dir, diff --git a/python/packages/jumpstarter-driver-network/README.md b/python/packages/jumpstarter-driver-network/README.md index aea2495df..e3ad19b9a 100644 --- a/python/packages/jumpstarter-driver-network/README.md +++ b/python/packages/jumpstarter-driver-network/README.md @@ -1,4 +1,4 @@ -# Network drivers +# Network Driver `jumpstarter-driver-network` provides functionality for interacting with network servers and connections, redirecting DUT network services to the client handling diff --git a/python/packages/jumpstarter-driver-noyito-relay/README.md b/python/packages/jumpstarter-driver-noyito-relay/README.md index 045717da8..e15768e4d 100644 --- a/python/packages/jumpstarter-driver-noyito-relay/README.md +++ b/python/packages/jumpstarter-driver-noyito-relay/README.md @@ -1,13 +1,13 @@ -# NoyitoPowerSerial / NoyitoPowerHID Driver +# Noyito Relay Driver `jumpstarter-driver-noyito-relay` provides Jumpstarter power drivers for NOYITO USB relay boards in 1, 2, 4, and 8-channel variants. Two hardware series are supported: -- **`NoyitoPowerSerial`** — 1/2-channel boards using a CH340 USB-to-serial chip +- **`NoyitoPowerSerial`** - 1/2-channel boards using a CH340 USB-to-serial chip (serial port, supports status query) -- **`NoyitoPowerHID`** — 4/8-channel "HID Drive-free" boards presenting as a +- **`NoyitoPowerHID`** - 4/8-channel "HID Drive-free" boards presenting as a USB HID device (no serial port, supports all-channels status query) Both use the same 4-byte binary command protocol (`A0` + channel + state + @@ -15,8 +15,9 @@ checksum). ## Installation -```shell -pip3 install --extra-index-url https://pkg.jumpstarter.dev/simple/ jumpstarter-driver-noyito-relay +```{code-block} console +:substitutions: +$ pip3 install --extra-index-url {{index_url}} jumpstarter-driver-noyito-relay ``` If you are using `NoyitoPowerHID`, the `hid` Python package requires the native @@ -28,29 +29,9 @@ If you are using `NoyitoPowerHID`, the `hid` Python package requires the native | Debian/Ubuntu | `sudo apt-get install libhidapi-hidraw0` | | Fedora/RHEL | `sudo dnf install hidapi` | -## Board Detection +## Configuration -To determine which driver to use, check whether the board appears as a serial -port or a HID device: - -- **Serial port** (`/dev/ttyUSB*`, `/dev/tty.usbserial-*`): Use `NoyitoPowerSerial` - (1/2-channel CH340 board) -- **No serial port / HID only**: Use `NoyitoPowerHID` (4/8-channel HID - Drive-free board). Confirm with `lsusb` — the NOYITO HID module appears with - VID `0x1409` / PID `0x07D7` (decimal: 5131 / 2007). - -## `NoyitoPowerSerial` (1/2-Channel Serial) - -### Hardware Notes - -- **Purchase**: [NOYITO 2-Channel USB Relay Module (Amazon)](https://www.amazon.com/NOYITO-2-Channel-Module-Control-Intelligent/dp/B081RM7PMY/) -- **Chip**: CH340 USB-to-serial -- **Baud rate**: 9600 -- **Default port**: `/dev/ttyUSB0` (Linux) — may appear as `/dev/tty.usbserial-*` on macOS -- **Channels**: 1 or 2 independent relay channels on one USB port -- **Supply voltage**: 5 V via USB - -### Configuration +### NoyitoPowerSerial (1/2-Channel Serial) | Parameter | Type | Default | Description | |-----------|------|---------|-------------| @@ -74,49 +55,7 @@ export: channel: 2 ``` -### API Reference - -Implements `PowerInterface` (provides `on`, `off`, `read`, and `cycle` via -`PowerClient`). - -| Method | Description | -|--------|-------------| -| `on()` | Energise the configured relay channel | -| `off()` | De-energise the configured relay channel | -| `read()` | Yields a single `PowerReading(voltage=0.0, current=0.0)` | -| `status()` | Returns the channel state string, e.g. `"on"`, `"off"`, or `"partial"` | - -### CLI Usage - -Inside a `jmp exporter shell`: - -```shell -# Power on relay 1 -j relay1 on - -# Query state of relay 1 -j relay1 status -# on - -# Power cycle relay 2 with a 3-second wait -j relay2 cycle --wait 3 - -# Power off relay 1 -j relay1 off -``` - -## `NoyitoPowerHID` (4/8-Channel HID Drive-free) - -### Hardware Notes - -- **Purchase (4-channel)**: [NOYITO 4-Channel HID Drive-free USB Relay (Amazon)](https://www.amazon.com/NOYITO-Drive-Free-Computer-2-Channel-Micro-USB/dp/B0B538N95Q) -- **Purchase (8-channel)**: [NOYITO 8-Channel HID Drive-free USB Relay (Amazon)](https://www.amazon.com/NOYITO-Drive-Free-Computer-2-Channel-Micro-USB/dp/B0B536M5MH) -- **Interface**: USB HID (no serial port) -- **Default VID/PID**: `5131` / `2007` (0x1409 / 0x07D7) -- **Channels**: 4 or 8 independent relay channels -- **Supply voltage**: 5 V via USB - -### Configuration +### NoyitoPowerHID (4/8-Channel HID) | Parameter | Type | Default | Description | |-----------|------|---------|-------------| @@ -144,32 +83,63 @@ export: all_channels: true ``` -### API Reference +### Board Detection -Implements `PowerInterface` (provides `on`, `off`, `read`, and `cycle` via -`PowerClient`). +To determine which driver to use, check whether the board appears as a serial +port or a HID device: -| Method | Description | -|--------|-------------| -| `on()` | Energise the configured relay channel(s) | -| `off()` | De-energise the configured relay channel(s) | -| `read()` | Yields a single `PowerReading(voltage=0.0, current=0.0)` | -| `status()` | Returns the channel state string, e.g. `"on"`, `"off"`, or `"partial"` | +- **Serial port** (`/dev/ttyUSB*`, `/dev/tty.usbserial-*`): Use `NoyitoPowerSerial` + (1/2-channel CH340 board) +- **No serial port / HID only**: Use `NoyitoPowerHID` (4/8-channel HID + Drive-free board). Confirm with `lsusb` - the NOYITO HID module appears with + VID `0x1409` / PID `0x07D7` (decimal: 5131 / 2007). + +#### Hardware Notes (Serial) + +- **Purchase**: [NOYITO 2-Channel USB Relay Module (Amazon)](https://www.amazon.com/NOYITO-2-Channel-Module-Control-Intelligent/dp/B081RM7PMY/) +- **Chip**: CH340 USB-to-serial +- **Baud rate**: 9600 +- **Default port**: `/dev/ttyUSB0` (Linux) - may appear as `/dev/tty.usbserial-*` on macOS +- **Channels**: 1 or 2 independent relay channels on one USB port +- **Supply voltage**: 5 V via USB -### CLI Usage +#### Hardware Notes (HID) + +- **Purchase (4-channel)**: [NOYITO 4-Channel HID Drive-free USB Relay (Amazon)](https://www.amazon.com/NOYITO-Drive-Free-Computer-2-Channel-Micro-USB/dp/B0B538N95Q) +- **Purchase (8-channel)**: [NOYITO 8-Channel HID Drive-free USB Relay (Amazon)](https://www.amazon.com/NOYITO-Drive-Free-Computer-2-Channel-Micro-USB/dp/B0B536M5MH) +- **Interface**: USB HID (no serial port) +- **Default VID/PID**: `5131` / `2007` (0x1409 / 0x07D7) +- **Channels**: 4 or 8 independent relay channels +- **Supply voltage**: 5 V via USB + +## Usage Inside a `jmp exporter shell`: ```shell -# Power on relay channel 1 of the 4-ch board -j relay_4ch_ch1 on +# Power on relay 1 +j relay1 on + +# Query state of relay 1 +j relay1 status +# on -# Power cycle with a 1-second wait -j relay_4ch_ch1 cycle --wait 1 +# Power cycle relay 2 with a 3-second wait +j relay2 cycle --wait 3 -# Power off -j relay_4ch_ch1 off +# Power off relay 1 +j relay1 off # Power on all 8 channels simultaneously j relay_8ch_all on ``` + +## API Reference + +```{eval-rst} +.. autoclass:: jumpstarter_driver_noyito_relay.driver.NoyitoPowerSerial() + :members: + +.. autoclass:: jumpstarter_driver_noyito_relay.driver.NoyitoPowerHID() + :members: +``` diff --git a/python/packages/jumpstarter-driver-noyito-relay/examples/exporter.yaml b/python/packages/jumpstarter-driver-noyito-relay/examples/exporter.yaml index 12bc659f6..6af63268b 100644 --- a/python/packages/jumpstarter-driver-noyito-relay/examples/exporter.yaml +++ b/python/packages/jumpstarter-driver-noyito-relay/examples/exporter.yaml @@ -22,25 +22,25 @@ export: config: port: "/dev/cu.usbserial-9120" all_channels: true - # 4-channel HID board — individual channel + # 4-channel HID board - individual channel relay_4ch_ch1: type: jumpstarter_driver_noyito_relay.driver.NoyitoPowerHID config: num_channels: 4 channel: 1 - # 4-channel HID board — all_channels=true fires all 4 channels simultaneously + # 4-channel HID board - all_channels=true fires all 4 channels simultaneously relay_4ch_all: type: jumpstarter_driver_noyito_relay.driver.NoyitoPowerHID config: num_channels: 4 all_channels: true - # 8-channel HID board — individual channel + # 8-channel HID board - individual channel relay_8ch_ch1: type: jumpstarter_driver_noyito_relay.driver.NoyitoPowerHID config: num_channels: 8 channel: 1 - # 8-channel HID board — all_channels=true fires all 8 channels simultaneously + # 8-channel HID board - all_channels=true fires all 8 channels simultaneously relay_8ch_all: type: jumpstarter_driver_noyito_relay.driver.NoyitoPowerHID config: diff --git a/python/packages/jumpstarter-driver-noyito-relay/jumpstarter_driver_noyito_relay/driver.py b/python/packages/jumpstarter-driver-noyito-relay/jumpstarter_driver_noyito_relay/driver.py index 95db40509..49addc410 100644 --- a/python/packages/jumpstarter-driver-noyito-relay/jumpstarter_driver_noyito_relay/driver.py +++ b/python/packages/jumpstarter-driver-noyito-relay/jumpstarter_driver_noyito_relay/driver.py @@ -69,23 +69,27 @@ def _channels(self) -> list[int]: @export def on(self) -> None: + """Energise the relay channel.""" for ch in self._channels(): self.logger.info("Relay channel %d ON", ch) self._send_command(_build_command(ch, 1)) @export def off(self) -> None: + """De-energise the relay channel.""" for ch in self._channels(): self.logger.info("Relay channel %d OFF", ch) self._send_command(_build_command(ch, 0)) @export def read(self) -> Generator[PowerReading, None, None]: + """Yield a power reading for the relay channel.""" raise NotImplementedError yield # makes this a generator function @export def status(self) -> str: + """Return the relay channel state as a string.""" all_channels = self._query_status() states = set() for ch in self._channels(): @@ -148,12 +152,14 @@ def _send_command(self, cmd: bytes) -> None: @export def on(self) -> None: + """Energise the relay channel.""" for ch in self._channels(): self.logger.info("HID Relay channel %d ON", ch) self._send_command(_build_command(ch, 1)) @export def off(self) -> None: + """De-energise the relay channel.""" for ch in self._channels(): self.logger.info("HID Relay channel %d OFF", ch) self._send_command(_build_command(ch, 0)) @@ -185,11 +191,13 @@ def _query_status(self) -> dict[str, str]: @export def read(self) -> Generator[PowerReading, None, None]: + """Yield a power reading for the relay channel.""" raise NotImplementedError yield # makes this a generator function @export def status(self) -> str: + """Return the relay channel state as a string.""" states = self._query_status() channel_states = [] for ch in self._channels(): diff --git a/python/packages/jumpstarter-driver-opendal/README.md b/python/packages/jumpstarter-driver-opendal/README.md index 234b13824..99199ebed 100644 --- a/python/packages/jumpstarter-driver-opendal/README.md +++ b/python/packages/jumpstarter-driver-opendal/README.md @@ -1,4 +1,4 @@ -# OpenDAL driver +# OpenDAL Driver `jumpstarter-driver-opendal` provides functionality for interacting with storages attached to the exporter. diff --git a/python/packages/jumpstarter-driver-opendal/jumpstarter_driver_opendal/client.py b/python/packages/jumpstarter-driver-opendal/jumpstarter_driver_opendal/client.py index f5a9597fc..22d195fbe 100644 --- a/python/packages/jumpstarter-driver-opendal/jumpstarter_driver_opendal/client.py +++ b/python/packages/jumpstarter-driver-opendal/jumpstarter_driver_opendal/client.py @@ -121,11 +121,11 @@ def seek(self, pos: int, whence: int = 0) -> int: Offset is interpreted relative to the position indicated by whence. The default value for whence is SEEK_SET. Values for whence are: - SEEK_SET or 0 – start of the file (the default); offset should be zero or positive + SEEK_SET or 0 - start of the file (the default); offset should be zero or positive - SEEK_CUR or 1 – current cursor position; offset may be negative + SEEK_CUR or 1 - current cursor position; offset may be negative - SEEK_END or 2 – end of the file; offset is usually negative + SEEK_END or 2 - end of the file; offset is usually negative Return the new cursor position """ diff --git a/python/packages/jumpstarter-driver-opendal/jumpstarter_driver_opendal/driver_test.py b/python/packages/jumpstarter-driver-opendal/jumpstarter_driver_opendal/driver_test.py index f4bc3cff4..9b8bb7e58 100644 --- a/python/packages/jumpstarter-driver-opendal/jumpstarter_driver_opendal/driver_test.py +++ b/python/packages/jumpstarter-driver-opendal/jumpstarter_driver_opendal/driver_test.py @@ -362,7 +362,7 @@ def log_message(self, format, *args): def _assert_encoding_preserved(received_paths): assert len(received_paths) >= 1 assert "%40" in received_paths[-1], ( - f"Server received decoded path {received_paths[-1]!r} — " + f"Server received decoded path {received_paths[-1]!r} - " f"original_url bypass did not activate with explicit operator" ) @@ -431,7 +431,7 @@ def log_message(self, format, *args): assert len(received_paths) >= 1 assert "%40" in received_paths[-1], ( - f"Server received decoded path {received_paths[-1]!r} — " + f"Server received decoded path {received_paths[-1]!r} - " f"_make_url is not preserving percent-encoding" ) finally: @@ -489,7 +489,7 @@ def log_message(self, format, *args): ) assert received_paths[0] == "/start" assert "%40" in received_paths[1], ( - f"Redirect target received decoded path {received_paths[1]!r} — " + f"Redirect target received decoded path {received_paths[1]!r} - " f"redirect following is not preserving percent-encoding" ) finally: diff --git a/python/packages/jumpstarter-driver-pi-pico/README.md b/python/packages/jumpstarter-driver-pi-pico/README.md index 58631b09b..aed21560f 100644 --- a/python/packages/jumpstarter-driver-pi-pico/README.md +++ b/python/packages/jumpstarter-driver-pi-pico/README.md @@ -1,15 +1,22 @@ -# PiPicoFlasher Driver +# Pi Pico Driver `jumpstarter-driver-pi-pico` flashes Raspberry Pi **Pico** (RP2040), **Pico W**, and **Pico 2** (RP2350) by copying a UF2 file onto the **BOOTSEL** USB mass-storage volume. The driver supports two methods for entering BOOTSEL mode programmatically: -1. **GPIO reset** — wire the Pico's BOOTSEL pad and RUN pin to host GPIO +1. **GPIO reset** - wire the Pico's BOOTSEL pad and RUN pin to host GPIO lines. -2. **1200-baud serial touch** — uses a USB CDC serial child. Only works when +2. **1200-baud serial touch** - uses a USB CDC serial child. Only works when the running firmware implements the convention (Pico SDK `pico_stdio_usb`, CircuitPython, Arduino). +## Installation + +```{code-block} console +:substitutions: +$ pip3 install --extra-index-url {{index_url}} jumpstarter-driver-pi-pico +``` + ## Configuration ### Serial-based BOOTSEL entry @@ -63,7 +70,7 @@ export: bootsel: type: jumpstarter_driver_gpiod.driver.DigitalOutput config: - device: "/dev/gpiochip4" # RPi5 GPIO chip — adjust for your host + device: "/dev/gpiochip4" # RPi5 GPIO chip - adjust for your host line: 17 # GPIO pin wired to BOOTSEL drive: open_drain active_low: true @@ -80,15 +87,14 @@ export: When both GPIO and serial children are present, GPIO reset is preferred. -## Shell commands +## Usage -- `j storage flash ...` — flash a UF2 file (auto-enters BOOTSEL if needed) -- `j storage bootloader` — request BOOTSEL mode without flashing -- `j serial ...` — USB CDC console (when serial child is configured) +- `j storage flash ...` - flash a UF2 file (auto-enters BOOTSEL if needed) +- `j storage bootloader` - request BOOTSEL mode without flashing +- `j serial ...` - USB CDC console (when serial child is configured) -## API +## API Reference -- **`flash(source, target=None)`** — Copies a UF2 from a Jumpstarter resource to the BOOTSEL volume. `target` is the destination filename (default `Firmware.uf2`). -- **`enter_bootloader()`** — Enters BOOTSEL mode via GPIO reset or 1200-baud serial touch. -- **`bootloader_info()`** — Parses `INFO_UF2.TXT` from the mounted volume. -- **`dump`** — Not supported over UF2 mass storage. +```{eval-rst} +.. autoclass:: jumpstarter_driver_pi_pico.driver.PiPicoFlasher() +``` diff --git a/python/packages/jumpstarter-driver-pi-pico/examples/pico-exporter.yaml b/python/packages/jumpstarter-driver-pi-pico/examples/pico-exporter.yaml index 08cadc36c..a6242f6bf 100644 --- a/python/packages/jumpstarter-driver-pi-pico/examples/pico-exporter.yaml +++ b/python/packages/jumpstarter-driver-pi-pico/examples/pico-exporter.yaml @@ -1,5 +1,5 @@ # Pico-only exporter (UF2 over BOOTSEL mass storage). Use this file when the host -# should export only the Pi Pico flasher—no other drivers on the same exporter. +# should export only the Pi Pico flasher - no other drivers on the same exporter. # # Register with a Jumpstarter controller (set endpoint and token from your environment): apiVersion: jumpstarter.dev/v1alpha1 diff --git a/python/packages/jumpstarter-driver-pi-pico/jumpstarter_driver_pi_pico/bootloader_mount.py b/python/packages/jumpstarter-driver-pi-pico/jumpstarter_driver_pi_pico/bootloader_mount.py index 6007f682e..a4afb816d 100644 --- a/python/packages/jumpstarter-driver-pi-pico/jumpstarter_driver_pi_pico/bootloader_mount.py +++ b/python/packages/jumpstarter-driver-pi-pico/jumpstarter_driver_pi_pico/bootloader_mount.py @@ -54,7 +54,7 @@ def iter_bootloader_mount_candidates() -> list[Path]: Unlike picotool (USB protocol), UF2 flashing needs a host path to the mounted FAT volume. We discover it by finding mount points whose root contains ``INFO_UF2.TXT`` - or ``INDEX.HTM`` — no volume *name* configuration required. + or ``INDEX.HTM`` - no volume *name* configuration required. """ if sys.platform == "linux": return _mount_points_linux() diff --git a/python/packages/jumpstarter-driver-pi-pico/jumpstarter_driver_pi_pico/driver.py b/python/packages/jumpstarter-driver-pi-pico/jumpstarter_driver_pi_pico/driver.py index 4138ec0cc..07037e119 100644 --- a/python/packages/jumpstarter-driver-pi-pico/jumpstarter_driver_pi_pico/driver.py +++ b/python/packages/jumpstarter-driver-pi-pico/jumpstarter_driver_pi_pico/driver.py @@ -39,11 +39,11 @@ class PiPicoFlasher(FlasherInterface, Driver): BOOTSEL entry methods (tried in priority order by ``enter_bootloader``): - 1. **GPIO reset** — ``bootsel`` + ``run`` children (DigitalOutput). + 1. **GPIO reset** - ``bootsel`` + ``run`` children (DigitalOutput). Assert BOOTSEL low, pulse RUN low, release. Works regardless of firmware. Requires two GPIO lines wired to the Pico BOOTSEL pad and RUN pin. - 2. **1200-baud serial touch** — ``serial`` child. Opens the USB CDC port + 2. **1200-baud serial touch** - ``serial`` child. Opens the USB CDC port at 1200 baud and toggles DTR. Only works when the running firmware implements the convention (Pico SDK ``pico_stdio_usb``, CircuitPython, MicroPython, Arduino). diff --git a/python/packages/jumpstarter-driver-power/README.md b/python/packages/jumpstarter-driver-power/README.md index a66e0ecbd..394330e8b 100644 --- a/python/packages/jumpstarter-driver-power/README.md +++ b/python/packages/jumpstarter-driver-power/README.md @@ -1,4 +1,4 @@ -# Power driver +# Power Driver `jumpstarter-driver-power` provides functionality for interacting with power control devices. diff --git a/python/packages/jumpstarter-driver-probe-rs/README.md b/python/packages/jumpstarter-driver-probe-rs/README.md index b477253b7..b70263fd9 100644 --- a/python/packages/jumpstarter-driver-probe-rs/README.md +++ b/python/packages/jumpstarter-driver-probe-rs/README.md @@ -1,4 +1,4 @@ -# probe-rs driver +# Probe-RS Driver `jumpstarter-driver-probe-rs` provides functionality for remote debugging and flashing of embedded devices using the [probe-rs](https://probe.rs) tools. diff --git a/python/packages/jumpstarter-driver-pyserial/README.md b/python/packages/jumpstarter-driver-pyserial/README.md index 908f72297..806775e0d 100644 --- a/python/packages/jumpstarter-driver-pyserial/README.md +++ b/python/packages/jumpstarter-driver-pyserial/README.md @@ -1,4 +1,4 @@ -# PySerial driver +# PySerial Driver `jumpstarter-driver-pyserial` provides functionality for serial port communication. @@ -47,19 +47,19 @@ export: | cps | Characters per second throttling limit. When set, data transmission will be throttled to simulate slow typing. Useful for devices that can't handle fast input | float | no | None | | disable_hupcl | Disable HUPCL on POSIX systems to avoid toggling DTR/RTS on close (can prevent MCU reset on serial disconnect) | bool | no | False | -## NVDemuxSerial Driver +### NVDemuxSerial Driver The `NVDemuxSerial` driver provides serial access to NVIDIA Tegra demultiplexed UART channels using the [nv_tcu_demuxer](https://docs.nvidia.com/jetson/archives/r38.2.1/DeveloperGuide/AT/JetsonLinuxDevelopmentTools/TegraCombinedUART.html) tool. It automatically handles device reconnection when the target device restarts. The nv_tcu_demuxer tool can be obtained from the NVIDIA Jetson BSP, at this path: `Linux_for_Tegra/tools/demuxer/nv_tcu_demuxer`. -### Multi-Instance Support +#### Multi-Instance Support Multiple driver instances can share a single demuxer process by specifying different target channels. This allows simultaneous access to multiple UART channels (CCPLEX, BPMP, SCE, etc.) from the same physical device. -### Configuration +#### Configuration -#### Single channel example: +##### Single channel example: ```yaml export: @@ -71,7 +71,7 @@ export: # chip defaults to T264 (Thor), use T234 for Orin ``` -#### Multiple channels example: +##### Multiple channels example: ```yaml export: @@ -97,7 +97,7 @@ export: chip: "T264" ``` -### Config parameters +#### Config parameters | Parameter | Description | Type | Required | Default | | -------------- | ----------------------------------------------------------------------------------------------- | ----- | -------- | ------------------------------------------------------------------------- | @@ -110,7 +110,7 @@ export: | timeout | Timeout in seconds waiting for demuxer to detect pts | float | no | 10.0 | | poll_interval | Interval in seconds to poll for device reappearance after disconnect | float | no | 1.0 | -### Device Auto-Detection +#### Device Auto-Detection The `device` parameter supports glob patterns for automatic device discovery: @@ -125,7 +125,7 @@ device: "/dev/serial/by-id/usb-NVIDIA_Tegra_On-Platform_Operator_ABC123-if01" device: "/dev/ttyUSB0" ``` -### Auto-Recovery +#### Auto-Recovery When the target device restarts (e.g., power cycle), the serial device disappears and the demuxer exits. The driver automatically: @@ -136,7 +136,7 @@ When the target device restarts (e.g., power cycle), the serial device disappear Active connections will receive errors when the device disconnects. Clients should reconnect, and the driver will wait for the device to be available again. -### Configuration Validation / Limitations +#### Configuration Validation / Limitations When using multiple driver instances, all instances must have compatible configurations: @@ -149,7 +149,7 @@ If these requirements are not met, the driver will raise a `ValueError` during i -## CLI Commands +## Usage The pyserial driver provides two CLI commands for interacting with serial ports: diff --git a/python/packages/jumpstarter-driver-pyserial/jumpstarter_driver_pyserial/driver_test.py b/python/packages/jumpstarter-driver-pyserial/jumpstarter_driver_pyserial/driver_test.py index f373fa117..5361c9b86 100644 --- a/python/packages/jumpstarter-driver-pyserial/jumpstarter_driver_pyserial/driver_test.py +++ b/python/packages/jumpstarter-driver-pyserial/jumpstarter_driver_pyserial/driver_test.py @@ -106,7 +106,7 @@ def test_cps_zero_disables_throttling(): end_time = time.perf_counter() elapsed_time = end_time - start_time - # With CPS=0, should be fast (no throttling) – allow headroom + # With CPS=0, should be fast (no throttling) - allow headroom assert elapsed_time < 0.5, f"Expected fast transmission with cps=0, got {elapsed_time}s" received = stream.receive() diff --git a/python/packages/jumpstarter-driver-qemu/README.md b/python/packages/jumpstarter-driver-qemu/README.md index 00484a57a..443519a3a 100644 --- a/python/packages/jumpstarter-driver-qemu/README.md +++ b/python/packages/jumpstarter-driver-qemu/README.md @@ -1,4 +1,4 @@ -# QEMU driver +# QEMU Driver `jumpstarter-driver-qemu` provides functionality for interacting with QEMU virtualization platform. @@ -24,4 +24,6 @@ export: ## API Reference -Add API documentation here. +```{eval-rst} +.. autoclass:: jumpstarter_driver_qemu.driver.Qemu() +``` diff --git a/python/packages/jumpstarter-driver-qemu/jumpstarter_driver_qemu/driver.py b/python/packages/jumpstarter-driver-qemu/jumpstarter_driver_qemu/driver.py index d22753d28..8b993fb11 100644 --- a/python/packages/jumpstarter-driver-qemu/jumpstarter_driver_qemu/driver.py +++ b/python/packages/jumpstarter-driver-qemu/jumpstarter_driver_qemu/driver.py @@ -427,6 +427,8 @@ class Hostfwd(BaseModel): @dataclass(kw_only=True) class Qemu(Driver): + """QEMU virtual machine management driver.""" + arch: str = field(default_factory=platform.machine) cpu: str | None = None diff --git a/python/packages/jumpstarter-driver-renode/README.md b/python/packages/jumpstarter-driver-renode/README.md index fe2430c15..bfcc6e60f 100644 --- a/python/packages/jumpstarter-driver-renode/README.md +++ b/python/packages/jumpstarter-driver-renode/README.md @@ -1,4 +1,4 @@ -# Renode driver +# Renode Driver `jumpstarter-driver-renode` provides a Jumpstarter driver for the [Renode](https://renode.io/) embedded systems emulation framework. It @@ -15,16 +15,6 @@ $ pip3 install --extra-index-url {{index_url}} jumpstarter-driver-renode Renode must be installed separately and available in `PATH`. See [Renode installation](https://renode.readthedocs.io/en/latest/introduction/installing.html). -## Architecture - -The driver follows the composite driver pattern: - -- **`Renode`** -- root composite driver, manages the simulation lifecycle -- **`RenodePower`** -- starts/stops the Renode process and controls the - simulation via the telnet monitor interface -- **`RenodeFlasher`** -- loads firmware (ELF/BIN/HEX) into the simulated MCU -- **`console`** -- UART output via PTY terminal, reusing the `PySerial` driver - ## Configuration Users define Renode targets entirely through YAML configuration. No @@ -110,7 +100,17 @@ response = renode.monitor_cmd("sysbus GetRegistrationPoints sysbus.usart2") The `monitor` CLI subcommand is also available inside a `jmp shell` session. -## Design Decisions +## Architecture + +The driver follows the composite driver pattern: + +- **`Renode`** - root composite driver, manages the simulation lifecycle +- **`RenodePower`** - starts/stops the Renode process and controls the + simulation via the telnet monitor interface +- **`RenodeFlasher`** - loads firmware (ELF/BIN/HEX) into the simulated MCU +- **`console`** - UART output via PTY terminal, reusing the `PySerial` driver + +### Design Decisions Key decisions: @@ -120,3 +120,9 @@ Key decisions: - **Configuration model**: Managed mode with `extra_commands` for target-specific customization - **Firmware loading**: `flash()` stores path, `on()` loads into simulation + +## API Reference + +```{eval-rst} +.. autoclass:: jumpstarter_driver_renode.driver.Renode() +``` diff --git a/python/packages/jumpstarter-driver-renode/examples/exporter.yaml b/python/packages/jumpstarter-driver-renode/examples/exporter.yaml index 4180a5985..beb96c059 100644 --- a/python/packages/jumpstarter-driver-renode/examples/exporter.yaml +++ b/python/packages/jumpstarter-driver-renode/examples/exporter.yaml @@ -2,7 +2,7 @@ # # Each example shows a different Renode target. The driver accepts any # .repl platform description (built-in or custom) and any UART peripheral -# path -- new targets require only YAML configuration, no code changes. +# path - new targets require only YAML configuration, no code changes. # --- STM32F407 Discovery (opensomeip FreeRTOS/ThreadX) --- # Uses Renode's built-in platform, USART2 for console output. diff --git a/python/packages/jumpstarter-driver-renode/jumpstarter_driver_renode/driver.py b/python/packages/jumpstarter-driver-renode/jumpstarter_driver_renode/driver.py index afcd4c0d6..ecd5f84b0 100644 --- a/python/packages/jumpstarter-driver-renode/jumpstarter_driver_renode/driver.py +++ b/python/packages/jumpstarter-driver-renode/jumpstarter_driver_renode/driver.py @@ -44,7 +44,7 @@ def _detect_load_command(firmware_path: str) -> str: def _find_free_port() -> int: - # NOTE: TOCTOU race — the port is released before Renode binds it, + # NOTE: TOCTOU race - the port is released before Renode binds it, # so another process could grab it first. Switching to Unix domain # sockets would eliminate this, but Renode does not yet support them. with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s: @@ -200,7 +200,7 @@ async def off(self) -> None: @export async def read(self) -> AsyncGenerator[PowerReading, None]: - """Not supported — Renode does not provide power readings.""" + """Not supported - Renode does not provide power readings.""" raise NotImplementedError def close(self): diff --git a/python/packages/jumpstarter-driver-ridesx/README.md b/python/packages/jumpstarter-driver-ridesx/README.md index 23081ad44..c268800b4 100644 --- a/python/packages/jumpstarter-driver-ridesx/README.md +++ b/python/packages/jumpstarter-driver-ridesx/README.md @@ -1,7 +1,10 @@ -# RideSX driver +# RideSX Driver `jumpstarter-driver-ridesx` provides functionality for Qualcomm RideSX devices, supporting fastboot flashing operations and power control through serial communication. +It includes automatic compression handling (`.gz`, `.gzip`, `.xz`), built-in storage +for firmware images with upload/download capabilities, and direct access to the +underlying serial interface for custom commands. This is mainly tailored towards images that were produced using [automotive-image-builder](https://sigs.centos.org/automotive/latest/getting-started/about-automotive-image-builder.html): @@ -85,23 +88,7 @@ Both drivers require: | ------ | ------------------------------------------------------------ | -------- | | serial | PySerial driver instance for communicating with the device | yes | -## API Reference - -### RideSXClient - -```{eval-rst} -.. autoclass:: jumpstarter_driver_ridesx.client.RideSXClient() - :members: flash, flash_images, boot_to_fastboot, cli -``` - -### RideSXPowerClient - -```{eval-rst} -.. autoclass:: jumpstarter_driver_ridesx.client.RideSXPowerClient() - :members: on, off, cycle, rescue, serial -``` - -## Usage Examples +## Usage ### Flash Single Partition @@ -144,10 +131,18 @@ power_client.off() power_client.cycle(wait=5) # Wait 5 seconds between off/on ``` -## Features +## API Reference + +### RideSXClient -- **Fastboot Support**: Automatically detects fastboot devices and flashes partitions -- **Compression Handling**: Supports automatic decompression of `.gz`, `.gzip`, and `.xz` files -- **Power Control**: Serial-based power control with on/off/cycle operations -- **Storage Management**: Built-in storage for firmware images with upload/download capabilities -- **Serial Communication**: Direct access to underlying serial interface for custom commands +```{eval-rst} +.. autoclass:: jumpstarter_driver_ridesx.client.RideSXClient() + :members: flash, flash_images, boot_to_fastboot, cli +``` + +### RideSXPowerClient + +```{eval-rst} +.. autoclass:: jumpstarter_driver_ridesx.client.RideSXPowerClient() + :members: on, off, cycle, rescue, serial +``` diff --git a/python/packages/jumpstarter-driver-ridesx/jumpstarter_driver_ridesx/client.py b/python/packages/jumpstarter-driver-ridesx/jumpstarter_driver_ridesx/client.py index 774caacdb..82f4a460e 100644 --- a/python/packages/jumpstarter-driver-ridesx/jumpstarter_driver_ridesx/client.py +++ b/python/packages/jumpstarter-driver-ridesx/jumpstarter_driver_ridesx/client.py @@ -195,11 +195,11 @@ def flash( if isinstance(path, str) and ":" in path: before_colon, after_colon = path.split(":", 1) if "/" in before_colon: - # registry/path:tag — likely an OCI ref missing the oci:// prefix + # registry/path:tag - likely an OCI ref missing the oci:// prefix raise click.ClickException( f"OCI URLs must start with oci://, got: {path}\nUsage: j storage flash oci://{path}" ) - # partition:something — likely a partition:path mapping + # partition:something - likely a partition:path mapping raise click.ClickException( f"'{path}' looks like a partition:path mapping.\n" f"Use the -t flag: j storage flash -t {path}\n" diff --git a/python/packages/jumpstarter-driver-sdwire/README.md b/python/packages/jumpstarter-driver-sdwire/README.md index 30d4a6644..31f5f19cc 100644 --- a/python/packages/jumpstarter-driver-sdwire/README.md +++ b/python/packages/jumpstarter-driver-sdwire/README.md @@ -1,4 +1,4 @@ -# SDWire driver +# SD Wire Driver `jumpstarter-driver-sdwire` provides functionality for using the SDWire storage multiplexer. This device multiplexes an SD card between the DUT and the exporter diff --git a/python/packages/jumpstarter-driver-shell/README.md b/python/packages/jumpstarter-driver-shell/README.md index aa552cec5..501ef1a9d 100644 --- a/python/packages/jumpstarter-driver-shell/README.md +++ b/python/packages/jumpstarter-driver-shell/README.md @@ -1,4 +1,4 @@ -# Shell driver +# Shell Driver `jumpstarter-driver-shell` provides functionality for shell command execution. @@ -83,37 +83,7 @@ For the dict format, each method supports: **Note:** You can mix both formats in the same configuration - use string format for simple commands and dict format when you want custom descriptions or timeouts. -## API Reference - -Assuming the exporter driver is configured as in the example above, the client -methods will be generated dynamically, and they will be available as follows: - -```{eval-rst} -.. autoclass:: jumpstarter_driver_shell.client.ShellClient - :members: - -.. function:: ls() - :noindex: - - :returns: A tuple(stdout, stderr, return_code) - -.. function:: method2() - :noindex: - - :returns: A tuple(stdout, stderr, return_code) - -.. function:: method3(arg1, arg2) - :noindex: - - :returns: A tuple(stdout, stderr, return_code) - -.. function:: env_var(arg1, arg2, ENV_VAR="value") - :noindex: - - :returns: A tuple(stdout, stderr, return_code) -``` - -## CLI Usage +## Usage The shell driver also provides a CLI when using `jmp shell`. All configured methods become available as CLI commands, except for methods starting with `_` which are considered private and hidden from the end user. @@ -196,3 +166,33 @@ Hello World second arg $ j shell env_var arg1 arg2 --env ENV_VAR=myvalue arg1,arg2,myvalue ``` + +## API Reference + +Assuming the exporter driver is configured as in the example above, the client +methods will be generated dynamically, and they will be available as follows: + +```{eval-rst} +.. autoclass:: jumpstarter_driver_shell.client.ShellClient + :members: + +.. function:: ls() + :noindex: + + :returns: A tuple(stdout, stderr, return_code) + +.. function:: method2() + :noindex: + + :returns: A tuple(stdout, stderr, return_code) + +.. function:: method3(arg1, arg2) + :noindex: + + :returns: A tuple(stdout, stderr, return_code) + +.. function:: env_var(arg1, arg2, ENV_VAR="value") + :noindex: + + :returns: A tuple(stdout, stderr, return_code) +``` diff --git a/python/packages/jumpstarter-driver-snmp/README.md b/python/packages/jumpstarter-driver-snmp/README.md index a4799f2a0..7b1d8bdc8 100644 --- a/python/packages/jumpstarter-driver-snmp/README.md +++ b/python/packages/jumpstarter-driver-snmp/README.md @@ -1,4 +1,4 @@ -# SNMP driver +# SNMP Driver `jumpstarter-driver-snmp` provides functionality for controlling power via SNMP-enabled PDUs (Power Distribution Units). diff --git a/python/packages/jumpstarter-driver-someip/README.md b/python/packages/jumpstarter-driver-someip/README.md index 0d6aead2a..5652d385d 100644 --- a/python/packages/jumpstarter-driver-someip/README.md +++ b/python/packages/jumpstarter-driver-someip/README.md @@ -68,33 +68,7 @@ export: remote_port: 30490 ``` -## API Reference - -### RPC - -- `rpc_call(service_id, method_id, payload, timeout=5.0)` — Make a SOME/IP RPC call and return the response - -### Raw Messaging - -- `send_message(service_id, method_id, payload)` — Send a raw SOME/IP message -- `receive_message(timeout=2.0)` — Receive a raw SOME/IP message - -### Service Discovery - -- `find_service(service_id, instance_id=0xFFFF, timeout=5.0)` — Find services via SOME/IP-SD; use `instance_id=0xFFFF` (default) to match any instance - -### Events - -- `subscribe_eventgroup(eventgroup_id)` — Subscribe to a SOME/IP event group -- `unsubscribe_eventgroup(eventgroup_id)` — Unsubscribe from a SOME/IP event group -- `receive_event(timeout=5.0)` — Receive next event notification - -### Connection Management - -- `close_connection()` — Close the SOME/IP connection -- `reconnect()` — Reconnect to the SOME/IP endpoint - -## Example Usage +## Usage ### RPC Call @@ -181,3 +155,9 @@ with env() as client: # Clean up someip.close_connection() ``` + +## API Reference + +```{eval-rst} +.. autoclass:: jumpstarter_driver_someip.driver.SomeIp() +``` diff --git a/python/packages/jumpstarter-driver-someip/examples/exporter.yaml b/python/packages/jumpstarter-driver-someip/examples/exporter.yaml index 018b5e29f..3f7d14b66 100644 --- a/python/packages/jumpstarter-driver-someip/examples/exporter.yaml +++ b/python/packages/jumpstarter-driver-someip/examples/exporter.yaml @@ -15,7 +15,7 @@ export: multicast_group: "239.127.0.1" multicast_port: 30490 --- -# Static endpoint (no Service Discovery) — for ECUs that don't run SOME/IP-SD +# Static endpoint (no Service Discovery) - for ECUs that don't run SOME/IP-SD apiVersion: jumpstarter.dev/v1alpha1 kind: ExporterConfig metadata: diff --git a/python/packages/jumpstarter-driver-someip/jumpstarter_driver_someip/conftest.py b/python/packages/jumpstarter-driver-someip/jumpstarter_driver_someip/conftest.py index 54d457a31..b847d72fe 100644 --- a/python/packages/jumpstarter-driver-someip/jumpstarter_driver_someip/conftest.py +++ b/python/packages/jumpstarter-driver-someip/jumpstarter_driver_someip/conftest.py @@ -99,7 +99,7 @@ def _read_someip_message(conn: socket.socket) -> tuple[int, int, int, int, int, # ========================================================================= -# MockSomeIpServer — minimal TCP server for wire-level integration tests +# MockSomeIpServer - minimal TCP server for wire-level integration tests # ========================================================================= @@ -195,7 +195,7 @@ def mock_someip_server(): # ========================================================================= -# StatefulOsipClient — drop-in for opensomeip.SomeIpClient +# StatefulOsipClient - drop-in for opensomeip.SomeIpClient # # Tracks connection state, service registry, event subscriptions, # message history, and enforces ordering rules. Designed to be @@ -307,7 +307,7 @@ def __init__(self, config=None) -> None: def _require_started(self): if not self._started: - raise SomeIpNotStarted("Client not started — call start() first") + raise SomeIpNotStarted("Client not started - call start() first") def start(self): self._started = True @@ -355,7 +355,7 @@ def unsubscribe_events(self, eventgroup_id: int): self._require_started() self._subscribed_eventgroups.discard(eventgroup_id) - # -- test helpers -- + # - test helpers -- def inject_event(self, service_id: int, event_id: int, payload: bytes): """Push a fake event notification into the event receiver queue.""" diff --git a/python/packages/jumpstarter-driver-someip/jumpstarter_driver_someip/driver_test.py b/python/packages/jumpstarter-driver-someip/jumpstarter_driver_someip/driver_test.py index 755fcf1a5..ec992b718 100644 --- a/python/packages/jumpstarter-driver-someip/jumpstarter_driver_someip/driver_test.py +++ b/python/packages/jumpstarter-driver-someip/jumpstarter_driver_someip/driver_test.py @@ -56,7 +56,7 @@ def _make_mock_osip_client(): # ========================================================================= -# Unit tests — happy paths +# Unit tests - happy paths # ========================================================================= @@ -560,7 +560,7 @@ def stateful_client(stateful_osip): yield from _stateful_client_ctx(stateful_osip) -# -- RPC workflows --------------------------------------------------------- +# - RPC workflows --------------------------------------------------------- def test_stateful_rpc_call_returns_canned_response(stateful_client, stateful_osip): @@ -600,7 +600,7 @@ def test_stateful_custom_rpc_response(stateful_client, stateful_osip): assert resp.payload == "cafe" -# -- send / receive messaging workflow ------------------------------------- +# - send / receive messaging workflow ------------------------------------- def test_stateful_send_then_receive(stateful_client, stateful_osip): @@ -637,7 +637,7 @@ def test_stateful_multiple_messages_fifo(stateful_client, stateful_osip): assert r3.service_id == 0x3333 -# -- service discovery workflow -------------------------------------------- +# - service discovery workflow -------------------------------------------- def test_stateful_find_service_all_instances(stateful_client): @@ -738,7 +738,7 @@ def test_stateful_discover_then_rpc_to_each_instance(stateful_client, stateful_o assert len(stateful_osip._rpc_history) == 2 -# -- event subscription workflow ------------------------------------------- +# - event subscription workflow ------------------------------------------- def test_stateful_subscribe_receive_unsubscribe(stateful_client, stateful_osip): @@ -793,7 +793,7 @@ def test_stateful_event_timeout_when_no_events(stateful_client): stateful_client.receive_event(timeout=0.1) -# -- connection management workflows --------------------------------------- +# - connection management workflows --------------------------------------- def test_stateful_reconnect_resets_subscriptions(stateful_client, stateful_osip): @@ -815,7 +815,7 @@ def test_stateful_close_then_reconnect(stateful_client, stateful_osip): assert stateful_osip._started is True -# -- end-to-end composite workflows ---------------------------------------- +# - end-to-end composite workflows ---------------------------------------- def test_stateful_full_rpc_session(stateful_client, stateful_osip): diff --git a/python/packages/jumpstarter-driver-ssh-mitm/README.md b/python/packages/jumpstarter-driver-ssh-mitm/README.md index 45acfdaa9..3364da69f 100644 --- a/python/packages/jumpstarter-driver-ssh-mitm/README.md +++ b/python/packages/jumpstarter-driver-ssh-mitm/README.md @@ -11,16 +11,6 @@ used as a child of `SSHWrapper`. $ pip3 install --extra-index-url {{index_url}} jumpstarter-driver-ssh-mitm ``` -## Architecture - -``` -SSHWrapper --> SSHMITM --> TcpNetwork --> DUT -``` - -- **SSHWrapper**: Handles SSH CLI and command execution -- **SSHMITM**: Provides authenticated proxy connection (stores the SSH key) -- **TcpNetwork**: Raw TCP connection to the DUT - ## Configuration The command name is determined by the key in the `export` section. Use `ssh_mitm` to get the `j ssh_mitm` command: @@ -104,6 +94,16 @@ j ssh_mitm -v hostname **Note**: The command name (`ssh_mitm`) is determined by the key in your exporter config's `export` section. You can use any name you prefer. +## Architecture + +``` +SSHWrapper --> SSHMITM --> TcpNetwork --> DUT +``` + +- **SSHWrapper**: Handles SSH CLI and command execution +- **SSHMITM**: Provides authenticated proxy connection (stores the SSH key) +- **TcpNetwork**: Raw TCP connection to the DUT + ## API Reference ```{eval-rst} diff --git a/python/packages/jumpstarter-driver-ssh/README.md b/python/packages/jumpstarter-driver-ssh/README.md index 91b708751..707c378ce 100644 --- a/python/packages/jumpstarter-driver-ssh/README.md +++ b/python/packages/jumpstarter-driver-ssh/README.md @@ -1,4 +1,4 @@ -# SSHWrapper Driver +# SSH Driver `jumpstarter-driver-ssh` provides SSH CLI functionality for Jumpstarter, allowing you to run SSH commands with configurable defaults and pass-through arguments. @@ -8,6 +8,10 @@ pip3 install --extra-index-url https://pkg.jumpstarter.dev/simple/ jumpstarter-driver-ssh ``` +### Dependencies + +- `ssh`: Standard SSH client (usually pre-installed) + ## Configuration Example configuration: @@ -49,7 +53,7 @@ j ssh ls -la ``` -## CLI Options +### CLI Options The SSH command supports the following options: @@ -57,7 +61,7 @@ The SSH command supports the following options: All other arguments are passed directly to the SSH command. The driver uses the configured SSH command and default username from the driver configuration. -### Username Handling +#### Username Handling The driver supports multiple ways to specify the username: @@ -66,10 +70,6 @@ The driver supports multiple ways to specify the username: If no `-l` flag or `user@hostname` format is provided, the default username from the driver configuration will be used automatically. -## Dependencies - -- `ssh`: Standard SSH client (usually pre-installed) - ## API Reference ### Driver Methods diff --git a/python/packages/jumpstarter-driver-stlink-msd/README.md b/python/packages/jumpstarter-driver-stlink-msd/README.md index c702b437a..6545d99b0 100644 --- a/python/packages/jumpstarter-driver-stlink-msd/README.md +++ b/python/packages/jumpstarter-driver-stlink-msd/README.md @@ -1,4 +1,4 @@ -# ST-LINK Mass Storage Flasher +# ST-LINK MSD Driver `jumpstarter-driver-stlink-msd` flashes STM32 **Nucleo** and **Discovery** boards by copying firmware to the **ST-LINK USB mass storage volume**. @@ -7,19 +7,6 @@ This is an alternative to probe-rs that avoids known [connect-under-reset issues with ST-Link V3](https://github.com/probe-rs/probe-rs/issues/3516). The ST-LINK's built-in mass storage interface handles all the flash programming. -## Supported Formats - -| Format | Handling | -|--------|----------| -| `.bin` | Copied directly to the ST-LINK volume | -| `.hex` | Copied directly to the ST-LINK volume | - -ELF files must be converted externally before flashing: - -```shell -arm-none-eabi-objcopy -O binary zephyr.elf zephyr.bin -``` - ## Installation ```shell @@ -40,7 +27,20 @@ export: |---------------|------------------------------------------------------------------|----------------|----------|--------------| | volume_name | Name of the mounted ST-LINK volume (e.g. `NOD_H755ZI`) | str \| None | no | auto-detect | -## Shell Commands +### Supported Formats + +| Format | Handling | +|--------|----------| +| `.bin` | Copied directly to the ST-LINK volume | +| `.hex` | Copied directly to the ST-LINK volume | + +ELF files must be converted externally before flashing: + +```shell +arm-none-eabi-objcopy -O binary zephyr.elf zephyr.bin +``` + +## Usage ```shell j flasher flash firmware.bin # flash a raw binary @@ -48,7 +48,8 @@ j flasher flash firmware.hex # flash an Intel HEX file j flasher info # show ST-LINK volume details ``` -## API +## API Reference -- **`flash(source, target=None)`** — Flash firmware to the board. Accepts `.bin` or `.hex` files. -- **`info()`** — Read `DETAILS.TXT` from the ST-LINK volume and return board metadata. +```{eval-rst} +.. autoclass:: jumpstarter_driver_stlink_msd.driver.StlinkMsdFlasher() +``` diff --git a/python/packages/jumpstarter-driver-stlink-msd/jumpstarter_driver_stlink_msd/client.py b/python/packages/jumpstarter-driver-stlink-msd/jumpstarter_driver_stlink_msd/client.py index 02fb0961f..470f01f7a 100644 --- a/python/packages/jumpstarter-driver-stlink-msd/jumpstarter_driver_stlink_msd/client.py +++ b/python/packages/jumpstarter-driver-stlink-msd/jumpstarter_driver_stlink_msd/client.py @@ -41,6 +41,6 @@ def flash(file, compression): name = Path(file).name click.echo(f"Flashing {name}...") self.flash(file, target=name, compression=compression) - click.echo("Flash complete — ST-LINK will program the target MCU.") + click.echo("Flash complete - ST-LINK will program the target MCU.") return base diff --git a/python/packages/jumpstarter-driver-stlink-msd/jumpstarter_driver_stlink_msd/driver.py b/python/packages/jumpstarter-driver-stlink-msd/jumpstarter_driver_stlink_msd/driver.py index 26d5955ef..4a9f472e7 100644 --- a/python/packages/jumpstarter-driver-stlink-msd/jumpstarter_driver_stlink_msd/driver.py +++ b/python/packages/jumpstarter-driver-stlink-msd/jumpstarter_driver_stlink_msd/driver.py @@ -87,7 +87,7 @@ def info(self) -> dict[str, str]: async def flash(self, source, target: str | None = None): """Flash firmware to the STM32 board via ST-LINK mass storage. - Accepts .bin or .hex files only. ELF files are rejected — convert + Accepts .bin or .hex files only. ELF files are rejected - convert them externally before flashing. :param source: Firmware resource (local path or storage handle). @@ -118,7 +118,7 @@ def _copy() -> None: await to_thread.run_sync(_copy) - self.logger.info("Flash complete — ST-LINK will program the target MCU") + self.logger.info("Flash complete - ST-LINK will program the target MCU") @export async def dump(self, target, partition: str | None = None): diff --git a/python/packages/jumpstarter-driver-tasmota/README.md b/python/packages/jumpstarter-driver-tasmota/README.md index 3b4c8bdc1..9b93ccfa6 100644 --- a/python/packages/jumpstarter-driver-tasmota/README.md +++ b/python/packages/jumpstarter-driver-tasmota/README.md @@ -1,4 +1,4 @@ -# Tasmota driver +# Tasmota Driver `jumpstarter-driver-tasmota` provides functionality for interacting with tasmota compatible devices. diff --git a/python/packages/jumpstarter-driver-tftp/README.md b/python/packages/jumpstarter-driver-tftp/README.md index 4133f821e..b0d8a59ef 100644 --- a/python/packages/jumpstarter-driver-tftp/README.md +++ b/python/packages/jumpstarter-driver-tftp/README.md @@ -1,4 +1,4 @@ -# TFTP driver +# TFTP Driver `jumpstarter-driver-tftp` provides functionality for a read-only TFTP server that can be used to serve files. diff --git a/python/packages/jumpstarter-driver-tmt/README.md b/python/packages/jumpstarter-driver-tmt/README.md index 15dd6e873..fb84e69ad 100644 --- a/python/packages/jumpstarter-driver-tmt/README.md +++ b/python/packages/jumpstarter-driver-tmt/README.md @@ -107,3 +107,9 @@ j tmt run --name /my/test/plan provision -h connect -g 192.168.1.100 -P 22 # Automatically transformed to use SSH connection # TMT receives: run --name /my/test/plan provision -h connect -g -P -u root -p password ``` + +## API Reference + +```{eval-rst} +.. autoclass:: jumpstarter_driver_tmt.driver.TMT() +``` diff --git a/python/packages/jumpstarter-driver-uboot/README.md b/python/packages/jumpstarter-driver-uboot/README.md index d8fa7644d..ef139faf0 100644 --- a/python/packages/jumpstarter-driver-uboot/README.md +++ b/python/packages/jumpstarter-driver-uboot/README.md @@ -1,4 +1,4 @@ -# U-Boot driver +# U-Boot Driver `jumpstarter-driver-uboot` provides functionality for interacting with the U-Boot bootloader. This driver does not interact with the DUT directly, instead diff --git a/python/packages/jumpstarter-driver-uds-can/README.md b/python/packages/jumpstarter-driver-uds-can/README.md index 90a5f92b2..ff951ff76 100644 --- a/python/packages/jumpstarter-driver-uds-can/README.md +++ b/python/packages/jumpstarter-driver-uds-can/README.md @@ -53,19 +53,8 @@ export: tx_data_length: 8 ``` -## Client API +## API Reference -The client API is shared by all UDS transport drivers via `jumpstarter-driver-uds`. -See the UDS driver documentation for the full API reference. - -| Method | Description | -|---------------------------------------|----------------------------------------------| -| `change_session(session)` | Change diagnostic session | -| `ecu_reset(reset_type)` | Reset ECU | -| `tester_present()` | Keep session alive | -| `read_data_by_identifier(did_list)` | Read DID values | -| `write_data_by_identifier(did, value)`| Write DID value | -| `request_seed(level)` | Request security access seed | -| `send_key(level, key)` | Send security access key | -| `clear_dtc(group)` | Clear diagnostic trouble codes | -| `read_dtc_by_status_mask(mask)` | Read DTCs matching status mask | +```{eval-rst} +.. autoclass:: jumpstarter_driver_uds_can.driver.UdsCan() +``` diff --git a/python/packages/jumpstarter-driver-uds-doip/README.md b/python/packages/jumpstarter-driver-uds-doip/README.md index 4a7fdffc0..cfb1446fd 100644 --- a/python/packages/jumpstarter-driver-uds-doip/README.md +++ b/python/packages/jumpstarter-driver-uds-doip/README.md @@ -37,29 +37,8 @@ export: request_timeout: 5 ``` -## Client API +## API Reference -| Method | Description | -|---------------------------------------|----------------------------------------------| -| `change_session(session)` | Change diagnostic session (default/extended/programming/safety) | -| `ecu_reset(reset_type)` | Reset ECU (hard/soft/key_off_on) | -| `tester_present()` | Keep session alive | -| `read_data_by_identifier(did_list)` | Read DID values | -| `write_data_by_identifier(did, value)`| Write DID value | -| `request_seed(level)` | Request security access seed | -| `send_key(level, key)` | Send security access key | -| `clear_dtc(group)` | Clear diagnostic trouble codes | -| `read_dtc_by_status_mask(mask)` | Read DTCs matching status mask | - -### Session Types - -- `default` -- Default diagnostic session -- `programming` -- Programming session -- `extended` -- Extended diagnostic session -- `safety` -- Safety system diagnostic session - -### Reset Types - -- `hard` -- Hard reset -- `key_off_on` -- Key off/on reset -- `soft` -- Soft reset +```{eval-rst} +.. autoclass:: jumpstarter_driver_uds_doip.driver.UdsDoip() +``` diff --git a/python/packages/jumpstarter-driver-uds/README.md b/python/packages/jumpstarter-driver-uds/README.md index 29f90d74b..8c5cb5aba 100644 --- a/python/packages/jumpstarter-driver-uds/README.md +++ b/python/packages/jumpstarter-driver-uds/README.md @@ -1,38 +1,33 @@ -# UDS Driver (Shared Interface) +# UDS Driver `jumpstarter-driver-uds` provides shared UDS (Unified Diagnostic Services, ISO-14229) models, client, and abstract interface for Jumpstarter UDS transport drivers. -This package is not used directly -- install a transport-specific driver instead: +This package is not used directly - install a transport-specific driver instead: -- `jumpstarter-driver-uds-doip` -- UDS over DoIP (automotive Ethernet) -- `jumpstarter-driver-uds-can` -- UDS over CAN/ISO-TP +- `jumpstarter-driver-uds-doip` - UDS over DoIP (automotive Ethernet) +- `jumpstarter-driver-uds-can` - UDS over CAN/ISO-TP -## Client API +## Installation -All UDS transport drivers share the same client interface: +```{code-block} console +:substitutions: +$ pip3 install --extra-index-url {{index_url}} jumpstarter-driver-uds +``` -| Method | Description | -|---------------------------------------|----------------------------------------------| -| `change_session(session)` | Change diagnostic session (default/extended/programming/safety) | -| `ecu_reset(reset_type)` | Reset ECU (hard/soft/key_off_on) | -| `tester_present()` | Keep session alive | -| `read_data_by_identifier(did_list)` | Read DID values | -| `write_data_by_identifier(did, value)`| Write DID value | -| `request_seed(level)` | Request security access seed | -| `send_key(level, key)` | Send security access key | -| `clear_dtc(group)` | Clear diagnostic trouble codes | -| `read_dtc_by_status_mask(mask)` | Read DTCs matching status mask | +## Configuration -### Session Types +`jumpstarter-driver-uds` provides the shared UDS interface and client. It does +not have its own exporter configuration because it is not used directly as a +driver. Configuration is done on the transport-specific drivers: -- `default` -- Default diagnostic session -- `programming` -- Programming session -- `extended` -- Extended diagnostic session -- `safety` -- Safety system diagnostic session +- `jumpstarter-driver-uds-can` - UDS over CAN/ISO-TP +- `jumpstarter-driver-uds-doip` - UDS over DoIP (automotive Ethernet) -### Reset Types +Refer to those driver READMEs for exporter configuration examples. -- `hard` -- Hard reset -- `key_off_on` -- Key off/on reset -- `soft` -- Soft reset +## API Reference + +```{eval-rst} +.. autoclass:: jumpstarter_driver_uds.driver.UdsInterface() +``` diff --git a/python/packages/jumpstarter-driver-uds/jumpstarter_driver_uds/driver.py b/python/packages/jumpstarter-driver-uds/jumpstarter_driver_uds/driver.py index e22beeabb..52f740912 100644 --- a/python/packages/jumpstarter-driver-uds/jumpstarter_driver_uds/driver.py +++ b/python/packages/jumpstarter-driver-uds/jumpstarter_driver_uds/driver.py @@ -172,7 +172,7 @@ def read_dtc_by_status_mask(self, mask: int = 0xFF) -> list[DtcInfo]: logger.warning("ReadDTCByStatusMask NRC 0x%02X (%s)", e.response.code, e.response.code_name) return [] - # -- RoutineControl (0x31) ------------------------------------------------ + # - RoutineControl (0x31) ------------------------------------------------ @export @validate_call(validate_return=True) @@ -234,7 +234,7 @@ def get_routine_result(self, routine_id: int, data_hex: str = "") -> RoutineCont nrc=e.response.code, nrc_name=e.response.code_name, ) - # -- Authentication (0x29) ------------------------------------------------ + # - Authentication (0x29) ------------------------------------------------ @export @validate_call(validate_return=True) @@ -282,7 +282,7 @@ def authentication( nrc=e.response.code, nrc_name=e.response.code_name, ) - # -- RequestFileTransfer (0x38) ------------------------------------------- + # - RequestFileTransfer (0x38) ------------------------------------------- @export @validate_call(validate_return=True) diff --git a/python/packages/jumpstarter-driver-ustreamer/README.md b/python/packages/jumpstarter-driver-ustreamer/README.md index 48be28e0a..22b0f3737 100644 --- a/python/packages/jumpstarter-driver-ustreamer/README.md +++ b/python/packages/jumpstarter-driver-ustreamer/README.md @@ -1,4 +1,4 @@ -# Ustreamer driver +# uStreamer Driver `jumpstarter-driver-ustreamer` provides functionality for using the ustreamer video streaming server driven by the jumpstarter exporter. This driver takes a diff --git a/python/packages/jumpstarter-driver-vnc/README.md b/python/packages/jumpstarter-driver-vnc/README.md index 58f8baebc..0ddcf6e50 100644 --- a/python/packages/jumpstarter-driver-vnc/README.md +++ b/python/packages/jumpstarter-driver-vnc/README.md @@ -1,4 +1,4 @@ -# Vnc Driver +# VNC Driver `jumpstarter-driver-vnc` provides functionality for interacting with VNC servers. It allows you to create a secure, tunneled VNC session in your browser. @@ -31,38 +31,6 @@ export: ## API Reference -The client class for this driver is `jumpstarter_driver_vnc.client.VNClient`. - -### `vnc.session()` - -This asynchronous context manager establishes a connection to the remote VNC server and provides a local web server to view the session. - -**Usage:** - -```python -async with vnc.session() as novnc_adapter: - print(f"VNC session available at: {novnc_adapter.url}") - # The session remains open until the context block is exited. - await novnc_adapter.wait() +```{eval-rst} +.. autoclass:: jumpstarter_driver_vnc.driver.Vnc() ``` - -### CLI: `j vnc session` - -This driver provides a convenient CLI command within the `jmp shell`. By default, it will open the session URL in your default web browser. - -**Usage:** - -```shell -# This will start the local server and open a browser. -j vnc session - -# To prevent it from opening a browser automatically: -j vnc session --no-browser - -# To force an encrypted (wss://) or unencrypted (ws://) connection, overriding -# the default set in the exporter configuration: -j vnc session --encrypt -j vnc session --no-encrypt -``` - -> **Note:** Using an encrypted connection is intended for advanced scenarios where the local proxy can be configured with a TLS certificate that your browser trusts. For standard local development, modern browsers will likely reject the self-signed certificate and the connection will fail. diff --git a/python/packages/jumpstarter-driver-xcp/README.md b/python/packages/jumpstarter-driver-xcp/README.md index 0fc7fd1bd..b4c007c15 100644 --- a/python/packages/jumpstarter-driver-xcp/README.md +++ b/python/packages/jumpstarter-driver-xcp/README.md @@ -70,7 +70,7 @@ export: config_file: /path/to/xcp_config.py ``` -## Configuration Parameters +### Configuration Parameters | Parameter | Type | Default | Description | |---|---|---|---| @@ -85,47 +85,7 @@ export: | `can_id_slave` | `int` | `None` | CAN ID for slave -> master (CAN only) | | `config_file` | `str` | `None` | Path to a pyXCP config file (overrides individual params) | -## API Reference - -### Session Management - -- `connect(mode=0)` - Connect to the XCP slave, returns negotiated properties -- `disconnect()` - Disconnect from the XCP slave -- `get_id(id_type=1)` - Get the slave identifier -- `get_status()` - Get session status and resource protection - -### Security - -- `unlock(resources=None)` - Perform seed & key unlock for protected resources - -### Memory Access (Measurement / Calibration) - -- `upload(length, address, ext=0)` - Read memory from the slave -- `download(address, data, ext=0)` - Write data to the slave memory -- `set_mta(address, ext=0)` - Set the Memory Transfer Address -- `build_checksum(block_size)` - Compute checksum over a memory block - -### DAQ (Data Acquisition) - -- `get_daq_info()` - Get DAQ processor, resolution, and event channel info -- `free_daq()` - Free all DAQ lists -- `alloc_daq(daq_count)` - Allocate DAQ lists -- `alloc_odt(daq_list_number, odt_count)` - Allocate ODTs -- `alloc_odt_entry(daq_list_number, odt_number, odt_entries_count)` - Allocate ODT entries -- `set_daq_ptr(daq_list, odt, entry)` - Set DAQ list pointer -- `write_daq(bit_offset, size, ext, address)` - Configure what to measure -- `set_daq_list_mode(mode, daq_list, event, prescaler, priority)` - Set DAQ list mode -- `start_stop_daq_list(mode, daq_list)` - Start/stop a single DAQ list -- `start_stop_synch(mode)` - Start/stop all DAQ lists synchronously - -### Programming (Flashing) - -- `program_start()` - Begin programming sequence -- `program_clear(clear_range, mode=0)` - Erase memory range -- `program(data, block_length=0)` - Download program data -- `program_reset()` - Reset slave after programming - -## Example Usage +## Usage ```python from jumpstarter.common.utils import env @@ -145,3 +105,9 @@ with env() as client: xcp.disconnect() ``` + +## API Reference + +```{eval-rst} +.. autoclass:: jumpstarter_driver_xcp.driver.Xcp() +``` diff --git a/python/packages/jumpstarter-driver-xcp/jumpstarter_driver_xcp/conftest.py b/python/packages/jumpstarter-driver-xcp/jumpstarter_driver_xcp/conftest.py index 184a8f3a9..8a2d8a20f 100644 --- a/python/packages/jumpstarter-driver-xcp/jumpstarter_driver_xcp/conftest.py +++ b/python/packages/jumpstarter-driver-xcp/jumpstarter_driver_xcp/conftest.py @@ -351,9 +351,9 @@ def __init__(self) -> None: def _require_connected(self): if not self._connected: - raise XcpNotConnected("Not connected – call connect() first") + raise XcpNotConnected("Not connected - call connect() first") - # -- session -------------------------------------------------------- + # - session -------------------------------------------------------- def connect(self, mode: int = 0): if self._connected: @@ -377,7 +377,7 @@ def getStatus(self): def getCurrentProtectionStatus(self) -> dict[str, bool]: return dict(self._protection) - # -- security ------------------------------------------------------- + # - security ------------------------------------------------------- def cond_unlock(self, resources=None): self._require_connected() @@ -385,7 +385,7 @@ def cond_unlock(self, resources=None): for key in self._protection: self._protection[key] = False - # -- memory access -------------------------------------------------- + # - memory access -------------------------------------------------- def setMta(self, address: int, ext: int = 0): self._require_connected() @@ -403,7 +403,7 @@ def download(self, data: bytes): self._require_connected() self._memory[self._mta_address] = data - # -- checksum ------------------------------------------------------- + # - checksum ------------------------------------------------------- def buildChecksum(self, block_size: int): self._require_connected() @@ -412,7 +412,7 @@ def buildChecksum(self, block_size: int): csum = sum(raw) & 0xFFFFFFFF return _SlaveProperties(checksumType=0x01, checksum=csum) - # -- DAQ ------------------------------------------------------------ + # - DAQ ------------------------------------------------------------ def getDaqInfo(self): self._require_connected() @@ -458,7 +458,7 @@ def startStopDaqList(self, mode: int, daq_list: int): def startStopSynch(self, mode: int): self._require_connected() - # -- programming ---------------------------------------------------- + # - programming ---------------------------------------------------- def programStart(self): self._require_connected() diff --git a/python/packages/jumpstarter-driver-xcp/jumpstarter_driver_xcp/driver_test.py b/python/packages/jumpstarter-driver-xcp/jumpstarter_driver_xcp/driver_test.py index 055746ff9..f5bb93d88 100644 --- a/python/packages/jumpstarter-driver-xcp/jumpstarter_driver_xcp/driver_test.py +++ b/python/packages/jumpstarter-driver-xcp/jumpstarter_driver_xcp/driver_test.py @@ -442,7 +442,7 @@ def stateful_client(stateful_master): yield from _stateful_client_ctx(stateful_master) -# -- session & identification -------------------------------------------------- +# - session & identification -------------------------------------------------- def test_stateful_connect_disconnect(stateful_client, stateful_master): @@ -470,7 +470,7 @@ def test_stateful_get_status_shows_protection(stateful_client): assert status.resource_protection["daq"] is False -# -- unlock flow --------------------------------------------------------------- +# - unlock flow --------------------------------------------------------------- def test_stateful_unlock_clears_protection(stateful_client): @@ -481,11 +481,11 @@ def test_stateful_unlock_clears_protection(stateful_client): assert result["dbg"] is False -# -- memory read / write round-trip ------------------------------------------- +# - memory read / write round-trip ------------------------------------------- def test_stateful_download_then_upload(stateful_client): - """Write data to an address and read it back — verifies memory state.""" + """Write data to an address and read it back - verifies memory state.""" stateful_client.connect() stateful_client.download(0x1000, b"\x0C\x0A", 0) @@ -502,7 +502,7 @@ def test_stateful_upload_unwritten_address_returns_zeros(stateful_client): def test_stateful_overwrite_memory(stateful_client): - """Download twice to the same address — second write wins.""" + """Download twice to the same address - second write wins.""" stateful_client.connect() stateful_client.download(0x2000, b"\x11\x22", 0) stateful_client.download(0x2000, b"\x33\x44", 0) @@ -525,7 +525,7 @@ def test_stateful_multiple_addresses(stateful_client): assert raw == expected, f"Mismatch at 0x{addr:X}" -# -- checksum ------------------------------------------------------------------ +# - checksum ------------------------------------------------------------------ def test_stateful_checksum_over_written_data(stateful_client): @@ -538,7 +538,7 @@ def test_stateful_checksum_over_written_data(stateful_client): assert result.checksum_value == 0x01 + 0x02 + 0x03 + 0x04 -# -- DAQ allocation workflow --------------------------------------------------- +# - DAQ allocation workflow --------------------------------------------------- def test_stateful_daq_alloc_flow(stateful_client, stateful_master): @@ -569,7 +569,7 @@ def test_stateful_daq_alloc_flow(stateful_client, stateful_master): assert stateful_master._daq_lists == 0 -# -- programming sequence ----------------------------------------------------- +# - programming sequence ----------------------------------------------------- def test_stateful_full_programming_flow(stateful_client, stateful_master): @@ -633,7 +633,7 @@ def test_stateful_program_before_clear_raises(stateful_master): c.program(b"\x00" * 8) -# -- end-to-end calibration workflow ------------------------------------------ +# - end-to-end calibration workflow ------------------------------------------ def test_stateful_calibration_workflow(stateful_client): @@ -663,7 +663,7 @@ def test_stateful_calibration_workflow(stateful_client): stateful_client.disconnect() -# -- connect-required enforcement --------------------------------------------- +# - connect-required enforcement --------------------------------------------- def test_stateful_operations_before_connect_raise(stateful_master): diff --git a/python/packages/jumpstarter-driver-yepkit/README.md b/python/packages/jumpstarter-driver-yepkit/README.md index 5b083d14c..309308546 100644 --- a/python/packages/jumpstarter-driver-yepkit/README.md +++ b/python/packages/jumpstarter-driver-yepkit/README.md @@ -1,4 +1,4 @@ -# Yepkit driver +# Yepkit Driver `jumpstarter-driver-yepkit` provides functionality for interacting with Yepkit products. diff --git a/python/packages/jumpstarter-mcp/README.md b/python/packages/jumpstarter-mcp/README.md index a9616b2d1..c331febe0 100644 --- a/python/packages/jumpstarter-mcp/README.md +++ b/python/packages/jumpstarter-mcp/README.md @@ -1,171 +1,49 @@ -# jumpstarter-mcp +# MCP -MCP (Model Context Protocol) server for AI agent interaction with Jumpstarter -hardware devices. +{term}`MCP` server for AI agent interaction with Jumpstarter hardware devices. -## Overview - -This package provides an MCP server that exposes Jumpstarter's lease management, -device connections, and command execution as structured tools accessible by AI -agents (e.g., via Cursor, Claude Code, or any MCP-compatible host). - -## IDE Integration - -### Cursor - -Add to `~/.cursor/mcp.json`: - -```json -{ - "mcpServers": { - "jumpstarter": { - "command": "jmp", - "args": ["mcp", "serve"] - } - } -} -``` - -### Claude Code - -Claude Code discovers MCP servers from its configuration. Add Jumpstarter -with: - -```bash -claude mcp add jumpstarter -- jmp mcp serve -``` - -Or manually add to `~/.claude/claude_desktop_config.json`: - -```json -{ - "mcpServers": { - "jumpstarter": { - "command": "jmp", - "args": ["mcp", "serve"] - } - } -} -``` - -### Claude Desktop - -Add to your Claude Desktop configuration file -(`~/Library/Application Support/Claude/claude_desktop_config.json` on macOS): - -```json -{ - "mcpServers": { - "jumpstarter": { - "command": "jmp", - "args": ["mcp", "serve"] - } - } -} -``` +For setup instructions and usage examples, see the +[Agentic Integration](../../getting-started/guides/integration-patterns/agentic.md) +guide. ## Available Tools -### Lease & Exporter Management +### Lease and Exporter Management | Tool | Description | |---|---| -| `jmp_list_exporters` | List exporters with online status and lease info | -| `jmp_list_leases` | List active leases | -| `jmp_create_lease` | Create a new lease by selector or exporter name | -| `jmp_delete_lease` | Release a lease | +| `jmp_list_exporters` | List {term}`exporter`s with online status and {term}`lease` info | +| `jmp_list_leases` | List active {term}`lease`s | +| `jmp_create_lease` | Create a new {term}`lease` by selector or {term}`exporter` name | +| `jmp_delete_lease` | Release a {term}`lease` | ### Connection Management | Tool | Description | |---|---| -| `jmp_connect` | Connect to a device (by lease, selector, or exporter) | -| `jmp_disconnect` | Disconnect from a device | +| `jmp_connect` | Connect to a {term}`device` (by {term}`lease`, selector, or {term}`exporter`) | +| `jmp_disconnect` | Disconnect from a {term}`device` | | `jmp_list_connections` | List active connections | ### Device Interaction | Tool | Description | |---|---| -| `jmp_run` | Execute CLI commands on a connected device | +| `jmp_run` | Execute CLI commands on a connected {term}`device` | | `jmp_get_env` | Get environment and code examples for direct access | -### Discovery & Introspection +### Discovery and Introspection | Tool | Description | |---|---| -| `jmp_explore` | Discover available CLI commands on a device | +| `jmp_explore` | Discover available CLI commands on a {term}`device` | | `jmp_drivers` | List driver objects and their methods | | `jmp_driver_methods` | Inspect driver method signatures and docstrings | -## Typical Workflow - -A typical interaction with an AI agent looks like this: - -1. **List exporters** to see what hardware is available: - > "What devices are available on the cluster?" - -2. **Create a lease** for a target device: - > "Get me a QEMU target" or "Lease a board with label board-type=qc8650" - -3. **Connect** to establish a persistent connection: - > "Connect to that lease" - -4. **Interact** with the device: - > "Power on the target and check what OS it's running via SSH" - -5. **Disconnect and release** when done: - > "Disconnect and delete the lease" - -## Writing Python with AI Assistance - -The MCP server is especially useful when writing Python code that interacts with -hardware. While connected to a device, the agent can introspect the live -connection to discover available drivers, methods, and their signatures -- then -use that knowledge to help you write correct code. - -**Ask the agent to explore what's available on your target:** - -> "I'm connected to an ARM board. What drivers and methods are available?" -> -> *The agent calls `jmp_drivers` and `jmp_driver_methods` to inspect the live -> connection and gives you a summary of power, ssh, serial, storage, etc.* - -**Ask for help writing automation scripts:** - -> "Write me a Python script that power-cycles the board, waits for it to boot, -> and grabs the kernel version over SSH." -> -> *The agent inspects the driver methods to discover exact signatures and -> generates a working script using the `env()` helper.* - -**Debug a failing interaction:** - -> "My serial expect is timing out. Can you read the serial output and tell me -> what the board is printing?" -> -> *The agent calls `jmp_run` with `["serial", "pipe"]` and a short timeout -> to capture what the console is outputting right now.* - -**Discover capabilities you didn't know about:** - -> "What can I do with the storage driver on this device?" -> -> *The agent calls `jmp_driver_methods` for the storage driver and shows you -> methods like `flash`, `write_local_file`, `read_to_local_file`, etc. with -> their full signatures and docstrings.* - -**Iterate on code with live hardware feedback:** - -> "Run my test script and tell me if the board boots successfully." -> -> *The agent uses `jmp_get_env` to get the shell environment, executes your -> script, and reports back with the actual device output.* - -## Logging - -The MCP server logs to `~/.jumpstarter/logs/mcp-server.log`. To monitor: +## API Reference -```bash -tail -f ~/.jumpstarter/logs/mcp-server.log +```{eval-rst} +.. automodule:: jumpstarter_mcp.server + :members: + :undoc-members: ``` diff --git a/python/packages/jumpstarter-mcp/jumpstarter_mcp/server.py b/python/packages/jumpstarter-mcp/jumpstarter_mcp/server.py index 5fefb2aa7..d9ccb1f23 100644 --- a/python/packages/jumpstarter-mcp/jumpstarter_mcp/server.py +++ b/python/packages/jumpstarter-mcp/jumpstarter_mcp/server.py @@ -41,13 +41,13 @@ - Power: ["power", "on"], ["power", "off"], ["power", "cycle"] - SSH: ["ssh", "--", "your", "command", "here"] - Storage: ["storage", "flash", "/path/to/image"] -- Serial: ["serial", "pipe"] (streaming -- use a short timeout_seconds, e.g. 10-15) +- Serial: ["serial", "pipe"] (streaming - use a short timeout_seconds, e.g. 10-15) jmp_run has a timeout_seconds parameter (default 120). For streaming/blocking commands like "serial pipe", set a short timeout_seconds so the command is killed after capturing available output rather than hanging. -Connections are persistent -- create once, run many commands against it. +Connections are persistent - create once, run many commands against it. For deeper inspection: - jmp_drivers shows the Python driver object tree (class names, descriptions, methods) @@ -62,12 +62,12 @@ installed, so `import jumpstarter` just works - Use jmp_driver_methods to get exact method signatures for Python code -IMPORTANT -- Python code examples: +IMPORTANT - Python code examples: When generating Python examples for the user, ALWAYS use the env() helper from jumpstarter.utils.env. This assumes the script runs under a jumpstarter shell where JUMPSTARTER_HOST is already set (via jmp_get_env or `j shell`). -NEVER use ClientConfigV1Alpha1, lease(), or connect() in examples -- those +NEVER use ClientConfigV1Alpha1, lease(), or connect() in examples - those are for standalone automation, not interactive use. Canonical pattern: @@ -134,7 +134,7 @@ async def _ensure_fresh_token(config: ClientConfigV1Alpha1) -> ClientConfigV1Alp refresh_token = config.refresh_token if not refresh_token: - logger.warning("Token is expired but no refresh_token stored — run 'jmp login --offline-access'") + logger.warning("Token is expired but no refresh_token stored - run 'jmp login --offline-access'") return config try: @@ -158,7 +158,7 @@ async def _ensure_fresh_token(config: ClientConfigV1Alpha1) -> ClientConfigV1Alp ClientConfigV1Alpha1.save(config) logger.info("Access token refreshed successfully") except Exception: - logger.warning("Token refresh failed — downstream call will likely fail", exc_info=True) + logger.warning("Token refresh failed - downstream call will likely fail", exc_info=True) return config diff --git a/python/packages/jumpstarter-mcp/jumpstarter_mcp/server_test.py b/python/packages/jumpstarter-mcp/jumpstarter_mcp/server_test.py index 29266a841..afb646232 100644 --- a/python/packages/jumpstarter-mcp/jumpstarter_mcp/server_test.py +++ b/python/packages/jumpstarter-mcp/jumpstarter_mcp/server_test.py @@ -659,11 +659,11 @@ def test_stray_writes_do_not_reach_saved_stdout(self): os.dup2(sys.stderr.fileno(), sys.stdout.fileno()) # fd 1 -> stderr sys.stdout = sys.stderr - # Stray write via sys.stdout -- should land in the stderr pipe + # Stray write via sys.stdout - should land in the stderr pipe sys.stdout.write("stray\n") sys.stdout.flush() - # MCP-only write via the saved fd -- should land in the stdout pipe + # MCP-only write via the saved fd - should land in the stdout pipe mcp_file = os.fdopen(mcp_fd, "w", closefd=True) mcp_file.write("mcp-json\n") mcp_file.flush() diff --git a/python/packages/jumpstarter/jumpstarter/client/core.py b/python/packages/jumpstarter/jumpstarter/client/core.py index 0612792be..065a56cb3 100644 --- a/python/packages/jumpstarter/jumpstarter/client/core.py +++ b/python/packages/jumpstarter/jumpstarter/client/core.py @@ -521,7 +521,7 @@ async def log_stream(): # noqa: C901 self.logger.debug("Log stream cancelled") break elif e.code() == StatusCode.UNIMPLEMENTED: - # Old exporters don't support LogStream — stop retrying permanently + # Old exporters don't support LogStream - stop retrying permanently self.logger.debug("Log stream not implemented (old exporter), skipping") break else: diff --git a/python/packages/jumpstarter/jumpstarter/client/lease.py b/python/packages/jumpstarter/jumpstarter/client/lease.py index 489e5a9f3..f0678556b 100644 --- a/python/packages/jumpstarter/jumpstarter/client/lease.py +++ b/python/packages/jumpstarter/jumpstarter/client/lease.py @@ -245,7 +245,7 @@ async def _acquire(self): message = condition_message(result.conditions, "Unsatisfiable") # Old controllers (pre-918d6341) mark offline-but-matching # exporters as Unsatisfiable with reason "NoExporter". - # This is transient — retry with a new lease. + # This is transient - retry with a new lease. if condition_present_and_equal(result.conditions, "Unsatisfiable", "True", "NoExporter"): await self._handle_no_exporter_retry(spinner, message) continue diff --git a/python/packages/jumpstarter/jumpstarter/common/oci.py b/python/packages/jumpstarter/jumpstarter/common/oci.py index 80630d898..361e81348 100644 --- a/python/packages/jumpstarter/jumpstarter/common/oci.py +++ b/python/packages/jumpstarter/jumpstarter/common/oci.py @@ -47,7 +47,7 @@ def parse_oci_registry(oci_url: str) -> str: # Remove tag/digest if someone passed just "registry:tag" with no path if "/" not in url and ":" in registry: - # Could be registry:port or image:tag — if the part after : is numeric + # Could be registry:port or image:tag - if the part after : is numeric # it's a port, otherwise it's a tag on a Docker Hub image host_port = registry.split(":", 1) if host_port[1].isdigit(): @@ -133,7 +133,7 @@ def _lookup_credentials_in_auth_data(auth_data: dict, registry: str) -> tuple[st if not auths: return None, None - # Try to find a matching entry — normalize all keys for comparison + # Try to find a matching entry - normalize all keys for comparison for key, value in auths.items(): if _normalize_registry(key) == registry: # The "auth" field is base64(username:password) diff --git a/python/packages/jumpstarter/jumpstarter/common/oci_test.py b/python/packages/jumpstarter/jumpstarter/common/oci_test.py index ece249194..39332e8a2 100644 --- a/python/packages/jumpstarter/jumpstarter/common/oci_test.py +++ b/python/packages/jumpstarter/jumpstarter/common/oci_test.py @@ -32,7 +32,7 @@ def test_standard_urls(self, oci_url, expected): assert parse_oci_registry(oci_url) == expected def test_bare_image_name_defaults_to_docker_hub(self): - # "ubuntu:latest" has no slash — it's a Docker Hub shorthand + # "ubuntu:latest" has no slash - it's a Docker Hub shorthand assert parse_oci_registry("oci://ubuntu:latest") == "docker.io" diff --git a/python/packages/jumpstarter/jumpstarter/config/exporter.py b/python/packages/jumpstarter/jumpstarter/config/exporter.py index 18ae6d400..efe4eb885 100644 --- a/python/packages/jumpstarter/jumpstarter/config/exporter.py +++ b/python/packages/jumpstarter/jumpstarter/config/exporter.py @@ -79,7 +79,7 @@ class FailureDetectionConfigV1Alpha1(BaseModel): rapid_failure_window: int = Field( default=60, alias="rapidFailureWindow", - description="Seconds – a child that exits faster than this counts as a rapid failure.", + description="Seconds - a child that exits faster than this counts as a rapid failure.", ) diff --git a/python/packages/jumpstarter/jumpstarter/exporter/exporter.py b/python/packages/jumpstarter/jumpstarter/exporter/exporter.py index 57191d740..f9b032a80 100644 --- a/python/packages/jumpstarter/jumpstarter/exporter/exporter.py +++ b/python/packages/jumpstarter/jumpstarter/exporter/exporter.py @@ -831,7 +831,7 @@ async def serve(self): # noqa: C901 self.stop, # Pass shutdown callback self._request_lease_release, # Pass lease release callback ) - # else: No hook configured — LEASE_READY is set inside handle_lease() + # else: No hook configured - LEASE_READY is set inside handle_lease() # after session and Listen stream are established else: logger.info("Currently not leased") diff --git a/python/packages/jumpstarter/jumpstarter/exporter/hooks.py b/python/packages/jumpstarter/jumpstarter/exporter/hooks.py index 7195f5a07..60fbaa74a 100644 --- a/python/packages/jumpstarter/jumpstarter/exporter/hooks.py +++ b/python/packages/jumpstarter/jumpstarter/exporter/hooks.py @@ -493,7 +493,7 @@ async def wait_for_process() -> int: cause = e logger.error(error_msg, exc_info=True) finally: - # Clean up file descriptors — only close those still open to avoid + # Clean up file descriptors - only close those still open to avoid # closing an unrelated fd that reused the same number. if pty_state.parent_fd_open: try: @@ -750,7 +750,7 @@ async def run_after_lease_hook( except Exception as e: # Unexpected errors: report failure but do not shut down. - # Same transient status — the lease is released and the exporter + # Same transient status - the lease is released and the exporter # accepts new leases after the finally block completes. logger.error("afterLease hook failed with unexpected error: %s", e, exc_info=True) await report_status( @@ -762,7 +762,7 @@ async def run_after_lease_hook( # Always delay to give client time to poll the final status await anyio.sleep(1.0) - # Don't release lease when exporter is shutting down — unregistration handles cleanup. + # Don't release lease when exporter is shutting down - unregistration handles cleanup. # Releasing here would report AVAILABLE to the controller right before shutdown. if request_lease_release and not shutdown_called: try: diff --git a/python/packages/jumpstarter/jumpstarter/exporter/hooks_test.py b/python/packages/jumpstarter/jumpstarter/exporter/hooks_test.py index a45bcf1d7..1b1386297 100644 --- a/python/packages/jumpstarter/jumpstarter/exporter/hooks_test.py +++ b/python/packages/jumpstarter/jumpstarter/exporter/hooks_test.py @@ -634,7 +634,7 @@ async def test_drain_reads_data_remaining_in_pty_buffer(self, lease_scope) -> No Patches os.read so that, once the main loop has consumed the initial subprocess output via EOF from the specific PTY fd, a subsequent read - returns additional data -- simulating the macOS scenario where the + returns additional data - simulating the macOS scenario where the kernel buffers output that arrives after the reader stop flag is set. """ import pty diff --git a/python/pyproject.toml b/python/pyproject.toml index 78623a341..f97dd3ebf 100644 --- a/python/pyproject.toml +++ b/python/pyproject.toml @@ -70,6 +70,7 @@ docs = [ "sphinxcontrib-programoutput>=0.19", "sphinx-copybutton>=0.5.2", "sphinx-inline-tabs>=2023.4.21", + "pyyaml>=6.0", ] dev = [ "ruff==0.15.10",