Package Specification and Ecosystem

[claymcleod]

Module Specification and Ecosystem

Overview

This RFC proposes a module specification and ecosystem for WDL. It defines the structure of a WDL module, the manifest format (module.json) that describes each module and its dependencies, and how modules are imported in consumer workflows.

As a motivating example, here is what importing a fastp task from a hypothetical BioWDL module might look like:

import Fastp from biowdl/fastp

workflow hello {
  # ...

  call Fastp.Fastp {
    # ...
  }

  # ...
}

{
  "version": "1.0.0",
  "license": "MIT",
  "dependencies": {
    "biowdl": { "git": "https://github.com/biowdl/tasks", "branch": "develop" }
  }
}

Motivation

Importing in WDL today is limited—far too limited to support distributed development and a module ecosystem. Only local file references and concrete URLs are supported, neither of which is sufficient.

• Importing of local files requires the files to be resident, which means you have to go out and manually download the files you want locally to your system before you run your workflow—a logistical nightmare.
• Importing of URLs is fine for simple imports that rarely change. However, it's still a far cry from enabling a real ecosystem of versioned modules.
  • Versioning is not inherently baked into or required to be present in URLs, so tasks and workflows are rarely versioned at all. In practice, imports typically just point to the most up-to-date document on a main/master branch. Since no conventions or backwards compatibility guarantees are made, documents can easily change out from under you at any time.
  • In cases where the package versions are baked into the URL (e.g., pointing to WDL documents on a particular branch or tag at https://raw.githubusercontent.com), one can implement some level of versioned importing. The result is still undesirable though, as the version of dependencies is baked into the source code itself. This means that a change from v2.0.1 to v2.0.2 of a dependency means that all URLs pointing to this dependency need to be updated. This is unnecessary work and added noise for the git commit history.
• Beyond the practical shortcomings, raw URL imports are a security and reproducibility hazard. URLs are mutable—the content at a given URL can change without notice, which silently breaks reproducibility. Plain http:// URLs are vulnerable to interception and tampering. The module system proposed here addresses all three problems through Git-based sources with immutable references (tags, commits) and a required lockfile.

All of this is made a bit easier through the use of relative imports (e.g., import "../task.wdl"), which ensure that imports are resolved relative to the current document location. This means that, whether you run a workflow from a local document (sprocket run ./workflow.wdl) or a remote document (sprocket run https://example.com/workflow.wdl), imports will be sourced correctly. That being said, this does not solve the issue of importing from external modules.

A more ideal state

The following are, in our view, the minimum requirements for a functioning module ecosystem.

• Symbolic resolution of module names to their sources should be supported across execution engines. Both the dependency file format and the lookup/resolution behavior should be defined by the WDL specification.
• Imports should support a broader set of sources, including:
  • Symbolically importing local directories.
  • Symbolically importing from git (new), with support for pulling a specific tag, branch, revision, and path.
• Creating and maintaining modules should be easy enough that it becomes the norm for WDL development. Each module is versioned, and versions must conform to semantic versioning so that resolution tooling can rely on its guarantees.

Goals

Given the above ideal state, this RFC sets out the following goals.

• Straightforward implementing and importing. It should be easy to both create a WDL module and import external tasks and sub-workflows in a downstream workflow without manually downloading files or managing URLs.
• Distributed responsibility. The system should be designed around individual repositories, though monorepos containing many modules are equally welcome. Tool authors should be incentivized to ship a WDL module alongside their tool, and the tools that do this well will have a better chance of adoption.
• Flexible access patterns. Symbolic importing from both Git repositories and local filesystem paths should be supported. Any Git hosting service (GitHub, GitLab, Bitbucket, self-hosted, etc.) should work—the system depends on Git, not on any particular host. For Git sources, importing by a released/tagged version, by a branch, or by a specific commit should all be supported.
• Reproducibility and immutability. A required lockfile (module-lock.json) should pin the fully resolved dependency tree so that builds are reproducible without additional effort. Immutable Git references (tags, commits) should guarantee that dependencies do not change out from under you.
• Software development lifecycle tooling. The system should facilitate automated tools for (a) pinning dependencies to a particular version, (b) resolution, (c) upgrading, and (d) machine-readable license communication (e.g., via SPDX identifiers).
• Upstream tool discovery and metadata. Modules should track not only their own WDL files but the upstream tools they wrap—including licenses, versions, publications, and other metadata. Consumers should be able to see what software a module wraps without reading its source, and tooling should be able to audit licenses across the full dependency chain.
• Supply chain security. Cryptographic signing should be supported so that consumers can verify both the integrity of module content and the identity of the publisher before executing a workflow.
• Community-maintained discoverability. Although modules themselves are distributed across independent repositories, a community-maintained index at openwdl.github.io/registry should aggregate metadata from registered repositories to make them searchable—without becoming a single point of failure for resolution.
• Import renaming. Renaming of imported items should be supported to avoid naming conflicts.

Antigoals

• This RFC is not intended to propose a centralized repository or a curated set of best-practices workflows (a la nf-core). It leans into one of WDL's core differentiators: distributed development by disparate teams and individuals. We want tool authors to ship WDL modules alongside their tools and workflow authors to pick the best ones. A system where authors maintain their own modules is far more scalable than a centralized, curated platform.
• This RFC does not specify engine-specific CLI commands (e.g., validation commands, module scaffolding). It defines the module format and resolution behavior; individual engines build their own tooling on top.

Prior Art

#226

As far as I can tell, this is the earliest official proposal to add some level of package management/versioning to WDL. The issue is rather short and describes a syntax like the following.

import "tools/published_tool" @ "v1.0"
import "tools/new_tool_version" @ "v11.3"
import "structs/alpha_struct_defs" @ "v0.5"

As previously stated, I'm not in support of stopping at the proposed mechanism because it assumes a centralized package system (something I don't think is feasible with our relatively small contributor team nor advisable based on our emphasis on enabling distributed development and maintenance).

#493

Revived in early 2022, this proposal similarly focuses on a centralized package repository. For the pieces that overlap with this proposal (essentially just import syntax), the two overlap quite a bit—just a difference in the order of the "import" and "from" clauses.

#499

#499 outlined a more concrete package format. We drew inspiration from it—particularly the metadata fields (name, author, version, license) and the reproducibility concerns, which we folded into this proposal's goals. Much of the remaining discussion in #499 centers on centralized distribution (e.g., tar vs. zip), which is irrelevant here given our Git-based approach.

#698

#698 proposes that we relax the constraints around what document versions can be imported, essentially advocating that any WDL v1.x version must be able to load any WDL documents with versions v1.x or lower. This adheres to common expectations regarding backwards compatibility of software with the same major version and would be required for this proposal to be manageable (else, the entire ecosystem could become deadlocked waiting for root modules in the ecosystem to update their WDL version when a new minor revision of WDL is released).

---

Proposal

Definition of a WDL module

A WDL module is a directory containing a module.json manifest and one or more .wdl files. There is no other organizational concept—no "workspace" type, no special grouping files, no hierarchy requirements.

When the resolver encounters a dependency, it scans the entire source tree for module.json files. Each one it finds is a module, registered at whatever path it sits at relative to the source root. This means a Git repository can contain one module at the root, many modules in subdirectories, or both. The resolver does not care about the shape of the repository—it discovers what is there.

A single-module repository:

fastp-wdl/
  module.json
  fastp.wdl

A multi-module repository:

biowdl-tasks/
  fastp/
    module.json
    fastp.wdl
  samtools/
    sort/
      module.json
      sort.wdl
    index/
      module.json
      index.wdl
  bwa/
    module.json
    bwa.wdl

Both are valid dependencies. The resolution logic is identical for both.

Manifest file

The manifest file for a module always lives at the root of the module directory with the name module.json. It contains the following fields.

Core fields

• name (string, required). A human-readable display name for the module (e.g., "fastp", "samtools-sort"). This is used by the registry and tooling for display purposes—it is not used for dependency resolution. Consumers still name dependencies in their own module.json, so there is no global namespace to manage and no squatting problem.
• version (string, required). The module version, following the SemVer v2.0.0 specification. The versioning contract is: if the version doesn't change, the expected output doesn't change. This means the module version must reflect changes to the WDL interface (inputs, outputs, behavior) and changes to the wrapped tool that alter expected output. The tools field (below) tracks the upstream tool version separately for metadata and provenance, but the module version is ultimately what consumers rely on for compatibility.
• license (string, required). An SPDX license expression (e.g., "MIT", "Apache-2.0", "MIT OR Apache-2.0", "MIT AND (Apache-2.0 WITH LLVM-exception)").
• authors (array of strings, optional). Author descriptions. The convention for individual authors is "First Last <first.last@example.com>", but this is not enforced.
• description (string, optional). A brief description of what the module does.
• repository (string, optional). The canonical Git URL for the module's source repository (e.g., "https://github.com/biowdl/tasks"). The registry uses this to link back to source.
• homepage (string, optional). A URL for the module's documentation or landing page, if distinct from the repository.
• readme (string, optional). Path to a markdown file relative to the module root. If omitted, engines and the registry should look for README.md in the module directory. If explicitly set to false, no readme is associated with the module.

Tools

The tools field is an array of objects that tracks the upstream software wrapped by the module. Each entry records:

• name (string, required). The tool name.
• version (string, required). The version of the tool.
• license (string, required). The tool's SPDX license identifier.
• homepage (string, optional). URL for the tool's homepage or repository.
• doi (string, optional). DOI for the tool's publication.
• biotools (string, optional). bio.tools registry identifier.

WDL tasks sit at the boundary between the workflow language and the tools they call. The tools array tracks which version of the upstream software the module wraps, but this is metadata—it does not replace the module's own semver version. Module authors are responsible for bumping the module version whenever the wrapped tool changes in a way that alters expected output. If you update fastp from 0.23.4 to 0.24.0 and that changes the default trimming behavior, the module version must change even if no WDL inputs or outputs were added or removed. The tools field exists so that downstream consumers have machine-readable provenance (which tool, which version, which license) without overloading the module version with that information. See also Guidance on API stability and optional inputs for how to minimize interface-breaking changes when upstream tools evolve.

Dependencies

The dependencies field is a JSON object that contains one key per dependency. Each key must be a valid WDL identifier. The key is a consumer-chosen name—it does not need to match the module's name field. The importer names the dependency however they like, and two consumers can refer to the same module by different local names without ambiguity.

Each dependency must specify a source and a version selector. A dependency with a git URL but no version, tag, branch, or commit is invalid—you must be explicit about what you want.

Version requirements (default)

The recommended way to declare a dependency is with a version field containing a semver version requirement. The resolver lists Git tags from the repository, parses them as semver (stripping a leading v if present, e.g., v1.2.0 → 1.2.0), and selects the highest version that satisfies the constraint.

The version requirement syntax is as follows:

• ^1.2.0 — compatible updates: >=1.2.0, <2.0.0. This is the default behavior if no operator is specified (i.e., "1.2.0" is equivalent to "^1.2.0").
• ~1.2.0 — patch-level updates only: >=1.2.0, <1.3.0.
• =1.2.0 — exactly this version.
• >=1.0.0, <2.0.0 — explicit range using comparison operators (>=, >, <=, <), combined with commas.
• * — any version. This is allowed but discouraged.

{
  "dependencies": {
    "samtools": { "git": "https://github.com/someone/samtools-wdl", "version": "^1.2.0" },
    "biowdl": { "git": "https://github.com/biowdl/tasks", "version": ">=2.0.0, <3.0.0" }
  }
}

Alternative version selectors

For cases where semver version requirements are not sufficient (e.g., pre-release testing, pinning to a specific commit for debugging, or tracking a development branch), the following alternatives are available:

• tag — a specific Git tag name (e.g., "v1.2.0-rc1"). Does not go through semver resolution.
• branch — a Git branch name (e.g., "main", "develop"). The resolved commit will vary over time; the lockfile pins the exact commit at resolution time.
• commit — a full Git commit SHA. The most precise and immutable selector.

The four selectors—version, tag, branch, and commit—are mutually exclusive. Specifying more than one on the same dependency is invalid.

{
  "dependencies": {
    "bleeding_edge": { "git": "https://github.com/org/tool", "branch": "main" },
    "pinned": { "git": "https://github.com/org/tool", "commit": "abc123d" },
    "prerelease": { "git": "https://github.com/org/tool", "tag": "v2.0.0-rc1" }
  }
}

Local path dependencies

A dependency with a path key points to a local filesystem directory. No version selector is needed—the module is used as-is from the local path.

{
  "dependencies": {
    "local_utils": { "path": "../../shared/utils" }
  }
}

Path within a repository

For any Git dependency, an optional path key can be included to set the root directory for module scanning within the repository. This is useful when WDL modules live in a subdirectory alongside other files (e.g., Docker files, CI configs, documentation).

{
  "dependencies": {
    "mytool": { "git": "https://github.com/org/mytool", "version": "^1.0.0", "path": "wdl" }
  }
}

Full example

{
  "name": "fastp",
  "version": "1.2.0",
  "license": "MIT OR Apache-2.0",
  "authors": ["Jane Doe <jane.doe@example.com>"],
  "description": "WDL wrapper for fastp quality control",
  "repository": "https://github.com/someone/fastp-wdl",
  "homepage": "https://someone.github.io/fastp-wdl",
  "tools": [
    {
      "name": "fastp",
      "version": "0.23.4",
      "license": "MIT",
      "homepage": "https://github.com/OpenGene/fastp",
      "doi": "10.1093/bioinformatics/bty560",
      "biotools": "fastp"
    }
  ],
  "dependencies": {}
}

Notes on the manifest format

• JSON was chosen because the format is already associated with the WDL specification and is readily used to express configuration information.
• The name field is for display only (e.g., in the registry and tooling output). It plays no role in dependency resolution—consumers choose their own local names for dependencies. This means there is no global namespace to manage and no squatting problem.
• The readme field defaults to README.md if omitted. The registry renders this file as the module's landing page.
• Engines must ignore unrecognized fields anywhere in module.json—at the top level, within tools entries, within dependencies entries, and in any other nested object—rather than treating them as errors. This allows the manifest format to evolve over time; new optional fields can be added without breaking older engines that don't understand them.

Symbolic imports

Symbolic imports use unquoted identifiers to refer to dependencies declared in module.json. Quoted imports remain for relative file paths. Because imports today must be enclosed in quotes, a backwards-compatible approach is that any non-quoted import is assumed to be a symbolic import.

# Symbolic import from a single-module dependency
import Fastp from fastp

# Symbolic import from a module within a multi-module dependency
import Fastp from biowdl/fastp
import Sort from biowdl/samtools/sort

# Relative file import (unchanged, fully supported)
import "../../shared/structs.wdl"

Resolution

When the parser encounters import X from foo/bar/baz, the resolution proceeds as follows:

1. Split on the first path component: foo is the dependency name, bar/baz is the module path within the dependency.
2. Look up foo in the current module's module.json dependencies.
3. Resolve the source (clone the Git repository, or read from the local path).
4. Scan the source for all module.json files, registering each module at its relative path.
5. Look up bar/baz in the discovered modules.
6. Parse that module's WDL files and make X available.

For import X from foo with no path component, the module must exist at the source root (i.e., a module.json at the top level of the dependency).

The steps above describe the logical resolution behavior that all compliant engines must produce. Engines are free to implement the mechanics however they choose (e.g., caching strategies, scan ordering, lazy vs. eager fetching).

Version discovery

How available versions are discovered depends on the source type.

For Git-based dependencies (the common case), the resolver lists the repository's Git tags and parses each as a semver version, stripping a leading v if present (e.g., tag v1.2.0 → version 1.2.0). Tags that do not parse as valid semver are ignored. The resulting set of versions is what the resolver matches against when evaluating a version requirement. This means module authors publish a new version by tagging a commit—there is no separate publish step, no upload, no registry submission. The Git tag is the release.

For local path dependencies, the resolver reads the version field from the module's module.json at the given path. If the dependency declaration includes a version requirement, the local module's version must satisfy it—otherwise resolution fails.

Transitive dependencies

Dependencies are fully transitive. If module A depends on module B, and B depends on C, the resolver walks the full tree.

Version precedence

Version precedence follows SemVer v2.0.0, section 11. When multiple tags satisfy a version requirement, the resolver selects the highest version according to semver precedence rules. Build metadata (i.e., anything after +) is ignored for precedence purposes.

Version resolution and conflicts

When multiple modules in the dependency tree require the same dependency with compatible version constraints (e.g., ^1.2.0 and ^1.5.0), the resolver should attempt to find a single version that satisfies all constraints (e.g., 1.5.0 or higher). This avoids unnecessary duplication.

When the constraints are incompatible (e.g., ^1.0.0 and ^2.0.0), both versions are fetched and used independently. No deduplication, no warnings. WDL modules are lightweight text files, and the tasks they define execute in isolated containers with no shared runtime state. There is nothing that can conflict. This sidesteps the diamond dependency problem entirely, at the cost of minor storage duplication—a few kilobytes of WDL source per duplicate.

Lockfile

The specification requires a module-lock.json file at the module root. This file pins the fully resolved dependency tree, including any duplicates. It must be committed to version control—it is what makes builds reproducible regardless of upstream changes.

Cached module sources (i.e., the local directories where resolved modules are downloaded or cloned) should not be committed. These are ephemeral and can be reconstructed from the lockfile. Engines should store cached modules in a location outside the project directory (e.g., a user-level cache) or in a directory that is .gitignored by convention.

Lockfile format

The module-lock.json file is a JSON object with the following structure:

{
  "version": 1,
  "dependencies": {
    "biowdl": {
      "source": {
        "git": "https://github.com/biowdl/tasks",
        "commit": "a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2"
      },
      "modules": {
        "fastp": {
          "version": "1.2.0",
          "checksum": "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
          "dependencies": {
            "common": {
              "source": {
                "git": "https://github.com/biowdl/common",
                "commit": "d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5"
              },
              "modules": {
                ".": {
                  "version": "0.3.0",
                  "checksum": "sha256:4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865",
                  "dependencies": {}
                }
              }
            }
          }
        }
      }
    },
    "samtools": {
      "source": {
        "git": "https://github.com/someone/samtools-wdl",
        "commit": "b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3"
      },
      "modules": {
        ".": {
          "version": "3.0.1",
          "checksum": "sha256:d7a8fbb307d7809469ca9abcb0082e4f8d5651e46d3cdb762d02d0bf37c9e592",
          "dependencies": {}
        }
      }
    },
    "local_utils": {
      "source": {
        "path": "../../shared/utils"
      },
      "modules": {
        ".": {
          "version": "0.5.0",
          "checksum": "sha256:9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08",
          "dependencies": {}
        }
      }
    }
  }
}

The structure is recursive—each module's dependencies field has the same shape as the top-level dependencies object, mirroring the full dependency tree.

The fields are as follows:

• version (integer, required). The lockfile format version. Currently 1. Engines should reject lockfiles with an unrecognized version.
• dependencies (object, required). A map from consumer-chosen dependency name (matching the key in module.json dependencies) to its resolved state.

Each dependency entry contains:

• source (object, required). The resolved source. For Git sources, this contains git (the repository URL) and commit (the full 40-character SHA that the tag, branch, or commit reference resolved to at lock time). For local path sources, this contains only path.
• modules (object, required). A map from module path within the dependency source to that module's locked state. The key is the relative path from the source root to the directory containing module.json. For modules at the source root, the key is ".".

Each module entry contains:

• version (string, required). The version from the module's module.json at lock time.
• checksum (string, required). The module's content hash in the format sha256:<hex_digest>, computed using the content hashing algorithm specified below.
• dependencies (object, required). The module's own transitive dependencies, in the same format as the top-level dependencies object. Empty if the module has no dependencies.

When two modules in the dependency tree require different versions of the same source, both resolved versions appear in the tree at whatever point in the nesting they were required.

Content hashing

Both the lockfile checksum and module signatures depend on the same deterministic content hash. All compliant engines must produce the same digest for the same module contents.

The algorithm, following the approach used by Sprocket for call caching:

1. Enumerate all files in the module directory, recursively. Exclude module.sig and module-lock.json.
2. Compute each file's relative path from the module root using / as the path separator, regardless of the host operating system.
3. Sort the file list lexicographically by relative path, comparing UTF-8 byte values.
4. Initialize a SHA-256 hasher.
5. For each file in sorted order:
   a. Hash the relative path (UTF-8 bytes).
   b. Hash the file contents (raw bytes).
6. Hash the total file count as a little-endian 64-bit unsigned integer.
7. Finalize. The resulting hex-encoded digest is the module's content hash.

The entry count in step 6 ensures that a module with files a and bc produces a different digest than a module with files ab and c, even if the concatenation of paths and contents happens to collide.

The lockfile records this digest in the format sha256:<hex_digest>.

Integrity: lockfile checksums

The module-lock.json checksum field provides tamper detection. Once a module is resolved and its checksum recorded, any modification to the cached content (whether by a compromised cache, a man-in-the-middle, or a corrupted download) is detectable. Engines must verify checksums against the lockfile before using cached modules. If the checksum does not match, the engine must refuse to proceed.

Module signing and supply chain security

Module ecosystems are targets for supply chain attacks—compromised repositories, force-pushed tags, impersonated maintainers. The signing model here addresses content tampering and maintainer impersonation without requiring centralized infrastructure.

Module signatures

Module authors can sign their modules by producing a module.sig file at the module root. This is a JSON file containing an Ed25519 signature computed over the module's content hash (i.e., the SHA-256 digest produced by the content hashing algorithm above).

Signature file format

{
  "algorithm": "ed25519",
  "public_key": "base64-encoded-32-byte-public-key",
  "signature": "base64-encoded-64-byte-signature"
}

The fields:

• algorithm (string, required). The signing algorithm. Currently the only permitted value is "ed25519". Future specification versions may add additional algorithms, at which point engines must reject unrecognized values.
• public_key (string, required). The signer's Ed25519 public key, base64-encoded.
• signature (string, required). The Ed25519 signature over the module's content hash (the raw 32-byte SHA-256 digest, not the hex-encoded string), base64-encoded.

A signed module looks like:

fastp/
  module.json
  module.sig
  fastp.wdl

Ed25519 was chosen because it is fast, produces small signatures (64 bytes) and small keys (32 bytes), and has mature implementations in every major language—engines can verify signatures in-process without shelling out to external tools or depending on any system keychain. No SSH infrastructure, no GPG, no platform-specific credential stores.

Signing is optional but encouraged. Unsigned modules (i.e., modules without a module.sig file) are valid—they simply skip the provenance verification flow.

Why out-of-band rather than Git-native signing?

Git tag and commit signing would be simpler today—authors already sign tags, and engines could verify them directly. But it couples the security model to the transport mechanism. If modules are ever distributed as tarballs, through a package server, or through any non-Git mechanism, Git signatures don't travel with the content. A module.sig file does. The cost is that authors need a separate signing step, but the benefit is a security model that survives changes to the distribution infrastructure.

Trust on first use (TOFU)

The trust model follows "trust on first use":

1. On first resolution, if module.sig is present, the engine verifies the signature and records the signer's public key in module-lock.json.
2. On subsequent resolutions, the engine verifies the signature matches the previously recorded key.
3. If the signing key has changed, the engine refuses to proceed and surfaces a clear warning explaining what happened. The user must explicitly accept the new key through an engine-specific command (e.g., sprocket module trust biowdl/fastp). This protects against compromised repositories where an attacker replaces both the content and the signature.
4. If a module was unsigned on first resolution and later becomes signed, the engine records the key going forward without disruption.

The lockfile records the signer's public key on each signed module entry:

{
  "version": 1,
  "dependencies": {
    "biowdl": {
      "source": {
        "git": "https://github.com/biowdl/tasks",
        "commit": "a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2"
      },
      "modules": {
        "fastp": {
          "version": "1.2.0",
          "checksum": "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
          "signer": "base64-encoded-32-byte-public-key",
          "dependencies": {}
        }
      }
    }
  }
}

For unsigned modules, the signer field is absent.

Engine policy

Engines are encouraged to:

• Surface a notice when resolving unsigned modules, nudging authors toward signing.
• Provide a configuration option (e.g., require_signed = true) that rejects unsigned modules entirely. Whether this defaults to on or off is left to the engine implementor.

Deprecation of remote URL imports

Remote URL imports (http://, https://) are deprecated. Compliant engines should emit a warning when encountering them. Removal is targeted for a future WDL specification version.

Raw URL imports have several problems: the content at a URL can change without notice (breaking reproducibility), there are no versioning guarantees, and plain http:// URLs are a security concern. The module system with Git-based sources and lockfile pinning addresses all of these. Relative file imports (import "../path/to/file.wdl") are unaffected by this deprecation.

Guidance on API stability and optional inputs

A common source of unnecessary major version bumps in WDL modules is the addition of new task inputs. Under semver, adding a new required input is a breaking change—every downstream consumer that calls the task must be updated. This leads to version churn that ripples through the dependency tree and erodes the usefulness of version constraints.

The remedy is straightforward: module authors should make liberal use of optional inputs with sensible defaults. WDL already supports this well. A task that wraps fastp, for example, might initially expose only the required flags. When fastp adds a new --cut_right option, the WDL wrapper can add an optional input with a default of false rather than a required input. Existing consumers continue to work without changes, and the version bump is minor rather than major.

We believe optional inputs are underused in WDL today. In a module ecosystem with transitive dependencies and semver constraints, the difference between a required and optional input is the difference between a breaking change and a compatible one. Module authors should default to optional inputs for any parameter that has a reasonable default value, reserving required inputs for the genuinely mandatory ones (e.g., input files).

Discoverability

Modules are resolved directly from Git repositories—there is no central package server. No single organization controls module availability, institutional and private modules work identically to public ones, and resolution has no external runtime dependency beyond the Git host.

The tradeoff is that distributed systems are harder to search. To address this, the community will maintain a module index at openwdl.github.io/registry, backed by the openwdl/registry GitHub repository.

1. A module author opens a PR to openwdl/registry adding their repository URL to the index.
2. CI validates the repository: it must exist, contain at least one module with a valid module.json, and pass structural validation.
3. The PR is merged.
4. CI scrapes all registered repositories, extracts metadata from every discovered module.json (version, description, tools, license, authors), and builds a static searchable site.

The index never hosts code. It points to code. If the index goes down, every existing import still works because imports resolve from Git, not from the index. The index is a convenience for discovery, not infrastructure for resolution.

Engine tooling

The WDL specification defines module format and resolution behavior; individual engines build their own CLI tooling on top. As an example, Sprocket plans a module validate command with four levels of checks:

1. Structural: module.json exists, required fields present, version is valid semver, license is a valid SPDX expression, tools entries have their required fields.
2. Syntactic: all .wdl files in the module parse without errors.
3. Cross-validation: tasks and workflows declared in the WDL are consistent with the manifest.
4. Dependency resolution: all declared dependencies resolve successfully.

Other engines may implement equivalent or different commands. The openwdl/registry CI could use any compliant engine's validation tooling to check submissions.

Design tradeoffs

Distributed hosting vs. centralized registry. We chose Git-based resolution over a central package server. The cost is discoverability, which we address with the community index. The benefit is that no single organization can become a bottleneck or point of failure for the ecosystem. Environments that cannot depend on third-party SaaS can use this system without modification.

Duplicate dependencies over conflict resolution. Other ecosystems (npm, Go) invest significant complexity in version resolution strategies. We skip all of that. WDL tasks run in isolated containers—there is no shared memory, no symbol table, no binary linking. Two versions of the same dependency coexist without interference. The duplication cost is negligible for text files.

Separate tool versioning. The module version and the upstream tool version are tracked in different fields, but they are not independent. The module version contract is that if the version doesn't change, the expected output doesn't change—so a tool update that alters output requires a module version bump. The tools array exists for provenance and license tracking, not as a substitute for proper semver on the module itself. The separation avoids conflating "which tool am I wrapping?" with "is my WDL interface stable?"—two questions that consumers care about for different reasons (cf. the discussion in the comments on this RFC).

Display name, not resolution name. The name field exists for human consumption—the registry, tooling output, search results. It is not used for dependency resolution. The importer names each dependency locally in their own module.json, so there is no global namespace to manage, no squatting problem, and no need for a naming authority. Two teams can independently wrap the same tool and consumers pick whichever they prefer.

Lockfile as a specification requirement. Making module-lock.json optional would undermine reproducibility. If the lockfile is required, every module is reproducible by default. Authors who want to live on the edge can regenerate it; authors who want stability commit it and move on.

Out-of-band signing over Git-native signing. Git tag signing would be simpler today, but it couples security to the transport mechanism. A module.sig file travels with the module regardless of how it's distributed—Git clone, tarball, or something we haven't built yet. The cost is a separate signing step for authors, but engines can make this a single command (e.g., sprocket module sign).

Trust on first use over a certificate authority. TOFU has known downsides: the first resolution is unverified (if the repo is already compromised, you trust the attacker's key), key rotation requires manual acceptance from every consumer, and there is no revocation mechanism. A PKI would address all of these but would require infrastructure and governance that a small open-source community cannot realistically sustain. TOFU protects against the most common attack—a repository compromised after you started using it—and we accept the tradeoff that it cannot protect against pre-existing compromise. The openwdl/registry can partially mitigate this by recording maintainer keys at submission time, giving new consumers a cross-reference point.

Optional signing with encouraged adoption. Requiring signatures would be more secure but would raise the friction bar for every module author. Making it optional with engine-level policy (e.g., require_signed_packages) lets security-conscious environments enforce signing while keeping the barrier to entry low for everyone else.

Soft URL deprecation. A hard break would strand existing workflows. Warnings give the ecosystem time to migrate while making the direction clear.

Auto-discovery over workspace configuration. We considered a separate workspace manifest (i.e., a wdl-workspace.json that lists member modules) and decided against it. One concept—the module—and one file format (module.json) is easier to explain, easier to implement, and sufficient for all repository layouts we examined.

Concerns left to address

• Should exported items have an associated visibility?
• Cascaded importing of WDL documents through the dependency tree while also decoupling version requirements from the source code.
• Credential management for private Git repositories (i.e., an .npmrc-style config or reliance on the user's existing Git credential helpers).

[markjschreiber]

I like the proposal overall. I would also suggest that this new feature also deprecate the import of random web URLs which is currently allowed, mainly because it breaks provenance if the URL doesn't encode a tag or commit id. It's also a very large security concern especially for http or ftp URLs.

Although you don't advocate for an NF-Core style repository, that effort does have the advantage that it is trusted to be curated and reasonably reliable. I suspect one reason that dockstore hasn't taken off is that there is no requirement for the workflows to be complete (all artifacts available) or syntactically correct (who knows how many are even runnable). I have been doing a lot of analysis of WDL workflows in dockstore and many are not runnable by miniwdl and therefore arguably not spec compliant, several of the repos don't correctly identify the "main" entrypoint wdl for the workflow and several don't contain required dependencies. I'd love to see something in this RFC or related RFCs to standardize ways to identify and import artifacts that can be relied on.

[patmagee]

This is a fantastic write up and I really feel like the spec is in need of something like this.

Dependency resolution has been a challenge in many cases allowing one of two options

1. Everything is public
2. Everything is local

In reality, the ecosystem is much more complex then that and there is no good mechanism to ensure dependencies are properly resolved.

I have a few general questions before launching into more ideas for things that I have encountered in the coding world

• do packages require adding more of an explicit namespace to wdl files?
• when you import a workflow, does the imported workflows manifest get used?
• when you import a task from within a package, does it use the manifest of the parent still?
• does a standalone task need a manifest?
• would we also support versioning, ie tag, or sha reference? I am less supportive of referencing a branch other than the implicit main branch because it is not fixed. I think main (or whatever is the repos main) can be assumed to auto incrementing.
• Can packages themselves declare a version outside of a commit sha,tag or release? If so, how do we resolve these and where are they stored from
• in the Java world the maven tool allows you to override dependency versions of imports completely. This is largely done to avoid conflicts between different versions of the same library. Would we need something similar?
• is there an idea of a common credential file or auth mechanism? Similar to an npmrc file

[rhpvorderman]

At BioWDL we did some prior packaging work in a tool called "wdl-packager" which made a zip file that could be used by cromwell. In miniwdl this was then further refined with the miniwdl zip command.

That however packages whole workflows and their dependencies. So it is not that "elegant". But it works well for distributing workflows as zip packages. We are very interested in having a better solution for WDL. I propose we do some hacking to get some things working alongside making the spec so we can quickly iterate when some things in the spec are either unwieldly or things need to be added. Perhaps there is some prior art for packaging repositories that we can already use to quickly get something running.

[a-frantz]

I am 100% on board with this concept and would love to see it reach fruition. However there is one major hurdle I have yet to see addressed in a way that satisfies me, which is the proper versioning schema the WDL ecosystem should adopt. Clay's proposal mentions semantic versioning (SEMVER) which is a popular schema adopted in many fields of software engineering, though I don't think it's an ideal fit for WDL.

I've found in my time writing and maintaining WDL that many of the relatively routine and minor changes to WDL source happen at the API layer (inputs and outputs), which according the SEMVER count as breaking changes that necessitate a major version bump. This contrasts with many other languages, where the API layer of packages is defined via things like exported functions, data structures, etc, which are more static and robust to breaking changes than a WDL task or workflow. WDL development is most often concerned with the API layer of the called tooling in a way that can break backward compatibility.

This is something me and @adthrasher have discussed at length while working on https://github.com/stjudecloud/workflows , which has been in development and in production for years, but has still not yet adopted an official versioning scheme. We typically run historic cemented git tags/commits that rarely (if ever) change, or just run whatever is on HEAD of main. I think we can all agree that neither of those modes are ideal.

I'd love to hear what other groups and WDL authors think of this. I am short on proposed solutions for WDL versioning, and would appreciate other ideas from the community! Maybe SEMVER is the best we can do and the WDL ecosystem will simply have more major version churn than other language's ecosystems, or maybe there's a solution I haven't thought of.

[rhpvorderman]

Maybe SEMVER is the best we can do and the WDL ecosystem will simply have more major version churn than other language's ecosystems, or maybe there's a solution I haven't thought of.

I think this is acceptable. Another way would be to use calender versioning. However calendar versioning is not that informative.

Given WDL's modularity, I think that for instance wrapping tools, would not see so much major version churn really. That would happen mostly at the workflow level. And even then, when mostly adding inputs the major version is not increased.

[emjbishop]

I love this idea, especially since we're working from a similar mindset with our WILDS WDL Library. Some thoughts:

• You say a module is a "directory containing a module.json manifest and one or more .wdl files". I think defining one task per .wdl and having multiple wdls is reasonable, all under one subfolder with a module.json and related files. I would not want to make a subfolder for each task, as appears to be the case in the "Definition of a WDL Module" diagram.
• Some tooling to automatically generate the lockfile will be helpful for adoption
• Encouraging the use of meta fields describing input/output file formats and what the task does (e.g., using the EDAM ontology which bio.tools and therefore nf-core modules do) could help users string modules together

[peterhuene]

Side note: I find GitHub discussions to be a really difficult format for design collaboration. Perhaps in the future we should solicit feedback on designs like this one via a pull request?

I'll try to give some high-level feedback here.

Reliance on git

This proposal will mandate that future WDL engines integrate a git library to support symbolic imports; I have reservations about this as it might become a barrier for implementation in some engines. On the surface, it also feels overly prescriptive of a language specification.

That said, I also don't see value in letting the resolution of symbolic imports be implementation-defined either as then there would be no consistency for building out an ecosystem of reusable WDL code.

So I guess you could summarize this specific feedback as a feeling of slight unease (i.e. nothing actionable).

How are multi-document modules "parsed"?

WDL currently imports by individual document. To resolve the import statement, the engine parses the imported document and its transitive imports. Top-level names in an individual document, along with custom type names introduced by the document's imports, are required to be unique by the specification (i.e. they are document namespaced).

The resolver proposed by this design includes this step:

Parse that module's WDL files and make X available

Here X is a name "exported" by the module. For multi-document modules, this name would then need to be unique across all documents in the module. This is effectively introducing a new namespace to the WDL specification: the "module" namespace.

The introduction of a new namespace means that engines must support modules regardless of their support for semantic imports as the engine must now check if the document is being parsed within the context of a module and, if so, that it only defines names that are unique across all documents in the module.

Given that WDL has no visibility specifiers for names, users will have no choice but to rename conflicting names. I also believe it may be confusing that names in a WDL document are namespaced differently depending on whether or not the document is contained in a module (i.e. the presence of a module.json file).

How do dependency versions work?

The proposal contains this paragraph regarding dependency version resolution:

For Git-based dependencies (the common case), the resolver lists the repository's Git tags and parses each as a semver version, stripping a leading v if present (e.g., tag v1.2.0 → version 1.2.0). Tags that do not parse as valid semver are ignored. The resulting set of versions is what the resolver matches against when evaluating a version requirement. This means module authors publish a new version by tagging a commit—there is no separate publish step, no upload, no registry submission. The Git tag is the release.

I get the reasoning for supporting this: some semblance of semver-compatible dependency version resolution without the need for an actual module registry.

However, where this starts to break down for me is that a tag may have no relevance to a version specified in a module.json file. If there is a mismatch between the two, how is it enforced?

Additionally, and more crucially, how does a repository-wide version tag work with a repository containing multiple independently-versioned modules?

Lastly, tags are mutable, unlike commit SHAs; intrinsically they aren't something tooling can rely on for version attestation, but definitely something we can detect has changed given that we digest the module's content as part of locking.

Using semver for modules

I'm onboard with this and I think it wouldn't be terribly difficult to create tooling to validate that a new version of a module conforms to semver; the tool would just need to know what the previous version was to compare against (i.e. parse the old version, parse the new version, compare "exported" names - tasks, workflows, custom types - for compatibility).

Module signing

I like that we're including signing as part of this proposal as provenance is an important part of any module/registry system.

Open questions

From the open questions list:

Credential management for private Git repositories (i.e., an .npmrc-style config or reliance on the user's existing Git credential helpers).

If we're taking a hard dependency on git, then I would greatly prefer that we rely on configured git credential helpers for managing access to private git repositories; effectively, if CLI git works, this should too and without the need for additional configuration.