Possible to build platform scoped features from the CLI API with rich platforms?

[matthewfeickert]

In Pixi v0.71.0 "rich platform" support was added:

We deprecated [system-requirements] in favor of the much more expressive rich platform support. On top of operating systems and processor architecture, platforms can now also include system requirements like CUDA or glibc version:

[workspace]
platforms = [
  "osx-arm64",
  { platform = "linux-64", cuda = "12.0", glibc = "2.28" },
  { name = "jetson-nano", platform = "linux-aarch64", cuda = "12.8" },
]

Speaking of CUDA, together with rich platforms we added support for __cuda_arch. We already supported __cuda which allows specifying the driver version you require. With __cuda_arch, you can specify the necessary hardware architecture.

along with a "Migrating from [system-requirements]" page.

I like to be able to use a lot of command line semantics as this helps with teaching and debugging with others. I want to be able to give Pixi CLI API calls to be able to create the Pixi v0.71.0+ version of this (old system-requirements) pixi.toml Pixi manifest

[workspace]
channels = ["conda-forge"]
name = "example"
platforms = ["linux-64"]
version = "0.1.0"

[tasks]

[dependencies]

[system-requirements]
cuda = "12"

[feature.gpu.dependencies]
pytorch-gpu = ">=2.12.1,<3"

[environments]
gpu = ["gpu"]

Details

which evaluates

$ pixi list --environment gpu torch
Environment: gpu
Name         Version  Build                                 Size  Kind   Source
libtorch     2.12.1   cuda129_mkl_h6b8258c_300        776.69 MiB  conda  https://conda.anaconda.org/conda-forge
pytorch      2.12.1   cuda129_mkl_py314_h89d6db0_300   25.83 MiB  conda  https://conda.anaconda.org/conda-forge
pytorch-gpu  2.12.1   cuda129_mkl_h0d04637_300         54.38 KiB  conda  https://conda.anaconda.org/conda-forge

which I believe is the following rich platform based Pixi manifest:

[workspace]
channels = ["conda-forge"]
name = "example"
platforms = ["linux-64", { name = "linux-64-cuda", platform = "linux-64", cuda = "12" }]
version = "0.1.0"

[tasks]

[dependencies]

[feature.gpu]
platforms = ["linux-64-cuda"]

[feature.gpu.dependencies]
pytorch-gpu = ">=2.12.1,<3"

[environments]
gpu = ["gpu"]

which gives me the ability to do

$ pixi list --environment gpu torch
Environment: gpu
Name         Version  Build                                 Size  Kind   Source
libtorch     2.12.1   cuda129_mkl_h6b8258c_300        776.69 MiB  conda  https://conda.anaconda.org/conda-forge
pytorch      2.12.1   cuda129_mkl_py314_h89d6db0_300   25.83 MiB  conda  https://conda.anaconda.org/conda-forge
pytorch-gpu  2.12.1   cuda129_mkl_h0d04637_300         54.38 KiB  conda  https://conda.anaconda.org/conda-forge

For Pixi version

$ pixi --version
pixi 0.72.0

I think the command line invocations that should create the functionally identical workspace on linux-64 with an NVIDIA GPU on the system are

cd /tmp && rm -rf /tmp/example && pixi init /tmp/example && cd /tmp/example
pixi workspace platform add --cuda 12 linux-64-cuda=linux-64
pixi add --feature gpu --platform linux-64-cuda --no-install pytorch-gpu
pixi workspace environment add --feature gpu gpu
pixi upgrade --no-install

which all in one line is:

cd /tmp && rm -rf /tmp/example && pixi init /tmp/example && cd /tmp/example && pixi workspace platform add --cuda 12 linux-64-cuda=linux-64 && pixi add --feature gpu --platform linux-64-cuda --no-install pytorch-gpu && pixi workspace environment add --feature gpu gpu && pixi upgrade --no-install

✔ Created /tmp/example/pixi.toml
✔ Added linux-64-cuda
✔ Added pytorch-gpu
Added these only for platform(s): linux-64-cuda
Added these only for feature: gpu
 WARN Encountered 1 warning while parsing the manifest:
  ⚠ The feature 'gpu' is defined but not used in any environment. Dependencies of unused features are not resolved or checked, and use wildcard (*) version specifiers by default,
  │ disregarding any set `pinning-strategy`
    ╭─[/tmp/example/pixi.toml:12:10]
 11 │
 12 │ [feature.gpu.target.linux-64-cuda.dependencies]
    ·          ───
 13 │ pytorch-gpu = "*"
    ╰────
  help: Remove the feature from the manifest or add it to an environment

✔ Added environment gpu
...

and produces

[workspace]
channels = ["conda-forge"]
name = "example"
platforms = ["linux-64", { name = "linux-64-cuda", platform = "linux-64", cuda = "12" }]
version = "0.1.0"

[tasks]

[dependencies]

[feature.gpu.target.linux-64-cuda.dependencies]
pytorch-gpu = ">=2.12.1,<3"

[environments]
gpu = ["gpu"]

However, now I have to pass the platform explicitly

$ pixi list --environment gpu torch
Error:   × No packages found in 'gpu' environment for 'linux-64' platform.
$ pixi list --environment gpu --platform linux-64-cuda torch
Environment: gpu
Name         Version  Build                                 Size  Kind   Source
libtorch     2.12.1   cuda129_mkl_h6b8258c_300        776.69 MiB  conda  https://conda.anaconda.org/conda-forge
pytorch      2.12.1   cuda129_mkl_py314_h89d6db0_300   25.83 MiB  conda  https://conda.anaconda.org/conda-forge
pytorch-gpu  2.12.1   cuda129_mkl_h0d04637_300         54.38 KiB  conda  https://conda.anaconda.org/conda-forge

Is there anyway from the CLI API to produce the Pixi manifest that has the platform scoped to the feature?

[workspace]
channels = ["conda-forge"]
name = "example"
platforms = ["linux-64", { name = "linux-64-cuda", platform = "linux-64", cuda = "12" }]
version = "0.1.0"

[tasks]

[dependencies]

[feature.gpu]
platforms = ["linux-64-cuda"]

[feature.gpu.dependencies]
pytorch-gpu = ">=2.12.1,<3"

[environments]
gpu = ["gpu"]

[matthewfeickert]

Cross-post to the question on the prefix.dev Discord: Possible to build platform scoped features from the CLI API with rich platforms?

[matthewfeickert]

Also, it is expected that defining rich platforms at the feature level will be possible? Currently

[workspace]
channels = ["conda-forge"]
name = "example"
platforms = ["linux-64"]
version = "0.1.0"

[tasks]

[dependencies]

[feature.gpu]
platforms = [{ name = "linux-64-cuda", platform = "linux-64", cuda = "12" }]

[feature.gpu.dependencies]
pytorch-gpu = ">=2.12.1,<3"

[environments]
gpu = { features = ["gpu"], no-default-feature = true }

fails with

$ pixi lock
Error:   × expected a string, found table
    ╭─[/tmp/example/pixi.toml:12:14]
 11 │ [feature.gpu]
 12 │ platforms = [{ name = "linux-64-cuda", platform = "linux-64", cuda = "12" }]
    ·              ──────────────────────────────────────────────────────────────
 13 │ 
    ╰────

and instead one needs to have platforms be part of the top-level [workspace] table and that you can't define feature specific rich platforms.

[workspace]
channels = ["conda-forge"]
name = "example"
platforms = ["linux-64", { name = "linux-64-cuda", platform = "linux-64", cuda = "12" }]
version = "0.1.0"

[tasks]

[dependencies]

[feature.gpu]
platforms = ["linux-64-cuda"]

[feature.gpu.dependencies]
pytorch-gpu = ">=2.12.1,<3"

[environments]
gpu = { features = ["gpu"], no-default-feature = true }

with pixi info

...
Environments
------------
        Environment: default
           Features: default
           Channels: conda-forge
   Dependency count: 0
   Target platforms: linux-64-cuda (cuda=12), linux-64
    Prefix location: /tmp/example/.pixi/envs/default

        Environment: gpu
           Features: gpu
           Channels: conda-forge
   Dependency count: 1
       Dependencies: pytorch-gpu
   Target platforms: linux-64-cuda (cuda=12)
    Prefix location: /tmp/example/.pixi/envs/gpu

where linux-64-cuda is now a top level target platform for everything.

In the [system-requirements] approach, this would have been

$ pixi self-update --version 0.70.0
$ pixi init /tmp/example && cd /tmp/example
$ pixi workspace system-requirements add --feature gpu cuda 12
$ pixi workspace environment add --feature gpu gpu --no-default-feature
$ pixi add --feature gpu pytorch-gpu --no-install

producing

[workspace]
channels = ["conda-forge"]
name = "example"
platforms = ["linux-64"]
version = "0.1.0"

[tasks]

[dependencies]

[feature.gpu.system-requirements]
cuda = "12"

[feature.gpu.dependencies]
pytorch-gpu = ">=2.12.1,<3"

[environments]
gpu = { features = ["gpu"], no-default-feature = true }

and pixi info

...
Environments
------------
        Environment: default
           Features: default
           Channels: conda-forge
   Dependency count: 0
   Target platforms: linux-64
    Prefix location: /tmp/example/.pixi/envs/default

        Environment: gpu
           Features: gpu
           Channels: conda-forge
   Dependency count: 1
       Dependencies: pytorch-gpu
   Target platforms: linux-64
    Prefix location: /tmp/example/.pixi/envs/gpu
System requirements: cuda = "12"

[hunger]

Just a remark for now:

[workspace]
platforms = ["linux-64", { name = "linux-64-cuda", platform = "linux-64", cuda = "12" }]

you probably want to swap the order here.

When looking up a platform to use, it will pick the first that "matches" the current machine. So if you have a machine with cuda, you would still use linux-64 in this setup, while you probably want the cuda flavor. This assumes both platforms are available in the environment you are working with of course: You will not get linux-64, when the environment is only available to linux-64-cuda.

It's really hard to come up with a sensible ranking for platforms from "best" to "worst", so I stuck with "top to bottom". At least that is easy to explain.

[hunger]

You also end up with all platforms in the default feature -- unless you define a list of platforms to use in the default feature. So your GPU feature does not work too well in many cases: It adds linux-64-cuda to an environment -- which probably has that via the default environment already.

This "all platforms in default environment" does tend to blow up the lock file quite a bit, as you end up solving (and locking) packages for platforms you defined for one special case. We need to improve in that area.

I have a feature request to allow for a "default=(true|false)" field in the platform definition, which says whether that platform should become part of the default environments platform list or not. I think that would improve things a bit.

[hunger]

About "feature platforms": Technically we need all platforms to be known at the top level, but that does of course not mean pixi.toml has to force you to define them like that.

[luciorq]

Rather than using a boolean default=(true|false) flag, would it make more sense to use a feature array (e.g., feature=["default", "cuda-gpu", "old-linux"]) that defaults to feature=["default"]?