Wip firmware exclusion refacto

[ycardaillac]

See the detailed commit log for a general description of the process.

Here's what the metadata will look like:

{
    "linux-firmware-nvidia-gpu": {
      "categories": ["GPU", "Storage"],
      "in_essential_category": true,
      "interfaces": []
    },
    "package": {
      "categories": ["Non-essential"],
      "in_essential_category": false,
      "interfaces": ["pci"]
    }
}

The created nonessential_firmware.txt looks like this:

linux-firmware-amdgpu : GPU
linux-firmware-ibt-misc : Bluetooth
linux-firmware-iwlwifi-8265 : UnsupportedInterfaces(pci,sdio)

---

Contributor checklist

• Changes have been tested
  • Covered in automated test suite
  • Manual test case recorded
• Change-type present on at least one commit
• Signed-off-by is present
• The PR complies with the Open Embedded Commit Patch Message Guidelines

Reviewer Guidelines

• When submitting a review, please pick:
  • 'Approve' if this change would be acceptable in the codebase (even if there are minor or cosmetic tweaks that could be improved).
  • 'Request Changes' if this change would not be acceptable in our codebase (e.g. bugs, changes that will make development harder in future, security/performance issues, etc).
  • 'Comment' if you don't feel you have enough information to decide either way (e.g. if you have major questions, or you don't understand the context of the change sufficiently to fully review yourself, but want to make a comment)

[flowzone-app]

Website deployed to CF Pages, 👀 preview link https://488b8571.balena-os.pages.dev

[ycardaillac]

@acostach and I, know this behavior is not what we would want in the end, we need to split all mixed package with an Essential category. nvidia-gpu and nvidia-tegra are of them because they incorporate some storage firmware as well.
This behavior was the one we had before the refacto, so I kept it but the wished behavior is available just bellow in comment.

[floion]

What is the electrical interface used by these chips? It is confusing since it does not seem to be part of any of the interfaces defined above.

[acostach]

@floion these use SDIO/UART and are soldered, the BeagleBone and VAR-SOM-MX6 devices are the only ones supported in balenaOS which would need to include these packages. To avoid including them for boards which don't need this firmware, we created a labeled feature which we could append in the device repo in order to keep the firmware only for the boards which need it

[acostach]

we could alternatively remove the wlcommon, wl12xx and wl18xx packages altogether from the 24 most used ones which are added generically by meta-balena, and add them back in the Beaglebone and VAR-SOM-MX6 repositories

[ycardaillac]

This is a bit awful tbh, we might want to get rid of it altogether but it helped me identify packages that have symlink to external packages but doesn't have any RDEPENDS to expres the dependency.

They are now identified, but I don't know what we should do about it. I guess we could say that if this is fine for Yocto it's fine for us.

[floion]

this is a duplicate, same file is listed a few lines further down this list

[floion]

You probably meant to add "iwlwifi-3160-11ucode" in between these?

[floion]

Have these categories been produced by manually parsing a WHENCE file?

[ycardaillac]

No it's derivated from OpenSuse classification, we probably had to add drivers to it as well.

[alexgg]

Let's see if I understand it correctly:

The purpose of the PR is to implement an automated linux-firmware classification and exclusion pipeline for balenaOS which replaces manually curated per-Yocto-release firmware lists with a metadata-driven policy.

With a dual goal:

1. Homogenizes firmware support — ensures all device types ship a consistent, well-defined set of essential firmware (Connectivity, Storage) that balenaOS can properly test and validate across the fleet.
2. Reduces image size — strips non-essential firmware blobs (GPU, Audio, Video, Bluetooth, Misc) that balenaOS doesn't use or test.

After the exploration I understand 1) is still worthwhile while 2) has been proven not to be the case.

The pull request uses a 3 stage pipeline:

1. Classification (balena-firmware-sort.bbclass)

Runs at do_package time on the linux-firmware recipe using the linux-firmware WHENCE file and a supplementary extra_WHENCE file for firmware not covered in upstream:

Produces firmware_metadata.json, per-package metadata with:

{
  "linux-firmware-ar5523": {
    "categories": ["Connectivity"],
    "in_essential_category": true,
    "interfaces": []
  }
}

This stage uses a manually curated flat python dict that maps every WHENCE driver name to a category:

firmware_sort_driver_categories = {
    "amdgpu": "GPU",
    "iwlwifi": "Connectivity",
    "snd-korg1212": "Audio",
    "atomisp": "Video",
    "isci": "Storage",
    "qat": "Misc",
    "btusb": "Bluetooth",
    # ...
}

This must be exhaustive, every driver in any WHENCE file needs an entry or the build fatals with "Uncategorized driver in WHENCE".

The categories are:

• Connectivity
• Storage
• GPU
• Audio
• Video
• Bluetooth
• Misc

2. Exclusion Policy (balena-firmware-exclusion.bbclass)

Runs at image time (before do_rootfs). Reads firmware_metadata.json and excludes packages that:

• Are NOT in an essential category (GPU, Audio, Video, Misc, Bluetooth)
• Declare hardware interfaces where NONE match the device's MACHINE_FEATURES.

Excluded packages are added to BAD_RECOMMENDATIONS, with BALENA_ALLOWED_FIRMWARE_PACKAGES for exceptions.

3. Post-build Audit (do_nonessential_firmware_check)

After do_rootfs, verifies the image manifest contains none of the excluded packages. Build fails on policy violation.

[alexgg]

The staged classification design makes sense, but the implementation as is now presents several problems that makes it not suitable to merge in its current form:

• Every linux-firmware version bump that introduces new drivers in WHENCE requires manually adding entries to this dict. This is a bigger maintenance burden than the current decentralized way to manage firmware imo.

• The extra_WHENCE file is another layer of manual maintenance overhead, also introducing an unwanted coupling with the upstream WHENCE file.

Initially we toyed with the idea of re-using upstream classification, like the ones done in openSuse, but this might have been found not possible. I did not find the shaping notes for this though. I think this stage should be automated, possibly using an LLM.

Interface-Based Filtering: This is a firmware-to-interface mapping via varflags in balena-os.inc:

• LINUX_FIRMWARE_PACKAGES[pci] — M.2 / PCIe WiFi modules
• LINUX_FIRMWARE_PACKAGES[sdio] — soldered SDIO/UART modules
• LINUX_FIRMWARE_PACKAGES[usbhost] — USB WiFi dongles
• LINUX_FIRMWARE_PACKAGES[features_TI_WiLink8] — TI WL1835/1837 (BeagleBone)
• LINUX_FIRMWARE_PACKAGES[features_TI_WiLink_6_7] — TI WL1271/1273 (VAR-SOM-MX6)

Device repos declare supported interfaces in MACHINE_FEATURES. Packages whose interfaces don't match are stripped at image time.

This was introduced as a way to reduce the required storage size by being able to filter out firmware by hardware capabilities. I currently think that this has proven not to bring enough size reduction benefits to make the extra complexity worth.

[alexgg]

Note that LINUX_FIRMWARE_PACKAGES is defined in several places with ?= and only the first assignment is actually done by bitbake.

[ycardaillac]

Every linux-firmware version bump that introduces new drivers in WHENCE requires manually adding entries to this dict. This is a bigger maintenance burden than the current decentralized way to manage firmware
Generally the firmwares are in the WHENCE already, we just found that some of them are not. And in such a case we have been adding them to the file extra_WHENCE. Also some firmware added by the vendor layers are not in WHENCE as well, so we had to add this to _WHENCE files. So I don't think this is going to be a big challenge in the end, but yes it comes with a bit of maintenance.

Initially we toyed with the idea of re-using upstream classification, like the ones done in openSuse
Yes we reused the classification, but the original file from openSuse added also a lot of complexity to how to generate a simple link between "drivers" (drivers are actually sets of firmwares as per Opensuse classification) and category. So it was quickly abandoned to use a flat and dumb table. (which might require a little bit of maintenance as well if new "drivers" are introduced.)

[ycardaillac]

Note that LINUX_FIRMWARE_PACKAGES is defined in several places with ?= and only the first assignment is actually done by bitbake.

Thanks @alexgg ! Good catch I'll be fixing it!

[ycardaillac]

@alexgg that's exactly that.

1. Homogenizes firmware support — ensures all device types ship a consistent, well-defined set of essential firmware (Connectivity, Storage) that balenaOS can properly test and validate across the fleet.

That has a lot of value, but I'm wondering if it starts with the work we are doing now?

2. Reduces image size — strips non-essential firmware blobs (GPU, Audio, Video, Bluetooth, Misc) that balenaOS doesn't use or test.

We can measure that, but indeed I don't feel like the change will be drastic in the DT that needs it. But that's what we are going to measure.