zed: Prefer dRAID distributed spares to regular ones

[amotin]

One of the main dRAID features is avoiding single drive bottlenecks by using distributed spares. Activation of regular spare will take more time, during which the vdev redundancy is much lower than in case of RAIDZ. But regular spares might still be added to the pool as a second line of defense, possibly shared by several vdevs.

The PR is built on top of #18597 .

How Has This Been Tested?

It wasn't. CI should test it.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Quality assurance (non-breaking change which makes the code more robust against bugs)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[amotin]

@andriytk What do you think about this?

[andriytk]

We do have some old pools with mixed spares, draid and regular ones, and yes - the fact that regular ones are selected before draid ones is a bit annoying. So the change makes perfect sense, but it would be nice also if the user would have the possibility to set the priority of the regular spare devices.

Having said that, some admins may prefer to use regular spares first because they leave them in the pool as replacement devices, so maybe it would be wise to add some sort of configuration for this to not break the existing behaviour.

[amotin]

Rebased on top of new #18597. The diff is smaller, but the idea is the same.

[behlendorf]

One tiny optimization would be to drop (or sort after) any distributed spares from the list which can't be used because they're part of dRAID 1 while the failed vdev is part of dRAID 2. A common configuration for our systems is to have two top-level dRAID vdevs in the pool. Not critical of course but it is a factor when selecting a preferred spare.

With #18597 merged this is ready for a rebase.

[amotin]

Rebased and added the top-level vdev GUID matching.

[behlendorf]

Using the top-guid is a nice way to handle this.