Fix: use MsBuild Locator rather than embed

[dpvreony]

This is a fairly fundamental change to MsBuild + Roslyn, so appreciate might want to take some time looking at it.

This changes DocFX to use Microsoft.Build.Locator to find the latest applicable version of the .NET SDK or Visual Studio. So the TFM used for the docfx runtime will only find the latest version it can use. (for example the NET8 version complains if you try to scan a net10 project with it as it can only locate net8 etc.) Also added a ForceBuild for cases where some output is not tied to Roslyn, or DesignTime builds.

Fixes #10848 #10808 #10781 #10731 #10901 #11046 #10542

potentially sorts #10772 and the underlying error seen in #10840 (but not the actual piece about enabling binary logging, though that could be done via a Directory.Build.rsp file next to a project \ solution and the ForceBuild option)

also would give the answer "It just works as long as you use the right TFM (or latest) version of the DocFX runtime" to #8251

Thanks :)

[filzrev]

docfx using Microsoft.CodeAnalysis.Workspaces.MSBuild package for metadata extractions.
And metadata extraction is executed on separated process.

Is registered MSBuild SDK on this PR is actually used for metadata extraction?

[dpvreony]

docfx using Microsoft.CodeAnalysis.Workspaces.MSBuild package for metadata extractions. And metadata extraction is executed on separated process.

Is registered MSBuild SDK on this PR is actually used for metadata extraction?

most the code remains the same for metadata, it's just the locator sets up MSBuild rather than it being included in the bin folder. the details are available in https://learn.microsoft.com/en-us/visualstudio/msbuild/find-and-use-msbuild-versions

i've tested metadata extraction using https://github.com/dpvreony/Vetuviem with the release version of docfx it fails to "build" the projects due to source generators being involved. with this PR:

• NET8 and NET9 versions of the docfx tool complain about being unable to Target NET10
• NET10 works and produces full metadata, including for generated code.

i'll look at the other adjustments above

[A-Loot]

Sorry to ask again, but will this be merged anytime soon?

[uo-uhbc]

@dpvreony Thank you very much for creating such a comprehensive solution for this problem.
I'm not sure, how this is working in this repository, but one required test for macos in the pipeline failed.
Maybe @yufeih waits for that to be fixed, before this PR is reviewed and approved.

[dpvreony]

The macos test issue has a pending fix in #11074 it's a download test that's failing.