PCI ROM BAR not exposed to guest when using xen-pci-passthrough

[timemaster5]

Description

When passing through a GPU to an HVM guest using xen-pci-passthrough on XCP-ng, the PCI Expansion ROM BAR is not exposed to the guest. As a result, the guest operating system cannot access the device VBIOS through the standard PCI ROM interface.

Inside the guest, the ROM BAR register (0x30) remains unset and the expansion ROM cannot be mapped:
setpci -s 00:08.0 30.l 00000000
For GPUs such as AMD Phoenix (Radeon 780M), the Linux amdgpu driver requires access to the VBIOS to initialize the device. Because the ROM BAR is not exposed, the driver fails with errors similar to:
amdgpu: Unable to locate a BIOS ROM amdgpu: Fatal error during GPU init
Even when specifying romfile and rombar properties on the xen-pci-passthrough device in QEMU, the ROM region is not mapped into guest memory.

---

Observed behavior

• xen-pci-passthrough device exposes romfile and rombar properties internally.
• The ROM BAR is not enabled or mapped for the guest.
• The guest driver cannot read the VBIOS via /sys/bus/pci/devices/.../rom.
• The PCI ROM BAR register (0x30) remains 0x00000000.

---

Expected behavior

If a passthrough device has a valid option ROM or a romfile is provided, the PCI Expansion ROM BAR should be exposed to the guest so that standard firmware discovery mechanisms work.

---

Impact

Modern GPU drivers (for example amdgpu) rely on the PCI ROM interface to retrieve the VBIOS. Without ROM BAR exposure, GPU passthrough may fail to initialize.

Question

Is the ROM BAR intentionally hidden for security reasons in xen-pci-passthrough, or is this a limitation or bug in the current implementation?

[DustyArmstrong]

While I'm not affiliated with the XCP team, I thought you may be interested in my own query that relates to this. May or may not be the exact same issue, but from reading your post it looks very similar at the very least.

https://xcp-ng.org/forum/topic/11918/amd-barcelo-passthrough-issues-any-success-stories/9

To quote Teddy from that thread:

So basically, there is a more annoying issue, as the device doesn't even have a ROMBAR, in this case, the VBIOS is likely in the VFCT ACPI table of host (which the guest can't see); which needs to be injected as a "fake" rombar for the guest to behave properly.

That doable on its own, but it's quite tricky to integrate (and you would e.g need to extract VBIOS from VFCT using external tools).

I just discussed with Xen/AMD people, and there are known issues regarding PCI Passthrough of integrated AMD GPUs (not specific to Xen AFAIU). There are some projects regarding alternative approaches to bring AMD GPUs to VMs (virtio-gpu native context) which is the current focus.

Basically, looks like there is some work going on with this, but we are probably a ways off from that as it's not a straightforward issue to resolve unfortunately. So in answer to your question, it's a limitation.

[stormi]

CC @xcp-ng/hypervisor-kernel

[fallen]

@stormi I'm aware of this issue and of the workarounds/PR opened by @timemaster5
But IIUC it's a bit more complex than it looks.
It's not straight forward to integrate/fix.
But the issue is tracked internally in plane (XCPNG-2988) it's just that since it's not straight forward and we already have a lot on our plate ... it's not been investigated yet.

[timemaster5]

@fallen sorry, I know the message wasn’t meant for me. Just to clarify, I’m trying to guide the workaround PRs so they make sense for broader use cases, not just this specific one. I think there’s real potential for them to be useful in other targeted scenarios, and since they’re quite simple, they could help unlock this functionality for a meaningful portion of AMD users before a proper fix is in place - without breaking anything or making it feel like a workaround.

BTW: BTW: I've been pointed to this issue xapi-project/xen-api@930ffb3 which resolves probably a big part of the AMD GPU passthrough issue, but I still don't see a way of setting up the ROMfile option via QMP once the machine is up. More here