Validate /host-root mount at VfioPciManager startup by johnahull · Pull Request #1077 · kubernetes-sigs/dra-driver-nvidia-gpu

johnahull · 2026-04-24T21:33:24Z

Summary

Adds a startup check in NewVfioPciManager that fails fast if /host-root is not mounted, instead of silently failing on every fuser call during VFIO bind
Improves error messages in WaitForGPUFree to include actionable diagnostics when fuser fails (suggests checking featureGates.PassthroughSupport) and when external processes are detected holding GPU device handles (references WaitForGPUFree blocks indefinitely when external processes hold GPU device handles during VFIO bind #1076)

Context

When deploying without the helm chart (or with PassthroughSupport not set), the /host-root hostPath volume is not mounted. WaitForGPUFree calls chroot /host-root fuser ... which fails with exit code 125 on every attempt, logging only Unexpected error checking if gpu device "0000:xx:00.0" is free: exit status 125 — with no indication that the mount is missing. This loops for 60 seconds until the gRPC deadline, making VFIO passthrough appear broken with no actionable error.

Tested

Verified on the following configuration:

Node: Dell PowerEdge R760xa, 2x NVIDIA A40 GPUs (0000:4a:00.0, 0000:61:00.0)
OS: Fedora 43 Server, kernel 6.19.13-200.fc43.x86_64
Kubernetes: v1.37.0-alpha.0 (single-node, kubeadm)
Container runtime: containerd 2.1.6
NVIDIA driver: 580.142
GPU Operator: v26.3.1

Without /host-root mount: NewVfioPciManager fails at startup with a clear error message referencing the missing mount and featureGates.PassthroughSupport. Previously, this produced only exit status 125 errors during VFIO bind with no indication of root cause.

With /host-root mount + dcgm-exporter disabled: VFIO GPU claim created via gpu-vfio.nvidia.com DeviceClass. GPU-0 successfully unbound from nvidia and bound to vfio-pci. Pod received /dev/vfio/10 (IOMMU group) and /dev/vfio/vfio. End-to-end time ~9 seconds.

With /host-root mount + dcgm-exporter running: WaitForGPUFree now logs the holder PIDs with an actionable message referencing #1076, instead of the previous generic error.

Test plan

Deploy kubelet-plugin without PassthroughSupport feature gate → NewVfioPciManager fails immediately with a clear message about the missing mount
Deploy with PassthroughSupport=true → startup succeeds, VFIO bind works
Trigger VFIO bind while dcgm-exporter holds GPU FDs → log message identifies the holder and references WaitForGPUFree blocks indefinitely when external processes hold GPU device handles during VFIO bind #1076

Ref: #1076

k8s-ci-robot · 2026-04-24T21:33:27Z

Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

netlify · 2026-04-24T21:33:29Z

✅ Deploy Preview for dra-driver-nvidia-gpu ready!

Name	Link
🔨 Latest commit	`8c67a5d`
🔍 Latest deploy log	https://app.netlify.com/projects/dra-driver-nvidia-gpu/deploys/6a02010f54904600088fa9c8
😎 Deploy Preview	https://deploy-preview-1077--dra-driver-nvidia-gpu.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

linux-foundation-easycla · 2026-04-24T21:33:32Z

The committers listed above are authorized under a signed CLA.

✅ login: johnahull / name: John Hull (8c67a5d)

k8s-ci-robot · 2026-04-24T21:33:33Z

Welcome @johnahull!

It looks like this is your first PR to kubernetes-sigs/dra-driver-nvidia-gpu 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/dra-driver-nvidia-gpu has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2026-04-24T21:33:34Z

Hi @johnahull. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

johnahull · 2026-04-24T21:49:01Z

/easycla

shivamerla · 2026-04-28T16:46:35Z

/assign @varunrsekar

shivamerla · 2026-04-28T16:50:49Z

 		return nil, fmt.Errorf("IOMMU is not enabled in the kernel")
 	}

+	if _, err := os.Stat(hostRoot); os.IsNotExist(err) {


When do you hit this case? If PassthroughSupport feature gate is enabled, then this host mount should be added through Helm templates. This function itself is only invoked when that feature gate is enabled here:

dra-driver-nvidia-gpu/cmd/gpu-kubelet-plugin/device_state.go

Line 136 in 7f0d03d

if featuregates.Enabled(featuregates.PassthroughSupport) {

I forked the repo (added some standard attributes to the driver) and i was using a custom-built image, which i deployed via Helm with some custom values. I agree that the stock Helm chart has the mount present. This was just something I hit doing some dev work.

… messages NewVfioPciManager now checks that the /host-root volume mount exists at startup. Without this mount, WaitForGPUFree silently fails with "exit status 125" on every fuser invocation, making VFIO passthrough appear to hang with no actionable diagnostics. Also improves error messages in WaitForGPUFree to: - Include the mount path and suggest checking the helm chart featureGates.PassthroughSupport setting when fuser fails - Reference issue kubernetes-sigs#1076 when external processes (e.g. dcgm-exporter) are detected holding GPU device handles Ref: kubernetes-sigs#1076

k8s-ci-robot · 2026-05-11T16:17:24Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: johnahull
Once this PR has been reviewed and has the lgtm label, please ask for approval from varunrsekar. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

varunrsekar · 2026-06-10T19:04:32Z

When deploying without the helm chart (or with PassthroughSupport not set), the /host-root hostPath volume is not mounted. WaitForGPUFree calls chroot /host-root fuser ... which fails with exit code 125 on every attempt, logging only Unexpected error checking if gpu device "0000:xx:00.0" is free: exit status 125 — with no indication that the mount is missing. This loops for 60 seconds until the gRPC deadline, making VFIO passthrough appear broken with no actionable error.

@johnahull Can you confirm you dont hit this issue if you deploy through the helm chart? This change is unnecessary if that's the case.

github-project-automation Bot added this to DRA Driver for NVIDIA GPUs Apr 24, 2026

github-project-automation Bot moved this to Backlog in DRA Driver for NVIDIA GPUs Apr 24, 2026

k8s-ci-robot added the do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. label Apr 24, 2026

k8s-ci-robot requested a review from cdesiniotis April 24, 2026 21:33

k8s-ci-robot added the needs-kind Indicates a PR lacks a `kind/foo` label and requires one. label Apr 24, 2026

k8s-ci-robot requested a review from varunrsekar April 24, 2026 21:33

k8s-ci-robot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Apr 24, 2026

k8s-ci-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Apr 24, 2026

johnahull force-pushed the fix/host-root-mount-validation branch from 62e3acd to a7d5a15 Compare April 24, 2026 21:51

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Apr 24, 2026

k8s-ci-robot assigned varunrsekar Apr 28, 2026

shivamerla reviewed Apr 28, 2026

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 7, 2026

johnahull force-pushed the fix/host-root-mount-validation branch from a7d5a15 to 8c67a5d Compare May 11, 2026 16:17

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 11, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Validate /host-root mount at VfioPciManager startup#1077

Validate /host-root mount at VfioPciManager startup#1077
johnahull wants to merge 1 commit into
kubernetes-sigs:mainfrom
johnahull:fix/host-root-mount-validation

johnahull commented Apr 24, 2026 •

edited

Loading

Uh oh!

k8s-ci-robot commented Apr 24, 2026

Uh oh!

netlify Bot commented Apr 24, 2026 •

edited

Loading

Uh oh!

linux-foundation-easycla Bot commented Apr 24, 2026 •

edited

Loading

Uh oh!

k8s-ci-robot commented Apr 24, 2026

Uh oh!

k8s-ci-robot commented Apr 24, 2026

Uh oh!

johnahull commented Apr 24, 2026

Uh oh!

shivamerla commented Apr 28, 2026

Uh oh!

shivamerla Apr 28, 2026

Uh oh!

johnahull Apr 28, 2026

Uh oh!

k8s-ci-robot commented May 11, 2026

Uh oh!

varunrsekar commented Jun 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

johnahull commented Apr 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Context

Tested

Test plan

Uh oh!

k8s-ci-robot commented Apr 24, 2026

Uh oh!

netlify Bot commented Apr 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for dra-driver-nvidia-gpu ready!

Uh oh!

linux-foundation-easycla Bot commented Apr 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Apr 24, 2026

Uh oh!

k8s-ci-robot commented Apr 24, 2026

Uh oh!

johnahull commented Apr 24, 2026

Uh oh!

shivamerla commented Apr 28, 2026

Uh oh!

shivamerla Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

johnahull Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

k8s-ci-robot commented May 11, 2026

Uh oh!

varunrsekar commented Jun 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

johnahull commented Apr 24, 2026 •

edited

Loading

netlify Bot commented Apr 24, 2026 •

edited

Loading

linux-foundation-easycla Bot commented Apr 24, 2026 •

edited

Loading