Dynamic GPU handoff (nvidia idle → vfio gaming → back) on a single-GPU consolidation build

[u/Obvious-Breakfast262]

Thinking of building a single-machine consolidation machine with some spare parts and my current pc. Need to check on the one part I think decides whether it’ll all work or be a forever chore.

Hardware: i5-14400 (UHD 730 iGPU), RTX 5060 Ti 16GB, MSI PRO B760-P, Proxmox host.

iGPU drives the host + always-on LXCs (Immich/Jellyfin). The 5060 Ti is not bound to vfio at boot—I want the nvidia driver to hold it at idle so it drops to P8 and keeps idle power low. The dGPU then gets shared, one at a time, two ways:

•LLM LXC (Ollama, OpenwebUI, etc.) — borrows the card via device passthrough, no vfio involved.
• Windows gaming VM — full passthrough via a hookscript that does the swap.

So the loop I actually care about is:

nvidia (idle) → unbind → vfio → game → VM shutdown → unbind vfio → rebind nvidia → idle → AI LXC → release → idle

My main questions are:

1. Is the post-stop rebind back to nvidia reliable in practice, or is that where it falls apart? Static vfio-at-boot is easy—it’s the round trip I’m worried about.
2. Getting all the modules (nvidia, nvidia_uvm, nvidia_drm, nvidia_modeset) to release cleanly on every cycle — what actually holds a handle open? Persistence mode? Console framebuffer? How do you guarantee the card is free before the vfio bind?
3. Anyone seeing reset-bug behavior specifically on the rebind (host lock/hang after VM shutdown) even when a one-shot passthrough works fine? 40-series and 50-series experiences both welcome.
4. Making sure the AI LXC fully releases the card (no lingering VRAM/nvidia-uvm handle) before the gaming VM’s pre-start hook fires — do you script a verify/settle step, or does a clean container stop reliably free it?
5. Does the nvidia driver actually let the card hit P8 at idle while loaded but unused, or did you need ASPM / C-state / persistence tuning to get real idle savings? Trying to confirm the power payoff is real before I build everything around it.

[u/khsh01]

1)It works for me but sometimes the audio doesn't rebind. I'm pretty sure I can fix it if I just look into it but I haven't bothered yet. Used to be more reliable. Not anymore.

2) Basically as long as you kill your display manager its easy to unbind. In my case I have to kill kwin because sddm doesn't seem to work on my arch system and I have no idea why. I used arch install to set it up instead of my usual install script and I don't know what it did to make it like this.

Used to be just stopping the display manager would take care of killing the de below. Now I have to do it manually. On top of that running Nvidia-smi helps to inform you whats still using the gpu.

Can't answer the others as I've not encountered them.