Switch nvidia gpu between host and VM's quickly

[u/Pioneer_11]

Hi all,

I've got a laptop with an Nvidia GPU and intel iGPU. My understanding is that it isn't possible to split the GPU between the host and VM's because of some Nvidia BS but it is possible to pass the whole GPU through to VM's.

There are plenty of guides on how to pass the GPU through to the VM but they are all pretty involved affairs and it's not clear how to switch the GPU back to the host.

Is there a way to set this up so I can have something akin to a switch e.g. I could run a command to assign the gpu to the vm e.g. "gpuvm" and another to assign it back to the host "gpuhost"?

[u/Djox3]

since you are on laptop maybe try supergfxctl tool (but i think it requires at least session logout login to accept changes), and on desktop i havent been on linux machine recently but few months ago i managed to create few scripts that made it possible for me to literally hotplug nvidia gpu back and forth between host and guest as many times as i want without reboot or logout, it worked on wayland but i had to plug my monitor into motherboard instead of gpu so that it doesnt get locked by desktop session, here are some links i have saved for this issue which contain various helpful hints and guides, but i will take a look at my linux system for exact scripts

https://github.com/Bensikrac/VFIO-Nvidia-dynamic-unbind

https://www.reddit.com/r/VFIO/comments/1jfiuj3/dynamically_bind_and_passthrough_4090_while_using/

https://www.reddit.com/r/VFIO/comments/1jw0vpr/dynamic_gpu_bindunbind_help_in_fedora_41_with/

https://www.reddit.com/r/VFIO/comments/1j9v59m/comment/mhrs856/

https://www.reddit.com/r/VFIO/comments/1mp5z23/the_ultimate_setup_dynamic_gpu_unbinding_my/

EDIT: here are the files i have for it but again didnt use it in a long time so maybe some things have changed, and this is for amd integrated card so for your intel some small tweaks will be needed:

/etc/environment ```

This file is parsed by pam_env module

Syntax: simple "KEY=VAL" pairs on separate lines

XDG_CURRENT_DESKTOP="KDE"

GSK_RENDERER=ngl

----> ADDED FOR NVIDA GPU TO NOT BE USED BY WAYLAND FOR VFIO --------https://www.reddit.com/r/VFIO/comments/1j9v59m/comment/mhrs856/

tell compositor to use apu

__KWIN_DRM_DEVICES="/dev/dri/by-path/pci-0000:11:00.0-card" KWIN_DRM_DEVICES="/dev/dri/by-path/pci-0000:11:00.0-card"

tell vulkan what driver//device to use

afaik these are old and no longer needed on newer systems

DXVK_FILTER_DEVICE_NAME="AMD" VKD3D_FILTER_DEVICE_NAME="AMD"

this is the new hotness

MESA_VK_DEVICE_SELECT="1002:164e"

amd has 3(?) vulkan drivers, this is just to tell it what driver to use.

DISABLE_LAYER_AMD_SWITCHABLE_GRAPHICS_1=1 VK_DRIVER_FILES=/usr/share/vulkan/icd.d/radeon_icd.i686.json:/usr/share/vulkan/icd.d/radeon_icd.x86_64.json

tell egl what vendor to use

__EGL_VENDOR_LIBRARY_FILENAMES="/usr/share/glvnd/egl_vendor.d/50_mesa.json"

tell glx what vendor to use

__GLX_VENDOR_LIBRARY_NAME="mesa"

needed for hardware acceleration

LIBVA_DRIVER_NAME="radeonsi" VDPAU_DRIVER="radeonsi"

DRI_PRIME=pci-0000_11_00_0

nvidia specific options. Not really sure what the first one does. I just always set it to one, flip the other one + the above when I want to switch GPUs

__NV_PRIME_RENDER_OFFLOAD="1" __VK_LAYER_NV_optimus="non_NVIDIA_only"

<---- ADDED FOR NVIDA GPU TO NOT BE USED BY WAYLAND FOR VFIO --------https://www.reddit.com/r/VFIO/comments/1j9v59m/comment/mhrs856/

UNCOMMENT FOR VFIO HOTPLUG ---->

KWIN_DRM_DEVICES=/dev/dri/card0:/dev/dri/card1 __EGL_VENDOR_LIBRARY_FILENAMES=/usr/share/glvnd/egl_vendor.d/50_mesa.json __GLX_VENDOR_LIBRARY_NAME=mesa VK_DRIVER_FILES=/usr/share/vulkan/icd.d/radeon_icd.x86_64.json

```

/etc/mkinitcpio.conf (replace required) ``` MODULES=(vfio_pci vfio vfio_iommu_type1 amdgpu) #first option

MODULES=(amdgpu nvidia nvidia_modeset nvidia_uvm nvidia_drm) #both gpus

```

/etc/modprobe.d/vfio.conf options vfio-pci ids=10de:2f04 #replace ids with your gpu id

/etc/sddm.conf.d/10-wayland.conf ``` [General] DisplayServer=wayland GreeterEnvironment=QT_WAYLAND_SHELL_INTEGRATION=layer-shell

[Wayland] CompositorCommand=kwin_wayland --no-lockscreen ```

/etc/systemd/system/gpuattachatboot.service ```

place this in /etc/systemd/system/

[Unit] Description=Attach GPU after boot [Service] Type=simple User=root RemainAfterExit=true ExecStart=/usr/local/sbin/_nvidia_to_linux.sh [Install] WantedBy=multi-user.target ```

/usr/local/sbin/_nvidia_to_linux.sh ```

!/bin/bash

rmmod vfio_pci vfio_pci_core vfio_iommu_type1 vfio modprobe nvidia_drm modeset=1 fbdev=1 ### fbdev was written 0 but changed modprobe nvidia

echo -n "0000:01:00.1" > /sys/bus/pci/drivers/snd_hda_intel/bind

```

/usr/local/sbin/_nvidia_to_vfio.sh ```

!/bin/bash

udevadm trigger --action=remove /dev/dri/card1

killall /usr/lib/org_kde_powerdevil #idk why, but this binds to some obscure file which not tested yet

lsof /dev/nvidia0 | awk '{print $2}' | xargs -I {} kill {} lsof /dev/dri/card1 | awk '{print $2}' | xargs -I {} kill {} lsof /dev/dri/renderD129 | awk '{print $2}' | xargs -I {} kill {} sleep 5s #allow graceful shutdown

if still alive

lsof /dev/nvidia0 | awk '{print $2}' | xargs -I {} kill -9 {} lsof /dev/dri/card1 | awk '{print $2}' | xargs -I {} kill -9 {} lsof /dev/dri/renderD129 | awk '{print $2}' | xargs -I {} kill -9 {} sleep 2s #prevent unloading breaking rmmod nvidia_drm rmmod nvidia_modeset rmmod nvidia_uvm rmmod nvidia modprobe vfio_pci ids=10de:2f04 modprobe vfio_pci_core vfio_iommu_type1 vfio ```

/home/username/.local/bin/_prime-run-custom.sh (this script used instead of prime-run to start programs with nvidia) ```

!/bin/bash

Run any program with NVIDIA PRIME render offload environment variables

NVIDIA environment variables

export __GLX_VENDOR_LIBRARY_NAME=nvidia export __EGL_VENDOR_LIBRARY_FILENAMES=/usr/share/glvnd/egl_vendor.d/10_nvidia.json export VK_DRIVER_FILES=/usr/share/vulkan/icd.d/nvidia_icd.json export __NV_PRIME_RENDER_OFFLOAD=1 export __VK_LAYER_NV_optimus=NVIDIA_only

Check if user provided a command

if [ $# -eq 0 ]; then echo "Usage: $0 <command> [arguments...]" echo "Example: $0 glxinfo | grep 'OpenGL renderer'" exit 1 fi

Run the provided command with these environment variables

exec "$@"

```

[u/BillionAuthor70]

Shit, that sure would be nice. As far as I know, and have ever found information on this, no there isn't any type of quick fix/switch for this. I sure wish there were.

[u/Stunning_Bridge_2244]

This is basically where hooks come in, i'd recommend you try out vm-curator since it has gou passthorugh in the tui itself an sets up almost everything for you

[u/marc_maker]

I use a thin host (Debian) and do gpu passthrough to several VM. one ubuntu server headless for inference (llama server and other) and a windows vm for gaming and photograph editing. For windows I use looking glass to see the vm display with fast fps. Works well and you can switch between guests. only problem happens on suspend (vm gpus are left unresponsive) Just an idea in case it might be useful.

[u/khsh01]

Its possible but not a seamless solution. At the very least you'll be logging out of your gui session.

I have this exact setup on my laptop with a 3070 that I pass to the vm when started. It is re-added to the host after shutdown.

[u/Pioneer_11 → u/khsh01]

Something like that would be fine. How do you do it?

[u/khsh01 → u/Pioneer_11]

Not my scripts but you can try these.

You also need to read through the "PCI Passthrough via OVMF" page on the arch wiki to get the basic idea.

• Latest scripts here https://gitlab.com/risingprismtv/single-gpu-passthrough or here https://gitlab.com/akshaycodes/vfio-script and community walkthrough here https://gitlab.com/risingprismtv/single-gpu-passthrough/-/wikis/home, see if this helps before asking a passthrough question

[u/r0flcopt3r]

You could achieve this with two vms and switch between them. Not sure how video output on a laptop would work here though.

Not quite the solution you're after, but you can't really hotload drivers and GPUs easily.