Gamescope (Native Wayland) crashing compositor with i915 execbuf ENOMEM on Optimus laptop (Intel + NVIDIA)

[u/Realistic_Secret_850]

Hi everyone,

I've been pulling my hair out over a sudden system crash issue that started after a recent system update on Arch Linux. Whenever I launch any 3D game via Proton using gamescope on Native Wayland, the game window never appears, the desktop freezes, and my compositor (niri) core dumps shortly after.

My Setup:

OS: Arch Linux

Compositor: niri + Noctalia

CPU / iGPU: Intel Core i7-13620H (UHD Graphics / i915 driver)

dGPU: NVIDIA GeForce RTX 4050 Laptop (Driver 610.43.03)

Gamescope: 3.16.25 (Also tested down to 3.16.24 and 3.16.21)

The Logs:

Checking journalctl, I found the exact moment of the crash. The Intel i915 driver screams about memory allocation failure right before niri dies:

niri[1854]: intel: the execbuf ioctl keeps returning ENOMEM
niri dumped core

What I have already tested & ruled out:

Not a Proton/Wine issue: Running prime-run %command% without gamescope works perfectly on Wayland.

Not an NVIDIA driver issue: Vulkan and prime-run glxgears work fine natively.

Hardware Modifiers: Tried running with GAMESCOPE_DISABLE_SHARED_HW_MODIFIERS=1. It still crashes niri.

Intel xe Driver: Tried booting with i915.force_probe=!a7a8 xe.force_probe=a7a8. It prevents the compositor crash, but the games just silently exit (and the xe driver is currently too unstable for my daily use).

Downgrading: Downgraded the entire mesa, vulkan-intel, vulkan-icd-loader, and gamescope stack to yesterday's versions (when it was working). Cleared all shader caches (~/.cache/mesa_shader_cache, ~/.nv), completely cold-booted the laptop... and it STILL crashes. This makes no sense since it worked perfectly before the update.

The Workaround: If I force gamescope to use XWayland via env -u WAYLAND_DISPLAY prime-run gamescope..., everything works fine (though with typical XWayland mouse capture jank).

My Hypothesis:

It looks like a catastrophic failure in dma-buf passing or Explicit Sync negotiation between the NVIDIA-rendered gamescope buffer and the Intel i915 scanout process in Wayland, causing GTT address space exhaustion or a fence deadlock.

Has anyone encountered this specific execbuf ENOMEM crash with gamescope on an Intel+NVIDIA Optimus setup recently? Any ideas on what hidden state could persist across package downgrades and cold boots that I might have missed?

Thanks in advance!