nvidia_drm remains in use for no apparent reason after Xorg shutdown.

Hi, i'm trying to Nvidia "Reverse Prime" with a gtx750 + intel hd4000
Displays are attached to the intel connectors.
What i've done so far:

My xorg.conf:
##########################
Section "Module"
	Load "modesetting"
EndSection

Section "Device"
	Identifier "nvidia"
	Driver "nvidia"
	#BusID ""
	Option "AllowEmptyConfiguration"
EndSection
##########################



my xinitrc:
##########################
export DISPLAY=:0.0 
xrandr --setprovideroutputsource modesetting NVIDIA-0
xrandr --auto
openbox --replace &
setxkbmap it
xset mouse 1.4 4
exec sudo -u koko lxterminal &
p1=$!
wait $p1
##########################



System starts with intel as the primary adapter, nvidia modules are NOT autoloaded.

So i do:
##########################
modprobe nvidia-drm
modprobe nvidia_modeset
modprobe nvidia
##########################


Prime synchronization is working fine:
##########################
cat /sys/module/nvidia_drm/parameters/modeset 
Y
##########################


All is good whitin the X server, and works as expected, but what seems wrong to me is that as i exit the Xorg session, nvidia_drm remains in use.
##########################
lsmod | grep nvidia
nvidia_drm             45056  1
nvidia_modeset        798720  1 nvidia_drm
nvidia              11476992  1 nvidia_modeset
drm_kms_helper        126976  2 i915,nvidia_drm
drm                   299008  7 i915,ttm,nvidia_drm,drm_kms_helper
##########################


It took me a while to understand, but there was an open handle to /dev/dri/card1 (nvidia):
##########################
# lsof -n | grep "/dev/dri/card/1"
systemd      1           root   19u      CHR              226,1      0t0            21785 /dev/dri/card1
##########################

It prevents me from unloading nvidia_drm and i need to do that to pass the gpu to a libvirt domain.

Manually attach to systemd and closing the handle makes it work, but i cannot do that ugly hacks all of the times.
Note that the open handles i am speaking about increases by on on every xorg restart.

Who is to blame? nvidia driver? Modesetting driver?