Bug 87819

Created attachment 111451 [details]
kern.log for X freeze with nouveau from linux-3.19.0-rc2

EQ overflowing == driver is stuck (in this case, at least). That "show state" shows:

[ 1014.288718] Xorg            R  running task        0  1638   1589 0x00400004
[ 1014.288720]  ffff8800bc98fb48 ffffffff82f158c0 ffff8800bc98ffd8 0000000000014200
[ 1014.288722]  ffff880138e7fb80 ffffffff82c1c500 ffff8800be2f8000 0000000000000000
[ 1014.288724]  ffff8800bc98fb80 000000010002d6d7 ffffffff82f158c0 ffffffff82f158c0
[ 1014.288727] Call Trace:
[ 1014.288729]  [<ffffffff827cd549>] schedule+0x29/0x70
[ 1014.288731]  [<ffffffff827d023c>] schedule_timeout+0x11c/0x210
[ 1014.288734]  [<ffffffff820deea0>] ? call_timer_fn+0x160/0x160
[ 1014.288736]  [<ffffffff827d11ae>] ? _raw_spin_unlock_irqrestore+0x1e/0x50
[ 1014.288739]  [<ffffffff8252f2c5>] fence_default_wait.part.10+0xe5/0x220
[ 1014.288741]  [<ffffffff8252e0e0>] ? fence_remove_callback+0x70/0x70
[ 1014.288744]  [<ffffffff8252f42c>] fence_default_wait+0x2c/0x30
[ 1014.288746]  [<ffffffff8252e13f>] fence_wait_timeout+0x3f/0x100
[ 1014.288748]  [<ffffffff8252fcf0>] reservation_object_wait_timeout_rcu+0xa0/0x2c0
[ 1014.288799]  [<ffffffffc03a15af>] nouveau_gem_ioctl_cpu_prep+0x6f/0x120 [nouveau]
[ 1014.288816]  [<ffffffffc022ce76>] drm_ioctl+0x2e6/0x590 [drm]
[ 1014.288843]  [<ffffffffc03a1540>] ? nouveau_gem_ioctl_pushbuf+0x9a0/0x9a0 [nouveau]
[ 1014.288846]  [<ffffffff82020ac6>] ? init_fpu+0x56/0xc0
[ 1014.288848]  [<ffffffff82021b49>] ? __restore_xstate_sig+0x99/0x6b0
[ 1014.288874]  [<ffffffffc0396c98>] nouveau_drm_ioctl+0x68/0xf0 [nouveau]
[ 1014.288877]  [<ffffffff820839c7>] ? __set_task_blocked+0x37/0x80
[ 1014.288879]  [<ffffffff82208b25>] do_vfs_ioctl+0x75/0x320
[ 1014.288881]  [<ffffffff82087408>] ? restore_altstack+0x18/0x30
[ 1014.288884]  [<ffffffff82015238>] ? sys_rt_sigreturn+0xb8/0xd0
[ 1014.288886]  [<ffffffff82208e61>] SyS_ioctl+0x91/0xb0
[ 1014.288889]  [<ffffffff827d176d>] system_call_fastpath+0x16/0x1b


Which basically just means that it's waiting for a fence to be reached, which in turn means that the GPU hung. GPU hang recovery is... lacking in nouveau.

Were you doing anything in particular in firefox? (Like running a WebGL testsuite or something)

Does booting with 'nouveau.config=NvMSI=0' help things? I would assume not, but it's an easy one to knock out.

Lastly, there are some patches eventually destined for 3.19 related to nvaa/nvac but which haven't made it to Linus's tree yet:

http://cgit.freedesktop.org/nouveau/linux-2.6/log/?h=linux-3.19

(specifically the 3 prefixed with drm/nouveau/fb/ram/mcp77.) Would be great if you could give those a shot.

Also it'd be nice to verify that 256MB is the correct quantity of stolen memory to be used as VRAM. This is often configurable from the bios, and you can probably get that information from the nvidia blob too.

In Firefox, I was NOT doing anything in particular.

I am NOT able to compile a Linux kernel, nor compile a custom module and include it inside a Linux Kernel.

The machine equipped with NVAC is an iMac 9,1 used in production under Ubuntu with package 'nvidia-331-updates'.
So, I am just able to test from time to time, at night, an upstream Linux Kernel published as an Ubuntu package at http://kernel.ubuntu.com/~kernel-ppa/mainline

Concerning the correct quantity of stolen memory to be used as VRAM :
Using the graphical driver coming from Ubuntu package 'nvidia-331-updates' :

The command 'nvidia-settings -q VideoRam -q GPUMemoryInterface' provides :
-  VideoRam           = 262144
-  GPUMemoryInterface = 128

$ sudo lspci -nn -vv -s 03:00
03:00.0 VGA compatible controller [0300]: NVIDIA Corporation C79 [GeForce 9400] [10de:0867] (rev b1) (prog-if 00 [VGA controller])
	Subsystem: Apple Inc. iMac 9,1 [106b:00ad]
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 46
	Region 0: Memory at d2000000 (32-bit, non-prefetchable) [size=16M]
	Region 1: Memory at c0000000 (64-bit, prefetchable) [size=256M]
	Region 3: Memory at d0000000 (64-bit, prefetchable) [size=32M]
	Region 5: I/O ports at 1000 [size=128]
	[virtual] Expansion ROM at d3000000 [disabled] [size=128K]
	Capabilities: [60] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 00000000fee0300c  Data: 4192
	Kernel driver in use: nvidia

Could you please try one of the images from nouveau.pmoreau.org (either the one from 12/01 or 11/01), using either the Latest or Nouveau boot entry? They contain the patch Ilia was referring too.

I'll put a "me too" up.  I can toss up logs for X.org however I've got a challenge trying to get dmesg out of the machine: local console becomes unresponsive and I have to hard-reboot to recover.

An exerpt from X.org logs:
(EE) [mi] EQ overflowing.  Additional events will be discarded until existing events are processed.
(EE) 
(EE) Backtrace:
(EE) 0: /usr/bin/X (xorg_backtrace+0x4f) [0x7f17a02c94bf]
(EE) 1: /usr/bin/X (mieqEnqueue+0x2b3) [0x7f17a02a81d3]
(EE) 2: /usr/bin/X (0x7f17a00e8000+0x73a6a) [0x7f17a015ba6a]
(EE) 3: /usr/bin/X (xf86PostMotionEvent+0xd0) [0x7f17a019e580]
(EE) 4: /usr/lib64/xorg/modules/input/synaptics_drv.so (0x7f17979fc000+0x5694) [0x7f1797a01694]
(EE) 5: /usr/lib64/xorg/modules/input/synaptics_drv.so (0x7f17979fc000+0x71f2) [0x7f1797a031f2]
(EE) 6: /usr/bin/X (0x7f17a00e8000+0xa3a57) [0x7f17a018ba57]
(EE) 7: /usr/bin/X (0x7f17a00e8000+0xd34a0) [0x7f17a01bb4a0]
(EE) 8: /lib64/libpthread.so.0 (0x7f179f27a000+0x11250) [0x7f179f28b250]
(EE) 9: /lib64/libc.so.6 (ioctl+0x7) [0x7f179dfab297]
(EE) 10: /usr/lib64/libdrm.so.2 (drmIoctl+0x38) [0x7f179f0704b8]
(EE) 11: /usr/lib64/libdrm.so.2 (drmCommandWrite+0x2b) [0x7f179f07335b]
(EE) 12: /usr/lib64/libdrm_nouveau.so.2 (nouveau_bo_wait+0x99) [0x7f179aa7a0e9]
(EE) 13: /usr/lib64/xorg/modules/drivers/nouveau_drv.so (0x7f179ac7f000+0xd0ae) [0x7f179ac8c0ae]
(EE) 14: /usr/lib64/xorg/modules/drivers/nouveau_drv.so (0x7f179ac7f000+0xd601) [0x7f179ac8c601]
(EE) 15: /usr/bin/X (DRI2SwapBuffers+0x27d) [0x7f17a029642d]
(EE) 16: /usr/bin/X (0x7f17a00e8000+0x1afa30) [0x7f17a0297a30]
(EE) 17: /usr/bin/X (0x7f17a00e8000+0x5bbbe) [0x7f17a0143bbe]
(EE) 18: /usr/bin/X (0x7f17a00e8000+0x6017a) [0x7f17a014817a]
(EE) 19: /lib64/libc.so.6 (__libc_start_main+0xf5) [0x7f179deead55]
(EE) 20: /usr/bin/X (0x7f17a00e8000+0x47fed) [0x7f17a012ffed]
(EE) 
(EE) [mi] These backtraces from mieqEnqueue may point to a culprit higher up the stack.
(EE) [mi] mieq is *NOT* the cause.  It is a victim.
(EE) [mi] EQ overflow continuing.  100 events have been dropped.


I don't have a second machine handy to use SSH with right now.

My host is running Gentoo AMD64 with Linux kernel 3.18.5, X.org server 1.15.2, xf86-video-nouveau 1.0.10 and mesa 10.2.6.  The machine is an Apple MacBook A1278 with a NVidia GeForce 9400M.

The kernel was built this morning from git sources (linux-stable git tree).

I'm finding OpenSCAD 2014.03 triggers the bug pretty reliably.  I can be booted in X running Firefox, Thunderbird, QTerminal and FVWM for *days*, but after about 5 minutes of fiddling with models in OpenSCAD, *boom*, nouveau accidentally says something insulting to the GPU and the GPU stops communicating.

An additional note, if you can suggest some patches to try cherry-picking into the kernel, I'm willing to give them a try.  By the sounds of things the video card is very similar to what I'm running here (both being from Apple; 256MB VRAM).

Created attachment 113151 [details]
Xorg.0.log for X freeze with nouveau from linux-3.19.0-rc7

Created attachment 113152 [details]
kern.log for X freeze with nouveau from linux-3.19.0-rc7

Module nouveau from linux-3.19.0-rc7 seems to be more resilient :
I had to perform video streaming under Firefox to get an X freeze.

I have attached :

-  Xorg.0.log,

-  kern.log, restricted to 'linux-3.19.0-rc7', with 'Loglevel set to 1' and 'SysRq : Show State'.

I hope that these log files will be useful.


I am NOT sure that I can test the latest 'Arch Linux' image from http://nouveau.pmoreau.org but I will try.

Thank you for the files.
I'll see if I can spend some time on it, but have unfortunately some other bugs to fix first.

(In reply to Etienne URBAH from comment #9)
> I am NOT sure that I can test the latest 'Arch Linux' image from
> http://nouveau.pmoreau.org but I will try.

The patch we were talking about made it into 3.19.0-rc4, so by testing 3.19.0-rc7 it's clear it wasn't enough to fix the issue.

Now running a kernel built from the Nouveau git tree at this commit:

commit ff4c0d5213b015e60aa87c1352604f10ba9c3e12
Author: Bruno Prémont <bonbons@linux-vserver.org>
Date:   Sun Dec 21 17:43:31 2014 +0100

    drm/nouveau/nouveau: Do not BUG_ON(!spin_is_locked()) on UP
    
    On !SMP systems spinlocks do not exist. Thus checking of they
    are active will always fail.
    
    Use
      assert_spin_locked(lock);
    instead of
      BUG_ON(!spin_is_locked(lock));
    to not BUG() on all UP systems.
    
    Signed-off-by: Bruno Prémont <bonbons@linux-vserver.org>
    Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

RC=0 stuartl@vk4msl-mb ~ $ uname -a
Linux vk4msl-mb 3.18.0-rc4-vk4msl-mb-00789-gff4c0d5 #1 SMP PREEMPT Wed Feb 4 20:19:39 EST 2015 x86_64 Intel(R) Core(TM)2 Duo CPU P7350 @ 2.00GHz GenuineIntel GNU/Linux
RC=0 stuartl@vk4msl-mb ~ $ uptime
 03:52:41 up 17 min,  1 user,  load average: 0.62, 0.49, 0.47

I've been tinkering in OpenSCAD for a while and it has not yet crashed.  Maybe this weekend I'll try stock kernel 3.18.0-rc4 then bisect until we figure out what patches fix it and try cherry-picking those.

Is it still an issue with 3.19? I tried Xonotic and video streaming with Firefox but was unable to trigger those isses using Nouveau's HEAD (currently at http://cgit.freedesktop.org/~darktama/nouveau/commit/?id=e616d549c78c27b6b4d4da25a2369b32bb2a99db).

(In reply to Pierre Moreau from comment #12)
> Is it still an issue with 3.19? I tried Xonotic and video streaming with
> Firefox but was unable to trigger those isses using Nouveau's HEAD
> (currently at
> http://cgit.freedesktop.org/~darktama/nouveau/commit/
> ?id=e616d549c78c27b6b4d4da25a2369b32bb2a99db).

I'll run it up the flagpole and see who salutes… Out of interest how much of current 'nouveau' HEAD is in the mainline kernel at present?

(I realise I meant to try a git bisect to possibly locate the needed patches, I apologise for not getting to that.)

I cannot get this to build:

RC=0 stuartl@vk4msl-mb /usr/src/linux-2.6-stable $ make menuconfig
make: *** No rule to make target `menuconfig'.  Stop.
RC=2 stuartl@vk4msl-mb /usr/src/linux-2.6-stable $ git describe
fatal: No tags can describe 'e616d549c78c27b6b4d4da25a2369b32bb2a99db'.
Try --always, or create some tags.
RC=128 stuartl@vk4msl-mb /usr/src/linux-2.6-stable $ git log -1 | cat
commit e616d549c78c27b6b4d4da25a2369b32bb2a99db
Author: Ben Skeggs <bskeggs@redhat.com>
Date:   Fri Feb 6 09:36:12 2015 +1000

    lib: fix drm backend

(In reply to Stuart Longland from comment #13)
> I'll run it up the flagpole and see who salutes… Out of interest how much of
> current 'nouveau' HEAD is in the mainline kernel at present?

Up to tag 3.19 of this repository: http://cgit.freedesktop.org/~darktama/nouveau/.


(In reply to Stuart Longland from comment #14)
> I cannot get this to build:
> 
> RC=0 stuartl@vk4msl-mb /usr/src/linux-2.6-stable $ make menuconfig
> make: *** No rule to make target `menuconfig'.  Stop.
> RC=2 stuartl@vk4msl-mb /usr/src/linux-2.6-stable $ git describe
> fatal: No tags can describe 'e616d549c78c27b6b4d4da25a2369b32bb2a99db'.
> Try --always, or create some tags.
> RC=128 stuartl@vk4msl-mb /usr/src/linux-2.6-stable $ git log -1 | cat
> commit e616d549c78c27b6b4d4da25a2369b32bb2a99db
> Author: Ben Skeggs <bskeggs@redhat.com>
> Date:   Fri Feb 6 09:36:12 2015 +1000
> 
>     lib: fix drm backend

I guess you're trying to compile Ben's repository (http://cgit.freedesktop.org/~darktama/nouveau/)? If that's the case, it is an out-of-tree repository, so you still need "regular" kernel sources.
To compile it, go to the drm/ directory and execute just a make - it will try to find where the sources are for the kernel you're actually running, or you can override this by setting the env variable LINUXDIR. This will only compile the nouveau.ko module, which you will then find at drm/nouveau/nouveau.ko.

Ahh right, I understood it was a fork of the kernel tree and so added it as a remote in my git repository, pulled it in and tried to build it as such.

So I've now built kernel 3.19 (commit bfa76d49576599a4b9f9b7a71f23d73d6dcff735).  I shall reboot and test with that in its stock configuration.

Then if I get problems, I'll try out the above git tree, building the nouveau.ko module out-of-tree, install it, do another reboot and see if the problem persists.

So far, so good.  I've been tinkering for an hour and haven't yet triggered the EQ overflow.

So the standard vanilla 3.19 kernel's nouveau module is talking nicely to the GPU and both are getting along well.  I haven't yet checked out the out-of-tree module.

I'm seeing no kernel messages regarding the video device either, other than what shows in `dmesg` on boot-up.  Whatever went upstream between 3.18 and 3.19 seems to be doing the trick.

Hello! I have the same issue, after restoring from memory suspend ("echo mem > /sys/power/state")

(EE) [mi] EQ overflowing.  Additional events will be discarded until existing events are processed.
(EE) 
(EE) Backtrace:
(EE) 0: /usr/bin/X (xorg_backtrace+0x42) [0x57bfe2]
(EE) 1: /usr/bin/X (mieqEnqueue+0x213) [0x55e83b]
(EE) 2: /usr/bin/X (QueuePointerEvents+0x5a) [0x44bf4a]
(EE) 3: /usr/lib64/xorg/modules/input/evdev_drv.so (0x7f227fb18000+0x5704) [0x7f227fb1d704]
(EE) 4: /usr/bin/X (0x400000+0x71558) [0x471558]
(EE) 5: /usr/bin/X (0x400000+0x98bd6) [0x498bd6]
(EE) 6: /lib64/libpthread.so.0 (0x7f2286ee8000+0x116b0) [0x7f2286ef96b0]
(EE) 7: /lib64/libc.so.6 (ioctl+0x7) [0x7f2285c15907]
(EE) 8: /usr/lib64/libdrm.so.2 (drmIoctl+0x34) [0x7f2286cdb984]
(EE) 9: /usr/lib64/libdrm.so.2 (drmCommandWrite+0x1e) [0x7f2286cddb36]
(EE) 10: /usr/lib64/libdrm_nouveau.so.2 (nouveau_bo_wait+0x89) [0x7f22827025c9]
(EE) 11: /usr/lib64/xorg/modules/drivers/nouveau_drv.so (0x7f2282908000+0x6a93) [0x7f228290ea93]
(EE) 12: /usr/lib64/xorg/modules/libexa.so (0x7f2281a78000+0xac47) [0x7f2281a82c47]
(EE) 13: /usr/bin/X (0x400000+0x16d363) [0x56d363]
(EE) 14: /usr/bin/X (0x400000+0xbcce5) [0x4bcce5]
(EE) 15: /usr/bin/X (0x400000+0x32bae) [0x432bae]
(EE) 16: /usr/bin/X (0x400000+0x3561e) [0x43561e]
(EE) 17: /usr/bin/X (0x400000+0x3937a) [0x43937a]
(EE) 18: /lib64/libc.so.6 (__libc_start_main+0xf5) [0x7f2285b54df5]
(EE) 19: /usr/bin/X (0x400000+0x24de1) [0x424de1]
(EE) 
(EE) [mi] These backtraces from mieqEnqueue may point to a culprit higher up the stack.
(EE) [mi] mieq is *NOT* the cause.  It is a victim.

Created attachment 113652 [details]
Xorg.0.log for X-1.15.2-r1 freeze with nouveau from linux-3.14.31

(In reply to Oleh Kravchenko from comment #18)
> Hello! I have the same issue, after restoring from memory suspend ("echo mem
> > /sys/power/state")

Could you please try if it is still a problem with kernel 3.19? It seems to have been fixed there, as per comments #11 and #16.


(In reply to Stuart Longland from comment #17)
> So far, so good.  I've been tinkering for an hour and haven't yet triggered
> the EQ overflow.

Good to hear! :)

> Whatever went upstream between 3.18 and 3.19 seems to be doing the trick.

There have some patches specific for NVAA/AC cards - see commits http://cgit.freedesktop.org/~darktama/nouveau/commit/?id=055c212ca03ea964599281211807c09c6cfb8eb5, http://cgit.freedesktop.org/~darktama/nouveau/commit/?id=f59e76f8720bbc8bf94579b9c9a119d518a4a64f and http://cgit.freedesktop.org/~darktama/nouveau/commit/?id=cb5cdd72027f90a9ed488e4d097b6e0a3911c8c9 - but it was fixing an issue where the laptop will hang during Nouveau's initialisation of the card.

(In reply to Pierre Moreau from comment #20)
> (In reply to Stuart Longland from comment #17)
> > So far, so good.  I've been tinkering for an hour and haven't yet triggered
> > the EQ overflow.
> 
> Good to hear! :)
> 
> > Whatever went upstream between 3.18 and 3.19 seems to be doing the trick.
> 
> There have some patches specific for NVAA/AC cards - see commits
> […] but it was fixing an issue
> where the laptop will hang during Nouveau's initialisation of the card.

Ahh, I think I've had that bug too.  Basically I'd power the laptop on and the machine would get as far as loading drivers then hang.  I had it on my TODO list to eventually track down which driver, but lately I've noticed this has stopped happening.

So slowly we're getting the upper hand in this fight.

It's a pity that Nvidia don't direct more resources to supporting the Nouveau driver.  I've vowed to never buy an Nvidia card again due to their poor support of their cards.  (I don't consider a proprietary driver as "support", more a cop-out and a support head-ache.)

It's thanks to teams like Nouveau that I can make reasonable use of the two Nvidia-based systems I have at all.
For this, I thank-you.

(In reply to Stuart Longland from comment #21)
> Ahh, I think I've had that bug too.  Basically I'd power the laptop on and
> the machine would get as far as loading drivers then hang.  I had it on my
> TODO list to eventually track down which driver, but lately I've noticed
> this has stopped happening.

That bug was corrected thanks to NVidia answers about how some specific registers worked.

> It's a pity that Nvidia don't direct more resources to supporting the
> Nouveau driver.  I've vowed to never buy an Nvidia card again due to their
> poor support of their cards.  (I don't consider a proprietary driver as
> "support", more a cop-out and a support head-ache.)

The NVidia team working on Tegra K1 support in Nouveau is growing and they are quite active. A bit more documentation was released last month - see http://lists.freedesktop.org/archives/nouveau/2015-January/019759.html. It might be slow, but they seem to head in the right direction. Let's see in September how the first two years went since they started releasing some documentation / contributing code.

> It's thanks to teams like Nouveau that I can make reasonable use of the two
> Nvidia-based systems I have at all.
> For this, I thank-you.

You are welcome!

Created attachment 114403 [details]
Xorg.0.log OK with nouveau from linux-3.19.1

Module nouveau from linux-3.19.1 seems to be fix the bug :

Even with video streaming under Firefox, I could NOT trigger any problem.

Lot of thanks to the developers.

(In reply to Etienne URBAH from comment #23)
> Created attachment 114403 [details]
> Xorg.0.log OK with nouveau from linux-3.19.1
> 
> Module nouveau from linux-3.19.1 seems to be fix the bug :
> 
> Even with video streaming under Firefox, I could NOT trigger any problem.
> 
> Lot of thanks to the developers.

Awesome! I'll close the bug report then.