Created attachment 43038 [details]
dmesg output

Created attachment 43040 [details]
2.6.37 Xorg.0.log

I'd consider this a regression since 2.6.36 worked perfectly with the same program and same libraries.

Created attachment 43052 [details]
dmesg with drm.debug=6

Created attachment 43058 [details]
2.6.37 dmesg

I managed to get a backtrace:

Program received signal SIGSEGV, Segmentation fault.
0xb72d7821 in nouveau_dri2_create_buffer (pDraw=0x845e918, attachment=1, format=0) at nouveau_dri2.c:72
72	nouveau_dri2.c: No such file or directory.
	in nouveau_dri2.c
(gdb) bt
#0  0xb72d7821 in nouveau_dri2_create_buffer (pDraw=0x845e918, attachment=1, format=0) at nouveau_dri2.c:72
#1  0xb7327935 in ?? () from /usr/lib/xorg/modules/extensions/libdri2.so
#2  0xb7329ca0 in ?? () from /usr/lib/xorg/modules/extensions/libdri2.so
#3  0x08071e7f in Dispatch () at dispatch.c:432
#4  0x0806577a in main (argc=6, argv=0xbfbd3824, envp=0xbfbd3840) at main.c:291

The offending line of code that caused the crash is:

    nouveau_bo_handle_get(nouveau_pixmap(ppix)->bo, &nvbuf->base.name);

I'm not sure why it's crashing at this point but my guess is that it's probably the ->bo that might be the cause. I'd be glad to help track the cause down. I just need some input on how to proceed.

Just a quick word to say I've traced it to memory allocation problems. The card is a 32MB adapter living in a 64MB GART, with 16MB allocated to a scratch buffer. Attempts to reduce the amount of memory allocated to the scratch buffer ended in failure. The biggest allocation was 17MB, not sure what this was allocated for. See http://ompldr.org/vN2x1eQ for some debugging in the nouveau ddx driver. 

Ideas welcome on how to get this to work.

http://cgit.freedesktop.org/nouveau/xf86-video-nouveau/commit/?id=ace98a492353e6de712f4f717e6d3f562e3591f0 fixes the crash in the X server. 
(thanks to pq)

There are still problems, but trying 2.6.38-rc6 at the moment.

There were some patches to nouveau-git kernel and xf86-video-nouveau which probably fix this bug. Please test it.

Funny though, I just tested an NV11, in old box I played with today.

System itself has 256MB or ram, thus totally unusable even in xubuntu, but nouveau does work and glxgears does work as well (glxinfo reports that nouveau, nv11 driver is used).
Kudos, guys!

That card was Nvidia Geforce2 MX 200, with 32 MB of ram.
I just installed ubuntu 11.10 there.

(In reply to comment #10)
> Funny though, I just tested an NV11, in old box I played with today.
> 
> System itself has 256MB or ram, thus totally unusable even in xubuntu, but
> nouveau does work and glxgears does work as well (glxinfo reports that nouveau,
> nv11 driver is used).
> Kudos, guys!
> 
> That card was Nvidia Geforce2 MX 200, with 32 MB of ram.
> I just installed ubuntu 11.10 there.

How much graphic memory did you have on your box, and which versions of the OpenGL stack did you use with which kernel, please?

I will be testing on my old laptop as well.

(In reply to comment #11)

> I will be testing on my old laptop as well.

Just tested, glxinfo still crashes with a segfault. 

OpenGL stack used:

xf86-video-nouveau 0.0.16_pre20111109
mesa 7.11.2
xorg-server 1.10.4

Kernel:
3.1.0

Alex: you need kernel from nouveau git tree (see http://nouveau.freedesktop.org/wiki/InstallDRM ).

(In reply to comment #13)
> Alex: you need kernel from nouveau git tree (see
> http://nouveau.freedesktop.org/wiki/InstallDRM ).

With 3.3.0, X no longer crashes. Tested on Gnome 3 in fallback mode. One small problem is that if I try to start Gnome 3, it will die with an error. Using fallback mode avoids the problem yet OpenGL programs seem to work just as well. 

Yay :)

All the comments seem to indicate that the issue is fixed, updating status to reflect that.