02:00.0 VGA compatible controller: ATI Technologies Inc RV350 AP [Radeon 9600] libdrm-2.4.13; mesa-7.5.1 kernel 2.6.31, but no modesetting xorg-server 1.6.3; xf86-video-ati-6.12.4 App crashes after a random period of time, doesn't seem to be related to any user action. PS.: "out of memory" would make a bit sense, but it worked correctly before (though I'm not sure which upgrade exactly broke it).
please attach your xorg log and the output of dmesg.
Created attachment 29633 [details] syslog of the crash
Created attachment 29634 [details] xorg log
Confirmed here. It was a lot more stable before -- but after I upgraded my radeon driver and xorg-server (and possibly the kernel), it crashes pretty much every time I play q3 (urbanterror), usually within a few minutes. 01:00.0 VGA compatible controller: ATI Technologies Inc Radeon Mobility X700 (PCIE) xf86-video-ati-6.12.4 mesa-7.5.2 libdrm-2.4.15 kernel 2.6.32-rc6 [drm] Num pipes: 2 urbanterror: page allocation failure. order:4, mode:0x40d0 Pid: 15033, comm: urbanterror Not tainted 2.6.32-rc6 #1 Call Trace: [<ffffffff81072846>] ? __alloc_pages_nodemask+0x576/0x650 [<ffffffff81072932>] ? __get_free_pages+0x12/0x60 [<ffffffff81227aa8>] ? radeon_cp_cmdbuf+0x1eb8/0x2250 [<ffffffff81030540>] ? default_wake_function+0x0/0x10 [<ffffffff8122a155>] ? radeon_mem_alloc+0x125/0x240 [<ffffffff812033a3>] ? drm_ioctl+0x173/0x3b0 [<ffffffff81225bf0>] ? radeon_cp_cmdbuf+0x0/0x2250 [<ffffffff81049960>] ? autoremove_wake_function+0x0/0x30 [<ffffffff810ad63d>] ? vfs_ioctl+0x7d/0xa0 [<ffffffff810adb10>] ? do_vfs_ioctl+0x410/0x5b0 [<ffffffff810adcf9>] ? sys_ioctl+0x49/0x80 [<ffffffff8100b1a8>] ? system_call_fastpath+0x16/0x1b Mem-Info: DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 0 active_anon:47218 inactive_anon:47285 isolated_anon:0 active_file:2111 inactive_file:3254 isolated_file:0 unevictable:1 dirty:14 writeback:0 unstable:0 free:1469 slab_reclaimable:1516 slab_unreclaimable:1817 mapped:6172 shmem:2606 pagetables:970 bounce:0 DMA free:2056kB min:84kB low:104kB high:124kB active_anon:5520kB inactive_anon:5604kB active_file:708kB inactive_fi le:1292kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15372kB mlocked:0kB dirty:4kB writeback:0kB mapped:184kB shmem:0kB slab_reclaimable:420kB slab_unreclaimable:96kB kernel_stack:0kB pagetables:12kB unstable:0k B bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 488 488 488 DMA32 free:3820kB min:2780kB low:3472kB high:4168kB active_anon:183352kB inactive_anon:183536kB active_file:7736kB inactive_file:11724kB unevictable:4kB isolated(anon):0kB isolated(file):0kB present:499884kB mlocked:4kB dirty:52kB writeback:0kB mapped:24504kB shmem:10424kB slab_reclaimable:5644kB slab_unreclaimable:7172kB kernel_stack:552kB pa getables:3868kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 DMA: 10*4kB 6*8kB 1*16kB 3*32kB 25*64kB 2*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2056kB DMA32: 811*4kB 26*8kB 5*16kB 1*32kB 4*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3820kB 8309 total pagecache pages 338 pages in swap cache Swap cache stats: add 678051, delete 677713, find 250357/286709 Free swap = 1259148kB Total swap = 1262076kB 130800 pages RAM 12213 pages reserved 10008 pages shared 111319 pages non-shared It sure looks like a memory bug.
I just bumped into this while playing Quake Live. Since it runs through Firefox (and it's variants), the crash takes the browser with it - which isn't nice. 01:00.0 VGA compatible controller: ATI Technologies Inc RV350 AP [Radeon 9600] libdrm-2.4.12 mesa-7.5.1 xorg-server-1.6.3 xf86-video-ati-6.12.2 kernel-2.6.32.2 drmRadeonCmdBuffer: -12 (icecat-bin:1844): GLib-GObject-WARNING **: invalid unclassed pointer in cast to `GObject' (icecat-bin:1844): GLib-GObject-CRITICAL **: g_object_ref: assertion `G_IS_OBJECT (object)' failed icecat-bin: page allocation failure. order:4, mode:0x40d0 Pid: 2139, comm: icecat-bin Not tainted 2.6.32.2 #1 Call Trace: [<c014ce6d>] ? 0xc014ce6d [<c014cebb>] ? 0xc014cebb [<e1191ca8>] ? 0xe1191ca8 [<c0118b01>] ? 0xc0118b01 [<c038dc60>] ? 0xc038dc60 [<c0118b01>] ? 0xc0118b01 [<c010e31b>] ? 0xc010e31b [<c0137ecb>] ? 0xc0137ecb [<c0138ca5>] ? 0xc0138ca5 [<c0138d56>] ? 0xc0138d56 [<c01321aa>] ? 0xc01321aa [<c03e4a02>] ? 0xc03e4a02 [<e0868b4e>] ? 0xe0868b4e [<c01293aa>] ? 0xc01293aa [<e0868e36>] ? 0xe0868e36 [<e0866360>] ? 0xe0866360 [<e08664a9>] ? 0xe08664a9 [<e1191b97>] ? 0xe1191b97 [<c03e347e>] ? 0xc03e347e [<c03871b7>] ? 0xc03871b7 [<c016911f>] ? 0xc016911f [<c0173418>] ? 0xc0173418 [<c017394c>] ? 0xc017394c [<c0134c11>] ? 0xc0134c11 [<c0134d4b>] ? 0xc0134d4b [<c01739af>] ? 0xc01739af [<c0102955>] ? 0xc0102955 Mem-Info: DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 Normal per-cpu: CPU 0: hi: 186, btch: 31 usd: 0 active_anon:42068 inactive_anon:42134 isolated_anon:0 active_file:15136 inactive_file:14967 isolated_file:0 unevictable:1 dirty:0 writeback:0 unstable:0 free:1839 slab_reclaimable:1326 slab_unreclaimable:1393 mapped:10564 shmem:581 pagetables:573 bounce:0 DMA free:2028kB min:84kB low:104kB high:124kB active_anon:2120kB inactive_anon:2288kB active_file:2928kB inactive_file:2160kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15804kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:52kB slab_unreclaimable:56kB kernel_stack:0kB pagetables:4kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 492 492 492 Normal free:5328kB min:2792kB low:3488kB high:4188kB active_anon:166152kB inactive_anon:166248kB active_file:57616kB inactive_file:57708kB unevictable:4kB isolated(anon):0kB isolated(file):0kB present:503872kB mlocked:4kB dirty:0kB writeback:0kB mapped:42256kB shmem:2324kB slab_reclaimable:5252kB slab_unreclaimable:5516kB kernel_stack:524kB pagetables:2288kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 DMA: 3*4kB 40*8kB 2*16kB 2*32kB 17*64kB 4*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2028kB Normal: 1116*4kB 50*8kB 9*16kB 2*32kB 4*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5328kB 34450 total pagecache pages 3762 pages in swap cache Swap cache stats: add 39475, delete 35713, find 42078/43867 Free swap = 993604kB Total swap = 1052216kB 131040 pages RAM 0 pages HighMem 2336 pages reserved 45231 pages shared 104783 pages non-shared
I think there's some inefficiency in a way r300_dri.so frees memory, compared to swrast_dri.so.
With 2.6.33-rc3, the problem still exists but the crash error is slightly more descriptive :s.. "drmRadeonCmdBuffer: -12. Kernel failed to parse or rejected command stream. See dmesg for more info." (P.S. If I run the offending program from an xterm, for example, by the way, it also crashes the xterm window and I have to kill it.)
Does mesa from git master or the 7.7 branch work any better? This commit might help: http://cgit.freedesktop.org/mesa/mesa/commit/?id=554043bff72ced41b2a5e03e61cbc087bb41bd3d
By now, I'm on 7.7 and this single patch doesn't seem to help. Now it's xorg-server 1.7.4 kernel 2.6.32 libdrm-2.4.17
Any other bright ideas? (I tried compiling older versions of xf86-video-ati, but they don't seem to be compatible with the new xorg-server's, complaining about "xf86Resources.h: No such file or directory".)
Created attachment 32755 [details] dmesg output (jamesbroadhead)
Created attachment 32756 [details] xorg log (jamesbroadhead)
Seeing this as well, running xbmc initially, but tested with extreme-tuxracer with the same effects. Environment: gentoo x86 2.6.31-gentoo-r6 media-tv/xbmc-9.11-r1 (Confluence skin, no unusual plugins/scripts) 01:00.0 VGA compatible controller: ATI Technologies Inc RV370 5B60 [Radeon X300 (PCIE)] media-libs/mesa-7.5.2 x11-base/xorg-server-1.6.5-r1 x11-drivers/xf86-video-ati-6.12.4 x11-libs/libdrm-2.4.15
Are there any NON-gentoo users seeing this bug? :P (Just to throw in a few more stats, I'm running vanilla kernel sources 2.6.33_rc3, and my vcard is: RADEON(0): Chipset: "ATI Mobility Radeon X700 (M26) (PCIE)" (ChipID = 0x5653) )
(In reply to comment #14) > Are there any NON-gentoo users seeing this bug? :P > > (Just to throw in a few more stats, I'm running vanilla kernel sources > 2.6.33_rc3, and my vcard is: RADEON(0): Chipset: "ATI Mobility Radeon X700 > (M26) (PCIE)" (ChipID = 0x5653) ) > Here's an example of it on a debian system. There are more mentions on some other mailing lists. http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=563257 This bug report seems to cover this and another problem; https://bugs.launchpad.net/ubuntu/+source/xserver-xorg-video-ati/+bug/446674
... and the ubuntu bug leads me to this : https://bugs.freedesktop.org/show_bug.cgi?id=24414
Just updated my system to see if it would go away, problem still exists. (although it seems to take longer to happen now both with xbmc and extreme-tuxracer) media-libs/mesa-7.7-r1 x11-base/xorg-server-1.7.4 x11-drivers/xf86-video-ati-6.12.4-r1 x11-libs/libdrm-2.4.17
Created attachment 32763 [details] syslog with updated system (jamesbroadhead)
That bug report referenced in Comment #17 suggests that downgrading mesa gets rid of the bug, I tried compiling many many git snapshots from the current 2010-Jan "1.5 Mesa 7.8-devel" all the way back to 2008-Dec "1.3 Mesa 7.3-devel", and all the builds had this bug. (I'm not sure if I can compile versions much older than that -- they complain about something regarding swrast.) So, I don't think it's a mesa problem. I also have problems compiling older versions of xf86-video-ati :|. Do the developers not have this bug? (I find the more graphics-intensive the opengl ap, like a huge q3 map, the sooner / more likely it will crash.)
Problem here is kernel part of radeon driver which tries to allocate more than one continues memory page which gets very hard under memory presure. So fix should go to kernel that it allocates only single non-continues memory pages. Possible work arounds for users are: 1. Free all possible memory so kernel has enough continues free pages for radeon driver. 2. Downgrade kernel to 2.6.29(?). I'm not exactly sure if that was the last version which used old kernel driver without this bug.
While comment 21 sounds like the real reason, neither 1. nor 2. seems a good solution. 1. doesn't solve anything and kernel downgrade seems a bad idea. However if that the real problem, what are the chances of getting it fixed by 2.6.34 (as 2.6.33 is probably out of the question now) ?
Unless we're lucky and it's only this commit, that creates problems: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=6abf66018f7fe231720e50f9a47b142182388869 Due to KMS migration, it's really hard to tell, what could be the real reason, as this is one of the dirs with most of the changes.
Sorry for the noise: comment 23 was a silly idea - that commit is probably off-topic here.
Created attachment 32899 [details] [review] Possible fix. Only tested with r200 card! Attached patch applies to current development kernel from Linus' tree (2.6.33-rc5 should work too). If you know how to compile your own kernel you can apply that patch to vanila kernel source and tes if it fixes problems for you. But notice that this only affects old kernel driver code and new KMS code is not changed anyway. If you have same dmesg error messange with KMS it is different bug. I tested patch shortly with r200 card (radeon 9200). Sauerbraten and compiz did run without crashing. IMPORTANT: I can't test r300-r500 (radeon 9500-x1950) code path so you must be extra carefull with those cards. Kernel might crash or corrupt memory if there is any hiden bugs which may destroy your data or eat your pet. But if you are still willing to test with one those cards I'm happy to hear about any errors you might see.
(In reply to comment #25) Well, it's r300 for me and that comment makes me oh-so-confident. Could somebody verify the correctness of that code for my model ?
I tested this possible fix and first of all: it seems to work now, at least I haven't experienced any problems yet. More information: I have here a radeon 9600 Mobility .. this is some sort of r300. The kernel I patched is the SuSe Kernel of 11.2 -- this is some 2.6.31 with those Suse patches applied -- and this patch applied cleanly (just some hunks..) I rebuilt only the radeon module and as it seems it works now for me! Thx for the patch, now the occasional Urban Terror game is enjoyable again!
So I went and tested that patch. No luck, right at the start: [drm:r300_do_cp_cmdbuf] *ERROR* bad cmd_type 0 at byte 1220 in the syslog and app goes bye-bye.
Thanks for testing. Can you reproduce the bug? If yes then booting with drm.debug=1 kernel parameter and providing dmesg log showing what is going wrong in command stream parsing would help fixing the bug. PS. The patch affects drm.ko and radeon.ko so both of them has to be installed if you only build the module instead of the full kernel. --- Comment #28 from Rafał Mużyło <galtgendo@o2.pl> 2010-01-31 19:03:20 PST > --- > So I went and tested that patch. > No luck, right at the start: > [drm:r300_do_cp_cmdbuf] *ERROR* bad cmd_type 0 at byte 1220 > in the syslog and app goes bye-bye. > > > >
Im sorry, your patch also doesn't work for me -- i see the same error as Rafał now. I hadn't checked before if the correct kernel module was loaded. So now I have rebuilt and loaded the correct modules drm.ko and radeon.ko, and the problem is simply reproduceable by running glxgears for instance. To help you debug I loaded the drm module with debug=1 and got the log which I will attach. Note: After loading this module produces a lot of debug messages, I skipped a lot of the standard messages in the beginning, and the log stops with the error, that appears when running glxgears (Nothing interesting happens after that ...). If I can provide more infos, I would be glad to help.
Created attachment 32963 [details] Log produced by drm.debug=1 This log shows the messages produced by the patched drm and radeon modules. The log ends with the error that occurs when running "glxgears".
Created attachment 32964 [details] [review] Second try to fix the memory allocation. Still needs r300 testing. Thanks for log. It helped locating the error. I hadn't conerted the old OUT_RING_TABLE to new OUT_RING_CMD_BUFFER macro. I also reviewed the changes again and found another bug too which fixed in this patch too.
Ok, that was fast! I applied the new patch and now glxgears runs fine! Of course I cannot say if the original problem will occur again, as I have never had a reliable way to reproduce it -- but after a few minutes of test gaming in Urban Terror (where for me the problem occured inregularly but often enough to be very annoying...) I can say at least it didn't occur again yet. I will come back here if I notice any problems, or if I hadn't any problem after trying it out for a longer period of time. Anyway, thx a lot for the work done so far, and especially for the fast reaction on the last report. This is why open source rocks!
(In reply to comment #32) For the moment, it looks like it works for me too. Though very little testing was done.
w00t! Patch in comment #32 fixes this for my R420. THANK YOU.
(In reply to comment #32) I've the same problem, but I don't know how to apply the patch. Can you please explain me what I've to do in details? Thank you in advance L
(In reply to comment #36) ..I forgot.. I've a ATI Mobility Radeon X1300. Is it suited for me? Thank you
Luca, if you know how to compile the kernel from source (I used the patch against a vanilla kernel 2.6.33-rc3, but other .33 and .31 etc kernels might do too), $ cd /usr/src/linux $ patch -p1 < "the patch file from comment #32" $ make $ mv arch/YOURARCH/boot/bzImage /YOUR/BOOTK/ERNEL
(if you compile "drm" and "radeon" as modules, of course you would also have to run "make modules_install" from there)
(In reply to comment #39) I'm not so expert.. I've never done this, I'll study and try! Thank you again.
It is best to do it your distro way when you compile kernel first time. There is good online documentations if you search for "compiling kernel <your distro>". Another easy instalation option is to user the package scrips provided with kernel, Example for debian packages is "make oldconfig && fakeroot make deb-pkg". > --- Comment #40 from Luca Tersi <lucatersi@gmail.com> 2010-02-04 06:36:49 PST --- > (In reply to comment #39) > > I'm not so expert.. I've never done this, I'll study and try! > Thank you again. > >
I applied the second patch against the 2.6.32.8 kernel, and the results were mixed. The good news is that the random crashes have all but disappeared, at least with Quake Live. The not so good news is that exiting the game (fullscreen or otherwise) freezes the display - although strangely not the cursor. I can still log in to the machine via ssh, and doing so scraped dmesg: BUG: unable to handle kernel NULL pointer dereference at 00000607 IP: [<f8988385>] 0xf8988385 *pde = 00000000 Oops: 0000 [#1] PREEMPT SMP last sysfs file: /sys/devices/pci0000:00/0000:00:12.0/net/eth0/carrier Modules linked in: pcmcia pcmcia_core snd_seq_oss radeon snd_seq_midi_event snd_seq ttm drm_kms_helper snd_pcm_oss snd_mixer_oss drm i2c_algo_bit iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables nls_iso8859_1 vfat fat video output lp fuse battery ac sg usb_storage snd_cs4236 snd_mpu401 snd_ens1371 snd_wavefront snd_ac97_codec snd_wss_lib ac97_bus snd_opl3_lib snd_pcm snd_hwdep snd_mpu401_uart snd_rawmidi snd_timer snd_seq_device fan snd via_agp parport_pc soundcore ns558 floppy parport i2c_viapro evdev gameport ehci_hcd via_rhine agpgart i2c_core snd_page_alloc uhci_hcd mii processor button thermal [last unloaded: pcmcia_core] Pid: 1506, comm: icecat-bin Not tainted (2.6.32.8 #1) KT600-8237 EIP: 0060:[<f8988385>] EFLAGS: 00010202 CPU: 0 EAX: 00000607 EBX: f5f36c00 ECX: ded2be04 EDX: 05a005a0 ESI: 00000607 EDI: f5f36f08 EBP: 00000003 ESP: ded2bd8c DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 Process icecat-bin (pid: 1506, ti=ded2b000 task=e637cdc0 task.ti=ded2b000) Stack: 00000000 e637cf4c c052ccf4 00000286 e637cf48 c0530590 c011725e 00000001 <0> ded2be88 00000001 00000001 e62cd520 e62cd5dc 00000286 e62cd520 00000020 <0> ded2be58 e6371c60 c03d2f61 00000000 00000246 f5ee5000 f5f36f08 00000000 Call Trace: [<c011725e>] ? 0xc011725e [<c03d2f61>] ? 0xc03d2f61 [<f898313e>] ? 0xf898313e [<c03d42dc>] ? 0xc03d42dc [<f8698acd>] ? 0xf8698acd [<c0127c61>] ? 0xc0127c61 [<f86964cc>] ? 0xf86964cc [<f8983002>] ? 0xf8983002 [<c015652d>] ? 0xc015652d [<c01375a5>] ? 0xc01375a5 [<c015652d>] ? 0xc015652d [<c01710d9>] ? 0xc01710d9 [<c0171600>] ? 0xc0171600 [<c0159860>] ? 0xc0159860 [<c01598eb>] ? 0xc01598eb [<c015a3e9>] ? 0xc015a3e9 [<c0171671>] ? 0xc0171671 [<c0102895>] ? 0xc0102895 Code: 89 d8 e8 31 f9 ff ff 85 c0 89 c6 c7 44 24 4c 00 00 00 00 0f 85 8e 1d 00 00 eb c8 ba 04 00 00 00 8d 4c 24 78 e8 b9 ae d0 ff 89 c6 <8a> 00 8d 50 ff 80 fa 08 0f 87 17 1d 00 00 0f b6 d2 ff 24 95 4c EIP: [<f8988385>] SS:ESP 0068:ded2bd8c CR2: 0000000000000607 ---[ end trace 9cffec1ef74538ac ]---
> --- Comment #42 from Juhana Uuttu <juhana.uuttu@gmail.com> 2010-02-11 13:40:46 PST --- > I applied the second patch against the 2.6.32.8 kernel, and the results were > mixed. The good news is that the random crashes have all but disappeared, at > least with Quake Live. The not so good news is that exiting the game > (fullscreen or otherwise) freezes the display - although strangely not the > cursor. I can still log in to the machine via ssh, and doing so scraped dmesg: > Can you build kernel with debug symbols and try to reproduce the bug? Also objdump -S -d radeon.ko would be helpful if the crash is in the radeon.ko module.
(In reply to comment #43) Sorry for the late reply, I had the case of the cold and couldn't contribute. Blergh. > Can you build kernel with debug symbols and try to reproduce the bug? > Also objdump -S -d radeon.ko would be helpful if the crash is in the > radeon.ko module. There may not be any need, since I did a silly thing in the beginning and tried the 2.6.32.8 kernel _without trying if it indeed worked without the patch_: And it does. I'll have to look into this and check with earlier versions of the kernel for good measurement.
...and on the topic of new kernels: I wonder if this patch will still be required/working with 2.6.33 (which will probably arrive shortly). KMS is no longer in staging there (though preformance is still not that good) if I read the news correctly. Though for the full experience we'll probably need to wait till new Mesa/xorg-server(1.8)/xf86-video-ati get released. Chances are whole set will be available around April Fools' ;)
The patch was applied to 2.6.34-rc1.
Any chance to backport the patches to 2.6.33.x?
(In reply to comment #47) > Any chance to backport the patches to 2.6.33.x? > Patches should be simple as applying to even very old drm. There hasn't been much changes in UMS code path. But size of changes is quite large. Dave already commented that he doesn't want to push it to stable because of invasive changes. (at least without extensive testing) But if any distro maintainer wants to backport changes to the stable kernel commits in Linus' tree are 7a9f0dd9c49425e2b0e39ada4757bc7a38c84873 drm: Add generic multipart buffer. b4fe945405e477cded91772b4fec854705443dd5 drm/radeon: Fix memory allocation failures in the preKMS command stream checking. 55a5cb5d594c18b3147a2288b00ea359c1a38cf8 drm/radeon: Fix printf type warning in 64bit system.
Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.