[Problem] Since upgrade from kernel 2.6.35 to kernel 3.2.0 (Ubuntu 12.04) we experience numerous kernel freezes (no keyboard/mouse, no kernel logging, no num key change, stop of server applications, e. g. dhcpd, postfix, bind9, magic keys not working, no serial console), that can only be resolved by switching power off (hard reset). There is no clear way to reproduce this bug. The likelihood of the kernel crash increases, if mail GUIs like Evolution or Thunderbird are open or Firefox is open and when switching between these windows. The kernel freezes latest within 6 h. Whereas if just the desktop and xterm is running, the system seems to be stable (48 h and more). The bug can be confirmed under lightdm/unity as desktop, as well as when using mdm/cinnamon. Log information can only be retrieved using netconsole. They show a GPU lockup and CP stall, from which the radeon driver cannot recover. [Configuration Specifics] We run two X Servers: one controlled by the display manager (on vt7 or vt8) and one controlled by xinit (on vt9). See attached process list. We use the DVI port of the integrated Radeon X1200 (RS690) display controller on an ASUS MSA mainboard. One monitor (Samsung) is connected. Radeon driver with KMS enabled is used. [Netconsole Output] The last kernel log messages that reach the netconsole receiver vary: a) The shortest log [212173.596044] radeon 0000:01:05.0: GPU lockup CP stall for more than 10000msec [212175.370234] radeon 0000:01:05.0: failed to reset GPU [212175.406899] [drm:radeon_ib_schedule] *ERROR* radeon: couldn't schedule IB(3). [212175.406912] [drm:radeon_cs_ioctl] *ERROR* Failed to schedule IB ! b) A longer one [286295.708052] radeon 0000:01:05.0: GPU lockup CP stall for more than 10020msec [286297.455900] radeon 0000:01:05.0: failed to reset GPU [286297.929150] [drm:radeon_ib_schedule] *ERROR* radeon: couldn't schedule IB(14). [286297.929174] [drm:radeon_cs_ioctl] *ERROR* Failed to schedule IB ! [286297.937321] [drm:radeon_ib_schedule] *ERROR* radeon: couldn't schedule IB(15). [286297.937349] [drm:radeon_cs_ioctl] *ERROR* Failed to schedule IB ! [286297.943050] [drm:radeon_ib_schedule] *ERROR* radeon: couldn't schedule IB(0). [286297.943074] [drm:radeon_cs_ioctl] *ERROR* Failed to schedule IB ! [286297.947188] [drm:radeon_ib_schedule] *ERROR* radeon: couldn't schedule IB(1). [286297.947213] [drm:radeon_cs_ioctl] *ERROR* Failed to schedule IB ! [286297.949490] [drm:radeon_ib_schedule] *ERROR* radeon: couldn't schedule IB(2). [286297.949509] [drm:radeon_cs_ioctl] *ERROR* Failed to schedule IB ! c) GPU reset attempt [179005.128038] radeon 0000:01:05.0: GPU lockup CP stall for more than 10032msec [179005.128068] GPU lockup (waiting for 0x0004E1FF last fence id 0x0004E1F4) [179005.268649] radeon: wait for empty RBBM fifo failed ! Bad things might happen. [179005.409044] Failed to wait GUI idle while programming pipes. Bad things might happen. [179005.410064] radeon 0000:01:05.0: (rs600_asic_reset:348) RBBM_STATUS=0x9401C100 [179005.908155] radeon 0000:01:05.0: (rs600_asic_reset:367) RBBM_STATUS=0x9401C100 [179006.405224] radeon 0000:01:05.0: (rs600_asic_reset:375) RBBM_STATUS=0x9400C100 [179006.902280] radeon 0000:01:05.0: (rs600_asic_reset:383) RBBM_STATUS=0x9400C100 [179006.902315] radeon 0000:01:05.0: restoring config space at offset 0x1 (was 0x100403, writing 0x100407) [179006.902346] radeon 0000:01:05.0: failed to reset GPU [179006.903346] radeon 0000:01:05.0: GPU reset failed d) Successful GPU reset but inaccessible CP [ 1775.356043] radeon 0000:01:05.0: GPU lockup CP stall for more than 10008msec [ 1775.356067] GPU lockup (waiting for 0x000124ED last fence id 0x000124EA) [ 1775.919383] radeon: wait for empty RBBM fifo failed ! Bad things might happen. [ 1776.059845] Failed to wait GUI idle while programming pipes. Bad things might happen. [ 1776.060872] radeon 0000:01:05.0: (rs600_asic_reset:348) RBBM_STATUS=0xB001C100 [ 1776.559021] radeon 0000:01:05.0: (rs600_asic_reset:367) RBBM_STATUS=0x90010140 [ 1777.056092] radeon 0000:01:05.0: (rs600_asic_reset:375) RBBM_STATUS=0x10000140 [ 1777.553160] radeon 0000:01:05.0: (rs600_asic_reset:383) RBBM_STATUS=0x10000140 [ 1777.553197] radeon 0000:01:05.0: restoring config space at offset 0x1 (was 0x100403, writing 0x100407) [ 1777.553232] radeon 0000:01:05.0: GPU reset succeed [ 1777.554232] radeon 0000:01:05.0: GPU reset succeed [ 1777.554263] sched: RT throttling activated [ 1777.749090] [drm] radeon: 1 quad pipes, 1 z pipes initialized. [ 1777.754590] [drm] PCIE GART of 512M enabled (table at 0x0000000036700000). [ 1777.754958] radeon 0000:01:05.0: WB enabled [ 1777.754991] [drm] radeon: ring at 0x0000000080001000 [ 1777.892767] [drm:r100_ring_test] *ERROR* radeon: ring test failed (scratch(0x15E4)=0xCAFEDEAD) [ 1777.892774] [drm:r100_cp_init] *ERROR* radeon: cp isn't working (-22). [ 1777.892783] radeon 0000:01:05.0: failed initializing CP (-22). [ 1786.390793] [drm:radeon_ib_schedule] *ERROR* radeon: couldn't schedule IB(11). [ 1786.390818] [drm:radeon_cs_ioctl] *ERROR* Failed to schedule IB ! A really verbose log with drm.debug set to 0xf has been attached. As well as the usually required information.
Created attachment 89569 [details] Dmesg Output
Created attachment 89570 [details] Lspci Output
Created attachment 89571 [details] GlxInfo
Created attachment 89572 [details] Process List
Created attachment 89573 [details] Xorg Log (X Server Controlled by MDM)
Created attachment 89574 [details] Xorg Log (X Server Controlled by xinit)
Created attachment 89575 [details] X Session Errors (X Server Controlled by MDM)
Created attachment 89576 [details] X Session Errors (X Server Controlled by xinit)
There is one more interesting log message, that I've found in my local kernel debug log: [29866.429121] radeon 0000:01:05.0: restoring config space at offset 0x1 (was 0x100403, writing 0x100407) [ 0.000000] DMI: System manufacturer System Product Name/M2A-VM HDMI, BIOS ASUS M2A-VM HDMI ACPI BIOS Revision 1001 06/26/2007 [ 0.000000] e820 update range: 0000000000000000 - 0000000000010000 (usable) ==> (reserved) [ 0.000000] e820 remove range: 00000000000a0000 - 0000000000100000 (usable) [ 0.000000] MTRR default type: uncachable
Is this still an issue with a newer kernel? Did you also update your userspace stack (mesa, xf86-video-ati)?
(In reply to comment #10) > Is this still an issue with a newer kernel? Did you also update your > userspace stack (mesa, xf86-video-ati)? We used Ubuntu update manager to upgrade to kernel 3.2. It's part of Ubuntu's long term support release 12.04. From the attached logs I can see following installed versions: 1. xserver-xorg-video-ati (Ubuntu's xf86-video-ati) [ 107.230] (II) Loading /usr/lib/xorg/modules/drivers/ati_drv.so [ 107.230] (II) Module ati: vendor="X.Org Foundation" [ 107.230] compiled for 1.11.3, module version = 6.14.99 [ 107.230] Module class: X.Org Video Driver [ 107.230] ABI class: X.Org Video Driver, version 11.0 [ 107.230] (II) LoadModule: "radeon" [ 107.230] (II) Loading /usr/lib/xorg/modules/drivers/radeon_drv.so [ 107.231] (II) Module radeon: vendor="X.Org Foundation" [ 107.231] compiled for 1.11.3, module version = 6.14.99 [ 107.231] Module class: X.Org Video Driver [ 107.231] ABI class: X.Org Video Driver, version 11.0 2. OpenGL version string: 2.1 Mesa 8.0.4 Are these the expected versions? I can try to install also a newer kernel. But as we talk about a 24/7 server, I need some guidance. Newer kernel also requires newer userspace stack? Which kernel version would you recommend? Do you have any ideas on possible root causes of this kernel oops?
(In reply to comment #11) > (In reply to comment #10) > > Is this still an issue with a newer kernel? Did you also update your > > userspace stack (mesa, xf86-video-ati)? > > We used Ubuntu update manager to upgrade to kernel 3.2. It's part of > Ubuntu's long term support release 12.04. > From the attached logs I can see following installed versions: > 1. xserver-xorg-video-ati (Ubuntu's xf86-video-ati) > [ 107.230] (II) Loading /usr/lib/xorg/modules/drivers/ati_drv.so > [ 107.230] (II) Module ati: vendor="X.Org Foundation" > [ 107.230] compiled for 1.11.3, module version = 6.14.99 > [ 107.230] Module class: X.Org Video Driver > [ 107.230] ABI class: X.Org Video Driver, version 11.0 > [ 107.230] (II) LoadModule: "radeon" > [ 107.230] (II) Loading /usr/lib/xorg/modules/drivers/radeon_drv.so > [ 107.231] (II) Module radeon: vendor="X.Org Foundation" > [ 107.231] compiled for 1.11.3, module version = 6.14.99 > [ 107.231] Module class: X.Org Video Driver > [ 107.231] ABI class: X.Org Video Driver, version 11.0 > 2. OpenGL version string: 2.1 Mesa 8.0.4 > > Are these the expected versions? I'm not sure what ubuntu ships off hand. > > I can try to install also a newer kernel. But as we talk about a 24/7 > server, I need some guidance. Newer kernel also requires newer userspace > stack? Which kernel version would you recommend? The kernels and userspace stacks are generally compatible, so for the most part you can mix and match userspace and kernel versions as long as both support your asics. > > Do you have any ideas on possible root causes of this kernel oops? Some combination of commands from the userspace components is causing a a GPU hang. The driver attempts to reset the GPU, but fails. I would suggest trying upgrading or downgrading your userspace gfx components to see if that helps.
(In reply to comment #12) > (In reply to comment #11) > > (In reply to comment #10) > > > Is this still an issue with a newer kernel? Did you also update your > > > userspace stack (mesa, xf86-video-ati)? Bad news: I confirm this bug also on linux kernel 3.8: Last night I upgraded the server to Ubuntu 12.04.3 with following kernel and userspace stack versions: Kernel: 3.8.0 Mesa: 9.1.4 xf86-video-ati: compiled for 1.13.3, module version = 7.1.0 Same configuration with two running X servers, one stand-alone on vt9, one controlled by mdm on vt7. Firefox and xterm running. After about 1 h striped screen, turning dark afterwards. Netconsole message a), then kernel oops. We will stay for now with kernel 3.8. Any ideas how to solve this issue? In a first step it would be sufficient, if keyboard and mouse control could be retained, so that the magic keys allow for a smooth reboot. The frequently frozen kernels are a nightmare.
(In reply to comment #13) > (In reply to comment #12) > > (In reply to comment #11) > > > (In reply to comment #10) > > > > Is this still an issue with a newer kernel? Did you also update your > > > > userspace stack (mesa, xf86-video-ati)? > > Bad news: I confirm this bug also on linux kernel 3.8: > Again a kernel crash: [ 1825.192038] radeon 0000:01:05.0: GPU lockup CP stall for more than 10000msec [ 1825.192072] radeon 0000:01:05.0: GPU lockup (waiting for 0x0000000000004ac4) [ 1825.192100] radeon 0000:01:05.0: failed to get a new IB (-35) [ 1825.192108] [drm:radeon_cs_ib_chunk] *ERROR* Failed to get ib ! [ 1825.333066] radeon: wait for empty RBBM fifo failed ! Bad things might happen. [ 1825.473624] Failed to wait GUI idle while programming pipes. Bad things might happen. [ 1825.484893] radeon 0000:01:05.0: Saved 5403 dwords of commands on ring 0. [ 1825.498544] radeon 0000:01:05.0: (rs600_asic_reset:401) RBBM_STATUS=0x9401C100 [ 1825.996650] radeon 0000:01:05.0: (rs600_asic_reset:421) RBBM_STATUS=0x9401C100 [ 1826.493732] radeon 0000:01:05.0: (rs600_asic_reset:429) RBBM_STATUS=0x9400C100 [ 1826.990803] radeon 0000:01:05.0: (rs600_asic_reset:437) RBBM_STATUS=0x9400C100 [ 1826.990854] radeon 0000:01:05.0: failed to reset GPU [ 1826.999117] radeon 0000:01:05.0: GPU reset failed [ 1827.000816] radeon 0000:01:05.0: couldn't schedule ib [ 1827.000831] [drm:radeon_cs_ib_chunk] *ERROR* Failed to schedule IB !
(In reply to comment #10) > Is this still an issue with a newer kernel? Did you also update your > userspace stack (mesa, xf86-video-ati)? The bug seems to have nothing to do with the actual two X server configuration and the used window manager/desktop combination. Last night we started an Ubuntu 12.04.3 Live CD session with kernel 3.8, whose 700 MB disk space just provide a simple standard one X server desktop (Unity, compiz, gdm). We experienced a few short (about half a second) black screens (GPU resets?) and after about 10 min. a kernel freeze with striped screen. Unfortunately, the Ubuntu Live CD does not come with a netconsole, so we have no logs.
(In reply to comment #14) > (In reply to comment #13) > > (In reply to comment #12) > > > (In reply to comment #11) > > > > (In reply to comment #10) > > > > > Is this still an issue with a newer kernel? Did you also update your > > > > > userspace stack (mesa, xf86-video-ati)? > > > > Bad news: I confirm this bug also on linux kernel 3.8: > > Current situation with kernel 3.8 seems to be worse than with kernel 3.2. It's almost impossible to use the desktop. As soon as we work on Cinnamon (e. g. Firefox), the system freezes latest after about 5 min. On the other hand, the process between GPU lockup and final kernel oops has slightly changed. It takes longer time until the kernel freezes. In the following there are two more netconsole logs: a) Late Kernel Freeze: [80933.891948] radeon: wait for empty RBBM fifo failed ! Bad things might happen. [80934.031367] Failed to wait GUI idle while programming pipes. Bad things might happen. [80934.046827] radeon 0000:01:05.0: Saved 699 dwords of commands on ring 0. [80934.046985] radeon 0000:01:05.0: (rs600_asic_reset:401) RBBM_STATUS=0x9401C100 [80934.545149] radeon 0000:01:05.0: (rs600_asic_reset:421) RBBM_STATUS=0x9401C100 [80935.042219] radeon 0000:01:05.0: (rs600_asic_reset:429) RBBM_STATUS=0x9400C100 [80935.539290] radeon 0000:01:05.0: (rs600_asic_reset:437) RBBM_STATUS=0x9400C100 [80935.539341] radeon 0000:01:05.0: failed to reset GPU [80935.547414] radeon 0000:01:05.0: GPU reset failed [80935.547563] radeon 0000:01:05.0: couldn't schedule ib [80935.547569] [drm:radeon_cs_ib_chunk] *ERROR* Failed to schedule IB ! [80935.548226] radeon 0000:01:05.0: couldn't schedule ib [80935.548241] [drm:radeon_cs_ib_chunk] *ERROR* Failed to schedule IB ! [80935.556425] radeon 0000:01:05.0: couldn't schedule ib [80935.556442] [drm:radeon_cs_ib_chunk] *ERROR* Failed to schedule IB ! [80936.548449] radeon 0000:01:05.0: couldn't schedule ib ... [86038.776434] radeon 0000:01:05.0: couldn't schedule ib [86038.776464] [drm:radeon_cs_ib_chunk] *ERROR* Failed to schedule IB ! [86039.057988] radeon 0000:01:05.0: couldn't schedule ib [86039.058017] [drm:radeon_cs_ib_chunk] *ERROR* Failed to schedule IB ! [86039.780421] radeon 0000:01:05.0: couldn't schedule ib [86039.780450] [drm:radeon_cs_ib_chunk] *ERROR* Failed to schedule IB ! b) Magic Key Reboot before Actual Kernel Freeze [ 1875.636067] radeon 0000:01:05.0: GPU lockup CP stall for more than 10000msec [ 1875.636099] radeon 0000:01:05.0: GPU lockup (waiting for 0x0000000000001602 last fence id 0x0000000000001601) [ 1875.776768] radeon: wait for empty RBBM fifo failed ! Bad things might happen. [ 1875.917215] Failed to wait GUI idle while programming pipes. Bad things might happen. [ 1875.925493] radeon 0000:01:05.0: Saved 379 dwords of commands on ring 0. [ 1875.938200] radeon 0000:01:05.0: (rs600_asic_reset:401) BBM_STATUS=0x9011C100 [ 1876.436314] radeon 0000:01:05.0: (rs600_asic_reset:421) BBM_STATUS=0x90010140 [ 1876.933384] radeon 0000:01:05.0: (rs600_asic_reset:429) BBM_STATUS=0x10000140 [ 1877.430455] radeon 0000:01:05.0: (rs600_asic_reset:437) BBM_STATUS=0x10000140 [ 1877.430502] radeon 0000:01:05.0: GPU reset succeed [ 1877.431503] radeon 0000:01:05.0: GPU reset succeeded, trying to resume [ 1877.432008] [sched_delayed] sched: RT throttling activated [ 1877.541132] [drm] radeon: 1 quad pipes, 1 z pipes initialized. [ 1877.546610] [drm] PCIE GART of 512M enabled (table at 0x0000000036280000). [ 1877.546690] radeon 0000:01:05.0: WB enabled [ 1877.546697] radeon 0000:01:05.0: fence driver on ring 0 use gpu addr 0x0000000080000000 and cpu addr 0xffff880036274000 [ 1877.546723] [drm] radeon: ring at 0x0000000080001000 [ 1877.684422] [drm:r100_ring_test] *ERROR* radeon: ring test failed (scratch(0x15E8)=0xCAFEDEAD) [ 1877.684426] [drm:r100_cp_init] *ERROR* radeon: cp isn't working (-22). [ 1877.684430] radeon 0000:01:05.0: failed initializing CP (-22). [ 1953.942015] SysRq : set to system default [ 1954.772761] SysRq : Tasks [ 1954.817276] init: upstart-udev-bridge main process ended, respawning [ 1954.817698] init: upstart-socket-bridge main process ended, respawning ... [ 1955.395903] init: dbus main process ended, respawning [ 1955.402682] init: mysql main process ended, respawning [ 1955.444430] SysRq : ... [ 1955.614015] init: Disconnected from system bus [ 1955.756307] init: plymouth-upstart-bridge main process (8471) terminated with status 1 [ 1959.555121] SysRq : [ 1960.080595] Emergency Sync complete [ 1962.056342] SysRq : [ 1962.139938] EXT4-fs (sdb1): re-mounted. Opts: (null) [ 1962.192168] EXT4-fs (sda1): re-mounted. Opts: (null) [ 1962.219457] EXT4-fs (sda2): re-mounted. Opts: (null) [ 1962.232572] EXT4-fs (sdb10): re-mounted. Opts: (null) [ 1962.378780] EXT4-fs (sdb7): re-mounted. Opts: (null) [ 1962.384871] EXT4-fs (sdb8): re-mounted. Opts: (null) [ 1962.385193] EXT4-fs (sdb9): re-mounted. Opts: (null) [ 1962.386199] Emergency Remount complete [ 1965.248738] SysRq : Resetting
I would suggest trying newer versions of the userspace gfx stack (mesa, xf86-video-ati). GPU resets are usually caused by bad command combinations sent by the userspace drivers.
(In reply to comment #17) > I would suggest trying newer versions of the userspace gfx stack (mesa, > xf86-video-ati). GPU resets are usually caused by bad command combinations > sent by the userspace drivers. Status with following kernel and userspace stack versions: Kernel: 3.8.0 Mesa: 9.1.7 xf86-video-ati: compiled for 1.13.3, module version = 7.1.0 The system is more stable again. We had only one kernel freeze and this was preceeded by a failed GPU reset. One root cause of the previously increased instability seems to be that the second X server runs wine-1.4. Wine requires also the 32-bit i386 userspace drivers, which had to be manually upgraded to the above mentioned latest versions. Is the wine required combined use of 32-bit (i386) and 64-bit (amd64) userspace drivers on an AMD platform supposed to be valid and stable approach?
(In reply to comment #18) > > Is the wine required combined use of 32-bit (i386) and 64-bit (amd64) > userspace drivers on an AMD platform supposed to be valid and stable > approach? Yes, that should work fine.
(In reply to comment #17) > I would suggest trying newer versions of the userspace gfx stack (mesa, > xf86-video-ati). GPU resets are usually caused by bad command combinations > sent by the userspace drivers. This bug still exists. Yesterday, we had three kernel freezes within 1h. Netconsole output was pretty much the same as in comment #14. The server had to be restarted using power off switch. Current versions: Kernel: 3.8.0 Mesa: 9.1.7 xf86-video-ati: compiled for 1.13.3, module version = 7.1.0 These are the latest stable versions for Ubuntu 12.04LTS. We would not update the server to XorgOnTheEdge versions due to the high risk of broken X system. In addition to the many and various attachments of this bug notification, what information is required to identify the root cause of this painful bug? And how can we retrieve it having in mind a freezing system, which cannot write information on the disks?
You might try a newer or older version of mesa to see if you can indenify a stable version. Once you do, you can use git to bisect what change caused of fixed the problem.
-- GitLab Migration Automatic Message -- This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity. You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/drm/amd/issues/407.
Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.