Bug 104352

Summary: [IGT] gem_spin_batch@spin-each Failed assertion: !"GPU hung"
Product: DRI Reporter: Octavio <octaviox.hernandez.lopez>
Component: IGTAssignee: Default DRI bug account <dri-devel>
Status: CLOSED FIXED QA Contact:
Severity: normal    
Priority: medium CC: intel-gfx-bugs
Version: DRI git   
Hardware: x86-64 (AMD64)   
OS: Linux (All)   
Whiteboard: ReadyForDev
i915 platform: BXT, CFL, CNL, GLK i915 features:
Attachments:
Description Flags
kernel log
none
dmesg
none
error state
none
Kernel log (comment 7)
none
error (comment 7) (/sys/class/drm/card0/error)
none
CFL-Kern-Log none

Description Octavio 2017-12-20 15:03:43 UTC 
Created attachment 136322 [details]
kernel log

Outpu
========================================
	
IGT-Version: 1.20-gfda6c10 (x86_64) (Linux: 4.15.0-rc4-drm-intel-qa-ww51-commit-e044c9e+ x86_64)
Stack trace:
  #0 [__igt_fail_assert+0x101]
  #1 [sig_abort+0x3a]
  #2 [killpg+0x40]
  #3 [__wait+0x1e]
  #4 [igt_waitchildren+0x68]
  #5 [__real_main92+0x107]
  #6 [main+0x23]
  #7 [__libc_start_main+0xf1]
  #8 [_start+0x29]
  #9 [<unknown>+0x29]
Subtest spin-each: FAIL (19.359s)

(gem_spin_batch:3962) igt-aux-CRITICAL: Test assertion failure function sig_abort, file igt_aux.c:482:
(gem_spin_batch:3962) igt-aux-CRITICAL: Failed assertion: !"GPU hung"
Subtest spin-each failed.
**** DEBUG ****
(gem_spin_batch:3962) igt-aux-CRITICAL: Test assertion failure function sig_abort, file igt_aux.c:482:
(gem_spin_batch:3962) igt-aux-CRITICAL: Failed assertion: !"GPU hung"
(gem_spin_batch:3962) igt-core-INFO: Stack trace:
(gem_spin_batch:3962) igt-core-INFO:   #0 [__igt_fail_assert+0x101]
(gem_spin_batch:3962) igt-core-INFO:   #1 [sig_abort+0x3a]
(gem_spin_batch:3962) igt-core-INFO:   #2 [killpg+0x40]
(gem_spin_batch:3962) igt-core-INFO:   #3 [__wait+0x1e]
(gem_spin_batch:3962) igt-core-INFO:   #4 [igt_waitchildren+0x68]
(gem_spin_batch:3962) igt-core-INFO:   #5 [__real_main92+0x107]
(gem_spin_batch:3962) igt-core-INFO:   #6 [main+0x23]
(gem_spin_batch:3962) igt-core-INFO:   #7 [__libc_start_main+0xf1]
(gem_spin_batch:3962) igt-core-INFO:   #8 [_start+0x29]
(gem_spin_batch:3962) igt-core-INFO:   #9 [<unknown>+0x29]
****  END  ****
Comment 1 Octavio 2017-12-20 15:04:11 UTC 
Created attachment 136323 [details]
dmesg
Comment 2 Octavio 2017-12-20 15:05:06 UTC 
Created attachment 136324 [details]
error state
Comment 3 Octavio 2017-12-20 15:05:41 UTC 
======================================
        Graphic stack
======================================

======================================
             Software
======================================
kernel version              : 4.15.0-rc4-drm-intel-qa-ww51-commit-e044c9e+
hostname                    : GLK-1-GLKRVP1DDR405
architecture                : x86_64
os version                  : Ubuntu 16.10
os codename                 : yakkety
kernel driver               : i915
bios revision               : 69.31
bios release date           : 10/11/2017
ksc                         : 1.41
hardware acceleration       : disabled
swap partition              : enabled on (/dev/sda3)

======================================
        Graphic drivers
======================================
grep: /opt/X11R7/var/log/Xorg.0.log: No such file or directory
libdrm                      : 2.4.70
cairo                       : 1.14.6

======================================
             Hardware
======================================
platform                   : Geminilake
motherboard model          : Geminilake
motherboard id             : GLKRVP1DDR4(05)
form factor                : Hand Held
manufacturer               : IntelCorp.
cpu family                 : Other
cpu family id              : 6
cpu information            : Genuine Intel(R) CPU @ 1.10GHz
gpu card                   : Intel Corporation Device 3185 (rev 03) (prog-if 00 [VGA controller])
memory ram                 : 3.69 GB
max memory ram             : 16 GB
cpu thread                 : 2
cpu core                   : 2
cpu model                  : 122
cpu stepping               : 1
socket                     : Other
hard drive                 : 223GiB (240GB)
current cd clock frequency : 79200 kHz
maximum cd clock frequency : 316800 kHz
displays connected         : eDP-1

======================================
             Firmware
======================================
dmc fw loaded             : yes
dmc version               : 1.4
guc fw loaded             : fetch SUCCESS, load SUCCESS
guc version wanted        : wanted 10.56, found 10.56
guc version found         : wanted 10.56, found 10.56

======================================
             kernel parameters
======================================
quiet i915.alpha_support=1 drm.debug=0xe log_buf_len=4M i915.enable_guc=-1 resume=/dev/sda3 fastboot
Comment 4 Octavio 2017-12-21 21:47:28 UTC 
This test fails on BXT 

igt@gem_spin_batch@spin-each

using IGT-Version: 1.20-gbeb26d8 (x86_64) (Linux: 4.15.0-rc4-drm-intel-qa-ww51-commit-b480e79+ x86_64)

output
=============================
(gem_spin_batch:2125) igt-aux-CRITICAL: Test assertion failure function sig_abort, file igt_aux.c:482:
(gem_spin_batch:2125) igt-aux-CRITICAL: Failed assertion: !"GPU hung"
Subtest spin-each failed.
**** DEBUG ****
(gem_spin_batch:2125) igt-aux-CRITICAL: Test assertion failure function sig_abort, file igt_aux.c:482:
(gem_spin_batch:2125) igt-aux-CRITICAL: Failed assertion: !"GPU hung"
(gem_spin_batch:2125) igt-core-INFO: Stack trace:
(gem_spin_batch:2125) igt-core-INFO:   #0 [__igt_fail_assert+0x101]
(gem_spin_batch:2125) igt-core-INFO:   #1 [sig_abort+0x3a]
(gem_spin_batch:2125) igt-core-INFO:   #2 [killpg+0x40]
(gem_spin_batch:2125) igt-core-INFO:   #3 [__wait+0x1e]
(gem_spin_batch:2125) igt-core-INFO:   #4 [igt_waitchildren+0x68]
(gem_spin_batch:2125) igt-core-INFO:   #5 [__real_main92+0x107]
(gem_spin_batch:2125) igt-core-INFO:   #6 [main+0x23]
(gem_spin_batch:2125) igt-core-INFO:   #7 [__libc_start_main+0xf1]
(gem_spin_batch:2125) igt-core-INFO:   #8 [_start+0x29]
(gem_spin_batch:2125) igt-core-INFO:   #9 [<unknown>+0x29]
****  END  ****
Comment 5 Octavio 2018-01-10 22:31:39 UTC 
This test still fails on GLK 

using 	
IGT-Version: 1.20-g19c6c04 (x86_64) (Linux: 4.15.0-rc7-drm-intel-qa-ww2-commit-17fd162+ x86_64)

(gem_spin_batch:1822) igt-aux-CRITICAL: Test assertion failure function sig_abort, file igt_aux.c:482:
(gem_spin_batch:1822) igt-aux-CRITICAL: Failed assertion: !"GPU hung"
(gem_spin_batch:1824) CRITICAL: Test assertion failure function spin, file gem_spin_batch.c:66:
(gem_spin_batch:1824) CRITICAL: Failed assertion: 100 * timeout_100ms * loops <= (100 + 5) * elapsed && 100 * timeout_100ms * loops >= (100 - 5) * elapsed
(gem_spin_batch:1824) CRITICAL: 'timeout_100ms * loops' != 'elapsed' (100000000 not within 5% tolerance of 17951977578)
Subtest spin-each failed.
Comment 6 Octavio 2018-01-15 22:33:50 UTC 
This test still fails on GLK 

igt@gem_spin_batch@spin-each

Configuration
=============================================
IGT-Version: 1.20-g84a3080 (x86_64) (Linux: 4.15.0-rc8-drm-intel-qa-ww3-commit-37eb04c+ x86_64)

Component: drm
    tag: libdrm-2.4.89-13-g57d3d4c
    commit: 57d3d4c96887cde016fe1d2766196accf77ca423

Component: cairo
    tag: 1.15.8-76-g6b05938
    commit: b7f313a8d2a3049e77b3497dd6040fcfab3b3c9b

Component: intel-gpu-tools
    tag: intel-gpu-tools-1.20-281-g84a3080
    commit: 84a308022028a55903a1916fcee516aab768ed48

Component: piglit
    tag: piglit-v1
    commit: caf60eece2d899cbfa5e92791326506630363ea4

Output 
=========================================
(gem_spin_batch:2774) igt-aux-CRITICAL: Test assertion failure function sig_abort, file igt_aux.c:482:
(gem_spin_batch:2774) igt-aux-CRITICAL: Failed assertion: !"GPU hung"
Subtest spin-each failed.
**** DEBUG ****
(gem_spin_batch:2774) igt-aux-CRITICAL: Test assertion failure function sig_abort, file igt_aux.c:482:
(gem_spin_batch:2774) igt-aux-CRITICAL: Failed assertion: !"GPU hung"
(gem_spin_batch:2774) igt-core-INFO: Stack trace:
(gem_spin_batch:2774) igt-core-INFO:   #0 [__igt_fail_assert+0x101]
(gem_spin_batch:2774) igt-core-INFO:   #1 [sig_abort+0x3a]
(gem_spin_batch:2774) igt-core-INFO:   #2 [killpg+0x40]
(gem_spin_batch:2774) igt-core-INFO:   #3 [__wait+0x1e]
(gem_spin_batch:2774) igt-core-INFO:   #4 [igt_waitchildren+0x61]
(gem_spin_batch:2774) igt-core-INFO:   #5 [__real_main92+0x11f]
(gem_spin_batch:2774) igt-core-INFO:   #6 [main+0x27]
(gem_spin_batch:2774) igt-core-INFO:   #7 [__libc_start_main+0xf1]
(gem_spin_batch:2774) igt-core-INFO:   #8 [_start+0x2a]
(gem_spin_batch:2774) igt-core-INFO:   #9 [<unknown>+0x2a]
****  END  ****
Comment 7 Hector Velazquez 2018-01-16 15:30:01 UTC 
This test has failed on CFL QA 

Tests List:

igt@gem_spin_batch@spin-each

using IGT-Version: 1.20-g84a3080 (x86_64) (Linux: 4.15.0-rc8-drm-tip-ww3-commit-37eb04c+ x86_64)

some tests has this output...
======================================
        output sample
======================================
. . .
**** DEBUG ****
(gem_spin_batch:3804) igt-aux-CRITICAL: Test assertion failure function sig_abort, file igt_aux.c:482:
(gem_spin_batch:3804) igt-aux-CRITICAL: Failed assertion: !"GPU hung"
(gem_spin_batch:3804) igt-core-INFO: Stack trace:
(gem_spin_batch:3804) igt-core-INFO:   #0 [__igt_fail_assert+0x101]
(gem_spin_batch:3804) igt-core-INFO:   #1 [sig_abort+0x3a]
(gem_spin_batch:3804) igt-core-INFO:   #2 [killpg+0x40]
(gem_spin_batch:3804) igt-core-INFO:   #3 [__wait+0x1e]
(gem_spin_batch:3804) igt-core-INFO:   #4 [igt_waitchildren+0x61]
(gem_spin_batch:3804) igt-core-INFO:   #5 [__real_main92+0x11f]
(gem_spin_batch:3804) igt-core-INFO:   #6 [main+0x27]
(gem_spin_batch:3804) igt-core-INFO:   #7 [__libc_start_main+0xf1]
(gem_spin_batch:3804) igt-core-INFO:   #8 [_start+0x2a]
(gem_spin_batch:3804) igt-core-INFO:   #9 [<unknown>+0x2a]
****  END  ****
. . .
Comment 8 Hector Velazquez 2018-01-16 15:30:38 UTC 
Created attachment 136771 [details]
Kernel log (comment 7)
Comment 9 Hector Velazquez 2018-01-16 15:31:17 UTC 
Created attachment 136772 [details]
error (comment 7) (/sys/class/drm/card0/error)
Comment 10 Ricardo Perez 2018-01-29 20:58:28 UTC 
Created attachment 137036 [details]
CFL-Kern-Log

CoffeeLake S UDIMM RVP Kernel Log file
Comment 11 Hector Velazquez 2018-02-12 14:07:29 UTC 
This test has failed on GLK QA 

Tests List:

igt@gem_spin_batch@spin-each

IGT-Version: 1.21-g94bd67c (x86_64) (Linux: 4.15.0-drm-tip-ww6-commit-078873d+ x86_64)

======================================
        output sample
======================================
. . .
**** DEBUG ****
(gem_spin_batch:1511) igt-aux-CRITICAL: Test assertion failure function sig_abort, file igt_aux.c:482:
(gem_spin_batch:1511) igt-aux-CRITICAL: Failed assertion: !"GPU hung"
(gem_spin_batch:1511) igt-core-INFO: Stack trace:
(gem_spin_batch:1511) igt-core-INFO:   #0 [__igt_fail_assert+0x101]
(gem_spin_batch:1511) igt-core-INFO:   #1 [sig_abort+0x3a]
(gem_spin_batch:1511) igt-core-INFO:   #2 [killpg+0x40]
(gem_spin_batch:1511) igt-core-INFO:   #3 [__wait+0x1e]
(gem_spin_batch:1511) igt-core-INFO:   #4 [igt_waitchildren+0x68]
(gem_spin_batch:1511) igt-core-INFO:   #5 [__real_main92+0x107]
(gem_spin_batch:1511) igt-core-INFO:   #6 [main+0x23]
(gem_spin_batch:1511) igt-core-INFO:   #7 [__libc_start_main+0xf0]
(gem_spin_batch:1511) igt-core-INFO:   #8 [_start+0x29]
(gem_spin_batch:1511) igt-core-INFO:   #9 [<unknown>+0x29]
****  END  ****
. . .
Comment 12 Jani Saarinen 2018-02-28 16:00:53 UTC 
Reference: https://patchwork.freedesktop.org/patch/207375/
Comment 13 Rodrigo Vivi 2018-02-28 22:51:15 UTC 
This works for me... got fixed recently?
Comment 14 Chris Wilson 2018-03-05 10:02:29 UTC 
commit c2af514a25ac6e94bc3dabf26cc7e755ce9fd2d6 (upstream/master)
Author: Chris Wilson <chris@chris-wilson.co.uk>
Date:   Wed Feb 21 14:42:12 2018 +0000

    igt/gem_spin_batch: Avoid waiting when running concurrently
    
    If we do a global wait while trying to execute spinners in parallel,
    it ends badly with a GPU hang.
    
    Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=104352
    Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>

Reviewed-by: Michał Winiarski <michal.winiarski@intel.com>
Comment 15 Lakshmi 2018-10-11 19:49:38 UTC 
Closing this bug as it was resolved/fixed.

Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.