Bug 106070

Summary: [CI][CFL only] igt@* - incomplete - timeout/system hang
Product: DRI Reporter: Marta Löfstedt <marta.lofstedt>
Component: DRM/IntelAssignee: Intel GFX Bugs mailing list <intel-gfx-bugs>
Status: RESOLVED INVALID QA Contact: Intel GFX Bugs mailing list <intel-gfx-bugs>
Severity: normal    
Priority: high CC: intel-gfx-bugs, jose.souza
Version: DRI git   
Hardware: Other   
OS: All   
Whiteboard: ReadyForDev
i915 platform: CFL i915 features: display/FBC
Bug Depends on:    
Bug Blocks: 105984    

Description Marta Löfstedt 2018-04-16 08:04:35 UTC
https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_21/fi-cfl-8700k/igt@gem_render_copy_redux@normal.html

run.log
running: igt/gem_render_copy_redux/normal

[63/75] skip: 23, pass: 40 \             
FATAL: command execution failed
...
Completed CI_IGT_test drmtip_21/fi-cfl-8700k/34 : FAILURE
CI_IGT_test runtime 543 seconds
Rebooting fi-cfl-8700k

Last dmesg:
<6>[  287.827513] Console: switching to colour frame buffer device 240x67
<6>[  287.966869] Console: switching to colour dummy device 80x25
<7>[  287.966905] [IGT] gem_render_copy_redux: executing
Comment 7 steven.j.hockemeier 2018-06-18 17:11:47 UTC
Next Steps?
Comment 8 James Ausmus 2018-06-18 17:42:02 UTC
How often does this issue occur?
Comment 9 Francesco Balestrieri 2018-06-19 16:31:27 UTC
According to CI buglogger, this was seen on 1 / 333 runs (0.3%). I wonder if the priority is correct.
Comment 10 Rodrigo Vivi 2018-06-19 23:15:24 UTC
The logs here has nothing in common. I wonder if this part is stable and reliable at all...
- Is it a production stepping?
- BIOS updated?
- LCK team contacted?
Comment 11 Martin Peres 2018-07-18 14:32:24 UTC
Random incompletes are also found on fi-cfl-8109u:

https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_82/fi-cfl-8109u/igt@kms_cursor_crc@cursor-128x128-suspend.html

https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_75/fi-cfl-8109u/igt@kms_flip@nonexisting-fb-interruptible.html

https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_76/fi-cfl-8109u/igt@kms_cursor_legacy@cursorb-vs-flipb-atomic-transitions-varying-size.html

https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_76/fi-cfl-8109u/igt@pm_rpm@gem-idle.html

https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_76/fi-cfl-8109u/igt@kms_frontbuffer_tracking@fbc-1p-primscrn-pri-indfb-draw-render.html

https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_76/fi-cfl-8109u/igt@kms_plane@plane-panning-top-left-pipe-c-planes.html

https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_85/fi-cfl-8109u/igt@gem_softpin@noreloc-s3.html

https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_82/fi-cfl-8109u/igt@kms_frontbuffer_tracking@fbc-2p-scndscrn-cur-indfb-move.html

https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_74/fi-cfl-8109u/igt@kms_busy@basic-flip-b.html

https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_87/fi-cfl-8109u/igt@kms_frontbuffer_tracking@fbc-1p-primscrn-spr-indfb-draw-pwrite.html

https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_78/fi-cfl-8109u/igt@kms_flip@blocking-absolute-wf_vblank.html

https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_87/fi-cfl-8109u/igt@drv_missed_irq.html

https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_74/fi-cfl-8109u/igt@kms_atomic_transition@plane-all-modeset-transition.html

https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_79/fi-cfl-8109u/igt@kms_vblank@pipe-c-ts-continuation-dpms-suspend.html

https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_4494/fi-cfl-8109u/igt@kms_pipe_crc_basic@suspend-read-crc-pipe-a.html

https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_83/fi-cfl-8109u/igt@drv_suspend@forcewake.html

https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_80/fi-cfl-8109u/igt@kms_frontbuffer_tracking@fbc-1p-primscrn-indfb-msflip-blt.html

https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_86/fi-cfl-8109u/igt@kms_color@pipe-a-ctm-0-25.html

https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_82/fi-cfl-8109u/igt@kms_plane@plane-panning-bottom-right-suspend-pipe-a-planes.html

https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_82/fi-cfl-8109u/igt@kms_plane@plane-panning-bottom-right-pipe-b-planes.html

https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4548/fi-cfl-8109u/igt@kms_pipe_crc_basic@suspend-read-crc-pipe-b.html

https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_82/fi-cfl-8109u/igt@kms_color@pipe-c-ctm-0-75.html


Theses ones have a pstore I can't get anything out of (but I'm sure it will look clear as day to Chris):

https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_75/fi-cfl-8109u/igt@kms_cursor_crc@cursor-256x256-suspend.html

https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_76/fi-cfl-8109u/igt@testdisplay.html

https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_4494/fi-cfl-8109u/igt@kms_pipe_crc_basic@suspend-read-crc-pipe-a.html
Comment 16 Martin Peres 2018-07-25 15:31:32 UTC
Seems like fi-cfl-s3 had a problem that was only visible on drmtip_80:

https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_80/fi-cfl-s3/igt@kms_frontbuffer_tracking@psr-rgb101010-draw-render.html

https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_80/fi-cfl-s3/igt@kms_frontbuffer_tracking@psr-1p-primscrn-cur-indfb-draw-render.html

https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_80/fi-cfl-s3/igt@kms_frontbuffer_tracking@fbcpsr-1p-primscrn-cur-indfb-draw-render.html

https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_80/fi-cfl-s3/igt@kms_draw_crc@draw-method-xrgb2101010-render-ytiled.html

Will monitor to see if this can be reproduced, but so far, no reproduction in any of the following runs.

There was however what appears to be an unrelated one here with some information:

https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_82/fi-cfl-s3/igt@kms_vblank@pipe-c-query-forked-busy-hang.html

<4>[  184.075646] ------------[ cut here ]------------
<4>[  184.075646] pipe_off wait timed out
<4>[  184.075678] WARNING: CPU: 2 PID: 1467 at drivers/gpu/drm/i915/intel_display.c:1098 intel_disable_pipe+0x131/0x170 [i915]
<4>[  184.075679] Modules linked in: vgem i915 cdc_ether usbnet r8152 mii x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel mei_me mei e1000e prime_numbers
<4>[  184.075687] CPU: 2 PID: 1467 Comm: kms_vblank Tainted: G     UD W         4.18.0-rc4-g02e578b7aace-drmtip_82+ #1
<4>[  184.075688] Hardware name: Intel Corporation CoffeeLake Client Platform/CoffeeLake S UDIMM RVP, BIOS CNLSFWR1.R00.X118.B19.1802080131 02/08/2018
<4>[  184.075709] RIP: 0010:intel_disable_pipe+0x131/0x170 [i915]
<4>[  184.075709] Code: 6a 00 8d b4 0a 08 00 07 00 31 c9 ba 00 00 00 40 e8 c4 2d fe ff 85 c0 5a 0f 84 55 ff ff ff 48 c7 c7 2a 1b 34 c0 e8 1f 05 e0 e6 <0f> 0b e9 42 ff ff ff 85 d2 0f 84 6c ff ff ff 48 83 c4 08 89 ee 48 
<4>[  184.075728] RSP: 0018:ffffaed400f8fab0 EFLAGS: 00010286
<4>[  184.075729] RAX: 0000000000000000 RBX: ffff89857f1a0000 RCX: 0000000000000001
<4>[  184.075730] RDX: 0000000080000001 RSI: ffffffffa8084e66 RDI: 00000000ffffffff
<4>[  184.075730] RBP: 000000000007f008 R08: 0000000000000000 R09: 0000000000000000
<4>[  184.075731] R10: ffffaed400f8f988 R11: ffffffffa8244e98 R12: 0000000000000003
<4>[  184.075731] R13: ffff89857cdf2548 R14: ffff89857cdf2e98 R15: ffff89857f1a0000
<4>[  184.075732] FS:  00007f82ed970980(0000) GS:ffff89858d280000(0000) knlGS:0000000000000000
<4>[  184.075733] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
<4>[  184.075733] CR2: 0000558218414b68 CR3: 00000004824c2006 CR4: 00000000003606e0
<4>[  184.075734] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<4>[  184.075734] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
<4>[  184.075735] Call Trace:
<4>[  184.075757]  haswell_crtc_disable+0xd9/0x140 [i915]
<4>[  184.075778]  intel_atomic_commit_tail+0x79f/0xd00 [i915]
<4>[  184.075800]  intel_atomic_commit+0x240/0x320 [i915]
<4>[  184.075803]  restore_fbdev_mode_atomic+0x1d3/0x1f0
<4>[  184.075807]  drm_fb_helper_restore_fbdev_mode_unlocked+0x42/0x90
<4>[  184.075808]  drm_fb_helper_set_par+0x24/0x50
<4>[  184.075829]  intel_fbdev_set_par+0x11/0x40 [i915]
<4>[  184.075831]  fbcon_init+0x4db/0x680
<4>[  184.075834]  visual_init+0xd0/0x130
<4>[  184.075835]  do_bind_con_driver+0x1ef/0x3f0
<4>[  184.075837]  store_bind+0x133/0x1a0
<4>[  184.075839]  kernfs_fop_write+0x104/0x190
<4>[  184.075842]  __vfs_write+0x31/0x180
<4>[  184.075844]  ? rcu_sync_lockdep_assert+0x9/0x50
<4>[  184.075845]  ? __sb_start_write+0x152/0x1f0
<4>[  184.075846]  ? __sb_start_write+0x168/0x1f0
<4>[  184.075848]  vfs_write+0xbd/0x1a0
<4>[  184.075849]  ksys_write+0x50/0xc0
<4>[  184.075851]  do_syscall_64+0x55/0x190
<4>[  184.075853]  entry_SYSCALL_64_after_hwframe+0x49/0xbe
<4>[  184.075854] RIP: 0033:0x7f82eceea281
<4>[  184.075854] Code: c3 0f 1f 84 00 00 00 00 00 48 8b 05 59 8d 20 00 c3 0f 1f 84 00 00 00 00 00 8b 05 8a d1 20 00 85 c0 75 16 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 57 f3 c3 0f 1f 44 00 00 41 54 55 49 89 d4 53 
<4>[  184.075873] RSP: 002b:00007ffcc45e3958 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
<4>[  184.075874] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f82eceea281
<4>[  184.075875] RDX: 0000000000000002 RSI: 00007f82ed569a21 RDI: 0000000000000005
<4>[  184.075875] RBP: 00007ffcc45e49a0 R08: 000055821840cbc3 R09: 0000000000000006
<4>[  184.075876] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f82eced3718
<4>[  184.075876] R13: 0000000000000003 R14: 00007f82eced8628 R15: 00007f82eced4d80
<4>[  184.075878] irq event stamp: 94934
<4>[  184.075880] hardirqs last  enabled at (94933): [<ffffffffa795719c>] _raw_spin_unlock_irqrestore+0x4c/0x60
<4>[  184.075881] hardirqs last disabled at (94934): [<ffffffffa794fcfa>] __schedule+0xaa/0xbf0
<4>[  184.075882] softirqs last  enabled at (94762): [<ffffffffa7c0034f>] __do_softirq+0x34f/0x505
<4>[  184.075884] softirqs last disabled at (94755): [<ffffffffa7090539>] irq_exit+0xa9/0xc0
<4>[  184.075905] WARNING: CPU: 2 PID: 1467 at drivers/gpu/drm/i915/intel_display.c:1098 intel_disable_pipe+0x131/0x170 [i915]
<4>[  184.075906] ---[ end trace 090efcfc132b8e1d ]---
Comment 19 Martin Peres 2018-09-03 12:20:44 UTC
https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_94/fi-cfl-s3/igt@gem_cpu_reloc@full.html

We have pstores, but they only contain the ftrace :s
Comment 26 Martin Peres 2018-11-06 09:44:05 UTC
https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_138/fi-cfl-8109u/igt@kms_frontbuffer_tracking@fbc-2p-primscrn-cur-indfb-draw-render.html

Starting subtest: fbc-2p-primscrn-cur-indfb-draw-render
(kms_frontbuffer_tracking:1311) igt_debugfs-CRITICAL: Test assertion failure function igt_assert_crc_equal, file ../lib/igt_debugfs.c:419:
(kms_frontbuffer_tracking:1311) igt_debugfs-CRITICAL: Failed assertion: !mismatch

(Funny that it died after that).
Comment 27 CI Bug Log 2019-01-04 14:04:26 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-cfl-8109u: random tests - incomplete -}
{+ fi-cfl-8109u: random tests - incomplete +}

New failures caught by the filter:

* https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_174/fi-cfl-8109u/igt@kms_frontbuffer_tracking@fbc-1p-rte.html
Comment 28 CI Bug Log 2019-01-29 10:08:16 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-cfl-8109u: random tests - incomplete -}
{+ fi-cfl-8109u: random tests - incomplete +}

New failures caught by the filter:

* https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_184/fi-cfl-8109u/igt@kms_chv_cursor_fail@pipe-b-64x64-bottom-edge.html
Comment 29 CI Bug Log 2019-02-12 08:36:39 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-whl-u: random tests - incomplete -}
{+ fi-whl-u: random tests - incomplete +}

New failures caught by the filter:

* https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_204/fi-whl-u/igt@kms_rotation_crc@multiplane-rotation-cropping-top.html
Comment 30 CI Bug Log 2019-03-20 12:08:07 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-cfl-8700k: random tests - incomplete -}
{+ fi-cfl-8700k: random tests - incomplete +}

New failures caught by the filter:

* https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_247/fi-cfl-8700k/igt@gem_create@create-clear.html
Comment 31 CI Bug Log 2019-04-26 07:19:32 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-cfl-8109u: random tests - incomplete -}
{+ fi-cfl-8109u: random tests - incomplete +}

New failures caught by the filter:

  * https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_263/fi-cfl-8109u/igt@kms_plane_alpha_blend@pipe-b-constant-alpha-max.html
  * https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_264/fi-cfl-8109u/igt@kms_addfb_basic@master-rmfb.html
Comment 32 CI Bug Log 2019-04-29 11:31:12 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-whl-u: random tests - incomplete -}
{+ fi-whl-u: random tests - incomplete +}

New failures caught by the filter:

  * https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_266/fi-whl-u/igt@kms_frontbuffer_tracking@fbc-1p-primscrn-pri-shrfb-draw-mmap-cpu.html
Comment 33 CI Bug Log 2019-04-29 11:31:28 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-whl-u: random tests - incomplete -}
{+ fi-whl-u: random tests - incomplete +}

New failures caught by the filter:

  * https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_266/fi-whl-u/igt@perf_pmu@busy-idle-no-semaphores-rcs0.html
Comment 34 CI Bug Log 2019-04-29 15:29:00 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-cfl-8109u: random tests - incomplete -}
{+ fi-cfl-8109u: random tests - incomplete +}

New failures caught by the filter:

  * https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_266/fi-cfl-8109u/igt@kms_frontbuffer_tracking@fbcpsr-1p-primscrn-pri-indfb-draw-mmap-cpu.html
Comment 35 CI Bug Log 2019-04-29 15:33:36 UTC
The CI Bug Log issue associated to this bug has been updated.

### New filters associated

* fi-cfl-guc: Random tests -  incomplete
  - https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_266/fi-cfl-guc/igt@kms_frontbuffer_tracking@fbc-1p-primscrn-pri-shrfb-draw-mmap-cpu.html
Comment 36 CI Bug Log 2019-04-30 07:16:10 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-whl-u: random tests - incomplete -}
{+ fi-whl-u: random tests - incomplete +}

New failures caught by the filter:

  * https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_269/fi-whl-u/igt@kms_frontbuffer_tracking@fbc-rgb101010-draw-pwrite.html
Comment 37 CI Bug Log 2019-04-30 10:49:05 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-cfl-8109u: random tests - incomplete -}
{+ fi-cfl-8109u: random tests - incomplete +}


  No new failures caught with the new filter
Comment 39 CI Bug Log 2019-05-26 19:18:50 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-whl-u: random tests - incomplete -}
{+ fi-whl-u: random tests - incomplete +}

New failures caught by the filter:

  * https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_293/fi-whl-u/igt@gem_tiled_swapping@non-threaded.html
Comment 40 CI Bug Log 2019-06-12 09:40:35 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-cfl-8109u: random tests - incomplete -}
{+ fi-cfl-8109u: random tests - incomplete +}

New failures caught by the filter:

  * https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_304/fi-cfl-8109u/igt@gem_linear_blits@interruptible.html
Comment 41 CI Bug Log 2019-06-18 06:33:48 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-whl-u: random tests - incomplete -}
{+ fi-whl-u: random tests - incomplete +}

New failures caught by the filter:

  * https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_308/fi-whl-u/igt@kms_flip@dpms-off-confusion.html
Comment 42 CI Bug Log 2019-07-01 08:27:11 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-cfl-8109u: random tests - incomplete -}
{+ fi-cfl-8109u: random tests - incomplete +}

New failures caught by the filter:

  * https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_319/fi-cfl-8109u/igt@gem_mmap_gtt@big-copy.html
Comment 43 CI Bug Log 2019-07-24 15:26:12 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-whl-u: random tests - incomplete -}
{+ fi-whl-u: random tests - incomplete +}

New failures caught by the filter:

  * https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_333/fi-whl-u/igt@gem_exec_suspend@basic-s3-devices.html
Comment 44 CI Bug Log 2019-07-24 15:26:46 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-whl-u: random tests - incomplete -}
{+ fi-whl-u: random tests - incomplete +}

New failures caught by the filter:

  * https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_333/fi-whl-u/igt@gem_eio@suspend.html
  * https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_333/fi-whl-u/igt@i915_pm_rpm@system-suspend-devices.html
Comment 45 CI Bug Log 2019-07-24 16:08:25 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-cfl-8109u: random tests - incomplete -}
{+ fi-cfl-8109u: random tests - incomplete +}

New failures caught by the filter:

  * https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_333/fi-cfl-8109u/igt@kms_flip@2x-flip-vs-suspend-interruptible.html
Comment 46 CI Bug Log 2019-08-07 14:48:43 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-cfl-8109u: random tests - incomplete -}
{+ fi-cfl-8109u: random tests - incomplete +}

New failures caught by the filter:

  * https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_337/fi-cfl-8109u/igt@kms_flip@flip-vs-suspend.html
Comment 47 Chris Wilson 2019-08-09 07:10:57 UTC
Very specific bugs are being lost here, e.g. the last is a resume failure that would be easy to debug in situ.
Comment 48 CI Bug Log 2019-09-13 16:05:57 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-cfl-guc: Random tests -  incomplete -}
{+ fi-cfl-guc: Random tests -  incomplete +}

New failures caught by the filter:

  * https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6886/fi-cfl-guc/igt@i915_selftest@live_gem_contexts.html

Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.