Bug 109632 - [CI][SHARDS] Logs are unavailable for some of the CI tests failures
Summary: [CI][SHARDS] Logs are unavailable for some of the CI tests failures
Status: CLOSED WORKSFORME
Alias: None
Product: DRI
Classification: Unclassified
Component: DRM/Intel (show other bugs)
Version: DRI git
Hardware: Other All
: high normal
Assignee: Intel GFX Bugs mailing list
QA Contact: Intel GFX Bugs mailing list
URL:
Whiteboard: ReadyForDev
Keywords:
Depends on:
Blocks:
 
Reported: 2019-02-14 16:35 UTC by Lakshmi
Modified: 2019-06-12 07:28 UTC (History)
2 users (show)

See Also:
i915 platform: KBL
i915 features: CI Infra


Attachments

Comment 2 CI Bug Log 2019-02-20 13:15:08 UTC
A CI Bug Log filter associated to this bug has been updated:

{- KBL: Random tests - incomplete - No Logs -}
{+ KBL: Random tests - incomplete - No Logs +}

New failures caught by the filter:

* https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_221/fi-icl-u3/igt@kms_cursor_legacy@cursorb-vs-flipb-atomic-transitions-varying-size.html
Comment 3 CI Bug Log 2019-02-20 13:15:30 UTC
A CI Bug Log filter associated to this bug has been updated:

{- KBL: Random tests - incomplete - No Logs -}
{+ KBL ICL: Random tests - incomplete - No Logs +}

 No new failures caught with the new filter
Comment 4 CI Bug Log 2019-02-21 12:39:03 UTC
A CI Bug Log filter associated to this bug has been updated:

{- KBL ICL: Random tests - incomplete - No Logs -}
{+ KBL ICL: Random tests - incomplete - No Logs +}

New failures caught by the filter:

* https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_221/fi-kbl-guc/igt@gem_ctx_sseu@engines.html
Comment 5 CI Bug Log 2019-02-25 09:21:07 UTC
A CI Bug Log filter associated to this bug has been updated:

{- KBL ICL: Random tests - incomplete - No Logs -}
{+ KBL ICL: Random tests - incomplete - No Logs +}

New failures caught by the filter:

* https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_228/fi-icl-u3/igt@kms_chv_cursor_fail@pipe-a-256x256-left-edge.html
Comment 6 CI Bug Log 2019-03-04 09:44:22 UTC
A CI Bug Log filter associated to this bug has been updated:

{- KBL ICL: Random tests - incomplete - No Logs -}
{+ KBL ICL: Random tests - incomplete - No Logs +}

New failures caught by the filter:

* https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_234/fi-icl-u3/igt@kms_frontbuffer_tracking@fbcpsr-2p-primscrn-shrfb-pgflip-blt.html
Comment 7 CI Bug Log 2019-03-13 09:51:21 UTC
A CI Bug Log filter associated to this bug has been updated:

{- KBL ICL: Random tests - incomplete - No Logs -}
{+ KBL ICL: Random tests - incomplete - No Logs +}

New failures caught by the filter:

* https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_242/fi-icl-u3/igt@kms_flip@2x-flip-vs-absolute-wf_vblank-interruptible.html
Comment 8 CI Bug Log 2019-03-13 09:56:11 UTC
A CI Bug Log filter associated to this bug has been updated:

{- KBL ICL: Random tests - incomplete - No Logs -}
{+ KBL ICL: Random tests - incomplete - No Logs +}

New failures caught by the filter:

* https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_240/fi-icl-u3/igt@perf_pmu@other-read-2.html
Comment 9 CI Bug Log 2019-03-13 09:56:29 UTC
A CI Bug Log filter associated to this bug has been updated:

{- KBL ICL: Random tests - incomplete - No Logs -}
{+ KBL ICL: Random tests - incomplete - No Logs +}

 No new failures caught with the new filter
Comment 10 CI Bug Log 2019-03-19 10:57:12 UTC
A CI Bug Log filter associated to this bug has been updated:

{- KBL ICL: Random tests - incomplete - No Logs -}
{+ KBL ICL: Random tests - incomplete - No Logs +}

New failures caught by the filter:

* https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_245/fi-icl-u3/igt@kms_frontbuffer_tracking@psr-2p-scndscrn-cur-indfb-draw-pwrite.html
* https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_246/fi-icl-u3/igt@gem_mmap_wc@write-cpu-read-wc-unflushed.html
* https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_246/fi-icl-u3/igt@gem_pwrite_pread@snooped-copy-correctness.html
Comment 11 CI Bug Log 2019-03-19 11:01:48 UTC
A CI Bug Log filter associated to this bug has been updated:

{- KBL ICL: Random tests - incomplete - No Logs -}
{+ KBL ICL: Random tests - incomplete - No Logs +}

New failures caught by the filter:

* https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_244/fi-icl-u3/igt@perf_pmu@rc6.html
Comment 12 CI Bug Log 2019-03-19 11:02:48 UTC
A CI Bug Log filter associated to this bug has been updated:

{- KBL ICL: Random tests - incomplete - No Logs -}
{+ KBL ICL: Random tests - incomplete - No Logs +}

New failures caught by the filter:

* https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_243/fi-icl-u3/igt@kms_draw_crc@draw-method-xrgb8888-mmap-cpu-untiled.html
Comment 13 CI Bug Log 2019-03-25 09:53:16 UTC
A CI Bug Log filter associated to this bug has been updated:

{- KBL ICL: Random tests - incomplete - No Logs -}
{+ KBL ICL: Random tests - incomplete - No Logs +}

New failures caught by the filter:

* https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4899/shard-iclb8/igt@prime_nv_test@nv_write_i915_cpu_mmap_read.html
Comment 14 CI Bug Log 2019-03-28 15:20:20 UTC
A CI Bug Log filter associated to this bug has been updated:

{- KBL ICL: Random tests - incomplete - No Logs -}
{+ KBL ICL: Random tests - incomplete - No Logs +}

New failures caught by the filter:

* https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4905/shard-iclb5/igt@kms_setmode@basic.html
Comment 15 Martin Peres 2019-04-23 12:12:53 UTC
Bumping to high since it affects the shard machines too!
Comment 16 Arek Hiler 2019-06-12 05:56:06 UTC
Seems like we have got better with those :-)

It used to be seen few times a week, but now we had 2 months break and it happened again just yesterday.

https://intel-gfx-ci.01.org/tree/drm-tip/IGT_5050/shard-iclb1/igt@kms_frontbuffer_tracking@psr-1p-offscren-pri-shrfb-draw-mmap-wc.html

From the dmesg log for that run:
<7>[   28.594271] [IGT] gem_render_copy_redux: executing
<7>[   28.602910] [IGT] gem_render_copy_redux: starting subtest flink
<7>[   29.111011] [IGT] gem_render_copy_redux: exiting, ret=0
<5>[   29.111169] Setting dangerous option reset - tainting kernel
<6>[   29.170487] Console: switching to colour frame buffer device 240x67
<2>[   29.192629] watchdog: watchdog0: watchdog did not stop!

Runner just created and fsynced journal/out/err/dmesg files for psr-1p-offscreen-pri-shrfb-draw-mmap-wc, but all the files are empty. Judging from the dmesg, the test has not even started yet and the watchdog kicked in.

So it seems like it's this one: https://bugs.freedesktop.org/show_bug.cgi?id=110246

@Lakshmi, is there is anything special about this one or can we deduplicate?

I think we should check the logic that is handling watchdog handling/pinging on a test boundaries.

As of user impact - seems like we have some issues either with watchdog handling or watchdogs themselves. From CI's POV it's important to fix this, but it does not affect people using i915 directly.
Comment 17 Arek Hiler 2019-06-12 06:43:57 UTC
After short chat with Lakshmi it seems like the reason for "duplicating" is capturing incompletes that does not have a result visualization page generated for them. 

Since then there were some vis and igt_runner changes and the most recent instance does have visualisation. I think it's good to de-duplicate those and if we will have another instance with no visualisation we should investigate what the tools do not like about it.
Comment 18 Lakshmi 2019-06-12 07:27:56 UTC
The most recent failure captured under this bug is not related to the original issue.
https://intel-gfx-ci.01.org/tree/drm-tip/IGT_5050/shard-iclb1/igt@kms_frontbuffer_tracking@psr-1p-offscren-pri-shrfb-draw-mmap-wc.html

The reproduction rate of this failure used to be once in 3.5 drmtip runs. Last seen on drmtip_246 and current drmtip run is 304.

Closing this issue as WORKSFORMME.
Comment 19 CI Bug Log 2019-06-12 07:28:15 UTC
The CI Bug Log issue associated to this bug has been archived.

New failures matching the above filters will not be associated to this bug anymore.


Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.