The CI Bug Log issue associated to this bug has been updated.

### New filters associated

* CML: random tests - incomplete
  - https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_267/fi-cml-u/igt@kms_busy@basic-modeset-c.html
  - https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_269/fi-cml-u/igt@kms_plane@pixel-format-pipe-b-planes.html
  - https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6001/re-cml-u/igt@kms_flip@flip-vs-panning-interruptible.html
  - https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_270/fi-cml-u/igt@kms_cursor_crc@cursor-128x42-random.html
  - https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_270/fi-cml-u/igt@gem_exec_big@single.html

A CI Bug Log filter associated to this bug has been updated:

{- CML: random tests - incomplete -}
{+ CML: random tests - incomplete +}

New failures caught by the filter:

  * https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6010/re-cml-u/igt@gem_tiled_wb.html
  * https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6012/re-cml-u/igt@gem_tiled_wb.html

A CI Bug Log filter associated to this bug has been updated:

{- CML: random tests - incomplete -}
{+ CML: random tests - incomplete +}

New failures caught by the filter:

  * https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6062/re-cml-u/igt@kms_chamelium@common-hpd-after-suspend.html

A CI Bug Log filter associated to this bug has been updated:

{- CML: random tests - incomplete -}
{+ CML: random tests - incomplete +}

New failures caught by the filter:

  * https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6061/re-cml-u/igt@kms_cursor_crc@cursor-64x64-offscreen.html

A CI Bug Log filter associated to this bug has been updated:

{- CML: random tests - incomplete -}
{+ CML: random tests - incomplete +}

New failures caught by the filter:

  * https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6067/re-cml-u/igt@kms_flip@flip-vs-fences.html
  * https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6067/re-cml-u/igt@kms_psr@psr2_sprite_plane_move.html

A CI Bug Log filter associated to this bug has been updated:

{- CML: random tests - incomplete -}
{+ CML: random tests - incomplete +}

New failures caught by the filter:

  * https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6072/re-cml-u/igt@gem_pwrite@big-gtt-backwards.html

A CI Bug Log filter associated to this bug has been updated:

{- CML: random tests - incomplete -}
{+ CML: random tests - incomplete +}

New failures caught by the filter:

  * https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_285/fi-cml-u/igt@gem_pwrite@big-cpu-backwards.html

A CI Bug Log filter associated to this bug has been updated:

{- CML: random tests - incomplete -}
{+ CML: random tests - incomplete +}

New failures caught by the filter:

  * https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6085/re-cml-u/igt@i915_pm_rpm@pm-caching.html

A CI Bug Log filter associated to this bug has been updated:

{- CML: random tests - incomplete -}
{+ CML: all tests - incomplete +}

New failures caught by the filter:

  * https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_288/fi-cml-u/igt@kms_frontbuffer_tracking@psr-2p-scndscrn-pri-shrfb-draw-blt.html
  * https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_288/fi-cml-u/igt@kms_cursor_crc@pipe-b-cursor-128x128-onscreen.html
  * https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_289/fi-cml-u/igt@kms_vblank@pipe-c-wait-busy.html
  * https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_289/fi-cml-u/igt@kms_vblank@pipe-c-ts-continuation-modeset-rpm.html
  * https://intel-gfx-ci.01.org/tree/drm-tip/IGT_5003/fi-cml-u/igt@core_auth@basic-auth.html
  * https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6121/re-cml-u/igt@gem_exec_parallel@bsd.html
  * https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6121/re-cml-u/igt@i915_pm_rpm@gem-execbuf.html
  * https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6121/re-cml-u/igt@kms_plane@pixel-format-pipe-b-planes-source-clamping.html
  * https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_292/fi-cml-u/igt@kms_plane_scaling@pipe-b-scaler-with-clipping-clamping.html
  * https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_292/fi-cml-u/igt@gem_exec_reloc@basic-wc-gtt-noreloc.html
  * https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_292/fi-cml-u2/igt@kms_cursor_crc@pipe-b-cursor-256x85-random.html

Putting to highest priority, since incompletes are now everywhere!

Different tests, but always same 2 machines(fi-cml-u, re-cml-u) - kinda suspicious. May be it would make sense to take it from CI to run the tests manually and check?

Lakhsmi, is this still valid?

(In reply to Jani Saarinen from comment #12)
> Lakhsmi, is this still valid?
Yes, it's still happening. But I see that Jenkins gives up in some cases.

https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_378/fi-cml-s/igt@gem_ctx_switch@vcs0-heavy.html

Starting subtest: vcs0-heavy
[32.199975] Abort requested by sudo LD_LIBRARY_PATH=/opt/igt/lib:/opt/igt/lib/x86_64-linux-gnu IGT_CI_META_TEST=yes IGT_REBOOT_ON_FATAL_ERROR=yes stdb [1183] via Terminated, terminating children
[32.200923] Abort requested by /sbin/init 3  [1] via Hangup, terminating children
[32.201059] Abort requested by sudo LD_LIBRARY_PATH=/opt/igt/lib:/opt/igt/lib/x86_64-linux-gnu IGT_CI_META_TEST=yes IGT_REBOOT_ON_FATAL_ERROR=yes stdb [1183] via Hangup, terminating children
[32.212075] Closing watchdogs
Done.

We need to go through all the failures and see if we need to split the issue.

I looked over about 20 logs for this issue. For about 60% of the logs there is a line saying: This is power.sh, remotely rebooting this machine.

It seems like the machine was reset remotely during the test - in a number of cases while the test was executing.

Tomi, is there something can be done to CI for this ?

https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_396/fi-cml-s/igt_runner6.txt
Subtest pipe-A-overlay-size-64: SUCCESS (2.454s)
[37.463866] [11/99] (945s left) kms_plane (plane-panning-top-left-pipe-b-planes)
Starting subtest: plane-panning-top-left-pipe-B-planes
[41.235917] Abort requested by /sbin/init 3 [1] via Hangup, terminating children
[41.237585] Abort requested by sudo LD_LIBRARY_PATH=/opt/igt/lib:/opt/igt/lib/x86_64-linux-gnu IGT_CI_META_TEST=yes IGT_REBOOT_ON_FATAL_ERROR=yes stdb [1215] via Terminated, terminating children
[42.417619] Closing watchdogs
Done.
These incompletes should be fixed as part of bug 111747.

-- GitLab Migration Automatic Message --

This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity.

You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/drm/intel/issues/283.