Bug 108767 - [CI][Runner] Abort on network down
Summary: [CI][Runner] Abort on network down
Status: NEW
Alias: None
Product: DRI
Classification: Unclassified
Component: IGT (show other bugs)
Version: XOrg git
Hardware: Other All
: high normal
Assignee: Petri Latvala
QA Contact:
URL:
Whiteboard: ReadyForDev
Keywords:
: 108680 108953 (view as bug list)
Depends on:
Blocks:
 
Reported: 2018-11-16 14:38 UTC by Martin Peres
Modified: 2019-02-21 15:19 UTC (History)
1 user (show)

See Also:
i915 platform:
i915 features:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Martin Peres 2018-11-16 14:38:02 UTC
When the network is down, the external watchdogs are bound to come and kill the machine.

Protect ourselves against these random incompletes by stopping the execution early and exiting.

The current plan was to allow the configuration of a host to ping before executing a new binary to make sure that the network is still up. The host to ping should be specified in igtrc. If none are specified, no pings should be made.
Comment 2 Jani Saarinen 2018-11-21 11:43:35 UTC
Dropping priority, new feature request.
Comment 3 Martin Peres 2018-12-03 16:26:11 UTC
https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_5244/fi-kbl-7560u/igt@gem_basic@create-close.html

No logs, but it seems like it got stuck very early on, as can be seen in the run log:

[001/332] (960s left) core_auth (basic-auth)
FATAL: command execution failed
java.io.EOFException
	at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2681)
	at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3156)
	at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:862)
	at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
	at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
	at hudson.remoting.Command.readFrom(Command.java:140)
	at hudson.remoting.Command.readFrom(Command.java:126)
	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:36)
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Caused: java.io.IOException: Unexpected termination of the channel
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
Caused: java.io.IOException: Backing channel 'fi-kbl-7560u' is disconnected.
	at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:214)
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283)
	at com.sun.proxy.$Proxy64.isAlive(Unknown Source)
	at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1144)
	at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1136)
	at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:155)
	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:109)
	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
	at hudson.model.Build$BuildExecution.build(Build.java:206)
	at hudson.model.Build$BuildExecution.doRun(Build.java:163)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504)
	at hudson.model.Run.execute(Run.java:1810)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:429)
FATAL: Unable to delete script file /tmp/jenkins3295738120970564664.sh
java.io.EOFException
	at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2681)
	at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3156)
	at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:862)
	at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
	at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
	at hudson.remoting.Command.readFrom(Command.java:140)
	at hudson.remoting.Command.readFrom(Command.java:126)
	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:36)
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Caused: java.io.IOException: Unexpected termination of the channel
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
Caused: hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on fi-kbl-7560u failed. The channel is closing down or has closed down
	at hudson.remoting.Channel.call(Channel.java:948)
	at hudson.FilePath.act(FilePath.java:1070)
	at hudson.FilePath.act(FilePath.java:1059)
	at hudson.FilePath.delete(FilePath.java:1563)
	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:123)
	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
	at hudson.model.Build$BuildExecution.build(Build.java:206)
	at hudson.model.Build$BuildExecution.doRun(Build.java:163)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504)
	at hudson.model.Run.execute(Run.java:1810)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:429)
Build step 'Execute shell' marked build as failure
Set build name.
Workspace is not accessible
ERROR: no workspace for CI_IGT_test #2662333
Notifying upstream projects of job completion
Finished: FAILURE
Completed CI_IGT_test CI_DRM_5244/fi-kbl-7560u/0 : FAILURE
CI_IGT_test runtime 6 seconds
Rebooting fi-kbl-7560u
Comment 4 Lakshmi 2018-12-07 16:16:09 UTC
*** Bug 108953 has been marked as a duplicate of this bug. ***
Comment 5 Lakshmi 2018-12-07 16:19:58 UTC
*** Bug 108680 has been marked as a duplicate of this bug. ***
Comment 6 CI Bug Log 2018-12-31 14:56:34 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-kbl-7560u: random tests - incomplete -}
{+ fi-kbl-7560u: random tests - incomplete +}

New failures caught by the filter:

* https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_172/fi-kbl-7560u/igt@pm_lpsp@edp-panel-fitter.html
Comment 7 CI Bug Log 2019-01-04 15:19:52 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-kbl-7560u: random tests - incomplete -}
{+ fi-kbl-7560u: random tests - incomplete +}

New failures caught by the filter:

* https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_175/fi-kbl-7560u/igt@kms_cursor_crc@cursor-128x128-offscreen.html
Comment 8 CI Bug Log 2019-01-14 17:20:53 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-kbl-7560u: random tests - incomplete -}
{+ fi-kbl-7560u: random tests - incomplete +}

New failures caught by the filter:

* https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_5403/fi-kbl-7560u/igt@gem_close_race@basic-threads.html
Comment 9 CI Bug Log 2019-01-24 13:17:09 UTC
The CI Bug Log issue associated to this bug has been updated.

### New filters associated

* SKL: random tests - incomplete
  - https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_5464/shard-skl8/igt@kms_flip@flip-vs-panning-vs-hang-interruptible.html
Comment 10 CI Bug Log 2019-01-24 13:18:28 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-kbl-7560u: random tests - incomplete -}
{+ fi-kbl-7560u: random tests - incomplete +}

New failures caught by the filter:

* https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4783/fi-kbl-7560u/igt@kms_pipe_crc_basic@suspend-read-crc-pipe-b.html
* https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_194/fi-kbl-7560u/igt@kms_pipe_crc_basic@suspend-read-crc-pipe-b.html
Comment 11 CI Bug Log 2019-01-28 10:37:58 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-kbl-7560u: random tests - incomplete -}
{+ fi-kbl-7560u: random tests - incomplete +}

New failures caught by the filter:

* https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_177/fi-kbl-7560u/igt@prime_busy@wait-hang-render.html
Comment 12 CI Bug Log 2019-01-28 10:43:06 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-kbl-7560u: random tests - incomplete -}
{+ fi-kbl-7560u: random tests - incomplete +}

New failures caught by the filter:

* https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_180/fi-kbl-7560u/igt@kms_pipe_b_c_ivb@enable-pipe-c-while-b-has-3-lanes.html
Comment 13 CI Bug Log 2019-01-28 12:45:59 UTC
A CI Bug Log filter associated to this bug has been updated:

{- CHAMELIUM: igt@kms_chamelium@* - fail - Last errno: 113, No route to host -}
{+ CHAMELIUM: igt@kms_chamelium@* - fail - Last errno: 113, No route to host +}

 No new failures caught with the new filter
Comment 14 CI Bug Log 2019-01-29 12:01:51 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-cfl-8109u: random tests - incomplete -}
{+ fi-cfl-8109u: random tests - incomplete +}

New failures caught by the filter:

* https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_178/fi-cfl-8109u/igt@kms_flip@2x-flip-vs-blocking-wf-vblank.html
Comment 15 CI Bug Log 2019-01-29 12:08:25 UTC
The CI Bug Log issue associated to this bug has been updated.

### New filters associated

* fi-blb-e6850: random tests - incomplete
  - https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_178/fi-blb-e6850/igt@kms_atomic_interruptible@universal-setplane-cursor.html
Comment 16 CI Bug Log 2019-01-29 14:53:44 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-kbl-7560u: random tests - incomplete -}
{+ fi-kbl-7560u: random tests - incomplete +}

New failures caught by the filter:

* https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_198/fi-kbl-7560u/igt@kms_frontbuffer_tracking@fbcpsr-1p-primscrn-cur-indfb-move.html
* https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_200/fi-kbl-7560u/igt@kms_frontbuffer_tracking@fbc-rgb101010-draw-mmap-gtt.html
Comment 17 CI Bug Log 2019-01-29 16:28:51 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-cfl-8109u: random tests - incomplete -}
{+ fi-cfl-8109u: random tests - incomplete +}

New failures caught by the filter:

* https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_187/fi-cfl-8109u/igt@kms_flip@2x-dpms-vs-vblank-race-interruptible.html
Comment 18 CI Bug Log 2019-01-31 11:22:38 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-cfl-8109u: random tests - incomplete -}
{+ fi-cfl-8109u: random tests - incomplete +}

New failures caught by the filter:

* https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_192/fi-cfl-8109u/igt@kms_busy@basic-modeset-c.html
Comment 19 CI Bug Log 2019-01-31 11:45:17 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-cfl-8109u: random tests - incomplete -}
{+ fi-cfl-8109u: random tests - incomplete +}

New failures caught by the filter:

* https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_195/fi-cfl-8109u/igt@perf_pmu@enable-race-vcs1.html
Comment 20 CI Bug Log 2019-01-31 14:11:16 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-blb-e6850: random tests - incomplete -}
{+ fi-blb-e6850: random tests - incomplete +}

New failures caught by the filter:

* https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_202/fi-blb-e6850/igt@kms_flip@flip-vs-suspend.html
Comment 21 CI Bug Log 2019-01-31 15:06:54 UTC
The CI Bug Log issue associated to this bug has been updated.

### New filters associated

* fi-snb-2520m: igt@gem_eio@throttle - incomplete - empty logs
  - https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_202/fi-snb-2520m/igt@gem_eio@throttle.html
Comment 22 CI Bug Log 2019-01-31 16:18:09 UTC
The CI Bug Log issue associated to this bug has been updated.

### New filters associated

* fi-kbl-guc: igt@kms_frontbuffer_tracking@fbcpsr-2p-indfb-fliptrack - incomplete - empty logs
  - https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_201/fi-kbl-guc/igt@kms_frontbuffer_tracking@fbcpsr-2p-indfb-fliptrack.html
Comment 23 CI Bug Log 2019-02-07 12:28:49 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-kbl-7560u: random tests - incomplete -}
{+ fi-kbl-7560u: random tests - incomplete +}

New failures caught by the filter:

* https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_203/fi-kbl-7560u/igt@gem_pwrite_pread@display-pwrite-blt-gtt_mmap-performance.html
Comment 24 CI Bug Log 2019-02-07 16:49:11 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-snb-2520m: igt@gem_eio@throttle - incomplete - empty logs -}
{+ fi-snb-2520m: igt@* - incomplete - empty logs +}

New failures caught by the filter:

* https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_203/fi-snb-2520m/igt@kms_color@pipe-a-legacy-gamma.html
Comment 25 CI Bug Log 2019-02-08 14:34:47 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-snb-2520m: igt@* - incomplete - empty logs -}
{+ fi-snb-2520m: igt@* - incomplete - empty logs +}

New failures caught by the filter:

* https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_211/fi-snb-2520m/igt@kms_busy@extended-pageflip-modeset-hang-oldfb-render-a.html
Comment 26 CI Bug Log 2019-02-08 14:40:56 UTC
The CI Bug Log issue associated to this bug has been updated.

### New filters associated

* fi-kbl-8809g: igt@* - incomplete - empty logs
  - https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_211/fi-kbl-8809g/igt@kms_busy@extended-pageflip-modeset-hang-oldfb-render-a.html
Comment 27 CI Bug Log 2019-02-11 12:36:45 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-kbl-guc: igt@kms_frontbuffer_tracking@fbcpsr-2p-indfb-fliptrack - incomplete - empty logs -}
{+ fi-kbl-guc: i random tests - incomplete +}

New failures caught by the filter:

* https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_213/fi-kbl-guc/igt@gem_exec_basic@basic-vebox.html
* https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_215/fi-kbl-guc/igt@gem_exec_basic@basic-vebox.html
Comment 28 CI Bug Log 2019-02-11 12:39:30 UTC
The CI Bug Log issue associated to this bug has been updated.

### New filters associated

* fi-skl-6770: random tests - incomplete
  - https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_215/fi-skl-6770hq/igt@gem_exec_basic@basic-vebox.html
Comment 29 CI Bug Log 2019-02-12 09:02:36 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-kbl-guc: i random tests - incomplete -}
{+ fi-kbl-guc: random tests - incomplete +}

 No new failures caught with the new filter
Comment 30 CI Bug Log 2019-02-12 09:03:39 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-kbl-guc: random tests - incomplete -}
{+ fi-kbl-guc: random tests - incomplete +}

 No new failures caught with the new filter
Comment 31 CI Bug Log 2019-02-12 09:03:52 UTC
The CI Bug Log issue associated to this bug has been updated.

### New filters associated

* fi-kbl-guc: random tests - incomplete - empty logs
  - https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_209/fi-kbl-guc/igt@kms_frontbuffer_tracking@fbc-rgb565-draw-pwrite.html
Comment 32 CI Bug Log 2019-02-12 09:27:24 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-cfl-8109u: random tests - incomplete -}
{+ fi-cfl-8109u: random tests - incomplete +}

New failures caught by the filter:

* https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_210/fi-cfl-8109u/igt@kms_busy@extended-modeset-hang-newfb-with-reset-render-b.html
Comment 33 CI Bug Log 2019-02-12 11:28:36 UTC
The CI Bug Log issue associated to this bug has been updated.

### New filters associated

* fi-ilk-650: random tests - incomplete
  - https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_217/fi-ilk-650/igt@gem_exec_await@wide-all.html
Comment 34 CI Bug Log 2019-02-15 14:16:06 UTC
The CI Bug Log issue associated to this bug has been updated.

### New filters associated

* BDW: random tests - incomplete
  - https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_218/fi-bdw-5557u/igt@perf_pmu@enable-race-vecs0.html
Comment 35 CI Bug Log 2019-02-15 14:18:50 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-cfl-8109u: random tests - incomplete -}
{+ fi-cfl-8109u: random tests - incomplete +}

New failures caught by the filter:

* https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_218/fi-cfl-8109u/igt@kms_busy@extended-modeset-hang-oldfb-with-reset-render-b.html
Comment 36 CI Bug Log 2019-02-15 14:20:33 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-kbl-guc: random tests - incomplete - empty logs -}
{+ fi-kbl-guc: random tests - incomplete - empty logs +}

New failures caught by the filter:

* https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_218/fi-kbl-guc/igt@kms_frontbuffer_tracking@fbc-2p-scndscrn-pri-shrfb-draw-mmap-gtt.html
Comment 37 CI Bug Log 2019-02-18 16:22:15 UTC
A CI Bug Log filter associated to this bug has been updated:

{- CHAMELIUM: igt@kms_chamelium@* - fail - Last errno: 113, No route to host -}
{+ CHAMELIUM: igt@kms_chamelium@* - fail - Last errno: 113, No route to host +}

New failures caught by the filter:

* https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_222/fi-kbl-7500u/igt@kms_chamelium@hdmi-cmp-yu16.html
* https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_223/fi-kbl-7500u/igt@kms_chamelium@hdmi-cmp-yu16.html
* https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_223/fi-kbl-7567u/igt@kms_chamelium@hdmi-cmp-yu16.html
Comment 38 CI Bug Log 2019-02-21 12:44:31 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-snb-2520m: igt@* - incomplete - empty logs -}
{+ fi-snb-2520m: igt@* - random tests - incomplete  +}

New failures caught by the filter:

* https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_221/fi-snb-2520m/igt@kms_plane@plane-position-covered-pipe-a-planes.html
Comment 39 CI Bug Log 2019-02-21 14:41:18 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-cfl-8109u: random tests - incomplete -}
{+ fi-cfl-8109u: random tests - incomplete +}

New failures caught by the filter:

* https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_224/fi-cfl-8109u/igt@kms_rotation_crc@multiplane-rotation.html
Comment 40 CI Bug Log 2019-02-21 15:19:11 UTC
A CI Bug Log filter associated to this bug has been updated:

{- fi-kbl-7560u: random tests - incomplete -}
{+ fi-kbl-7560u: random tests - incomplete +}

New failures caught by the filter:

* https://intel-gfx-ci.01.org/tree/drm-tip/drmtip_225/fi-kbl-7560u/igt@gem_exec_parallel@default.html


Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.