Bug 105373 - [kbl-h] GPU HANG: ecode 9:0:0xfedffffa, in Xorg [1345], reason: Hang on rcs0, action: reset
Summary: [kbl-h] GPU HANG: ecode 9:0:0xfedffffa, in Xorg [1345], reason: Hang on rcs0,...
Status: REOPENED
Alias: None
Product: DRI
Classification: Unclassified
Component: DRM/Intel (show other bugs)
Version: XOrg git
Hardware: x86-64 (AMD64) Linux (All)
: high normal
Assignee: Mika Kuoppala
QA Contact: Intel GFX Bugs mailing list
URL:
Whiteboard: ReadyForDev
Keywords:
Depends on:
Blocks:
 
Reported: 2018-03-06 22:44 UTC by Vasil Kolev
Modified: 2018-10-19 19:49 UTC (History)
4 users (show)

See Also:
i915 platform: KBL
i915 features: GPU hang


Attachments
/sys/class/drm/card0/error (16.83 KB, application/x-bzip)
2018-03-06 22:44 UTC, Vasil Kolev
no flags Details
dmesg (57.19 KB, text/plain)
2018-03-06 22:45 UTC, Vasil Kolev
no flags Details
/sys/class/drm/card0/error with 1.04 (4.16 KB, application/x-bzip)
2018-03-07 16:20 UTC, Vasil Kolev
no flags Details
dmesg with 1.04 (56.72 KB, text/plain)
2018-03-07 16:20 UTC, Vasil Kolev
no flags Details
dmesg with drim.debug (97.48 KB, text/plain)
2018-03-07 18:15 UTC, Vasil Kolev
no flags Details
/sys/class/drm/card0/error with drm-tip (26.33 KB, application/x-bzip)
2018-03-29 13:43 UTC, Vasil Kolev
no flags Details
dmesg with drm-tip (117.97 KB, text/plain)
2018-03-29 13:43 UTC, Vasil Kolev
no flags Details
dmesg with drm-tip 4.17.0-rc2 (d04fd4f6d93cea918521059db8358ff9e7a4a03b) (118.07 KB, text/plain)
2018-04-24 15:54 UTC, Vasil Kolev
no flags Details
/sys/class/drm/card0/error with drm-tip 4.17.0-rc2 (d04fd4f6d93cea918521059db8358ff9e7a4a03b) (26.39 KB, application/x-bzip)
2018-04-24 15:55 UTC, Vasil Kolev
no flags Details
dmesg with drm-tip 4.18.0+ (d53f119472fc7daa532e46ea77098e9e9db2ac10) (25.27 KB, application/x-bzip)
2018-08-21 09:05 UTC, Vasil Kolev
no flags Details
/sys/class/drm/card0/error with drm-tip 4.18.0+ (d53f119472fc7daa532e46ea77098e9e9db2ac10) (25.00 KB, application/x-bzip)
2018-08-21 09:06 UTC, Vasil Kolev
no flags Details
/sys/class/drm/card0/error with drm-tip 4.18.0+ (d53f119472fc7daa532e46ea77098e9e9db2ac10), second try (25.09 KB, application/x-bzip)
2018-08-21 14:25 UTC, Vasil Kolev
no flags Details
dmesg with drm-tip 4.18.0+ (d53f119472fc7daa532e46ea77098e9e9db2ac10), second try (19.12 KB, application/x-bzip)
2018-08-21 14:25 UTC, Vasil Kolev
no flags Details
results from intel-gpu-tools (53.28 KB, application/x-bzip-compressed-tar)
2018-10-18 20:11 UTC, Vasil Kolev
no flags Details
results with disable_display (35.97 KB, application/x-bzip)
2018-10-19 19:49 UTC, Vasil Kolev
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Vasil Kolev 2018-03-06 22:44:46 UTC
Created attachment 137844 [details]
/sys/class/drm/card0/error

issue: GPU doesn't even start working

On every boot, as soon as I login and compiz tries to start, the above message shows up in dmesg.
Comment 1 Vasil Kolev 2018-03-06 22:45:39 UTC
Created attachment 137845 [details]
dmesg
Comment 2 Chris Wilson 2018-03-07 08:45:31 UTC
You need to update the dmc firmware:

commit 4f0aa1fa3e3849caee450ee5d14fcc289cf16703
Author: Anusha Srivatsa <anusha.srivatsa@intel.com>
Date:   Thu Nov 9 10:51:43 2017 -0800

    drm/i915/dmc: DMC 1.04 for Kabylake
    
    There is a new version of DMC available for KBL.
    
    The release notes mentions:
    1. Fix for the issue where DC_STATE was getting enabled even
    when disabled by driver causing data corruption.
    
    v2: Remove pull request from commit message (Rodrigo).
    
    Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
    Signed-off-by: Anusha Srivatsa <anusha.srivatsa@intel.com>
    Reviewed-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
    Signed-off-by: Jani Nikula <jani.nikula@intel.com>
    Link: https://patchwork.freedesktop.org/patch/msgid/1510253503-12634-1-git-send-email-anusha.srivatsa@intel.com
Comment 3 Vasil Kolev 2018-03-07 16:19:22 UTC
Same happens with 1.04. Attaching dmesg, /sys/class/drm/card0/error.

Also, echo 1 > /sys/kernel/debug/dri/0/i915_wedged doesn't seem to have any effect, it's still unable to reset the GPU.
Comment 4 Vasil Kolev 2018-03-07 16:20:09 UTC
Created attachment 137865 [details]
/sys/class/drm/card0/error with 1.04
Comment 5 Vasil Kolev 2018-03-07 16:20:35 UTC
Created attachment 137866 [details]
dmesg with 1.04
Comment 6 Elizabeth 2018-03-07 17:33:49 UTC
Hi, could you attach dmesg with debug info, drm.debug=0xe parameter in grub. Thanks.
Comment 7 Vasil Kolev 2018-03-07 18:15:06 UTC
Created attachment 137869 [details]
dmesg with drim.debug

Here's the dmesg with debug.
Comment 8 Jani Saarinen 2018-03-29 07:10:51 UTC
First of all. Sorry about spam.
This is mass update for our bugs. 

Sorry if you feel this annoying but with this trying to understand if bug still valid or not.
If bug investigation still in progress, please ignore this and I apologize!

If you think this is not anymore valid, please comment to the bug that can be closed.
If you haven't tested with our latest pre-upstream tree(drm-tip), can you do that also to see if issue is valid there still and if you cannot see issue there, please comment to the bug.
Comment 9 Vasil Kolev 2018-03-29 07:42:59 UTC
Jani, yes, this is still valid - there doesn't seem to have been a new release of anything related to this, and the bug persists (my GPU is hung and I still use some very slow driver/access to my video card).

The stuff I'm currently running is 4.15.0 with the path for the 1.04 firmware. Is there any new work in the drm-tip, and how do I fetch that?
Comment 10 Jani Saarinen 2018-03-29 08:09:31 UTC
You can get drm-tip from: https://cgit.freedesktop.org/drm-tip.
Comment 11 Vasil Kolev 2018-03-29 13:43:20 UTC
Created attachment 138427 [details]
/sys/class/drm/card0/error with drm-tip
Comment 12 Vasil Kolev 2018-03-29 13:43:57 UTC
Created attachment 138428 [details]
dmesg with drm-tip
Comment 13 Vasil Kolev 2018-03-29 13:45:33 UTC
(In reply to Jani Saarinen from comment #10)
> You can get drm-tip from: https://cgit.freedesktop.org/drm-tip.

Attached are the dmesg (with debug enabled) and the error from the card with drm-tip. The issue persists.
Comment 14 Jani Saarinen 2018-04-24 07:00:58 UTC
Mika, Chris, any advice here?
Comment 15 Mika Kuoppala 2018-04-24 14:07:24 UTC
Looked at this and discussed with Chris on irc and here are the findings:

HW RING START is not from request that was queued to hardware. And the
gpu is dormant on a previous requests tail.

Please retest with fetching a up-to-date drm-tip from https://cgit.freedesktop.org/drm-tip and prevent driver from loading a dmc firmware by moving dmc firmware binaries out from /lib/firmware/i915.
Comment 16 Vasil Kolev 2018-04-24 15:54:30 UTC
Created attachment 139062 [details]
dmesg with drm-tip 4.17.0-rc2 (d04fd4f6d93cea918521059db8358ff9e7a4a03b)
Comment 17 Vasil Kolev 2018-04-24 15:55:06 UTC
Created attachment 139063 [details]
/sys/class/drm/card0/error with drm-tip 4.17.0-rc2 (d04fd4f6d93cea918521059db8358ff9e7a4a03b)
Comment 18 Vasil Kolev 2018-04-24 15:56:46 UTC
Retested with the latest drm-tip, the issue looks the same.

Is there anything else besides the dmesg and /sys/class/drm/card0/error I can help with? I can see to provide access to the laptop in question.
Comment 19 Vasil Kolev 2018-08-19 15:41:27 UTC
Just a note, I've been using the latest drm-tip (updating it around once a month) and the issue doesn't get any different. Any pointers where I should look?
Comment 20 Mika Kuoppala 2018-08-21 07:43:44 UTC
Please update your drm-tip again and test with following
kernel parameters.

drm.debug=0xe intel_iommu=off i915.enable_dc=0 i915.disable_power_well=0

Upload dmesg and error state. Thanks
Comment 21 Vasil Kolev 2018-08-21 08:00:09 UTC
Rebuilding at the moment. Should I have the DMC firmware loaded or not?
Comment 22 Mika Kuoppala 2018-08-21 08:04:54 UTC
Should not matter as the command line takes care of it.
Comment 23 Vasil Kolev 2018-08-21 09:05:58 UTC
Created attachment 141211 [details]
dmesg with drm-tip 4.18.0+ (d53f119472fc7daa532e46ea77098e9e9db2ac10)
Comment 24 Vasil Kolev 2018-08-21 09:06:37 UTC
Created attachment 141212 [details]
/sys/class/drm/card0/error with drm-tip 4.18.0+ (d53f119472fc7daa532e46ea77098e9e9db2ac10)
Comment 25 Vasil Kolev 2018-08-21 09:08:27 UTC
I've attached the new ones. Currently I see a difference - before glxinfo was just saying i/o error, now it has proper output.
Comment 26 Mika Kuoppala 2018-08-21 12:59:29 UTC
Not much has changed. The gpu has not even picked up the work item given for it by Xorg to execution.

Can you ssh to the box after the boot has failed and do a following:

# intel_reg read 0x2358
# intel_reg read 0x2358

it is a timestamp register so let's see it the gpu is completely dead at
this point.

Also please enable everything under "drm/i915 Debugging" kernel config
for more detailed traces.

You could also add the following to the previous kernel command line
options: intel_idle.max_cstate=1 intel_pstate=disable
Comment 27 Vasil Kolev 2018-08-21 14:25:07 UTC
Created attachment 141215 [details]
/sys/class/drm/card0/error with drm-tip 4.18.0+ (d53f119472fc7daa532e46ea77098e9e9db2ac10), second try
Comment 28 Vasil Kolev 2018-08-21 14:25:34 UTC
Created attachment 141216 [details]
dmesg with drm-tip 4.18.0+ (d53f119472fc7daa532e46ea77098e9e9db2ac10), second try
Comment 29 Vasil Kolev 2018-08-21 14:29:19 UTC
I've attached updated dmesg and error with all the debug options enabled in the kernel and the new parameters.

I've tried the register read, and the register changes between reads:

(0x00002358): 0x6b619e3d
(0x00002358): 0x6dddb91a
(0x00002358): 0x6f0d705a

Also, it seems I haven't described the situation properly, so here it goes:

After boot, when the display manager starts, it takes some time to start reacting to anything else than cursor moves, and I can't even switch to the text console. After a while it starts working, but anything GPU related either crashes or blocks (for example, I'm running chromium with --disable-gpu). Compiz was refusing to work, and glxinfo was saying "i/o error".

Now glxinfo shows a normal result, but in any case anything GPU-related blocks, otherwise I'm able to (somewhat slowly) use X.
Comment 30 Lakshmi 2018-09-10 13:50:21 UTC
Mika, any updates on this issue?
Comment 31 Mika Kuoppala 2018-10-15 14:01:47 UTC
Vasil, can you boot the machine without gui (adding '3' to kernel cmdline), and then run some gem tests from intel-gpu-tools?

For starters: gem_exec_basic, gem_exec_nop, gem_ctx_switch

And if they seem to work fine, then the full fastfeedback
suite with
'./scripts/run-tests.sh -F ./tests/intel-ci/fast-feedback.testlist'

Basically to see if tests will also encounter nonresponsive gpu right from the start.
Comment 32 Vasil Kolev 2018-10-18 06:31:49 UTC
Hi Mika,

I've been a bit swamped in the last few days, but I'll rebuild with the latest drm-tip tonight and will run these tests.
Comment 33 Vasil Kolev 2018-10-18 20:11:25 UTC
Created attachment 142085 [details]
results from intel-gpu-tools

The generated results are in results/, in res/ there's the output of the three tests and a dmesg file from since the second one (had to reboot after the third one, then ran the barrage).

I seem to be unable to turn off the frame buffer, can it be the reason for the issue?
Comment 34 Mika Kuoppala 2018-10-19 09:15:46 UTC
Seems that atleast some gpu processing have been achieved
at some point (gem_ctx_switch).

Care to do with kernel params 'drm.debug=0xe i915.disable_display=1'
Thanks
Comment 35 Vasil Kolev 2018-10-19 09:28:14 UTC
Does disable_display mean that I should do this over ssh?
Comment 36 Mika Kuoppala 2018-10-19 12:22:46 UTC
Yes, you need to do it through ssh.

There is bug currently in display side which will oops the kernel
if you run the whole fastfeedback testlist.

But if you could run the 3 specific gem tests provided through ssh, without display enabled
Comment 37 Vasil Kolev 2018-10-19 19:49:21 UTC
Created attachment 142107 [details]
results with disable_display


Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.