Bug 105373

Created attachment 137845 [details]
dmesg

You need to update the dmc firmware:

commit 4f0aa1fa3e3849caee450ee5d14fcc289cf16703
Author: Anusha Srivatsa <anusha.srivatsa@intel.com>
Date:   Thu Nov 9 10:51:43 2017 -0800

    drm/i915/dmc: DMC 1.04 for Kabylake
    
    There is a new version of DMC available for KBL.
    
    The release notes mentions:
    1. Fix for the issue where DC_STATE was getting enabled even
    when disabled by driver causing data corruption.
    
    v2: Remove pull request from commit message (Rodrigo).
    
    Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
    Signed-off-by: Anusha Srivatsa <anusha.srivatsa@intel.com>
    Reviewed-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
    Signed-off-by: Jani Nikula <jani.nikula@intel.com>
    Link: https://patchwork.freedesktop.org/patch/msgid/1510253503-12634-1-git-send-email-anusha.srivatsa@intel.com

Same happens with 1.04. Attaching dmesg, /sys/class/drm/card0/error.

Also, echo 1 > /sys/kernel/debug/dri/0/i915_wedged doesn't seem to have any effect, it's still unable to reset the GPU.

Created attachment 137865 [details]
/sys/class/drm/card0/error with 1.04

Created attachment 137866 [details]
dmesg with 1.04

Hi, could you attach dmesg with debug info, drm.debug=0xe parameter in grub. Thanks.

Created attachment 137869 [details]
dmesg with drim.debug

Here's the dmesg with debug.

First of all. Sorry about spam.
This is mass update for our bugs. 

Sorry if you feel this annoying but with this trying to understand if bug still valid or not.
If bug investigation still in progress, please ignore this and I apologize!

If you think this is not anymore valid, please comment to the bug that can be closed.
If you haven't tested with our latest pre-upstream tree(drm-tip), can you do that also to see if issue is valid there still and if you cannot see issue there, please comment to the bug.

Jani, yes, this is still valid - there doesn't seem to have been a new release of anything related to this, and the bug persists (my GPU is hung and I still use some very slow driver/access to my video card).

The stuff I'm currently running is 4.15.0 with the path for the 1.04 firmware. Is there any new work in the drm-tip, and how do I fetch that?

You can get drm-tip from: https://cgit.freedesktop.org/drm-tip.

Created attachment 138427 [details]
/sys/class/drm/card0/error with drm-tip

Created attachment 138428 [details]
dmesg with drm-tip

(In reply to Jani Saarinen from comment #10)
> You can get drm-tip from: https://cgit.freedesktop.org/drm-tip.

Attached are the dmesg (with debug enabled) and the error from the card with drm-tip. The issue persists.

Mika, Chris, any advice here?

Looked at this and discussed with Chris on irc and here are the findings:

HW RING START is not from request that was queued to hardware. And the
gpu is dormant on a previous requests tail.

Please retest with fetching a up-to-date drm-tip from https://cgit.freedesktop.org/drm-tip and prevent driver from loading a dmc firmware by moving dmc firmware binaries out from /lib/firmware/i915.

Created attachment 139062 [details]
dmesg with drm-tip 4.17.0-rc2 (d04fd4f6d93cea918521059db8358ff9e7a4a03b)

Created attachment 139063 [details]
/sys/class/drm/card0/error with drm-tip 4.17.0-rc2 (d04fd4f6d93cea918521059db8358ff9e7a4a03b)

Retested with the latest drm-tip, the issue looks the same.

Is there anything else besides the dmesg and /sys/class/drm/card0/error I can help with? I can see to provide access to the laptop in question.

Just a note, I've been using the latest drm-tip (updating it around once a month) and the issue doesn't get any different. Any pointers where I should look?

Please update your drm-tip again and test with following
kernel parameters.

drm.debug=0xe intel_iommu=off i915.enable_dc=0 i915.disable_power_well=0

Upload dmesg and error state. Thanks

Rebuilding at the moment. Should I have the DMC firmware loaded or not?

Should not matter as the command line takes care of it.

Created attachment 141211 [details]
dmesg with drm-tip 4.18.0+ (d53f119472fc7daa532e46ea77098e9e9db2ac10)

Created attachment 141212 [details]
/sys/class/drm/card0/error with drm-tip 4.18.0+ (d53f119472fc7daa532e46ea77098e9e9db2ac10)

I've attached the new ones. Currently I see a difference - before glxinfo was just saying i/o error, now it has proper output.

Not much has changed. The gpu has not even picked up the work item given for it by Xorg to execution.

Can you ssh to the box after the boot has failed and do a following:

# intel_reg read 0x2358
# intel_reg read 0x2358

it is a timestamp register so let's see it the gpu is completely dead at
this point.

Also please enable everything under "drm/i915 Debugging" kernel config
for more detailed traces.

You could also add the following to the previous kernel command line
options: intel_idle.max_cstate=1 intel_pstate=disable

Created attachment 141215 [details]
/sys/class/drm/card0/error with drm-tip 4.18.0+ (d53f119472fc7daa532e46ea77098e9e9db2ac10), second try

Created attachment 141216 [details]
dmesg with drm-tip 4.18.0+ (d53f119472fc7daa532e46ea77098e9e9db2ac10), second try

I've attached updated dmesg and error with all the debug options enabled in the kernel and the new parameters.

I've tried the register read, and the register changes between reads:

(0x00002358): 0x6b619e3d
(0x00002358): 0x6dddb91a
(0x00002358): 0x6f0d705a

Also, it seems I haven't described the situation properly, so here it goes:

After boot, when the display manager starts, it takes some time to start reacting to anything else than cursor moves, and I can't even switch to the text console. After a while it starts working, but anything GPU related either crashes or blocks (for example, I'm running chromium with --disable-gpu). Compiz was refusing to work, and glxinfo was saying "i/o error".

Now glxinfo shows a normal result, but in any case anything GPU-related blocks, otherwise I'm able to (somewhat slowly) use X.

Mika, any updates on this issue?

Vasil, can you boot the machine without gui (adding '3' to kernel cmdline), and then run some gem tests from intel-gpu-tools?

For starters: gem_exec_basic, gem_exec_nop, gem_ctx_switch

And if they seem to work fine, then the full fastfeedback
suite with
'./scripts/run-tests.sh -F ./tests/intel-ci/fast-feedback.testlist'

Basically to see if tests will also encounter nonresponsive gpu right from the start.

Hi Mika,

I've been a bit swamped in the last few days, but I'll rebuild with the latest drm-tip tonight and will run these tests.

Created attachment 142085 [details]
results from intel-gpu-tools

The generated results are in results/, in res/ there's the output of the three tests and a dmesg file from since the second one (had to reboot after the third one, then ran the barrage).

I seem to be unable to turn off the frame buffer, can it be the reason for the issue?

Seems that atleast some gpu processing have been achieved
at some point (gem_ctx_switch).

Care to do with kernel params 'drm.debug=0xe i915.disable_display=1'
Thanks

Does disable_display mean that I should do this over ssh?

Yes, you need to do it through ssh.

There is bug currently in display side which will oops the kernel
if you run the whole fastfeedback testlist.

But if you could run the 3 specific gem tests provided through ssh, without display enabled

Created attachment 142107 [details]
results with disable_display

Mika, did the logs reveal anything?

Only that everytime the GPU goes to sleep it doesn't wake up until a reset is issued. Still strongly suggesting the dmc.

FWIW, I did some more tests with drm-tim 3ac901085a9fae8699716ac44579dab1dec546c3 and the latest BIOS for this MB. The problem persists, but there's something strange, this time mpv -vo opengl blocks for 10-14 seconds and then starts showing the video. Running glxgears doesn't show the gears, just reports FPS.

How does this look? If it could be a hardware issue, I can look into some workarounds...

Vasil, sorry for the delay. Do you still have the issue?
If so, can you reproduce the issue with current drmtip? (https://cgit.freedesktop.org/drm-tip).

If problem persists with current drmtip, Can you attach the dmesg and crash dump file?

Hi Lakshimi,

The problem existed with drm-tip from about two months ago. I'll rebuild and will let you know, as I'll need to reenable DRI.

Created attachment 144449 [details]
/sys/class/drm/card0/error with drm-tip 5.2.0-rc2 (aff09dc14e1d9f03f9e6c8c157d4abccf4ca2b14)

Created attachment 144451 [details]
dmesg with with drm-tip 5.2.0-rc2 (aff09dc14e1d9f03f9e6c8c157d4abccf4ca2b14)

Lakshimi, the problem persists, I've attached dmesg and the error.

(In reply to Vasil Kolev from comment #45)
> Lakshimi, the problem persists, I've attached dmesg and the error.

Thanks Vasil. I will come back to you when soon.

@Mika, any updates here?

@Mika, since this is a high priority bug this issue needs an update at least once in a week. Any progress here?

-- GitLab Migration Automatic Message --

This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity.

You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/drm/intel/issues/80.