Bug 99028 - [KVM][GVT-d] [BDW & SKL ]Ubuntu 16.04 guest boot up with GPU Hang with drm-intel kernel
Summary: [KVM][GVT-d] [BDW & SKL ]Ubuntu 16.04 guest boot up with GPU Hang with drm-in...
Status: CLOSED NOTOURBUG
Alias: None
Product: DRI
Classification: Unclassified
Component: DRM/Intel (show other bugs)
Version: DRI git
Hardware: x86-64 (AMD64) Linux (All)
: highest blocker
Assignee: Intel GFX Bugs mailing list
QA Contact: Intel GFX Bugs mailing list
URL:
Whiteboard: ReadyForDev
Keywords: bisected, regression
: 99025 (view as bug list)
Depends on:
Blocks:
 
Reported: 2016-12-08 11:20 UTC by Terrence Xu
Modified: 2017-06-27 14:20 UTC (History)
5 users (show)

See Also:
i915 platform: BDW, SKL
i915 features: GEM/Other, GPU hang


Attachments
dmesg-guest.log (28.83 KB, text/plain)
2016-12-08 14:37 UTC, Terrence Xu
no flags Details
error.log (712.91 KB, text/plain)
2017-01-03 14:00 UTC, Terrence Xu
no flags Details
guest igd pci config (1.48 KB, text/plain)
2017-04-06 10:55 UTC, XiongZhang
no flags Details
host_dmesg_with_stolen_memory_enabled (8.09 KB, text/plain)
2017-04-06 11:03 UTC, XiongZhang
no flags Details
host_dmesg_with_stolen_memory_disabled (2.75 KB, text/plain)
2017-04-06 11:08 UTC, XiongZhang
no flags Details
host_dmesg_boot_guest_4.8_kernel (3.85 KB, text/plain)
2017-04-06 11:17 UTC, XiongZhang
no flags Details
/sys/class/drm/card0/error when gpu hang (12.63 KB, text/plain)
2017-04-13 02:46 UTC, XiongZhang
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Terrence Xu 2016-12-08 11:20:41 UTC
System Environment
=======
Host kernel repo: kvm.git
Host commit: master-813ae37e
Guest repo: drm-intel.git
Guest commit: drm-intel-next-queued-312c3c46

Regression?
=======
No

Bug detailed description
=======
The guest boot up with the drm-intel 4.8.0-rc2+ kernel, with GPU hang be found.
The guest boot up with the latest drm-intel 4.9.0-rc4+ kernel with kernel panic (as https://bugs.freedesktop.org/show_bug.cgi?id=99025), but if we try to remove the kernel panic error handling manually, the GPU hang issue also won existed.
This is KVM GVT-d environment issue.

Reproduce Steps
==============
Boot up Ubuntu 16.04 guest with the drm-intel kernel, the command as below:
qemu-system-x86_64 --enable-kvm -m 2048 -smp 4 -hda /root/ubuntu-16.04.img -usb -usbdevice tablet -device virtio-net-pci,netdev=nic0,mac=00:16:3e:60:0a:50 -netdev tap,id=nic0,script=/etc/kvm/qemu-ifup -serial stdio

Expected Result
=============
Guest boot up without any issues.

Actual Result
===========
Guest boot up with GPU hang.

Analysis & Root Cause
===================
[    6.939377] [drm] GPU HANG: ecode 8:0:0xfffffffe, reason: No progress on rend er ring, action: reset
[    6.939378] [drm] GPU hangs can indicate a bug anywhere in the entire gfx stack, including userspace.
[    6.939379] [drm] Please file a _new_ bug report on bugs.freedesktop.org against DRI -> DRM/Intel
[    6.939379] [drm] drm/i915 developers can then reassign to the right compone t if it's not a kernel issue.
[    6.939380] [drm] The gpu crash dump is required to analyze gpu hangs, so please always attach it.
[    6.939380] [drm] GPU crash dump saved to /sys/class/drm/card0/error
[    6.939429] drm/i915: Resetting chip after gpu hang
[    7.869346] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts:    (null)
[    7.964728] systemd[1]: systemd 229 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +                                                                                                                     XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN)
[    7.964756] systemd[1]: Detected virtualization qemu.
[    7.964758] systemd[1]: Detected architecture x86-64.
[    7.965211] systemd[1]: Set hostname to <gvt-ub16>.
[    8.092829] systemd[1]: Listening on fsck to fsckd communication Socket.
[    8.092875] systemd[1]: Reached target User and Group Name Lookups.
[    8.092882] systemd[1]: Reached target Remote File Systems (Pre).
[    8.092902] systemd[1]: Listening on udev Kernel Socket.
[    8.092945] systemd[1]: Created slice System Slice.
[    8.092976] systemd[1]: Listening on Syslog Socket.
[    8.278376] EXT4-fs (sda1): re-mounted. Opts: errors=remount-ro
[    8.349290] systemd-journald[205]: Received request to flush runtime journal from PID 1
[    8.481575] random: crng init done
[    8.889146] drm/i915: Resetting chip after gpu hang
[    9.060874] Adding 4000764k swap on /dev/sda5.  Priority:-1 extents:1 across:  4000764k
[   10.364525] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[   10.873269] drm/i915: Resetting chip after gpu hang
[   12.923317] drm/i915: Resetting chip after gpu hang
[   14.905157] drm/i915: Resetting chip after gpu hang
[   16.900984] drm/i915: Resetting chip after gpu hang
[   18.885876] drm/i915: Resetting chip after gpu hang
[   20.921190] drm/i915: Resetting chip after gpu hang
[   22.905154] drm/i915: Resetting chip after gpu hang
Comment 1 Terrence Xu 2016-12-08 14:37:14 UTC
Created attachment 128381 [details]
dmesg-guest.log

Attach the full guest dmesg log.
Comment 2 yann 2016-12-09 08:40:04 UTC
(In reply to Terrence Xu from comment #1)
> Created attachment 128381 [details]
> dmesg-guest.log
> 
> Attach the full guest dmesg log.

Terence can you also attach gpu crash dump located at /sys/class/drm/card0/error ?
Comment 3 Terrence Xu 2017-01-03 14:00:03 UTC
Created attachment 128724 [details]
error.log

error.log from /sys/class/drm/card0/error.
Comment 4 Terrence Xu 2017-01-03 14:02:25 UTC
(In reply to yann from comment #2)
> (In reply to Terrence Xu from comment #1)
> > Created attachment 128381 [details]
> > dmesg-guest.log
> > 
> > Attach the full guest dmesg log.
> 
> Terence can you also attach gpu crash dump located at
> /sys/class/drm/card0/error ?

Hi yann, sorry I was uploaded the error log failed and I didn't catch it previously, I just upload the error log successfully.
Comment 5 Jari Tahvanainen 2017-01-31 09:23:49 UTC
Yann - please take a look on gpu crash dump.
Comment 6 yann 2017-01-31 12:51:24 UTC
Terence, please use latest drm-tip ()

it looks like we have a loop of MI_SEMAPHORE_MBOX command (wait upon the semaphore value remained ?) with blitter ring (any relationship with bug 54226?)

You may also try to add drm.debug=0xe in your boot command line to collect more data.
Comment 7 Chris Wilson 2017-01-31 12:57:44 UTC
Still a kvm bug.
Comment 8 Terrence Xu 2017-02-03 06:28:56 UTC
(In reply to yann from comment #6)
> Terence, please use latest drm-tip ()
> 
> it looks like we have a loop of MI_SEMAPHORE_MBOX command (wait upon the
> semaphore value remained ?) with blitter ring (any relationship with bug
> 54226?)
> 
> You may also try to add drm.debug=0xe in your boot command line to collect
> more data.

Hi Yann, use the newest drm-tip, it still blocked by the Bug99025 :(, I just updated the newest status on it.
Comment 9 Chuanxiao Dong 2017-02-07 03:39:30 UTC
This issue actually happened together with bug#99025. It is also a regression. As no GVT code is involved (updated in comment: https://bugs.freedesktop.org/show_bug.cgi?id=99025#c10), suggest i915 team to help to investigate.
Comment 10 Daniel Vetter 2017-02-09 12:16:53 UTC
Regression = we need the bisect.
Comment 11 XiongZhang 2017-02-12 06:41:24 UTC
I use upstream 4.9 kernel as host and guest kernel, I also see GPU hang in guest.
git bisect tell me the first bad commit is:
commit 49ef5294cda256aa5496ba56bbf859d3c7a17e07
Author: Chris Wilson <chris@chris-wilson.co.uk>
Date:   Thu Aug 18 17:17:00 2016 +0100

    drm/i915: Move fence tracking from object to vma

    In order to handle tiled partial GTT mmappings, we need to associate the
    fence with an individual vma.

    v2: A couple of silly drops replaced spotted by Joonas

    Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
    Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Comment 12 Chris Wilson 2017-02-14 09:32:23 UTC
It appears to be using stolen memory in the guest, which is not mediated by gvt-d nor does gvt-d fixup the reported size of stolen memory to be zero.
Comment 13 XiongZhang 2017-02-15 07:28:15 UTC
Indeed guest are using stolen memory. After disabling stolen memory in i915, gpu hang disappear.
While only one guest is using GPU in gvt-d, why guest couldn't use it like host ?
Comment 14 XiongZhang 2017-02-27 01:47:10 UTC
*** Bug 99025 has been marked as a duplicate of this bug. ***
Comment 15 XiongZhang 2017-02-27 01:49:37 UTC
Fixed this issue in Qemu with: 
c2b2e15 vfio/pci-quirks.c: Disable stolen memory for igd VFIO
Comment 16 XiongZhang 2017-04-06 10:50:59 UTC
As the qemu patch in commit#15 breaks windows igd driver load and has been reverted. So this bug will exist again.
Comment 17 XiongZhang 2017-04-06 10:55:26 UTC
Created attachment 130723 [details]
guest igd pci config

From guest igd pci config.txt, we could know stolen memory base is 0xda000000, size is 32M, it's  range is 0xda000000 ~ 0xdbffffff
Comment 18 XiongZhang 2017-04-06 11:03:45 UTC
Created attachment 130726 [details]
host_dmesg_with_stolen_memory_enabled

This message is gotten from host when I boot a guest drm-intel-nightly kernel with stolen memory enabled, I saw gpu hang in guest and reproduced this issue. From this message we could see there are tons of IOMMU read/write fault at 0xdaxxxxx where are stolen memory. So igd are accessing stolen memory.
Comment 19 XiongZhang 2017-04-06 11:08:59 UTC
Created attachment 130727 [details]
host_dmesg_with_stolen_memory_disabled

This message is gotten from host when I boot the same guest drm-intel-nightly kernel with stolen memory disabled.  I didn't see a gpu hang in guest and this bug disappear. From This message we didn't see any IOMMU interrupt fault.
Comment 20 XiongZhang 2017-04-06 11:17:26 UTC
Created attachment 130728 [details]
host_dmesg_boot_guest_4.8_kernel

This host message is gotten when I boot a default ubuntu 16.10 kernel based on v4.8 with stolen memory enabled. From this dmesg, we could see there are several IOMMU interrupt faults at stolen memory. But there are tons of IOMMU interrupt faults at 0xff20c000 where I don't know who owns. while I didn't see gpu hang in this guest during the boot process.  
The access to 0xff20c0000 should be a bug and it has been fixed in drm-intel-nightly.
Comment 21 XiongZhang 2017-04-06 11:22:56 UTC
Anyway once guest enable stolen memory, we could see iommu fault which means igd fail in read/write the target address and couldn't successfully finish the command. So it is better that we could disable stolen memory in kvm guest to prevent this iommu error.
Comment 22 yann 2017-04-11 15:00:31 UTC
Reference to XiongZhang's patch: https://patchwork.freedesktop.org/series/21818/
Comment 23 XiongZhang 2017-04-13 02:46:16 UTC
Created attachment 130820 [details]
/sys/class/drm/card0/error when gpu hang

the error message:
[    9.853024] [drm] GPU HANG: ecode 9:1:0xeeffefa9, in Xorg [871], reason: No progress on blitter ring, action: reset
[    9.853027] [drm] GPU hangs can indicate a bug anywhere in the entire gfx stack, including userspace.
[    9.853028] [drm] Please file a _new_ bug report on bugs.freedesktop.org against DRI -> DRM/Intel
[    9.853030] [drm] drm/i915 developers can then reassign to the right component if it's not a kernel issue.
[    9.853031] [drm] The gpu crash dump is required to analyze gpu hangs, so please always attach it.
[    9.853032] [drm] GPU crash dump saved to /sys/class/drm/card0/error
[    9.853223] [drm:i915_reset_and_wakeup [i915]] resetting chip

[   16.896147] drm/i915: Resetting chip after gpu hang
[   16.898713] [drm:i915_gem_reset [i915]] context Xorg[871]/0 marked guilty (score 10) banned? no
[   16.898756] [drm:i915_gem_reset [i915]] resetting blitter ring to restart from tail of request 0x1
[   16.898764] BUG: unable to handle kernel NULL pointer dereference at 0000000000000070
[   16.901051] IP: reset_common_ring+0x8b/0x120 [i915]
[   16.902392] PGD 0

[   16.903508] Oops: 0000 [#1] SMP
[   16.904262] Modules linked in: crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper cryptd joydev input_leds serio_raw i2c_piix4 qemu_fw_cfg mac_hid parport_pc ppdev lp parport ip_tables x_tables autofs4 hid_generic usbhid hid i915(OE) video i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm e1000 floppy psmouse pata_acpi
[   16.907024] CPU: 1 PID: 257 Comm: kworker/1:3 Tainted: G           OE   4.11.0-rc3nightly+ #2
[   16.907710] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.1-0-g8891697-prebuilt.qemu-project.org 04/01/2014
[   16.908587] Workqueue: events_long i915_hangcheck_elapsed [i915]
[   16.909124] task: ffff89e25e6dcb00 task.stack: ffffa6d2807e4000
[   16.909582] RIP: 0010:reset_common_ring+0x8b/0x120 [i915]
[   16.910081] RSP: 0018:ffffa6d2807e7b60 EFLAGS: 00010206
[   16.910474] RAX: 0000000000003f00 RBX: ffff89e2bbd1f680 RCX: ffff89e2777539f4
[   16.911098] RDX: 0000000000003f40 RSI: ffff89e25d92e000 RDI: ffff89e2777539c0
[   16.911656] RBP: ffffa6d2807e7b78 R08: 0000000000000001 R09: 0000000000000001
[   16.912202] R10: 00000000ffffffff R11: 0000000000000448 R12: ffff89e25df56000
[   16.912749] R13: 0000000000000000 R14: ffff89e2bbd1f680 R15: ffff89e25dba4d38
[   16.913250] FS:  0000000000000000(0000) GS:ffff89e2bfd00000(0000) knlGS:0000000000000000
[   16.913864] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   16.914322] CR2: 0000000000000070 CR3: 00000000434a0000 CR4: 00000000003406e0
[   16.914922] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   16.915520] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   16.916097] Call Trace:
[   16.916324]  i915_gem_reset+0xbe/0x380 [i915]
[   16.916679]  ? intel_uncore_forcewake_put+0x48/0x60 [i915]
[   16.917111]  ? bit_wait_io+0x60/0x60
[   16.917374]  i915_reset+0xde/0x170 [i915]
[   16.917754]  i915_reset_and_wakeup+0x17c/0x190 [i915]
[   16.918164]  i915_handle_error+0x1db/0x220 [i915]
[   16.918536]  ? scnprintf+0x4d/0x90
[   16.918790]  hangcheck_declare_hang+0xdb/0x100 [i915]
[   16.919211]  i915_hangcheck_elapsed+0x2a9/0x2d0 [i915]
[   16.919613]  process_one_work+0x1fc/0x4b0
[   16.919937]  worker_thread+0x4b/0x500
[   16.920246]  kthread+0x101/0x140
[   16.920478]  ? process_one_work+0x4b0/0x4b0
[   16.920775]  ? kthread_create_on_node+0x60/0x60
[   16.921148]  ret_from_fork+0x2c/0x40
[   16.921450] Code: c8 01 00 00 89 50 14 48 8b 83 80 00 00 00 8b 93 c8 01 00 00 89 50 28 48 8b bb 80 00 00 00 e8 6d 2c 00 00 4d 8b ac 24 60 02 00 00 <49> 8b 45 70 48 39 43 70 74 5d 4d 85 ed 74 20 48 c7 c0 e0 70 61
[   16.922917] RIP: reset_common_ring+0x8b/0x120 [i915] RSP: ffffa6d2807e7b60
[   16.923488] CR2: 0000000000000070
[   16.923762] ---[ end trace 2a911e958217bc24 ]---
error
Comment 24 XiongZhang 2017-05-03 09:07:44 UTC
I did the git bisect again, and the target commit is:
commit c58b735fc762e891481e92af7124b85cb0a51fce
Author: Chris Wilson <chris@chris-wilson.co.uk>
Date:   Thu Aug 18 17:16:57 2016 +0100

    drm/i915: Allocate rings from stolen

    If we have stolen available, make use of it for ringbuffer allocation.
    Previously this was restricted to !llc platforms, as writing to stolen
    requires a GGTT mapping - but now that we have partial mappable support,
    the mappable aperture isn't quite so precious so we can use it more
    freely and ringbuffers are a good user for the otherwise wasted stolen.

    Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
    Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>


After reverting this patch from drm-intel-nightly, I didn't see gpu hang during boot process. So we could confirm this issue is caused by i915 access stolen memory when i915 is as kvm guest where kvm doesn't supply ept and guest domain iommu mapping for stolen memory.
Comment 25 Ricardo 2017-05-09 16:32:11 UTC
Adding tag into "Whiteboard" field - ReadyForDev
The bug still active
*Status is correct
*Platform is included
*Feature is included
*Priority and Severity correctly set
*Logs included
Comment 26 Joonas Lahtinen 2017-05-30 10:44:47 UTC
The i915 driver behavior is correct, if stolen memory is reported, it'll be used. If it's reported as zero, it shall not be used. It was agreed that QEMU/KVM will be changed to correctly report stolen memory as zero and bugs will be fixed outside of i915 to accommodate that. A step two will be to add support to QEMU/KVM for stolen memory passthrough and after that report the stolen memory size as non-zero. Both changes do not require any modifications to i915.


Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.