Bug 109939

Summary: After upgrade mesa to 19.0.0 stop working the game Rise of the Tomb Raider
Product: Mesa Reporter: mikhail.v.gavrilov
Component: Drivers/Vulkan/radeonAssignee: mesa-dev
Status: RESOLVED NOTOURBUG QA Contact: mesa-dev
Severity: normal    
Priority: medium CC: massabnaeem160, mike
Version: unspecified   
Hardware: Other   
OS: All   
Whiteboard:
i915 platform: i915 features:
Attachments: htop screenshot
for i in {1..10}; do gdb --batch --ex "t a a bt" -pid=`pidof RiseOfTheTombRaider` &>bt$i.txt; sleep 0.1; done
Rise of the Tomb Raider

Description mikhail.v.gavrilov 2019-03-08 09:18:08 UTC
Created attachment 143585 [details]
htop screenshot

After game menu when I select continue.
The splash screen show infinitely loading.
In task manager `CBackgroundWork` threads eat all CPU cores.
Comment 1 mikhail.v.gavrilov 2019-03-08 09:19:10 UTC
Created attachment 143586 [details]
for i in {1..10}; do gdb --batch --ex "t a a bt" -pid=`pidof RiseOfTheTombRaider` &>bt$i.txt; sleep 0.1; done
Comment 2 Timothy Arceri 2019-03-08 09:32:36 UTC
Mesa 19.0 has not been released yet. You need to provide more detail on what exactly you are testing with.
Comment 3 mikhail.v.gavrilov 2019-03-08 09:36:17 UTC
I am tested versions rc1 - rc6
Comment 4 mikhail.v.gavrilov 2019-03-08 09:51:31 UTC
PS Yet another native Linux Vulkan game "F1 2017" that's made by Feral is works, but first time the game starts too long around of 10 minutes, but it not consuming all CPU cores.
Comment 5 Timur Kristóf 2019-03-08 10:26:49 UTC
Mikhail, are you sure this is a mesa issue? It could be a bug in the game. On my machine (RX 570, with mesa 18.3 RADV) Rise of the Tomb Raider can spend a lot of time (sometimes 10-15 minutes) on the loading screen, but it loads eventually.

Could we ask someone from Feral to look into this?
Comment 6 Alex Smith 2019-03-08 11:32:28 UTC
The loading screen will block for a while on pipeline creation for the level that you are loading (that's what the CBackgroundWork threads are doing). If you see much faster loading times on an older version, that sounds like there has been some regression in pipeline creation times.
Comment 7 Mike Lothian 2019-03-08 12:49:17 UTC
Out of interest, if you wait long enough does it eventually complete and it it faster the next time you load if you get it to complete

Might be that you're relying on caching and each time you install a newer RC you have to rebuild the cache
Comment 8 Alex Smith 2019-03-08 14:18:06 UTC
What LLVM version are you using and what CPU do you have? I'm unable to reproduce this on current 19.0 branch (8ab2fc8c963955818d6cbc1db47b76d9bffb9eb6) using LLVM 7, loading times seem as expected to me with the disk cache cleared (on an i7 6700, a couple of minutes to load Soviet Installation, which has the most pipelines of all levels in the game).
Comment 9 fin4478 2019-03-09 04:35:31 UTC
The Rise of the Tomb Raider game works fine with my
System:
  Host: ryzenpc Kernel: 5.0.0 x86_64 bits: 64 Desktop: Xfce 4.13.2 
  Distro: Debian GNU/Linux buster/sid 
Machine:
  Type: Desktop Mobo: ASUSTeK model: PRIME B350M-K v: Rev X.0x 
  serial: <root required> UEFI [Legacy]: American Megatrends v: 4207 
  date: 12/07/2018 
CPU:
  6-Core: AMD Ryzen 5 1600 type: MT MCP speed: 2967 MHz 
Graphics:
  Device-1: AMD Ellesmere [Radeon RX 470/480] driver: amdgpu v: kernel 
  Display: x11 server: X.Org 1.20.4 driver: amdgpu 
  resolution: 3840x2160~60Hz 
  OpenGL: 
  renderer: Radeon RX 570 Series (POLARIS10 DRM 3.27.0 5.0.0 LLVM 9.0.0) 
  v: 4.5 Mesa 19.1.0-devel - padoka PPA
Comment 10 Timur Kristóf 2019-03-10 14:09:32 UTC
(In reply to Alex Smith from comment #8)
> What LLVM version are you using and what CPU do you have? I'm unable to
> reproduce this on current 19.0 branch
> (8ab2fc8c963955818d6cbc1db47b76d9bffb9eb6) using LLVM 7, loading times seem
> as expected to me with the disk cache cleared (on an i7 6700, a couple of
> minutes to load Soviet Installation, which has the most pipelines of all
> levels in the game).

I have an Intel i7-8550U, 16 GB RAM and an AMD RX 570 with 4 GB or VRAM (connected through a TB3 port). On this system, RotTR is mostly limited by the GPU, so once the game is loaded it hardly uses any CPU, but it utilizes the GPU to nearly 100%.

On some heavy areas (Soviet Installation and Geothermal Valley are the most heavy ones in my experience), the game takes a long time to load, but sometimes (regardless of which area I'm in), it never loads, just keeps showing the loading screen and spins up the CPU. (By never, I mean about 20-30 minutes when I run out of patience and kill it.) When this happens, it helps to clear the contents of "~/.local/share/feral-interactive/Rise of the Tomb Raider" (except for the savegames of course). After that the game always loads (though takes a bit longer than usual, I assume it has to rebuild some cache).

I'm running Fedora 29 with mesa 18.3.4 and LLVM 7.0.1.
Comment 11 mikhail.v.gavrilov 2019-03-11 05:48:03 UTC
(In reply to Mike Lothian from comment #7)
> Out of interest, if you wait long enough does it eventually complete and it
> it faster the next time you load if you get it to complete
> 
> Might be that you're relying on caching and each time you install a newer RC
> you have to rebuild the cache

I was wait a bit more and level are was loaded after while. But this is abnormal to wait 20 minutes for loading each level on high end level hardware.
CPU: Ryzen 2700X
RAM: 32GB
GPU: Vega 64
SSD: Optane 905P

I remember that when the game out on Linux i am not experienced this problem on worse hardware.
Comment 12 Alex Smith 2019-03-11 14:00:21 UTC
Sounds like this is potentially a game issue.

Could both of you zip up the contents of the preferences folder ("~/.local/share/feral-interactive/Rise of the Tomb Raider") once you're getting the issue, and send it to us at support@feralinteractive.com so that we can investigate?

Also, Mikhail - since Timur said that clearing out that folder (except for the save games) makes the problem go away temporarily, could you try doing so as well to see if that helps? Please make a copy of the old contents to send to us before trying.

Thanks!
Comment 13 mikhail.v.gavrilov 2019-03-11 19:04:19 UTC
Created attachment 143627 [details]
Rise of the Tomb Raider
Comment 14 mikhail.v.gavrilov 2019-03-11 19:10:01 UTC
Alex, I am attach "Rise of the Tomb Raider" archive here.
For reproducing problem after continue the game.
Try fast travel to locations: "The Gulag" and "Valley Farmstead"
Comment 15 Timur Kristóf 2019-03-11 20:46:32 UTC
Will attach the contents of the preferences folder next time I encounter this problem.
Comment 16 Alex Smith 2019-03-25 14:51:02 UTC
I think I have an idea what's happening here. Pipeline compilation will be a lot slower using debug builds of Mesa.

Mikhail, it looks from your backtraces like you're using a debug build you've done yourself? If so could you use a release build instead (pass -Dbuildtype=release to meson, the default is debugoptimized).

Timur, are you by any chance using Fedora's packages? It appears that they are currently shipping a debug build in there, and I can reproduce this when using those :( I will file a bug with Fedora.
Comment 17 Alex Smith 2019-03-25 15:28:33 UTC
Reported: https://bugzilla.redhat.com/show_bug.cgi?id=1692426
Comment 18 Mike Lothian 2019-03-25 15:42:44 UTC
Will --buildtype=plane cause any issues? It's the default here on Gentoo

meson --buildtype plain --libdir lib64 --localstatedir /var/lib --prefix /usr --sysconfdir /etc --wrap-mode nodownload -Dplatforms=x11,surfaceless,wayland,drm -Dllvm=true -Dlmsensors=true -Dlibunwind=false -Dgallium-nine=true -Dgallium-va=true -Dva-libs-path=/usr/lib64/va/drivers -Dgallium-vdpau=true -Dgallium-xa=false -Dgallium-xvmc=false -Dgallium-opencl=disabled -Dosmesa=none -Dbuild-tests=false -Dglx=dri -Dshared-glapi=true -Ddri3=true -Degl=true -Dgbm=true -Dgles1=false -Dgles2=true -Dglvnd=false -Dselinux=false -Dvalgrind=false -Ddri-drivers=i965 -Dgallium-drivers=radeonsi,swrast -Dvulkan-drivers=amd,intel -Dvulkan-overlay-layer=true --buildtype plain -Db_ndebug=true /var/tmp/portage/media-libs/mesa-9999/work/mesa-9999 /var/tmp/portage/media-libs/mesa-9999/work/mesa-9999-abi_x86_64.amd64
Comment 19 Alex Smith 2019-03-25 15:48:17 UTC
You have -Db_ndebug=true which I think is what Fedora is missing - their build definitely does not have that as I see code in the binaries that is only compiled when NDEBUG is not set.

However you don't get any optimization flag set, unless Gentoo is setting it with CFLAGS?
Comment 20 Mike Lothian 2019-03-25 16:05:40 UTC
Seems to be setting it with CFLAGS, though I guess that might need to change
Comment 21 mikhail.v.gavrilov 2019-03-25 19:52:31 UTC
Thanks, Alex.
I tested new mesa Fedora build (mesa-19.0.0-2.fc31)
I noted that "-Db_ndebug=true" also helps another games rinning by DXVK (steam play):
- Dishonored series becomes playeble, yes shutters accidently may occurs but without "-Db_ndebug=true" shutters occurs every time then protogonist change angle of view, with "-Db_ndebug=true" they occurs much rarely.

- Homefront: The Revolution without "-Db_ndebug=true" just stucking in main menu. Start game button not woking not on keyboard <Enter> neither on gamepad.

About Rise of the Tomb Raider I can say that the game started after 3 minutes. I suppose there is some more space for optimizations here.
Comment 22 Dylan Baker 2019-03-25 20:56:17 UTC
There is a meson bug here, I've proposed a patch here: https://github.com/mesonbuild/meson/pull/5143
Comment 23 Alex Smith 2019-03-26 09:28:24 UTC
Thanks Dylan! I'll close this as not a Mesa bug now.
Comment 24 Timur Kristóf 2019-04-01 12:21:14 UTC
Sorry for my late reply. Yes, I use the Fedora packages, so it might be the same problem. The issue is still there with 19.0.1 on Fedora 30. In fact, I can confirm that the game overall performs much worse with this than 18.x on Fedora 29.

However, besides the packaging problem there appears to be at least some issue with the game itself; in the cases when it actually never loads. I'll try to follow up on that with the Feral devs, once this packaging issue is gone.
Comment 25 mikhail.v.gavrilov 2019-04-02 06:23:32 UTC
Alex, can you look please at this issue:
https://bugzilla.redhat.com/show_bug.cgi?id=1694938
There about warning "Current CPU governor may impact performance" even when package gamemode is installed.
Looks like affected only "Rise of the Tomb Raider" because with "DiRT 4" all ok.

Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.