Bug 75276

Created attachment 94675 [details] [review]
SGPR spilling fix

Some of these crashed may be caused by a bug in SGPR spilling rather than lack of VGPR spilling.

Tom put up an LLVM Git branch: http://cgit.freedesktop.org/~tstellar/llvm/log/?h=si-spill-fixes

(In reply to comment #2)
> Tom put up an LLVM Git branch:
> http://cgit.freedesktop.org/~tstellar/llvm/log/?h=si-spill-fixes

Newest mesa git needs a newer llvm than your branch:


gallivm/lp_bld_debug.cpp: In function 'size_t disassemble(const void*, llvm::raw_ostream&)':
gallivm/lp_bld_debug.cpp:255:79: error: no matching function for call to 'llvm::Target::createMCDisassembler(const llvm::MCSubtargetInfo&, llvm::MCContext&) const'

Whom do I have to buy a beer or three to get this implemented yesterday?
Tom is there anything I could do to help you along with this bug?

Updated branch for testing:

http://cgit.freedesktop.org/~tstellar/llvm/log/?h=si-spill-fixes-v2

Updated v3 branch here:
http://cgit.freedesktop.org/~tstellar/llvm/log/?h=si-spill-fixes-v3

Created attachment 98982 [details]
tesseract segfault when setting morphological AA to ultra

Okay, the good news is:  works pretty well with your v3 branch.

With current latest svn, Tesseract 2014-05-11 just segfaults on loading maps somewhere.

But with your v3 branch it works pretty well.

The only crash with your v3 branch I got (every time) when setting morphological anti aliasing to ultra either ingame, or when starting a game. bt full attached. Maybe it's unrelated, but maybe not...?

Created attachment 99169 [details] [review]
VGPR spilling work-around

*** Bug 75361 has been marked as a duplicate of this bug. ***

*** Bug 75211 has been marked as a duplicate of this bug. ***

*** Bug 73320 has been marked as a duplicate of this bug. ***

Created attachment 99186 [details] [review]
VGPR Spill Work Around v2

This patch should build.

Created attachment 99187 [details] [review]
VGPR Spill Work Around v3 with possible antichamber crash fix

Here is an updated version of the patch, which may fix the antichamber crash.

The v3 patch doesnt fix Antichamber's crash to desktop.
But with this patch it actually shows the VGPR messages:

LLVM triggered Diagnostic Handler: SIInstrInfo::storeRegToStackSlot - Can't spill VGPR!
LLVM triggered Diagnostic Handler: SIInstrInfo::loadRegToStackSlot - Can't retrieve spilled VGPR!

Here's the start of the backtrace for reference:

Program received signal SIGSEGV, Segmentation fault.
0xf504f671 in std::pair<llvm::SlotIndex, llvm::SlotIndex>::operator= (this=0x450d2ff4, __p=...)
    at /usr/lib/gcc/x86_64-pc-linux-gnu/4.7.3/include/g++-v4/bits/stl_pair.h:153
153             second = __p.second;
(gdb) bt
#0  0xf504f671 in std::pair<llvm::SlotIndex, llvm::SlotIndex>::operator= (this=0x450d2ff4, __p=...)
    at /usr/lib/gcc/x86_64-pc-linux-gnu/4.7.3/include/g++-v4/bits/stl_pair.h:153
#1  0xf505e8c5 in llvm::IntervalMapImpl::NodeBase<std::pair<llvm::SlotIndex, llvm::SlotIndex>, llvm::LiveInterval*, 16u>::copy<16u> (
    this=0x44ee9644, Other=..., i=250679, j=250678, Count=4294967295)
    at /mnt/daten/Daten/Repositories/llvm/include/llvm/ADT/IntervalMap.h:231
#2  0xf505f3ad in llvm::IntervalMapImpl::NodeBase<std::pair<llvm::SlotIndex, llvm::SlotIndex>, llvm::LiveInterval*, 16u>::moveLeft (
    this=0x44ee9644, i=1, j=0, Count=4294967295) at /mnt/daten/Daten/Repositories/llvm/include/llvm/ADT/IntervalMap.h:242
#3  0xf505ec81 in llvm::IntervalMapImpl::NodeBase<std::pair<llvm::SlotIndex, llvm::SlotIndex>, llvm::LiveInterval*, 16u>::erase (
    this=0x44ee9644, i=0, j=1, Size=0) at /mnt/daten/Daten/Repositories/llvm/include/llvm/ADT/IntervalMap.h:263
#4  0xf505e175 in llvm::IntervalMapImpl::NodeBase<std::pair<llvm::SlotIndex, llvm::SlotIndex>, llvm::LiveInterval*, 16u>::erase (
    this=0x44ee9644, i=0, Size=0) at /mnt/daten/Daten/Repositories/llvm/include/llvm/ADT/IntervalMap.h:270
#5  0xf505d1ee in llvm::IntervalMap<llvm::SlotIndex, llvm::LiveInterval*, 16u, llvm::IntervalMapInfo<llvm::SlotIndex> >::iterator::erase (
    this=0xffff6348) at /mnt/daten/Daten/Repositories/llvm/include/llvm/ADT/IntervalMap.h:1876
#6  0xf505c846 in llvm::LiveIntervalUnion::extract (this=0x44ee9640, VirtReg=...) at LiveIntervalUnion.cpp:68
#7  0xf50655ef in llvm::LiveRegMatrix::unassign (this=0x44d26e60, VirtReg=...) at LiveRegMatrix.cpp:96
#8  0xf511e81f in (anonymous namespace)::RAGreedy::tryLastChanceRecoloring (this=0x44243000, VirtReg=..., Order=..., NewVRegs=..., 
    FixedRegisters=..., Depth=0) at RegAllocGreedy.cpp:2067
#9  0xf511f224 in (anonymous namespace)::RAGreedy::selectOrSplitImpl (this=0x44243000, VirtReg=..., NewVRegs=..., FixedRegisters=..., 
    Depth=0) at RegAllocGreedy.cpp:2285
#10 0xf511eb60 in (anonymous namespace)::RAGreedy::selectOrSplit (this=0x44243000, VirtReg=..., NewVRegs=...) at RegAllocGreedy.cpp:2144

Tesseract does still crash with si-spill-fixes-v4 if I set "Shadow resolution" to "Ultra" or "Global Illumination" to "High" or "Morphological AA" to "Ultra".

If someone is interested here is my rebased si-spill-fixes-v4: https://github.com/darkbasic/llvm/tree/master-si-spill-fixes-v4

I will try to rebase it from time to time, you cal also find the matching compiler-rt, clang and clang-tools-extra: https://github.com/darkbasic

(In reply to comment #15)
> Tesseract does still crash with si-spill-fixes-v4 if I set "Shadow
> resolution" to "Ultra" or "Global Illumination" to "High" or "Morphological
> AA" to "Ultra".

Have you tested with the patch attached to this bug?

(In reply to comment #14)
> The v3 patch doesnt fix Antichamber's crash to desktop.
> But with this patch it actually shows the VGPR messages:
> 

With the patch applied can you run the game with R600_DEBUG=ps,vs,gs and post the output?

Created attachment 99236 [details]
antichamber log with r600_debug

> Have you tested with the patch attached to this bug?

Still not because I was late and llvm takes a long time to compile, but I will test it tomorrow.

Created attachment 99254 [details] [review]
VGPR Spill Work Around v4 with possible antichamber crash fix

Can you try this patch?  It should fix the antichamber crash.

Patch v4 fixes the crash in Antichamber:
LLVM triggered Diagnostic Handler: SIInstrInfo::storeRegToStackSlot - Can't spill VGPR!
LLVM triggered Diagnostic Handler: SIInstrInfo::loadRegToStackSlot - Can't retrieve spilled VGPR!
radeon_llvm_compile: Processing Diag Flag
LLVM failed to compile shader
EE si_state.c:2133 si_shader_select - Failed to build shader variant (type=1) 1

Here is tesseract with R600_DEBUG=ps,vs,gs:
http://bpaste.net/show/287559/

I forgot to say I'm using attachment 99254 [details] [review].

With si-spill-fixes-v4 Painkiller Hell & Damnation does not crash, but it still stutters alot making the game completely unplayable. What's the problem?

I've pushed a work-arond for the crash to LLVM trunk, so if you use the latest version of LLVM from svn/git these games should not crash, but there may be some mis-rendering.

When I come up with a proper fix, I will post it to this bug (this could be a while).

(In reply to comment #26)
> I've pushed a work-arond for the crash to LLVM trunk, so if you use the
> latest version of LLVM from svn/git these games should not crash, but there
> may be some mis-rendering.

It's a bit unclear what that means and because of that it's unclear what are actual rendering bugs in other parts of the driver.

When you talk about mis-rendering, are these the kind of problems we should see:

https://www.youtube.com/watch?v=6i0vJW5k5js&hd=1
(The setting is "Texture Quality" and "Medium" works better with less artifacts and "High" produces much more artifacts)

Or is this something different that should be reported separately?
(I'm seeing the same problems in the "Distance" game, closed alpha version)

Also, when starting a map a rendering thread in Sanctum 2 segfaults, but I also don't know if I can blame it on this workaround, maybe someone else can try whether this can be reproduced first?

> LLVM ERROR: ran out of registers during register allocation

I get this error when trying out Unreal Engine demo buid. Coredumps always around the same moment for me, from 9th to 11th second. Here's the link to the demo in case anyone wants to apitrace and analyze it. http://ue4linux.raxxy.com/effects_cave_demo.zip
(Demo comes from https://wiki.unrealengine.com/Linux_Demos)

I don't know if will help a lot, but a crash is "always reproducible" with LLVM 3.4.2 using my OLAND 8750M with "Civilization V", always on a leader scene.

Using LLVM from GIT I don't reproduce this, but is VERY slow, like darkbasic said. Is only on the leader scene.

(In reply to comment #28)
> > LLVM ERROR: ran out of registers during register allocation

I do not get this message on upstream llvm recent revisions.

But every demo segfaults. Mostly in LLVMBuildBitCast().

Running a demo looks like this:


$ DRI_PRIME=1 ./Effects
Using binned.
4.3.0-0+UE4 7038 3077 379 0
Signal 11 caught.
EngineCrashHandler: Signal=11


Exiting due to error
Starting ../../../Engine/Binaries/Linux/CrashReportClient
[1]    10932 abort (core dumped)  DRI_PRIME=1 ./Effects


This is still a problem with register spilling that just looks different, right?
Should I compile with debug symbols and get a complete backtrace or wouldn't that provide any new information?


(By the way, applying this small patch makes it render almost completely correct on intel: https://bugs.freedesktop.org/show_bug.cgi?id=78716#c10)

(In reply to comment #30)
> But every demo segfaults. Mostly in LLVMBuildBitCast().
[...]
> This is still a problem with register spilling that just looks different,
> right?

No, that sounds like bug 81834.

Created attachment 103583 [details]
backtrace of unreal engine effects demo with debug

(In reply to comment #31)
> (In reply to comment #30)
> > But every demo segfaults. Mostly in LLVMBuildBitCast().
> [...]
> > This is still a problem with register spilling that just looks different,
> > right?
> 
> No, that sounds like bug 81834.

So I built mesa with debug symbols and I guess debugging enables some assertions because now fails at some assertion about Register.Index stuff. Full backtrace attached.

(bug 81834 doesn't say what versions he uses. I use upstream llvm 214022 and mesa git with the small patch I mentioned)

Of course it could be both problems at the same time.

(In reply to comment #32)
> > No, that sounds like bug 81834.
> 
> So I built mesa with debug symbols and I guess debugging enables some
> assertions because now fails at some assertion about Register.Index stuff.

Yep, that looks like bug 81834. I'm actually not sure why I'm not failing the assertion in Mesa before the one in LLVM.


> (bug 81834 doesn't say what versions he uses.

I normally use current Git of everything.

(In reply to comment #33)
> > So I built mesa with debug symbols and I guess debugging enables some
> > assertions because now fails at some assertion about Register.Index stuff.
> 
> Yep, that looks like bug 81834.

BTW, you may be able to work around this by reverting Mesa commit f4b0ab7afd83c811329211eae8167c9bf238870c, but then you may run into bug 80880 instead.

Hello. I'm just wondering if this has been resolved in Mesa 10.3, and if not, what needs to be done to get it there. Thank you!

It must be solved in LLVM, not mesa. You can use my forward-ported llvm git branch if you need register spilling: https://github.com/darkbasic/llvm

Comment on attachment 103583 [details]
backtrace of unreal engine effects demo with debug

If with "this" you mean the Unreal Engine troubles, they are solved and they should run.

Looks like I stepped on similar issue at http://llvm.org/bugs/show_bug.cgi?id=20738 when dealing with OpenCL and rather filed bug against LLVM (not that like if they have right sections for AMD GPUs in theie bugzilla, but it seems to be LLVM issue, yeah?...).

Basically, running simple CL memory benchmark causes huge flood by error messages from LLVM and then GPU locks up, which is not fun at all, obviously.

This bug appears to be resolved for me now, as of LLVM 3.5 and Mesa 10.2.7 on Arch Linux. Thanks for the hard work!!!

It's still reproducible in Painkiller H&D with LLVM r202464 (ie. current trunk).

While some previously broken PP effects now work, dynamic lighting is still messed up and there's plenty of this in the log:

LLVM triggered Diagnostic Handler: SIInstrInfo::storeRegToStackSlot - Can't spill VGPR!
LLVM triggered Diagnostic Handler: SIInstrInfo::loadRegToStackSlot - Can't retrieve spilled VGPR!

s/r202464/commit 3666e7f4c161c50e5f6dcb0e015ca16bf69fb941/ ;-)

The Witcher 2 is full of spilling warnings too.

FYI, while other games doesn't crash anymore with llvm-3.5/mesa-10.2.7 (painkiller, the cave, serious sam), brütal legends horribly crash's my system with this versions. With older versions it just closes/crash's the game, but now it crash's the whole system and i always have to reset it.

(In reply to comment #43)
> FYI, while other games doesn't crash anymore with llvm-3.5/mesa-10.2.7
> (painkiller, the cave, serious sam), brütal legends horribly crash's my
> system with this versions.

The Brütal Legend issue might be unrelated to this bug report then and should be tracked in a separate report for now.

Created attachment 106527 [details] [review]
Possible fix

Here is a patch to test.

Created attachment 106569 [details]
Build failure with "Implement VGPR register spilling v3" patch

What llvm version is that patch supposed to apply to?

Build fails with a current checkout of http://llvm.org/git/llvm.git master - see the attached log.

(In reply to comment #45)
> Created attachment 106527 [details] [review] [review]
> Possible fix
> 
> Here is a patch to test.

Hi Tom,

The Mesa driver doesn't set LDS_SIZE for VS, GS, ES, and HS shaders, so I don't think the spilling to LDS will work on those.

Reproducible crash in Pioneer (remake of Elite 2) with LLVM-3.5/Mesa-10.3

Hello,

I don't know if that helps, but the bug is still present on the brand new openSUSE 13.2 beta1, for example. (With libLLVM 3.4.2 and mesa 10.3.0)

In my case, it can be easily reproduced when trying to launch Tomb Raider 2013 through Wine or Playonlinux. The game seems to start normally but crashes after a few seconds and I get this error message :
'LLVM ERROR: ran out of registers during register allocation'

For the record, I have an AMD APU A8-7600 with Radeon R7 Graphics.

(In reply to comment #49)
> Hello,
> 
> I don't know if that helps, but the bug is still present on the brand new
> openSUSE 13.2 beta1, for example. (With libLLVM 3.4.2 and mesa 10.3.0)

This is LLVM bug and it is partially fixed in LLVM 3.5

Not sure if it is exactly related to this bug, but I can't play NeverWinterNight, I get a lot of blocs like these:

LLVM triggered Diagnostic Handler: SIInstrInfo::storeRegToStackSlot - Do not know how to spill register
LLVM triggered Diagnostic Handler: SIInstrInfo::loadRegFromStackSlot - Do not know how to restore register
LLVM triggered Diagnostic Handler: Ran out of VGPRs for spilling SGPR
then:
radeon_llvm_compile: Processing Diag Flag
LLVM failed to compile shader
EE si_state.c:2282 si_shader_select - Failed to build shader variant (type=1) 1
and then back to above

I'm on git for llvm, clang, mesa (radeonsi).
Using Linux 3.17.2, x64 if that matters.

(In reply to John from comment #51)
> Not sure if it is exactly related to this bug, but I can't play
> NeverWinterNight, I get a lot of blocs like these:
> 

What do you mean can't play?  Does it crash, or is nothing being rendered?

It stays on a loading page, and I see those lines in my console.
The game does not freeze, it just "loads" forever.

Of course wine being what it is, maybe the main issue lies in wine and not the driver, but I see no special wine error on the console.

Please forget my last 2 comments, I still get these messages in the terminal, so there's obviously an issue there, but I found some game workarounds that allow me to play the game, so I guess these messages are unrelated to my issue.

Thanks!

I have this working for some simple cases, can you test these mesa and llvm trees together:

http://cgit.freedesktop.org/~tstellar/llvm/log/?h=vgpr-spilling-Jan07-2014

http://cgit.freedesktop.org/~tstellar/mesa/log/?h=vgpr-spilling-Jan07-2014

(In reply to Tom Stellard from comment #55)
> I have this working for some simple cases, can you test these mesa and llvm
> trees together:
> 
> http://cgit.freedesktop.org/~tstellar/llvm/log/?h=vgpr-spilling-Jan07-2014
> 
> http://cgit.freedesktop.org/~tstellar/mesa/log/?h=vgpr-spilling-Jan07-2014

 I tried those for bug 79155 both medium/high global illumination now works fine there.

Oh nice. +20% in Unigine and +30% in unreal 4.5 demos with no any image corruptions with vgpr-spilling-Jan07-2014 branch on my R9 280X.

I have tested perf-Jan-08-2015 tstellar branch,
and I get a failure to compile a Unigine heaven shader
under nine.
There is the same error when reverting the last patch of the branch (subreg liveness).

LLVM ERROR: Not supported instr: <MCInst 1908 <MCOperand Reg:3210> <MCOperand Imm:8> <MCOperand Reg:2044> <MCOperand Reg:74>>

1908 seems to correspond to SI_SPILL_V96_RESTORE

I'll include the TGSI shader, and next the llvm IR

Created attachment 112178 [details]
TGSI failing shader

Created attachment 112179 [details]
llvm failing shader

The VGPR spilling patches have been pushed.  Is anyone still having problems with latest mesa and llvm from git?

I am not able to do proper testing right now nor in the next couple of days, but I guess you can close this bug report and let peoples open a new one if they enncounter issues with a specific game.

Great plan. :) Awesome work, Tom!