Bug 68224

Summary: [radeonsi] Serious Sam3 is segfaulting (LLVM assert)
Product: Mesa Reporter: Laurent carlier <lordheavym>
Component: Drivers/Gallium/radeonsiAssignee: Default DRI bug account <dri-devel>
Status: RESOLVED FIXED QA Contact:
Severity: blocker    
Priority: medium CC: arek.rusi, darkbasic, grantipak
Version: git   
Hardware: x86-64 (AMD64)   
OS: Linux (All)   
Whiteboard:
i915 platform: i915 features:
Attachments: shader dump from Serious Sam 3 with RADEON_DUMP_SHADERS=1
Possible Fix
Shader dump with patch applied
Patch #2
LLVM Patch #3
Shader dump with 3 patches applied
SGPR register spilling patch
SGPR register spilling patch v2
Work Around
dmesg with fix (stalker)
steam's log (SS3)
log with R600_DEBUG=ps,vs
Debugging Patch
v2 log with R600_DEBUG=ps,vs
v3 log with R600_DEBUG=ps,vs
Possible Fix
v4 log with R600_DEBUG=ps,pv
Patch 1
Patch 2
screenshot

Description Laurent carlier 2013-08-17 15:35:11 UTC
Created attachment 84175 [details]
shader dump from Serious Sam 3  with RADEON_DUMP_SHADERS=1

* mesa:
  OpenGL renderer string: Gallium 0.4 on AMD PITCAIRN
  OpenGL version string: 2.1 Mesa 9.3.0-devel (git-703a2f4)
* llvm-3.4svn r188604

The game "segfault" just before the logo shows up:

Starting program: /home/lordh/.local/share/Steam/SteamApps/common/Serious Sam 3/Bin/Sam3.orig 
warning: Could not load shared library symbols for linux-gate.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
cat: /etc/debian_version: Aucun fichier ou dossier de ce type
Setting breakpad minidump AppID = 41070
Steam_SetMinidumpSteamID:  Caching Steam ID:  76561197992653503 [API loaded no]
[New Thread 0xf5914b40 (LWP 6689)]
[New Thread 0xf14acb40 (LWP 6693)]
AL lib: pulseaudio.c:612: Context did not connect: Access denied
[New Thread 0xeff0ab40 (LWP 6698)]
[Thread 0xeff0ab40 (LWP 6698) exited]
AL lib: pulseaudio.c:612: Context did not connect: Access denied
[New Thread 0xeff0ab40 (LWP 6699)]
[Thread 0xeff0ab40 (LWP 6699) exited]
AL lib: pulseaudio.c:612: Context did not connect: Access denied
[New Thread 0xeff0ab40 (LWP 6700)]
[New Thread 0xef0e9b40 (LWP 6701)]
[New Thread 0xee832b40 (LWP 6702)]
[Thread 0xee832b40 (LWP 6702) exited]
Installing breakpad exception handler for appid(gameoverlayui)/version(20130816170723_client)
Installing breakpad exception handler for appid(gameoverlayui)/version(1.0_client)
Installing breakpad exception handler for appid(gameoverlayui)/version(1.0_client)
Installing breakpad exception handler for appid(gameoverlayui)/version(1.0_client)
[0817/170723:WARNING:proxy_service.cc(958)] PAC support disabled because there is no system implementation
Sam3.orig: AMDGPUInstrInfo.cpp:109: virtual void llvm::AMDGPUInstrInfo::storeRegToStackSlot(llvm::MachineBasicBlock&, llvm::MachineBasicBlock::iterator, unsigned int, bool, int, const llvm::TargetRegisterClass*, const llvm::TargetRegisterInfo*) const: Assertion `!"Not Implemented"' failed.

* gdb backstrace:

#0  0xf7fdb425 in __kernel_vsyscall ()
#1  0xf7d6f7e6 in raise () from /usr/lib32/libc.so.6
#2  0xf7d70f73 in abort () from /usr/lib32/libc.so.6
#3  0xf7d687d7 in __assert_fail_base () from /usr/lib32/libc.so.6
#4  0xf7d68887 in __assert_fail () from /usr/lib32/libc.so.6
#5  0xf17a86a9 in llvm::AMDGPUInstrInfo::storeRegToStackSlot(llvm::MachineBasicBlock&, llvm::MachineBasicBlock::bundle_iterator<llvm::MachineInstr, llvm::ilist_iterator<llvm::MachineInstr> >, unsigned int, bool, int, llvm::TargetRegisterClass const*, llvm::TargetRegisterInfo const*) const () from /usr/lib32/libLLVM-3.4svn.so
#6  0xf1d50389 in ?? () from /usr/lib32/libLLVM-3.4svn.so
#7  0xf1d52773 in ?? () from /usr/lib32/libLLVM-3.4svn.so
#8  0xf1e6aafe in ?? () from /usr/lib32/libLLVM-3.4svn.so
#9  0xf1e595ad in llvm::RegAllocBase::allocatePhysRegs() () from /usr/lib32/libLLVM-3.4svn.so
#10 0xf1e6e099 in ?? () from /usr/lib32/libLLVM-3.4svn.so
#11 0xf1dd2d52 in llvm::MachineFunctionPass::runOnFunction(llvm::Function&) () from /usr/lib32/libLLVM-3.4svn.so
#12 0xf1a0973f in llvm::FPPassManager::runOnFunction(llvm::Function&) () from /usr/lib32/libLLVM-3.4svn.so
#13 0xf1a09b68 in llvm::FPPassManager::runOnModule(llvm::Module&) () from /usr/lib32/libLLVM-3.4svn.so
#14 0xf1a09e1c in llvm::MPPassManager::runOnModule(llvm::Module&) () from /usr/lib32/libLLVM-3.4svn.so
#15 0xf1a0a05b in llvm::PassManagerImpl::run(llvm::Module&) () from /usr/lib32/libLLVM-3.4svn.so
#16 0xf1a0a166 in llvm::PassManager::run(llvm::Module&) () from /usr/lib32/libLLVM-3.4svn.so
#17 0xf1f42937 in ?? () from /usr/lib32/libLLVM-3.4svn.so
#18 0xf1f42cc0 in LLVMTargetMachineEmitToMemoryBuffer () from /usr/lib32/libLLVM-3.4svn.so
#19 0xf36c9b47 in radeon_llvm_compile (M=M@entry=0x14da15b8, binary=binary@entry=0xffff5ed4, gpu_family=0xf38653d2 "pitcairn", dump=dump@entry=0) at radeon_llvm_emit.c:124
#20 0xf36bbd82 in si_compile_llvm (rctx=rctx@entry=0x9845218, shader=shader@entry=0x14da88f8, mod=0x14da15b8) at radeonsi_shader.c:1478
#21 0xf36bc5cd in si_pipe_shader_create (ctx=ctx@entry=0x9845218, shader=shader@entry=0x14da88f8) at radeonsi_shader.c:1620
#22 0xf36c5576 in si_shader_select (ctx=ctx@entry=0x9845218, sel=sel@entry=0x14e27610, dirty=dirty@entry=0x0) at si_state.c:2393
#23 0xf36c58bb in si_create_shader_state (ctx=0x9845218, state=<optimized out>, pipe_shader_type=1) at si_state.c:2435
#24 0xf34f9313 in st_translate_fragment_program (st=st@entry=0x98c45a8, stfp=stfp@entry=0x14ed0c58, key=key@entry=0xffffb148) at ../../src/mesa/state_tracker/st_program.c:768
#25 0xf34fa6f6 in st_get_fp_variant (st=st@entry=0x98c45a8, stfp=stfp@entry=0x14ed0c58, key=key@entry=0xffffb148) at ../../src/mesa/state_tracker/st_program.c:805
#26 0xf34c1f33 in update_fp (st=0x98c45a8) at ../../src/mesa/state_tracker/st_atom_shader.c:92
#27 0xf34be81e in st_validate_state (st=st@entry=0x98c45a8) at ../../src/mesa/state_tracker/st_atom.c:221
#28 0xf34d5eac in st_draw_vbo (ctx=0x98e7760, prims=0xffffb298, nr_prims=1, ib=0xffffb288, index_bounds_valid=1 '\001', min_index=0, max_index=3, tfb_vertcount=0x0) at ../../src/mesa/state_tracker/st_draw.c:210
#29 0xf34a92f9 in vbo_handle_primitive_restart (ctx=ctx@entry=0x98e7760, prim=prim@entry=0xffffb298, nr_prims=nr_prims@entry=1, ib=ib@entry=0xffffb288, index_bounds_valid=index_bounds_valid@entry=1 '\001', min_index=min_index@entry=0, 
    max_index=max_index@entry=3) at ../../src/mesa/vbo/vbo_exec_array.c:585
#30 0xf34aa4a9 in vbo_validated_drawrangeelements (ctx=ctx@entry=0x98e7760, mode=mode@entry=4, index_bounds_valid=index_bounds_valid@entry=1 '\001', start=start@entry=0, end=end@entry=3, count=count@entry=6, type=type@entry=5123, 
    indices=indices@entry=0x0, basevertex=basevertex@entry=0, numInstances=numInstances@entry=1, baseInstance=baseInstance@entry=0) at ../../src/mesa/vbo/vbo_exec_array.c:1004
#31 0xf34aa8a4 in vbo_exec_DrawRangeElementsBaseVertex (mode=mode@entry=4, start=start@entry=0, end=end@entry=3, count=count@entry=6, type=type@entry=5123, indices=indices@entry=0x0, basevertex=basevertex@entry=0)
    at ../../src/mesa/vbo/vbo_exec_array.c:1112
#32 0xf34aa99f in vbo_exec_DrawRangeElements (mode=4, start=0, end=3, count=6, type=5123, indices=0x0) at ../../src/mesa/vbo/vbo_exec_array.c:1132
Comment 1 Tom Stellard 2013-08-22 20:16:35 UTC
Created attachment 84473 [details] [review]
Possible Fix

Can you try this mesa patch.  If it doesn't fix the crash, can you re-post the shader dump with the patch applied.
Comment 2 Laurent carlier 2013-08-22 21:06:10 UTC
(In reply to comment #1)
> Created attachment 84473 [details] [review] [review]
> Possible Fix
> 
> Can you try this mesa patch.  If it doesn't fix the crash, can you re-post
> the shader dump with the patch applied.

With the patch applied it still fail
Comment 3 Laurent carlier 2013-08-22 21:07:07 UTC
Created attachment 84475 [details]
Shader dump with patch applied
Comment 4 Tom Stellard 2013-08-24 00:19:04 UTC
Created attachment 84543 [details] [review]
Patch #2
Comment 5 Tom Stellard 2013-08-24 00:19:46 UTC
Created attachment 84544 [details] [review]
LLVM Patch #3

If you apply all three of these patches, does it fix the crash?
Comment 6 Laurent carlier 2013-08-24 07:25:49 UTC
(In reply to comment #5)
> Created attachment 84544 [details] [review] [review]
> LLVM Patch #3
> 
> If you apply all three of these patches, does it fix the crash?

It's better, now it crash before the intro start (so latter), new shader dump attached
Comment 7 Laurent carlier 2013-08-24 08:59:21 UTC
Created attachment 84551 [details]
Shader dump with 3 patches applied
Comment 8 Laurent carlier 2013-08-28 14:51:05 UTC
Same problem with Brütal Legend (after the intro):

Buddha.bin.x86: AMDGPUInstrInfo.cpp:109: virtual void llvm::AMDGPUInstrInfo::storeRegToStackSlot(llvm::MachineBasicBlock&, llvm::MachineBasicBlock::iterator, unsigned int, bool, int, const llvm::TargetRegisterClass*, const llvm::TargetRegisterInfo*) const: Assertion `!"Not Implemented"' failed.
Stack dump:
0.      Running pass 'Function Pass Manager' on module 'tgsi'.
1.      Running pass 'Greedy Register Allocator' on function '@main'
Comment 9 Tom Stellard 2013-08-29 18:55:35 UTC
Created attachment 84880 [details] [review]
SGPR register spilling patch

Can you test this patch?  It should fix the crash, but it may hang your GPU.
Comment 10 Laurent carlier 2013-08-29 20:24:03 UTC
(In reply to comment #9)
> Created attachment 84880 [details] [review] [review]
> SGPR register spilling patch
> 
> Can you test this patch?  It should fix the crash, but it may hang your GPU.

It's not really better; instead of the assert, it gives a GPU lockup
Comment 11 Michel Dänzer 2013-08-30 16:37:45 UTC
(In reply to comment #9)
> .addImm(Lane);
	
This results in the lane number being encoded verbatim in the VSRC1 instruction field, which I don't think is correct?

I tried adding 0x80 to make it an inline literal, but Sanctuary still locks up.

Another potential gotcha I noticed in the ISA documentation is that the V_READ/WRITELANE instructions don't respect the EXEC mask.
Comment 12 Tom Stellard 2013-09-06 19:39:16 UTC
Created attachment 85372 [details] [review]
SGPR register spilling patch v2

Can you try this v2 patch?  It fixes the bug Michel found plus another one.
Comment 13 Laurent carlier 2013-09-06 20:15:32 UTC
(In reply to comment #12)
> Created attachment 85372 [details] [review] [review]
> SGPR register spilling patch v2
> 
> Can you try this v2 patch?  It fixes the bug Michel found plus another one.

Same result as patch v1, GPU lockup instead of llvm assert; Sanctuary demo gives also a GPU lockup
Comment 14 Tom Stellard 2013-09-13 22:39:49 UTC
(In reply to comment #13)
> (In reply to comment #12)
> > Created attachment 85372 [details] [review] [review] [review]
> > SGPR register spilling patch v2
> > 
> > Can you try this v2 patch?  It fixes the bug Michel found plus another one.
> 
> Same result as patch v1, GPU lockup instead of llvm assert; Sanctuary demo
> gives also a GPU lockup

Sanctuary doesn't lockup for me with this patch, but the only thing visible is the torch.  Everything else is black.  What settings are you using with Sanctuary?
Comment 15 Laurent carlier 2013-09-14 13:21:09 UTC
(In reply to comment #14)
> (In reply to comment #13)
> > (In reply to comment #12)
> > > Created attachment 85372 [details] [review] [review] [review] [review]
> > > SGPR register spilling patch v2
> > > 
> > > Can you try this v2 patch?  It fixes the bug Michel found plus another one.
> > 
> > Same result as patch v1, GPU lockup instead of llvm assert; Sanctuary demo
> > gives also a GPU lockup
> 
> Sanctuary doesn't lockup for me with this patch, but the only thing visible
> is the torch.  Everything else is black.  What settings are you using with
> Sanctuary?

It's locking up just at the end of the loading screen, so it's the default settings
Comment 16 Tom Stellard 2013-09-14 20:15:36 UTC
(In reply to comment #15)
> (In reply to comment #14)
> > (In reply to comment #13)
> > > (In reply to comment #12)
> > > > Created attachment 85372 [details] [review] [review] [review] [review] [review]
> > > > SGPR register spilling patch v2
> > > > 
> > > > Can you try this v2 patch?  It fixes the bug Michel found plus another one.
> > > 
> > > Same result as patch v1, GPU lockup instead of llvm assert; Sanctuary demo
> > > gives also a GPU lockup
> > 
> > Sanctuary doesn't lockup for me with this patch, but the only thing visible
> > is the torch.  Everything else is black.  What settings are you using with
> > Sanctuary?
> 
> It's locking up just at the end of the loading screen, so it's the default
> settings

Are you using libtxc-dxtn and/or do you have floating-point textures enabled?
Comment 17 Laurent carlier 2013-09-14 20:32:26 UTC
(In reply to comment #16)
> 
> Are you using libtxc-dxtn and/or do you have floating-point textures enabled?

I have both enabled, (lib32-)libtxc-dxtn and floating-point textures.
Comment 18 Michel Dänzer 2013-09-17 10:04:37 UTC
(In reply to comment #12)
>  - Use correct register flags, so we don't overwrite the low dword
>    when restoring multi-dword registers.

This doesn't seem to result in any difference for the code generated for the first Sanctuary shader which spills registers.
Comment 19 Laurent carlier 2013-11-05 15:13:22 UTC
Now Sanctuary works fine with llvm-3.4svn r194070 / mesa-git 86cdff5 @ PITCAIN

Brutal Legend / Serious Sam 3 are still problematic
Comment 20 Arek Ruśniak 2013-11-20 21:39:30 UTC
With Mesa 10.1.0-devel (git-0601598) & LLVM 3.4 (svn-r195266) still quits in the same moment but log is a little bit different:
Sam3: AMDGPUInstrInfo.cpp:113: virtual void llvm::AMDGPUInstrInfo::storeRegToStackSlot(llvm::MachineBasicBlock&, llvm::MachineBasicBlock::iterator, unsigned int, bool, int, const llvm::TargetRegisterClass*, const llvm::TargetRegisterInfo*) const: Assertion `!"Not Implemented"' failed.

I don't try Brutal Legent, but Stalker game(by the wine) has the same issue and log changes from AMDGPUInstrInfo.cpp:113 to AMDGPUInstrInfo.cpp:109 after update mesa/llvm too.
Comment 21 Tom Stellard 2013-11-22 04:44:51 UTC
Created attachment 89616 [details] [review]
Work Around

Hopefully this patch will fix the crash, but it will likely result in some things being rendered incorrectly.  Can you test it and let me know if it makes the game playable.
Comment 22 Arek Ruśniak 2013-11-22 12:05:33 UTC
Created attachment 89631 [details]
dmesg with fix (stalker)

Hi, i can't get log from SS3 because GPU crashes and can't get tty or ssh.
Wine doesn't give me any usefull logs but gpu was restarted succesfully and i have dmesg output. 

I applied fix on top on latest llvm(3.5). Should i use 3.4 branch instead?

I'll try redirect steam's logs to non-temporary filesystem and maybe that gives me some info.
Comment 23 Arek Ruśniak 2013-11-22 14:39:11 UTC
Created attachment 89640 [details]
steam's log (SS3)

My PC hung. This is log from the game - nothing exciting as far as i know.
When i find out anything useful i will call you.
Comment 24 Tom Stellard 2013-11-22 20:50:14 UTC
(In reply to comment #23)
> Created attachment 89640 [details]
> steam's log (SS3)
> 
> My PC hung. This is log from the game - nothing exciting as far as i know.
> When i find out anything useful i will call you.

If you could get a dump with the environment variable

R600_DEBUG=ps,vs

That would help.
Comment 25 Arek Ruśniak 2013-11-23 00:30:25 UTC
Created attachment 89659 [details]
log with R600_DEBUG=ps,vs

llvm with Tom's fix
Comment 26 Tom Stellard 2013-11-23 03:34:12 UTC
Created attachment 89671 [details] [review]
Debugging Patch

Do you still have a GPU hang with this patch?  Can you post the output of R600_DEBUG=ps,vs with this patch?
Comment 27 Arek Ruśniak 2013-11-23 13:54:40 UTC
Created attachment 89680 [details]
v2 log with R600_DEBUG=ps,vs

yes it still hung.
Comment 28 Arek Ruśniak 2013-11-24 19:36:35 UTC
Created attachment 89719 [details]
v3 log with R600_DEBUG=ps,vs

i have to downgrade libdrm to 2.4.48 because it gives me randomly gpu crash.
Comment 29 Alexandre Demers 2013-11-25 03:19:21 UTC
may be related to bug 71859
Comment 30 Tom Stellard 2013-11-26 04:32:11 UTC
Created attachment 89806 [details] [review]
Possible Fix

Can you try this patch?  If you get a lockup or a hang, please post the output of R600_DEBUG=ps,vs
Comment 31 Arek Ruśniak 2013-11-26 10:41:12 UTC
Created attachment 89825 [details]
v4 log with R600_DEBUG=ps,pv

it's not hung anymore, but new log:

Sam3: SIInstrInfo.cpp:98: virtual void llvm::SIInstrInfo::copyPhysReg(llvm::MachineBasicBlock&, llvm::MachineBasicBlock::iterator, llvm::DebugLoc, unsigned int, unsigned int, bool) const: Assertion `AMDGPU::SReg_32RegClass.contains(SrcReg)' failed.

BTW stalker & xonotic works perfectly with this fix. 
(32bit 10.1 git-ddc77c5 + llvm 3.5 r195722+fix)
Comment 32 darkbasic 2013-11-26 11:58:20 UTC
Stalker? When did they release stalker for linux???
Comment 33 Arek Ruśniak 2013-11-26 12:26:48 UTC
We have wine don't you forget;) Now [thanks Tom] it works with dynamic lighting etc.
Comment 34 Tom Stellard 2013-11-26 18:29:36 UTC
Created attachment 89853 [details] [review]
Patch 1
Comment 35 Tom Stellard 2013-11-26 18:30:01 UTC
Created attachment 89854 [details] [review]
Patch 2

Can you try these two patches?
Comment 36 Tom Stellard 2013-11-26 18:30:43 UTC
(In reply to comment #35)
> Created attachment 89854 [details] [review] [review]
> Patch 2
> 
> Can you try these two patches?

You should apply them both together.
Comment 37 Arek Ruśniak 2013-11-27 12:07:30 UTC
yes, it works. Some textures flicker, but i don't know how it work before. 
I don't see visual regressions in unigine-*, xonotic, lightmarks.
I check driver performance with/without patches later. 

thanks a lot Tom
Comment 38 Arek Ruśniak 2013-11-27 16:28:37 UTC
performance is fine, slow but not slower than before :)
Comment 39 Tom Stellard 2013-11-28 00:03:34 UTC
The fix was committed to the LLVM tree: r195880
Comment 40 darkbasic 2013-11-28 13:12:51 UTC
Created attachment 89954 [details]
screenshot

It works but it's unplayable, 25 fps with shit graphic settings and 5 fps with Ultra graphic settings.

Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.