Created attachment 121127 [details]
Screenshot of the glitch
When enabling both soft shadows and antialiasing in Knights of the Old Republic 2 , there are lines visible in addition to the shadows.
Here is an apitrace showing the problem:
http://constexpr.org/tmp/KotOR2-shadows-radeonsi.trace.xz (37 MiB)
The same trace (and the game) render fine with Catalyst 15.12.
Disabling either soft shadows or antialiasing also fixes the rendering.
GPU: Radeon HD 7950 (TAHITI)
Mesa 11.2.0-devel (git-d018619)
From what I can tell the problem first appears in e.g. draw call 1919333 in my trace - the stencil buffer has value 127 everywhere except for parts covered by the shadow volumes. The color buffer however contains jagged lines afterwards in addition to the pixels covered by the shadows.
NB: The game also has fog issues like those worked around in the ATI_fs implementation  for the first KotOR game, but they are also present with Catalyst and even in one of the official screenshots  on the game's Steam store page. I think its fair to assume that that is a game bug, but if anyone wants to try it out with another driver / HW, here is another apitrace:
http://constexpr.org/tmp/KotOR2-fog-radeonsi.trace.xz (35 MiB)
With the ATI_fs patch series v4 applied (even the ones that are not in master yet), I see no such glitches in KotOR 2. Shadows are correct, and fog is correct. The only problem I see is that enabling soft shadows makes the shadows so faint they are barely visible, but I think that's a bug in the game.
I'm using latest Mesa from git, llvm 3.7.1, Wine 1.9.6.
Note that Steam only offers the remastered version of KotOR2 now, which doesn't use ATI_fs anymore. The screenshots on the store page may be from the old version though.
> Note that Steam only offers the remastered version of KotOR2 now, which
> doesn't use ATI_fs anymore.
Yes, I'm talking about the remastered version (the only one that has a native linux release). Did you test the original?
Anyway, I'm still seeing the glitches here. Soft shadows aren't any fainter than hard shadows as far as I can tell.
GPU: R9 380X (tonga)
Mesa 11.3.0-devel (git-9076e04)
(In reply to Daniel Scharrer from comment #2)
> > Note that Steam only offers the remastered version of KotOR2 now, which
> > doesn't use ATI_fs anymore.
> Yes, I'm talking about the remastered version (the only one that has a
> native linux release). Did you test the original?
> Anyway, I'm still seeing the glitches here. Soft shadows aren't any fainter
> than hard shadows as far as I can tell.
> GPU: R9 380X (tonga)
> Mesa 11.3.0-devel (git-9076e04)
> LLVM r265023
My last comment was about the original version. Now I checked the remastered, and it seems fine, although I haven't gotten very far yet. I see that they fixed the faint soft shadows in the remaster.
I took a look at your traces. The shadow trace renders fine here. I examined both call 1919333 and the final frame, and I see no corruption. In the fog trace some surfaces get no fog, while others get too much. I'm very far from reaching that point in the remastered version, so I can't verify this in-game.
Hi Daniel, thanks for the apitrace and preliminary investigation. I can confirm that the trace shows a rendering error on Verde, Bonaire, and Tonga. A workaround for now is to set R600_DEBUG=nohyperz.
Could you please test whether the problem appears also on lower MSAA levels?
almos, I'm surprised that you're not seeing the glitch. What's your system setup and hardware?
A proper fix may involve disabling ALLOW_EXPCLEAR for MSAA stencil.
Hi, thanks for looking into this.
The problem happens on all MSAA levels the game offers: 2, 4 and 6 (I assume this maps to 8). The there are less lines with lower MSAA levels though, or at least they are less noticeable.
R600_DEBUG=nohyperz does get rid of the glitches for all MSAA levels.
Created attachment 122748 [details]
(In reply to Nicolai Hähnle from comment #5)
> almos, I'm surprised that you're not seeing the glitch. What's your system
> setup and hardware?
I attached the saved image of frame 296 (which contains call 1919333 mentioned in the original report) from the shadows trace. I don't see corruption on it, even though MSAA seems to be enabled in glretrace. My card is a R9 270x, and I have Mesa 11.1.2, LLVM 3.7.1.
This makes sense. Mesa 11.1 does not contain the stencil fast clear changes. I'm going to try to get some clarification on what's going wrong here.
I confirm that the glitch occurs with Mesa 11.3-dev, and the patch "radeonsi: work around an MSAA fast stencil clear problem" posted to mesa-dev fixes it.
I can also confirm that the patch fixes the issue with the game.
The workaround is now in Mesa master, commit 8c43c06e0463515c1339d44cbb8f78169e6a06fb