Bug 84570 - Borderlands 2/Pre-Sequel: Constant frame rate drops while playing; really bad with additionl lighting
Summary: Borderlands 2/Pre-Sequel: Constant frame rate drops while playing; really bad...
Status: RESOLVED FIXED
Alias: None
Product: Mesa
Classification: Unclassified
Component: Mesa core (show other bugs)
Version: git
Hardware: x86-64 (AMD64) Linux (All)
: medium normal
Assignee: mesa-dev
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks: 77449
  Show dependency treegraph
 
Reported: 2014-10-01 22:25 UTC by Kai
Modified: 2015-09-12 15:16 UTC (History)
4 users (show)

See Also:
i915 platform:
i915 features:


Attachments
Screenshot with GALLIUM_HUD=fps showing FPS drops (DynamicLights=false) (2.77 MB, image/png)
2014-10-06 10:52 UTC, Kai
Details
Screenshot with GALLIUM_HUD=fps showing FPS drops (DynamicLights=true) (2.06 MB, image/png)
2014-10-06 13:09 UTC, Kai
Details
Shader dump with R600_DEBUG=gs,vs,fs,ps (DynamicLights=false) (1.22 MB, application/x-xz)
2014-10-06 16:07 UTC, Kai
Details
Screenshot with GALLIUM_HUD=fps,requested-VRAM+VRAM-usage,requested-GTT+GTT-usage showing FPS drops (DynamicLights=true) (2.61 MB, image/png)
2014-10-07 15:44 UTC, Kai
Details
Screenshot with GALLIUM_HUD=fps,requested-VRAM+VRAM-usage,requested-GTT+GTT-usage showing FPS drops (DynamicLights=false) (2.81 MB, image/png)
2014-10-07 15:45 UTC, Kai
Details
Screenshot with GALLIUM_HUD=fps,requested-VRAM+VRAM-usage,requested-GTT+GTT-usage showing FPS drops (DynamicLights=true) (139.14 KB, image/png)
2014-10-08 18:46 UTC, Kai
Details
Screenshot with GALLIUM_HUD=fps,requested-VRAM+VRAM-usage,requested-GTT+GTT-usage showing FPS drops (DynamicLights=true) (126.53 KB, image/png)
2014-10-15 15:31 UTC, Kai
Details

Description Kai 2014-10-01 22:25:19 UTC
Running Borderlands 2 with the option "DynamicLights" set to true, causes constant frame rate drops (GALLIUM_HUD=fps shows ~30 as the usual lower end, when there are many new objects, it drops to single digit numbers), when new objects come into the visible area, ie. you see a new enemy or a team mate is running into your LOS. After a second or so, you're back to ~40-60 FPS.

I'm seeing this on (Debian testing as a base):
GPU: Hawaii PRO [Radeon R9 290] (ChipID = 0x67b1)
Mesa: Git:master/5ccdc23a86
libdrm: Git:master/00847fa48b
LLVM: SVN:trunk/r218506 (3.6 devel)
X.Org: 2:1.16.1-1
Linux: Git:~agdf5/linux:drm-next-3.17-rebased-on-fixes:fa78380797 (calls itself 3.16-rc6)
Firmware: <http://people.freedesktop.org/~agd5f/radeon_ucode/>
> 9e05820da42549ce9c89d147cf1f8e19  /lib/firmware/updates/3.16.0-rc6-citadel/radeon/hawaii_ce.bin
> c8bab593090fc54f239c8d7596c8d846  /lib/firmware/updates/3.16.0-rc6-citadel/radeon/hawaii_mc.bin
> 3618dbb955d8a84970e262bb2e6d2a16  /lib/firmware/updates/3.16.0-rc6-citadel/radeon/hawaii_me.bin
> c000b0fc9ff6582145f66504b0ec9597  /lib/firmware/updates/3.16.0-rc6-citadel/radeon/hawaii_mec.bin
> 0643ad24b3beff2214cce533e094c1b7  /lib/firmware/updates/3.16.0-rc6-citadel/radeon/hawaii_pfp.bin
> ba6054b7d78184a74602fd81607e1386  /lib/firmware/updates/3.16.0-rc6-citadel/radeon/hawaii_rlc.bin
> 11288f635737331b69de9ee82fe04898  /lib/firmware/updates/3.16.0-rc6-citadel/radeon/hawaii_sdma.bin
> 284429675a5560e0fad42aa982965fc2  /lib/firmware/updates/3.16.0-rc6-citadel/radeon/hawaii_smc.bin
libclc: Git:master/5b48f170c8
DDX: Git:master/fbf575cb01 + Patch from <http://lists.x.org/archives/xorg-driver-ati/2014-August/026534.html>

But according to Aspyr (the developer of the Linux port), this is also a problem with the Intel driver (see <http://steamcommunity.com/app/49520/discussions/0/616189106749909144/#c616189742697847998>), therefore I selected Mesa Core as the component.

Let me know, if you need anything else to debug this.
Comment 1 Ian C. Bullard 2014-10-03 16:38:41 UTC
A little more information (I'm a developer at Aspyr Media):

DynamicLights' name is misleading, when set to false *all* lighting except for ambient light is turned off.  That's why everything looks so flat when it's off.  As you can imagine, that turns off a large part of the rendering engine.  From our experience, most Intel hardware just can't handle it.  Most newer AMD hardware should be able to handle DynamicLights=true.

If you need more information, please let me know.
Comment 2 Kai 2014-10-03 17:52:52 UTC
After playing a bit more, I'm also seeing framerate drops every time I meet at least one of the following conditions:
- moving around fast, preferrably turning (examples: driving, running or jumping and turning)
- lots of new objects/overlays spawn or become visible/active (example: enemies jump down in front of you from a ledge)
- LOD gets above a certain point in an area with many objects/enemy (example: you see a hord of enemies in a camp far ahead and charge ahead, shortly before you reach the point where you want to engage)

Activating DynamicLights in the configuration just makes it worse. Interestingly enough, in the non-dropped case, my FPS are roughly the same ~50 with DynamicLighs and almost 60 without. When the rate drops it's like the game halts for a second and GALLIUM_HUD shows a single-digit FPS rate.

Without DynamicLights the game stays playable, even if the drops are annoying. At least I haven't managed to die due to such a framerate drop yet. ;-)


(In reply to Ian C. Bullard from comment #1)
> DynamicLights' name is misleading, when set to false *all* lighting except
> for ambient light is turned off.

I changed the bug title. Let me know, whether this is more correct or if I should change it to something else.

> From our experience, most Intel hardware just can't handle it.

Does that mean, this bug should be reassigned to the Radeon driver?

> If you need more information, please let me know.

Just guessing here, but a test case (ie. an ApiTrace) demonstrating the problem might help. Otherwise the devs would need access to the game.
Comment 3 Tapani Pälli 2014-10-06 04:54:03 UTC
For Intel, you can try to run game with environment variable 'INTEL_DEBUG=perf' to see some hints what might be going on, as example you can see if shader recompilation happens and lots of other useful information. A log output from that and apitrace would be the most ideal to start debugging this.
Comment 4 Kai 2014-10-06 10:52:10 UTC
Created attachment 107417 [details]
Screenshot with GALLIUM_HUD=fps showing FPS drops (DynamicLights=false)

(In reply to Tapani Pälli from comment #3)
> For Intel, you can try to run game with environment variable
> 'INTEL_DEBUG=perf' to see some hints what might be going on, as example you
> can see if shader recompilation happens and lots of other useful
> information. A log output from that and apitrace would be the most ideal to
> start debugging this.

I can't do the INTEL_DEBUG part, since I've got a Radeon GPU, but I did make an Apitrace (without DynamicLights). That is however 1.5 GB in size and the apitrace trim command is now running for > 20 minutes (I checked with qapitrace which frames are outside the menu and passed that with --frames=<START>-<END> to apitrace trim). Is there a way to get a reasonably sized chunk, that would help you, in a speedier manner? Otherwise, this might take a bit longer, especially since I guess you'll want an Apitrace with DynamicLights as well.

In the meantime, I've attached a screenshot, which shows the framerate drops. It seems to get worse, the longer you play, but that might just be imagination. The screenshot is scaled down from my native resolution, because the bug tracker won't allow the larger file.
Comment 5 Kai 2014-10-06 13:09:33 UTC
Created attachment 107424 [details]
Screenshot with GALLIUM_HUD=fps showing FPS drops (DynamicLights=true)

Ok, trimming the trace didn't really work. So I've uploaded a trace with the DynamicLights option set to true and another with it set to false to:
- <http://dev.carbon-project.org/debian/mesa.bugs/84570/borderlands2.dynlights.trace.7z>
- <http://dev.carbon-project.org/debian/mesa.bugs/84570/borderlands2.trace.7z>

Because those files are > 500 MB, even when packed with 7z, I password-protected the directory to prevent needless downloads. Known Mesa devs wanting to investigate this, can contact me by e-mail and I send them the login.


As you can see from the attached screenshot, the framerate gets a lot less stable, when DynamicLights is set to true. But the basic problem is there even without it.


My current stack is:
GPU: Hawaii PRO [Radeon R9 290] (ChipID = 0x67b1)
Mesa: Git:master/ca824e6940
libdrm: Git:master/00847fa48b
LLVM: SVN:trunk/r219049 (3.6 devel)
X.Org: 2:1.16.1-1
Linux: 3.17.0
Firmware: <http://people.freedesktop.org/~agd5f/radeon_ucode/>
> 9e05820da42549ce9c89d147cf1f8e19  /lib/firmware/updates/3.17.0-citadel/radeon/hawaii_ce.bin
> c8bab593090fc54f239c8d7596c8d846  /lib/firmware/updates/3.17.0-citadel/radeon/hawaii_mc.bin
> 3618dbb955d8a84970e262bb2e6d2a16  /lib/firmware/updates/3.17.0-citadel/radeon/hawaii_me.bin
> c000b0fc9ff6582145f66504b0ec9597  /lib/firmware/updates/3.17.0-citadel/radeon/hawaii_mec.bin
> 0643ad24b3beff2214cce533e094c1b7  /lib/firmware/updates/3.17.0-citadel/radeon/hawaii_pfp.bin
> ba6054b7d78184a74602fd81607e1386  /lib/firmware/updates/3.17.0-citadel/radeon/hawaii_rlc.bin
> 11288f635737331b69de9ee82fe04898  /lib/firmware/updates/3.17.0-citadel/radeon/hawaii_sdma.bin
> 284429675a5560e0fad42aa982965fc2  /lib/firmware/updates/3.17.0-citadel/radeon/hawaii_smc.bin
libclc: Git:master/7f6f5bff1f
DDX: Git:master/xf86-video-ati-7.5.0


Let me know, if you need something else.
Comment 6 Kai 2014-10-06 16:07:13 UTC
Created attachment 107434 [details]
Shader dump with R600_DEBUG=gs,vs,fs,ps (DynamicLights=false)

Forgot to upload this shader dump before. This is the entire output for R600_DEBUG=gs,vs,fs,ps for about ten minutes of Borderlands 2 (including menus). Maybe somebody familiar with the ISA can tell if one of the shaders generates problematic instructions? Just in case this is faster than going through the ApiTraces from comment #5.
Comment 7 Michel Dänzer 2014-10-07 01:51:57 UTC
Can you add 'requested-VRAM+VRAM-usage,requested-GTT+GTT-usage' to GALLIUM_HUD?
Comment 8 Kai 2014-10-07 15:44:32 UTC
Created attachment 107502 [details]
Screenshot with GALLIUM_HUD=fps,requested-VRAM+VRAM-usage,requested-GTT+GTT-usage showing FPS drops (DynamicLights=true)

(In reply to Michel Dänzer from comment #7)
> Can you add 'requested-VRAM+VRAM-usage,requested-GTT+GTT-usage' to
> GALLIUM_HUD?

Sure, here you go. One time with DynamicLights=true and one with DynamicLights=false.
Comment 9 Kai 2014-10-07 15:45:05 UTC
Created attachment 107503 [details]
Screenshot with GALLIUM_HUD=fps,requested-VRAM+VRAM-usage,requested-GTT+GTT-usage showing FPS drops (DynamicLights=false)
Comment 10 Michel Dänzer 2014-10-08 02:26:20 UTC
Does the kernel patch I attached to bug 84662 help? Note that you may need a newer kernel for it to apply.
Comment 11 Kai 2014-10-08 15:04:25 UTC
(In reply to Michel Dänzer from comment #10)
> Does the kernel patch I attached to bug 84662 help? Note that you may need a
> newer kernel for it to apply.

Which one? Attachment 107451 [details], attachment 107544 [details] [review] or both? Do I need the other two patches for Mesa, attachment 107542 [details] [review] and attachment 107543 [details] [review], you've attached to that report, as well?
Comment 12 Michel Dänzer 2014-10-08 15:13:18 UTC
(In reply to Kai from comment #11)

Please try with all the patches you mentioned.
Comment 13 Kai 2014-10-08 18:46:31 UTC
Created attachment 107574 [details]
Screenshot with GALLIUM_HUD=fps,requested-VRAM+VRAM-usage,requested-GTT+GTT-usage showing FPS drops (DynamicLights=true)

Ok, with the following stack detailed at the end of this comment, the "DynamicLights=false" case starts to become stable. I am, however, down from the ~50 FPS during "normal" operations to ~42-45 FPS. The GTT usage stays < 10 MB and the requested GTT size varies between a few hundred K to ~4 MB. VRAM usage is up to about 950 MB and the requested VRAM is somewhere above 520-530 MB. That said, there are still some drops FPS drops, especially, when you come around a corner (ie. lots of new stuff to draw). Or you turn really fast. But after the first turn in the general area the drops are almost gone. The non-DynamicLights case is now more or less flawless, even if I would expect a few more FPS again.

The "DynamicLights=True" case is however still a mess. The attached screenshot was taken with the stack detailed below and DynamicLigts turned on. Again, you see the ~10 FPS drop compared to the DynamicLights=False case (what is to be expeceted, considering what Ian said in comment #1, but it still hurts, becuase ~45 FPS is not high enough for a 10 FPS drop to go by unnoticed). And in addition – even while GTT and VRAM look exactly like the non-DynamicLights case – you get the constant drops when turning, running, etc.


My current stack is (Debian testing as a base):
GPU: Hawaii PRO [Radeon R9 290] (ChipID = 0x67b1)
Mesa: Git:master/581418585e + attachment 107542 [details] [review] and attachment 107543 [details] [review]
libdrm: Git:master/00847fa48b
LLVM: SVN:trunk/r219288 (3.6 devel)
X.Org: 2:1.16.1-1
Linux: Git:~agd5f/linux:drm-next-3.18:369283bfbd + attachment 107451 [details] [review] and attachment 107544 [details] [review] (identifies itself as 3.17.0-rc5)
Firmware: <http://people.freedesktop.org/~agd5f/radeon_ucode/>
> 9e05820da42549ce9c89d147cf1f8e19  /lib/firmware/updates/3.17.0-citadel/radeon/hawaii_ce.bin
> c8bab593090fc54f239c8d7596c8d846  /lib/firmware/updates/3.17.0-citadel/radeon/hawaii_mc.bin
> 3618dbb955d8a84970e262bb2e6d2a16  /lib/firmware/updates/3.17.0-citadel/radeon/hawaii_me.bin
> c000b0fc9ff6582145f66504b0ec9597  /lib/firmware/updates/3.17.0-citadel/radeon/hawaii_mec.bin
> 0643ad24b3beff2214cce533e094c1b7  /lib/firmware/updates/3.17.0-citadel/radeon/hawaii_pfp.bin
> ba6054b7d78184a74602fd81607e1386  /lib/firmware/updates/3.17.0-citadel/radeon/hawaii_rlc.bin
> 11288f635737331b69de9ee82fe04898  /lib/firmware/updates/3.17.0-citadel/radeon/hawaii_sdma.bin
> 284429675a5560e0fad42aa982965fc2  /lib/firmware/updates/3.17.0-citadel/radeon/hawaii_smc.bin
libclc: Git:master/7f6f5bff1f
DDX: Git:master/xf86-video-ati-7.5.0

Let me know, if you need something else (e-mail me, if you want access to the ApiTraces).
Comment 14 Michel Dänzer 2014-10-10 01:27:50 UTC
People reported that Mesa commit 7b4276d7acf2e0f77044cb50caa6ad936fa78786 ('r600g,radeonsi: Always use GTT again for PIPE_USAGE_STREAM buffers') helped for Borderlands 2.
Comment 15 Kai 2014-10-10 15:12:03 UTC
(In reply to Michel Dänzer from comment #14)
> People reported that Mesa commit 7b4276d7acf2e0f77044cb50caa6ad936fa78786
> ('r600g,radeonsi: Always use GTT again for PIPE_USAGE_STREAM buffers')
> helped for Borderlands 2.

THAT is a LOT better! Even with DynamicLights on I only get occasional FPS drops. Usually directly after entering a new area. Sometimes, when there's a lot to draw, that moves, you can get the drops again as well. I played for about an hour (with DynamicLights=true), and it didn't get worse with time. It's just dependent on whether there is lots of new stuff to draw or there are lots of moving parts with lighting eg. from fires or effects.

TL;DR: Almost there. There are still drops, but not as bad as before and almost no complete "1 second freezes" any longer.


My current stack is (Debian testing as a base):
GPU: Hawaii PRO [Radeon R9 290] (ChipID = 0x67b1)
Mesa: Git:master/ac557b4c12 + attachment 107542 [details] [review] and attachment 107543 [details] [review]
libdrm: Git:master/00847fa48b
LLVM: SVN:trunk/r219409 (3.6 devel)
X.Org: 2:1.16.1-1
Linux: Git:~agd5f/linux:drm-next-3.18:369283bfbd + attachment 107451 [details] [review] and attachment 107544 [details] [review] (identifies itself as 3.17.0-rc5)
Firmware: <http://people.freedesktop.org/~agd5f/radeon_ucode/>
> 9e05820da42549ce9c89d147cf1f8e19  /lib/firmware/updates/3.17.0-citadel/radeon/hawaii_ce.bin
> c8bab593090fc54f239c8d7596c8d846  /lib/firmware/updates/3.17.0-citadel/radeon/hawaii_mc.bin
> 3618dbb955d8a84970e262bb2e6d2a16  /lib/firmware/updates/3.17.0-citadel/radeon/hawaii_me.bin
> c000b0fc9ff6582145f66504b0ec9597  /lib/firmware/updates/3.17.0-citadel/radeon/hawaii_mec.bin
> 0643ad24b3beff2214cce533e094c1b7  /lib/firmware/updates/3.17.0-citadel/radeon/hawaii_pfp.bin
> ba6054b7d78184a74602fd81607e1386  /lib/firmware/updates/3.17.0-citadel/radeon/hawaii_rlc.bin
> 11288f635737331b69de9ee82fe04898  /lib/firmware/updates/3.17.0-citadel/radeon/hawaii_sdma.bin
> 284429675a5560e0fad42aa982965fc2  /lib/firmware/updates/3.17.0-citadel/radeon/hawaii_smc.bin
libclc: Git:master/7f6f5bff1f
DDX: Git:master/xf86-video-ati-7.5.0
Comment 16 Michel Dänzer 2014-10-15 06:40:41 UTC
(In reply to Kai from comment #15)
> THAT is a LOT better! Even with DynamicLights on I only get occasional FPS
> drops. Usually directly after entering a new area. Sometimes, when there's a
> lot to draw, that moves, you can get the drops again as well.

And those remaining drops don't happen with other drivers? Lower FPS when more things are happening doesn't seem unexpected, and e.g. when entering a new area, the game code itself might slow down due to loading stuff etc.
Comment 17 Michel Dänzer 2014-10-15 06:41:44 UTC
Can you attach a new GALLIUM_HUD screenshot corresponding to a remaining drop?
Comment 18 Kai 2014-10-15 15:31:57 UTC
Created attachment 107887 [details]
Screenshot with GALLIUM_HUD=fps,requested-VRAM+VRAM-usage,requested-GTT+GTT-usage showing FPS drops (DynamicLights=true)

(In reply to Michel Dänzer from comment #16)
> (In reply to Kai from comment #15)
> > THAT is a LOT better! Even with DynamicLights on I only get occasional FPS
> > drops. Usually directly after entering a new area. Sometimes, when there's a
> > lot to draw, that moves, you can get the drops again as well.
> 
> And those remaining drops don't happen with other drivers?

I can't tell you that. I have only a R9 290 and I'm only using radeonsi.

> Lower FPS when
> more things are happening doesn't seem unexpected, and e.g. when entering a
> new area, the game code itself might slow down due to loading stuff etc.

That's quite possible, though I can see drops (as shown in the attached screenshot), when I do nothing particularily intersting besides moving around in an area which I've been in for some time. The really bad drops (single-digit) indeed seem only to happen upon entering an area, ie. after a loading screen.
Though I did hear from a friend – using a nVidia card with the proprietary driver and DynamicLights=false – that she's seeing occasional drops/micro-freezes as well.

Please note that this is all with the DynamicLights option set to true.

If there's some kind of info I can provide you with, which would make this clear, let me know. Maybe Ian from Aspyr can help as well (especially: might any option listed at <http://udn.epicgames.com/Three/ConsoleCommands.html> help in debugging this? Are there other options that might produce helpful output?)?

(In reply to Michel Dänzer from comment #17)
> Can you attach a new GALLIUM_HUD screenshot corresponding to a remaining drop?

Done. As you can see those aren't as bad as they were initially and that's with DynamicLights=true. Still they are noticeable. This graph shows a drop during a fight (first drop till those smaller drops before the last big drop) and afterwards I was running around in that same small area I've been fighting in and was collecting the loot (from those smaller drops till the last big drop). If you stand still the FPS usually go up to ~40 FPS and stay there with only minor fluctuations.

Jumping and turning on the spot can trigger a big drop as well.


Overall though, I would call the game playable now. Not sure though, if we should split this bug, so one can stay open for Intel and the Radeon version can be closed (maybe with a follow-up along the lines of "Borderlands 2 shows FPS drops – more optimization needed")?

Any chance those kernel patches will be applied to the 3.17 tree (point releases)?
Comment 19 Andreas Hartmetz 2014-10-15 15:46:24 UTC
I've noticed that disabling CPU frequency scaling makes a big difference in the severity of micro-hangs (3.17, dynamic lights off). My explanation is that, when threads are waiting for each other, the scheduler doesn't know that the apparently "idle-blocked" threads would all run faster overall if the CPU went faster. Doesn't matter if that's wrong, the difference is there...

At least on AMD (some Intel CPUs use a special frequency scaling driver), you can effectively disable CPU frequency scaling like so:
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

Maybe that also helps a bit with dynamic lights on?
Comment 20 Ian C. Bullard 2014-10-15 15:51:23 UTC
(In reply to Kai from comment #18)
> That's quite possible, though I can see drops (as shown in the attached
> screenshot), when I do nothing particularily intersting besides moving
> around in an area which I've been in for some time. The really bad drops
> (single-digit) indeed seem only to happen upon entering an area, ie. after a
> loading screen.

The drops can occur even if you've been in an area for a while.  Borderlands is a heavily streamed game so new textures/shaders can cause a hiccup.  We're looking at ways to smooth this out.

> Though I did hear from a friend – using a nVidia card with the proprietary
> driver and DynamicLights=false – that she's seeing occasional
> drops/micro-freezes as well.

That fits with what I expected and matches what I said above. 

> If there's some kind of info I can provide you with, which would make this
> clear, let me know. Maybe Ian from Aspyr can help as well (especially: might
> any option listed at <http://udn.epicgames.com/Three/ConsoleCommands.html>
> help in debugging this? Are there other options that might produce helpful
> output?)?

I don't know of any command that can assist, sorry.  Most of the commands are disabled in release builds.
Comment 21 Kai 2014-10-15 16:13:35 UTC
(In reply to Andreas Hartmetz from comment #19)
> I've noticed that disabling CPU frequency scaling makes a big difference in
> the severity of micro-hangs (3.17, dynamic lights off). My explanation is
> that, when threads are waiting for each other, the scheduler doesn't know
> that the apparently "idle-blocked" threads would all run faster overall if
> the CPU went faster. Doesn't matter if that's wrong, the difference is
> there...
> 
> At least on AMD (some Intel CPUs use a special frequency scaling driver),
> you can effectively disable CPU frequency scaling like so:
> echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
> 
> Maybe that also helps a bit with dynamic lights on?

I have that Intel pstate driver for my CPU, but after feeding the performance setting to scaling_governor the amount of drops goes down, the drops aren't as severe (usually around 17 FPS and not single-digits; only exception: the first five to ten seconds after loading screens) and the overall FPS rate is up by ~5-8 FPS. Thanks a lot for that tip!


(In reply to Ian C. Bullard from comment #20)
> (In reply to Kai from comment #18)
> > That's quite possible, though I can see drops (as shown in the attached
> > screenshot), when I do nothing particularily intersting besides moving
> > around in an area which I've been in for some time. The really bad drops
> > (single-digit) indeed seem only to happen upon entering an area, ie. after a
> > loading screen.
> 
> The drops can occur even if you've been in an area for a while.  Borderlands
> is a heavily streamed game so new textures/shaders can cause a hiccup. 
> We're looking at ways to smooth this out.

Thanks for clearing this up. Though I like to state that using the radeonsi driver with the stack detailed in comment #15 gives you a very playable game (almost no drops) with "DynamicLights=false" and only the smaller drops I described in comment #18 with DynamicLights=true. From my POV that sounds like "supporting" Radeons with the FLOSS driver should be ok or at least not worse than nVidia cards. Especially when combined with Andreas' suggestion.
I've been playing for a few hours (alone and co-op) with this stack and DynamicLights=true and it works for me; if you keep DynamicLights off it should work for most, shouldn't it?

> > Though I did hear from a friend – using a nVidia card with the proprietary
> > driver and DynamicLights=false – that she's seeing occasional
> > drops/micro-freezes as well.
> 
> That fits with what I expected and matches what I said above. 

Oh, I thought your previous statements only related to Intel and AMD GPUs (especially since nVidia is the only officially supported GPU vendor).
Comment 22 Ian C. Bullard 2014-10-15 16:28:43 UTC
(In reply to Kai from comment #21)
> From my POV that sounds
> like "supporting" Radeons with the FLOSS driver should be ok or at least not
> worse than nVidia cards. Especially when combined with Andreas' suggestion.
> I've been playing for a few hours (alone and co-op) with this stack and
> DynamicLights=true and it works for me; if you keep DynamicLights off it
> should work for most, shouldn't it?

I'll ask QA to take a look when they have a chance (not soon) and get new performance numbers.  If it does well enough we'll add support.

> Oh, I thought your previous statements only related to Intel and AMD GPUs
> (especially since nVidia is the only officially supported GPU vendor).

__GL_THREADED_OPTIMIZATIONS=1 smooths things out for Nvidia but there are hitches every once in a while.

"Supported" for Aspyr means that we have tested it on that hardware/driver and feel like the experience is good for players using that hardware/driver.  Nvidia's proprietary driver did that, everything else didn't.  

If you want to stress test the driver and see the performance under high load, I suggest using the Deploy Beacon part of the "Bright Lights, Flying City" mission (http://borderlands.wikia.com/wiki/Bright_Lights,_Flying_City) when the first round of robots arrive near the entrance to Overlook.
Comment 23 Kai 2014-10-17 19:03:12 UTC
Not too surprising "Borderlands: The Pre-Sequel" is also affected and at this point behaves (with DynamicLights=true) as Borderlands 2. Ie. with the patches from comment #15 (the Mesa patches have landed on master and the Kernel patches should become part of 3.18, at least I've seen the pull request from Alex to that effect).
Setting the CPUfreq governor to performance helps as well.

Only difference is that the GPU fan goes to 100 % for a second every five seconds.
Comment 24 Kai 2014-10-17 21:53:40 UTC
I've made some screenshots with the Pre-Sequel as well: <https://imgur.com/a/DnoC1>. As you can see, big FPS drops coincide usually with buffer wait time and/or draw call spikes. Some of these spikes can be correlated to "stuff happening on the screen", but in other cases drops can come out of nowhere, so to speak.

With VBlank on, FPS capping at 60 FPS, all video settings to max, a 2560×1440 resolution, CPUfreq governor set to performance (really helps reducing the frequency of the FPS drops/buffer wait time spikes) and the stack detailed below, I get 40 to 60 FPS most of the time. The images show the "error" case.

My current stack is (Debian testing as a base):
GPU: Hawaii PRO [Radeon R9 290] (ChipID = 0x67b1)
Mesa: Git:master/2883aff3be
libdrm: Git:master/00847fa48b
LLVM: SVN:trunk/r219920 (3.6 devel)
X.Org: 2:1.16.1-1
Linux: Git:~agd5f/linux:drm-next-3.18:369283bfbd + attachment 107451 [details] [review] and attachment 107544 [details] [review] (identifies itself as 3.17.0-rc5)
Firmware: <http://people.freedesktop.org/~agd5f/radeon_ucode/>
> 9e05820da42549ce9c89d147cf1f8e19  /lib/firmware/updates/3.17.0-citadel/radeon/hawaii_ce.bin
> c8bab593090fc54f239c8d7596c8d846  /lib/firmware/updates/3.17.0-citadel/radeon/hawaii_mc.bin
> 3618dbb955d8a84970e262bb2e6d2a16  /lib/firmware/updates/3.17.0-citadel/radeon/hawaii_me.bin
> c000b0fc9ff6582145f66504b0ec9597  /lib/firmware/updates/3.17.0-citadel/radeon/hawaii_mec.bin
> 0643ad24b3beff2214cce533e094c1b7  /lib/firmware/updates/3.17.0-citadel/radeon/hawaii_pfp.bin
> ba6054b7d78184a74602fd81607e1386  /lib/firmware/updates/3.17.0-citadel/radeon/hawaii_rlc.bin
> 11288f635737331b69de9ee82fe04898  /lib/firmware/updates/3.17.0-citadel/radeon/hawaii_sdma.bin
> 284429675a5560e0fad42aa982965fc2  /lib/firmware/updates/3.17.0-citadel/radeon/hawaii_smc.bin
libclc: Git:master/7f6f5bff1f
DDX: 1:7.5.0-1
Comment 25 Michel Dänzer 2014-10-20 07:51:54 UTC
(In reply to Kai from comment #24)
> As you can see, big FPS drops coincide usually with buffer wait time

The buffer wait time might be at least partially due to a game bug (though there might be a workaround in Mesa at some point): http://lists.freedesktop.org/archives/mesa-dev/2014-October/069354.html


> and/or draw call spikes.

The number of draw calls is entirely up to the game. More draw calls generally result in more time spent per frame, i.e. fewer frames per seconds.
Comment 26 Kai 2014-10-20 15:45:09 UTC
(In reply to Michel Dänzer from comment #25)
> (In reply to Kai from comment #24)
> > As you can see, big FPS drops coincide usually with buffer wait time
> 
> The buffer wait time might be at least partially due to a game bug (though
> there might be a workaround in Mesa at some point):
> http://lists.freedesktop.org/archives/mesa-dev/2014-October/069354.html
> 
> > and/or draw call spikes.
> 
> The number of draw calls is entirely up to the game. More draw calls
> generally result in more time spent per frame, i.e. fewer frames per seconds.

ok, then this looks like it is fixed for Radeons. Maybe this should be reassigned to the Intel driver? But that's up to you, Michel.

Thanks for all your help!
Comment 27 Ian C. Bullard 2014-10-20 16:42:17 UTC
> The buffer wait time might be at least partially due to a game bug (though
> there might be a workaround in Mesa at some point):
> http://lists.freedesktop.org/archives/mesa-dev/2014-October/069354.html

Thank you for finding this!  We'll look into our glDrawRangeElements calls this week and make sure we're not causing the stall.  I would reply to the mailing list but I'm currently not subscribed.
Comment 28 Kai 2014-10-20 16:51:35 UTC
(In reply to Ian C. Bullard from comment #27)
> > The buffer wait time might be at least partially due to a game bug (though
> > there might be a workaround in Mesa at some point):
> > http://lists.freedesktop.org/archives/mesa-dev/2014-October/069354.html
> 
> Thank you for finding this!  We'll look into our glDrawRangeElements calls
> this week and make sure we're not causing the stall.  I would reply to the
> mailing list but I'm currently not subscribed.

AFAIR you don't need to be subscribed to be able to send e-mails, though you should prefix you e-mails with the "I'm not subscribed, please CC me" line.
Comment 29 Kenneth Graunke 2014-10-20 17:07:30 UTC
(In reply to Ian C. Bullard from comment #27)
> > The buffer wait time might be at least partially due to a game bug (though
> > there might be a workaround in Mesa at some point):
> > http://lists.freedesktop.org/archives/mesa-dev/2014-October/069354.html
> 
> Thank you for finding this!  We'll look into our glDrawRangeElements calls
> this week and make sure we're not causing the stall.  I would reply to the
> mailing list but I'm currently not subscribed.

Awesome - thanks for looking into it!  It makes a huge difference in performance on Intel.
Comment 30 Kai 2014-10-24 07:17:23 UTC
Michel, is there any chance attachment 107544 [details] [review] will be part of 3.18? I saw attachment 107451 [details] [review] hitting Linus' tree, but AFAICT attachment 107544 [details] [review] is missing. Is this intentional or was it just missed?
Comment 31 Michel Dänzer 2014-10-29 07:41:51 UTC
(In reply to Kai from comment #30)
> Michel, is there any chance attachment 107544 [details] [review] [review] will be
> part of 3.18?

No, but it's in Alex's queue for 3.19.
Comment 32 Kai 2015-02-09 08:39:31 UTC
(In reply to Michel Dänzer from comment #31)
> (In reply to Kai from comment #30)
> > Michel, is there any chance attachment 107544 [details] [review] will be
> > part of 3.18?
> 
> No, but it's in Alex's queue for 3.19.

3.19 has been released. As I'm not sure, what the situation with Intel GPUs is, it's not clear to me, whether this bug should be closed, renamed or duplicated (with this one being closed afterwards for Radeon GPUs and the other renamed to Intel: …). But for me the game ran pretty well over the past weeks with the various RC versions of 3.19. Michel/Tapani, what do you think?
Comment 33 Kenneth Graunke 2015-02-09 08:50:40 UTC
Aspyr's latest release contains bugfixes related to this, and that's solved the problem on Intel.
Comment 34 Kai 2015-02-09 14:48:33 UTC
Then I'm closing this bug. Thanks to everybody for your various contributions!
Comment 35 Andreas Hartmetz 2015-06-29 23:29:06 UTC
The bug is still present in Borderlands 2 and in fact seems stronger than ever. I saw something in the changelog of Borderlands: The Pre-Sequel that looked like a fix for this bug, but nothing in Borderlands 2.
Latest LLVM, DRM and Mesa, radeonsi, Radeon HD 7750 / Cape Verde.
That the bug has become worse may or may not be because performance of Mesa has also regressed a bit recently, so let's not pay too much attention to that.
Comment 36 Andreas Hartmetz 2015-06-29 23:43:52 UTC
Forgot to mention: Ubuntu kernel 3.19.
Comment 37 Ian C. Bullard 2015-06-30 15:20:15 UTC
Just so you're aware...

The glDrawRangeElements fix mentioned above should be live.  I've checked our depot's history and no rendering code changes have been made since January.  If there are changes on Aspyr's end that will help with performance let me know and I'll see if I can get them into the schedule (no promises).
Comment 38 Andreas Hartmetz 2015-09-12 14:31:23 UTC
The other bug seems to be about the same problem, but more helpful because it contains Apitrace traces.

*** This bug has been marked as a duplicate of bug 91969 ***
Comment 39 Kai 2015-09-12 15:16:56 UTC
Andreas, this bug was fixed, when I closed it. Also this bug does contain an Apitrace (comment #5).

You reopened it and now want to move everybody over to the "new" bug about this issue? Come on, it should have been the other way around or not at all. Anyway, as far as I'm concerned, my bug was fixed. Bug 91969 might be similar/the same or a different beast at all. I think we should just let this one closed and you can continue in the other bug as you see fit.


Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.