Bug 109371

Summary: Textures seem to be byteswapped on big endian architectures
Product: Mesa Reporter: Adam Stylinski <kungfujesus06>
Component: Drivers/DRI/nouveauAssignee: Nouveau Project <nouveau>
Status: RESOLVED MOVED QA Contact: Nouveau Project <nouveau>
Severity: normal    
Priority: medium    
Version: 18.3   
Hardware: PowerPC   
OS: Linux (All)   
Whiteboard:
i915 platform: i915 features:
Attachments: Trace demoing the issue

Description Adam Stylinski 2019-01-16 00:36:08 UTC
When using an NV43 based GPU on a PPC64 kernel with a 64 bit userspace, it seems applications will have byte swapped textures when specified with GL_RGBA and GL_UNSIGNED_BYTE.  Matplotlib will do this when blit=True for the plot routines, and several games will have this issue.  In particular, for Legacy Doom, I've found that if I swap the textures so that they are A,B,G,R just before the textures are mapped, everything is correct.  

I also found that doing glPixelStorei(GL_UNPACK_BYTES, GL_TRUE) and choosing GL_UNSIGNED_INT_8_8_8_8 as the representation seemed to also remedy this.  I found that the _REV variant didn't byte swap the textures back to the incorrect ordering (this is probably a different bug).

Easiest way to demonstrate the bug:
svn co https://svn.code.sf.net/p/doomlegacy/svn/legacy_one/trunk doomlegacy
cd doomlegacy
make OS=LINUX
make
cd bin
Grab this archive: https://sourceforge.net/projects/doomlegacy/files/1.47.2/doomlegacy_1.47.2_common.zip/download, unzip it, extract legacy.wad into the bin directory
Grab this archive:
https://github.com/freedoom/freedoom/releases/download/v0.11.3/freedoom-0.11.3.zip, extract, move freedoom1.wad into the bin directory.
Then do:
./doomlegacy -iwad freedoom1.wad -opengl
Comment 1 Ilia Mirkin 2019-01-16 00:40:17 UTC
You'd save me a bunch of effort by performing these steps yourself, and tracing using apitrace (https://github.com/apitrace/apitrace). Ideally record an as-short-as-possible trace, and check that it still looks bad on replay, and then xz -9 it and attach here.
Comment 2 Adam Stylinski 2019-01-16 05:41:23 UTC
Created attachment 143138 [details]
Trace demoing the issue

Heh, the issue is there by the time I hit the title screen.
Comment 3 Ilia Mirkin 2019-01-19 20:31:58 UTC
Unfortunately it appears that my G5 is dead. It half-booted once (died right around nfsroot load time). Then for a bit it would turn on, with fans going but no light and no chime. Now the fans don't even turn on. So I think that's the end.

The things I would have done:

1. Check whether the textures look OK in qapitrace's inspector
2. Look at the transfer methods being used (in nv30_transfer.c). Try commenting some out, although that can also lead to failures.
3. Try to create a simple program based on the apitrace which reproduces the issue, and debug it step by step, to figure out where the byteswap might be happening.

A few bits of info:

The GPUs are LE deep down inside. However they have BE modes which byteswap "some stuff". So like MMIO accesses, FIFO commands, etc. So they can still be packed like integers as usual, without an extra byteswap, and the GPU will "take care of it". I don't have a clean idea of whether and how byteswapping happens between GART and VRAM, as well as in various "blit"/copy methods, where the byteswap would be different depending on whether it's u8, u16, or u32 datatype. This is most relevant to index buffers though, not textures. [Like let's say you're feeding the data via FIFO commands, there's an implicit byteswap there, etc.]

Have a look at nv30_format.c for the supported texture formats/etc.

As always, feel free to ask stuff in #nouveau.

It should be noted that with a patch to apitrace (https://github.com/apitrace/apitrace/issues/601#issuecomment-455019551), the attached trace replays fine on both nv42 and nv34 on x86.
Comment 4 GitLab Migration User 2019-09-18 20:47:20 UTC
-- GitLab Migration Automatic Message --

This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity.

You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/mesa/mesa/issues/1167.

Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.