A few questions:

1. Which hardware was this tested on? (lspci -nn -d 10de: should produce the requisite info)
2. What mesa version
3. Is glReadPixels reading into a PBO or a client-side buffer?

The downstream bug is at [1].

1. So far I've received bug reports for these cards:

NVIDIA Quadro FX 580
NVIDIA GT218M NVS 3100M
NVIDIA GT216M GeForce GT 320M

2. Mesa 18.2.6

3. glReadPixels is reading into a client-side buffer

[1]: https://github.com/swaywm/wlroots/issues/1438

I'm seeing this issue on the following hardware too:

NVIDIA Corporation GP107 [GeForce GTX 1050] [10de:1c81] (rev a1)

OK, so the 3 GPUs listed by emersion are all nv50, while the GP107 would be covered by the nvc0 driver backend. Unfortunately I have a nv42 plugged in ATM, but I'll have a look when I'm back on something a bit more modern.

If someone with C/C++ development experience and the requisite hardware is interested, feel free to join in #nouveau on irc.freenode.net and I can provide advice for things to investigate.

Lastly, if someone can provide an apitrace of the application doing glReadPixels and getting unexpected results, that would avoid the need to have to reproduce the software setup. I believe it's possible to look at glReadPixels results in qapitrace ... would have to double-check though.

One guess is that we're copying from a RGBX surface to a RGBA surface, and not whacking alpha to 1. However that should generally work, so one would have to identify the precise code-path followed whereby that does not work.

Another guess is that the result of the rendering really does produce an alpha of 0, either due to an earlier bug (unrelated to glReadPixels) happening, or due to something specific to the drm formats exposed by nouveau and/or order of gl configs, which causes something funny to happen..

As per my comments on IRC, I think this is a wlroots bug: drm_connector_set_mode calls init_drm_plane_surfaces (which in turn creates a gbm surface) with GBM_FORMAT_XRGB8888. Flipping that to ARGB8888 resolves the issue.

As per my understanding, the config has to match the surface format, otherwise you get stuff like this. (At the extreme, imagine one said RGB10A2.)

The more detailed issue is that the winsys fb calls dri2_drm_image_get_buffers (via the getBuffers loader API call), which in turn uses the surface's format to create a backing bo. That format is XRGB8888, which becomes PIPE_FORMAT_BGRX8888_UNORM. The pipe_surface also has this format. However the GL believes that the format should be MESA_FORMAT_BGRA8888 (since that's what the config said), and since that matches the glReadPixels format, decides it can just do a memcpy.

Perhaps the pipe_surface should ignore the resource's format in this case, but that's not how the current logic flows. In st_framebuffer_validate, it calls u_surface_default_template which copies the format out of the given resource. AFAIK having mismatches between surface format and config format is really bad though, and not really supported. But I'm not a gbm API expert.

Yeah, I think it's reasonable. The render target is XRGB8888, and X really does mean undefined, including 'throw content away on render' and 'invent fake content on sample'.

I'd suggest allocating the gbm_surface as GBM_SURFACE_ARGB8888, then taking those BOs and passing DRM_FORMAT_XRGB8888 to AddFB2. This is totally safe to do, and will ensure that drivers preserve the alpha channel when you write out.

(I was going to say about matching EGL_NATIVE_VISUAL_ID as well given your short example, but you already do that properly - nice!)

Thanks for the explanation, this makes sense. We'll fix this in wlroots.

Again, sorry for filling an invalid bug!