Bug 100710

Summary: GLAMOR: glamor-xv crashes in glamor_xv_render() on nouveau
Product: xorg Reporter: Olivier Fourdan <fourdan>
Component: Server/Acceleration/glamorAssignee: Xorg Project Team <xorg-team>
Status: RESOLVED FIXED QA Contact: Xorg Project Team <xorg-team>
Severity: normal    
Priority: medium    
Version: git   
Hardware: Other   
OS: All   
See Also: https://bugzilla.redhat.com/show_bug.cgi?id=1412814
Whiteboard:
i915 platform: i915 features:

Description Olivier Fourdan 2017-04-18 14:51:51 UTC
Description:

glamor-xv crashes in glamor_xv_render() on nouveau when playing mp4 file.

How reproducible:

Always

Steps to reproduce:

1. Xephyr -glamor -screen 1280x1024 :10
2. DISPLAY=:10 mplayer -vo xv somefile.mp4

Actual result:

Xephyr crashes in glamor_xv_render()

Additional info:

This issue is affecting Xwayland the same.

GLAMOR_DEBUG:

         glamor_pixmap_ensure_fb:	glamor: Failed to create fbo, incomplete attachment
         glamor_pixmap_ensure_fb:	glamor: Failed to create fbo, incomplete attachment
         glamor_pixmap_ensure_fb:	glamor: Failed to create fbo, incomplete attachment

Backtrace signature (similar to Xwayland)

(EE) 
(EE) Backtrace:
(EE) 0: Xephyr (OsLookupColor+0x139) [0x5b36a9]
(EE) 1: /lib64/libpthread.so.0 (__restore_rt+0x0) [0x7f4998e915bf]
(EE) 2: Xephyr (glamor_pixmap_exchange_fbos+0x11b6) [0x45c616]
(EE) 3: Xephyr (glamor_pixmap_exchange_fbos+0x1bff) [0x45d9df]
(EE) 4: Xephyr (RemoveInputDeviceTraces+0x2153) [0x4ac473]
(EE) 5: Xephyr (XvFillColorKey+0x327) [0x4f7517]
(EE) 6: Xephyr (SendErrorToClient+0x405) [0x463ef5]
(EE) 7: Xephyr (InitFonts+0x428) [0x467e38]
(EE) 8: /lib64/libc.so.6 (__libc_start_main+0xf1) [0x7f4998ada401]
(EE) 9: Xephyr (_start+0x2a) [0x42787a]
(EE) 10: ? (?+0x2a) [0x2a]
(EE) 
(EE) Segmentation fault at address 0x8
(EE) 
Fatal server error:
(EE) Caught signal 11 (Segmentation fault). Server aborting
(EE) 
Aborted (core dumped)

Debugging:

(gdb) bt
#0  0x000000000045b746 in glamor_xv_render (port_priv=port_priv@entry=0x119ccc0) at glamor_xv.c:291
#1  0x000000000045c18f in glamor_xv_put_image (port_priv=0x119ccc0, pDrawable=0x1a4bc20, src_x=<optimized out>, src_y=<optimized out>, drw_x=<optimized out>, 
    drw_y=<optimized out>, src_w=1280, src_h=720, drw_w=1280, drw_h=720, id=842094169, 
    buf=0x7fae87d7403c, width=1280, height=720, sync=0, clipBoxes=0x7ffca2294290) at glamor_xv.c:508
#2  0x00000000004aa5f3 in KdXVPutImage (pDraw=0x1a4bc20, pPort=0x119dd00, pGC=<optimized out>, src_x=<optimized out>, src_y=<optimized out>, src_w=<optimized out>, 
    src_h=720, drw_x=0, drw_y=0, drw_w=1280, drw_h=720, format=0x119da90, 
    data=0x7fae87d7403c, sync=0, width=1280, height=720) at kxv.c:1413
#3  0x00000000004f73d7 in ProcXvPutImage (client=0x1a3d980) at xvdisp.c:865
#4  0x0000000000463e75 in Dispatch () at dispatch.c:479
#5  0x0000000000467df8 in dix_main (argc=5, argv=0x7ffca2294548, envp=<optimized out>) at main.c:287
#6  0x00007fae926e1401 in __libc_start_main () from /lib64/libc.so.6
#7  0x000000000042787a in _start ()

(gdb) list
286	
287	    for (i = 0; i < 3; i++) {
288	        if (port_priv->src_pix[i]) {
289	            src_pixmap_priv[i] =
290	                glamor_get_pixmap_private(port_priv->src_pix[i]);
291	            pixmap_priv_get_scale(src_pixmap_priv[i], &src_xscale[i],
292	                                  &src_yscale[i]);
293	        }
294	    }
295	    glamor_make_current(glamor_priv);
(gdb) p src_pixmap_priv
$1 = {0x7fae87c92060, 0x0, 0x500}

(gdb) p port_priv->src_pix
$2 = {0x7fae87c92010, 0x1a4c1f0, 0x1a846c0}
Comment 1 Olivier Fourdan 2017-04-18 15:30:09 UTC
Problem is the pixmap has no fbo, therefore (_pixmap_priv_)->fbo->width and (_pixmap_priv_)->fbo->height will be a null-pointer dereference in pixmap_priv_get_scale() from glamor_utils.h

p *src_pixmap_priv[0]
$2 = {type = GLAMOR_MEMORY, gl_fbo = GLAMOR_FBO_UNATTACHED, map_access = GLAMOR_ACCESS_RO, fbo = 0x0, box = {x1 = 0, y1 = 0, x2 = 0, y2 = 0}, pbo = 0, prepare_region = {
    extents = {x1 = 0, y1 = 0, x2 = 0, y2 = 0}, data = 0x0}, prepared = 0, image = 0x0, block_w = 0, block_h = 0, block_wcnt = 0, block_hcnt = 0, box_array = 0x0, 
  fbo_array = 0x0}
Comment 2 Olivier Fourdan 2017-04-19 07:34:11 UTC
fbo creation fails with "glamor: Failed to create fbo, incomplete attachment", this is from:

#0  glamor_pixmap_ensure_fb (glamor_priv=glamor_priv@entry=0xd62570, fbo=fbo@entry=0x86ab60) at glamor_fbo.c:89
#1  0x00000000004b6760 in glamor_create_fbo_from_tex (glamor_priv=glamor_priv@entry=0xd62570,
    w=w@entry=1280, h=h@entry=720, format=format@entry=6406, tex=2, flag=flag@entry=0) at glamor_fbo.c:112
#2  0x00000000004b67cd in glamor_create_fbo (glamor_priv=glamor_priv@entry=0xd62570, w=w@entry=1280,
    h=h@entry=720, format=format@entry=6406, flag=flag@entry=0) at glamor_fbo.c:163
#3  0x000000000049d39c in glamor_create_pixmap (screen=screen@entry=0x868970, w=w@entry=1280,
    h=h@entry=720, depth=depth@entry=8, usage=usage@entry=0) at glamor.c:224
#4  0x00000000004b8612 in glamor_xv_put_image (port_priv=0x86ae20, pDrawable=0x10cc200,
    src_x=<optimized out>, src_y=<optimized out>, drw_x=<optimized out>, 
    drw_y=<optimized out>, src_w=1280, src_h=720, drw_w=1280, drw_h=720, id=842094169, 
    buf=0x7fffe90a003c, width=1280, height=720, sync=0, clipBoxes=0x7fffffffddc0) at glamor_xv.c:434
#5  0x00000000004c088d in KdXVPutImage (pDraw=0x10cc200, pPort=0x86bde0, pGC=<optimized out>,
    src_x=<optimized out>, src_y=<optimized out>, src_w=<optimized out>, 
    src_h=720, drw_x=0, drw_y=0, drw_w=1280, drw_h=720, format=0x86bb70, 
    data=0x7fffe90a003c, sync=0, width=1280, height=720) at kxv.c:1413
#6  0x000000000050293b in ProcXvPutImage (client=0xf21c80) at xvdisp.c:865
#7  0x000000000044c1f6 in Dispatch () at dispatch.c:479
#8  0x0000000000450188 in dix_main (argc=5, argv=0x7fffffffe058, envp=<optimized out>) at main.c:276
#9  0x00007ffff37f6401 in __libc_start_main () from /lib64/libc.so.6
#10 0x0000000000432c9a in _start ()

format is 6406 == 0x1906 == GL_ALPHA

format = gl_iformat_for_pixmap(pixmap) which is defined in glamor_utils.h as:

static inline GLenum
gl_iformat_for_pixmap(PixmapPtr pixmap)
{
    glamor_screen_private *glamor_priv =
        glamor_get_screen_private((pixmap)->drawable.pScreen);

    if (glamor_priv->gl_flavor == GLAMOR_GL_DESKTOP &&
        ((pixmap)->drawable.depth == 1 || (pixmap)->drawable.depth == 8)) {
        return glamor_priv->one_channel_format;
    } else {
        return GL_RGBA;
    }
}

(gdb) p glamor_priv->gl_flavor
$2 = GLAMOR_GL_DESKTOP

(gdb) p glamor_priv->one_channel_format
$3 = 6406 == GL_ALPHA
Comment 3 Olivier Fourdan 2017-04-19 07:47:37 UTC
I am not sure GL_ALPHA is valid there (at least everywhere, and stackoverflow seems to concur [1]) and indeed, using GL_RGBA fixes the issue (without noticeable rendering issue).

So, I am not sure how to fix that, it seems some driver are more permissive than others and allow GL_ALPHA as format there, whereas others such a nouveau (on this hardware, dunno if that's the case everywhere) will fail with an GL_FRAMEBUFFER_INCOMPLETE_ATTACHMENT if GL_ALPHA is used...

[1] http://stackoverflow.com/questions/3613889/gl-framebuffer-incomplete-attachment-when-trying-to-attach-texture
Comment 4 Olivier Fourdan 2017-04-21 08:45:17 UTC
https://patchwork.freedesktop.org/series/23309/

Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.