Bug 102006

Summary: gstreamer vaapih264enc segfault
Product: DRI Reporter: rataj28
Component: DRM/RadeonAssignee: Default DRI bug account <dri-devel>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: medium    
Version: XOrg git   
Hardware: x86-64 (AMD64)   
OS: Linux (All)   
Whiteboard:
i915 platform: i915 features:
Attachments:
Description Flags
proposed patch by Julien Isorce none

Description rataj28 2017-08-01 20:00:53 UTC
Created attachment 133180 [details] [review]
proposed patch by Julien Isorce

Based on following bug:

https://bugzilla.gnome.org/show_bug.cgi?id=785085

In Mesa the picture_id is a frame number which is in the range 0-31 here:

https://cgit.freedesktop.org/mesa/mesa/tree/src/gallium/state_trackers/va/picture.c#n430

In gst vaapi it is used as surface handles and that can be larger than 31 thus causing SEGFAULT.
Comment 1 Julien Isorce 2017-08-14 13:00:13 UTC
Also see https://bugzilla.gnome.org/show_bug.cgi?id=785085
Comment 2 Julien Isorce 2017-08-14 13:00:38 UTC
Comment on attachment 133180 [details] [review]
proposed patch by Julien Isorce

commit 91d93aa62162f98d6377e5c796b63faa263f2c18
Author: Julien Isorce <julien.isorce@gmail.com>
Date:   Tue Jul 25 15:31:28 2017 +0100

    st/va: change frame_idx from array to hash table
    
    The picture_id was assumed to be a frame number so in 0-31.
    But the vaapi client gstreamer-vaapi uses the surfaces handles
    as identifier which are unsigned int.
    
    This bug can happen when using a lot of vaapi surfaces within
    the same process. Indeed Mesa/st/va increments a counter for the
    surface ID: mesa/util/u_handle_table.c::handle_table_add which
    starts from 0 and incremented by 1 at each call.
    So creating more than 32 surfaces was a problem.
    
    The following bug contains a test that reproduces the problem
    by running a couple of vaapih264enc in the same process. The
    above also explains why there was no pb when running them in
    separated processes.
    
    Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=102006
    Signed-off-by: Julien Isorce <jisorce@oblong.com>
    Tested-by: Tomas Rataj <rataj28@gmail.com>
    Acked-by: Christian K├Ânig <christian.koenig@amd.com>
    Reviewed-and-tested-by: Boyuan Zhang <Boyuan.Zhang@amd.com>

Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.