Bug 102006 - gstreamer vaapih264enc segfault
Summary: gstreamer vaapih264enc segfault
Alias: None
Product: DRI
Classification: Unclassified
Component: DRM/Radeon (show other bugs)
Version: XOrg git
Hardware: x86-64 (AMD64) Linux (All)
: medium normal
Assignee: Default DRI bug account
QA Contact:
Depends on:
Reported: 2017-08-01 20:00 UTC by rataj28
Modified: 2017-08-14 13:01 UTC (History)
0 users

See Also:
i915 platform:
i915 features:

proposed patch by Julien Isorce (4.97 KB, patch)
2017-08-01 20:00 UTC, rataj28
no flags Details | Splinter Review

Description rataj28 2017-08-01 20:00:53 UTC
Created attachment 133180 [details] [review]
proposed patch by Julien Isorce

Based on following bug:


In Mesa the picture_id is a frame number which is in the range 0-31 here:


In gst vaapi it is used as surface handles and that can be larger than 31 thus causing SEGFAULT.
Comment 1 Julien Isorce 2017-08-14 13:00:13 UTC
Also see https://bugzilla.gnome.org/show_bug.cgi?id=785085
Comment 2 Julien Isorce 2017-08-14 13:00:38 UTC
Comment on attachment 133180 [details] [review]
proposed patch by Julien Isorce

commit 91d93aa62162f98d6377e5c796b63faa263f2c18
Author: Julien Isorce <julien.isorce@gmail.com>
Date:   Tue Jul 25 15:31:28 2017 +0100

    st/va: change frame_idx from array to hash table
    The picture_id was assumed to be a frame number so in 0-31.
    But the vaapi client gstreamer-vaapi uses the surfaces handles
    as identifier which are unsigned int.
    This bug can happen when using a lot of vaapi surfaces within
    the same process. Indeed Mesa/st/va increments a counter for the
    surface ID: mesa/util/u_handle_table.c::handle_table_add which
    starts from 0 and incremented by 1 at each call.
    So creating more than 32 surfaces was a problem.
    The following bug contains a test that reproduces the problem
    by running a couple of vaapih264enc in the same process. The
    above also explains why there was no pb when running them in
    separated processes.
    Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=102006
    Signed-off-by: Julien Isorce <jisorce@oblong.com>
    Tested-by: Tomas Rataj <rataj28@gmail.com>
    Acked-by: Christian König <christian.koenig@amd.com>
    Reviewed-and-tested-by: Boyuan Zhang <Boyuan.Zhang@amd.com>

Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.