Bug 102006

Summary:

gstreamer vaapih264enc segfault

Product:

DRI

Reporter:

rataj28

Component:

DRM/Radeon

Assignee:

Default DRI bug account <dri-devel>

Status:

RESOLVED FIXED

QA Contact:

Severity:

normal

Priority:

medium

Version:

XOrg git

Hardware:

x86-64 (AMD64)

OS:

Linux (All)

Whiteboard:

i915 platform:

i915 features:

Attachments:

Description	Flags
proposed patch by Julien Isorce	none

Description rataj28 2017-08-01 20:00:53 UTC

Created attachment 133180 [details] [review]
proposed patch by Julien Isorce

Based on following bug:

https://bugzilla.gnome.org/show_bug.cgi?id=785085

In Mesa the picture_id is a frame number which is in the range 0-31 here:

https://cgit.freedesktop.org/mesa/mesa/tree/src/gallium/state_trackers/va/picture.c#n430

In gst vaapi it is used as surface handles and that can be larger than 31 thus causing SEGFAULT.

Comment 1 Julien Isorce 2017-08-14 13:00:13 UTC

Also see https://bugzilla.gnome.org/show_bug.cgi?id=785085

Comment 2 Julien Isorce 2017-08-14 13:00:38 UTC

Comment on attachment 133180 [details] [review]
proposed patch by Julien Isorce

commit 91d93aa62162f98d6377e5c796b63faa263f2c18
Author: Julien Isorce <julien.isorce@gmail.com>
Date:   Tue Jul 25 15:31:28 2017 +0100

    st/va: change frame_idx from array to hash table
    
    The picture_id was assumed to be a frame number so in 0-31.
    But the vaapi client gstreamer-vaapi uses the surfaces handles
    as identifier which are unsigned int.
    
    This bug can happen when using a lot of vaapi surfaces within
    the same process. Indeed Mesa/st/va increments a counter for the
    surface ID: mesa/util/u_handle_table.c::handle_table_add which
    starts from 0 and incremented by 1 at each call.
    So creating more than 32 surfaces was a problem.
    
    The following bug contains a test that reproduces the problem
    by running a couple of vaapih264enc in the same process. The
    above also explains why there was no pb when running them in
    separated processes.
    
    Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=102006
    Signed-off-by: Julien Isorce <jisorce@oblong.com>
    Tested-by: Tomas Rataj <rataj28@gmail.com>
    Acked-by: Christian König <christian.koenig@amd.com>
    Reviewed-and-tested-by: Boyuan Zhang <Boyuan.Zhang@amd.com>

Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.