Created attachment 133180 [details] [review] proposed patch by Julien Isorce Based on following bug: https://bugzilla.gnome.org/show_bug.cgi?id=785085 In Mesa the picture_id is a frame number which is in the range 0-31 here: https://cgit.freedesktop.org/mesa/mesa/tree/src/gallium/state_trackers/va/picture.c#n430 In gst vaapi it is used as surface handles and that can be larger than 31 thus causing SEGFAULT.
Also see https://bugzilla.gnome.org/show_bug.cgi?id=785085
Comment on attachment 133180 [details] [review] proposed patch by Julien Isorce commit 91d93aa62162f98d6377e5c796b63faa263f2c18 Author: Julien Isorce <julien.isorce@gmail.com> Date: Tue Jul 25 15:31:28 2017 +0100 st/va: change frame_idx from array to hash table The picture_id was assumed to be a frame number so in 0-31. But the vaapi client gstreamer-vaapi uses the surfaces handles as identifier which are unsigned int. This bug can happen when using a lot of vaapi surfaces within the same process. Indeed Mesa/st/va increments a counter for the surface ID: mesa/util/u_handle_table.c::handle_table_add which starts from 0 and incremented by 1 at each call. So creating more than 32 surfaces was a problem. The following bug contains a test that reproduces the problem by running a couple of vaapih264enc in the same process. The above also explains why there was no pb when running them in separated processes. Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=102006 Signed-off-by: Julien Isorce <jisorce@oblong.com> Tested-by: Tomas Rataj <rataj28@gmail.com> Acked-by: Christian König <christian.koenig@amd.com> Reviewed-and-tested-by: Boyuan Zhang <Boyuan.Zhang@amd.com>
Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.