I noticed a constant increase of rss memory while playing a video using vdpau + radeonsi, caused by X events accumulating in read_packet (c=0x8275a0) at xcb_in.c:333, and never flushed, and corresponding to this backtrace :

Breakpoint 6, read_packet (c=0x8275a0) at xcb_in.c:333
(gdb) bt
#0  0x00007fffec99b281 in read_packet (c=0x8275a0) at xcb_in.c:333
#1  0x00007fffec99c742 in _xcb_in_read (c=0x8275a0) at xcb_in.c:936
#2  0x00007fffec999ae4 in _xcb_conn_wait (c=0x8275a0, cond=0x7fffe1636960, vector=0x0, count=0x0) at xcb_conn.c:495
#3  0x00007fffec99b78b in wait_for_reply (c=0x8275a0, request=207, e=0x0) at xcb_in.c:493
#4  0x00007fffec99b8a4 in xcb_wait_for_reply (c=0x8275a0, request=207, e=0x0) at xcb_in.c:523
#5  0x00007fffea7c5ac8 in xcb_dri2_swap_buffers_reply (c=0x8275a0, cookie=..., e=0x0) at dri2.c:893
#6  0x00007fffe6a982f0 in vl_dri2_get_flush_reply (scrn=0x83cab0) at ../../../../src/gallium/auxiliary/vl/vl_winsys_dri.c:103
#7  0x00007fffe6a984b4 in vl_dri2_destroy_drawable (scrn=0x83cab0) at ../../../../src/gallium/auxiliary/vl/vl_winsys_dri.c:146
#8  0x00007fffe6a98524 in vl_dri2_set_drawable (scrn=0x83cab0, drawable=41943041) at ../../../../src/gallium/auxiliary/vl/vl_winsys_dri.c:162
#9  0x00007fffe6a988ed in vl_screen_get_timestamp (vscreen=0x83cab0, drawable=41943041)	at ../../../../src/gallium/auxiliary/vl/vl_winsys_dri.c:269
#10 0x00007fffe6aa193d in vlVdpPresentationQueueGetTime	(presentation_queue=11,	current_time=0x7fffe1636b80) at presentation.c:189
#11 0x00007fffe6aa1f04 in vlVdpPresentationQueueBlockUntilSurfaceIdle (presentation_queue=11, surface=13, first_presentation_time=0x7fffe1636b80) at presentation.c:335
#12 0x00007fffe719fb51 in put_surface () at /usr/lib64/dri/radeonsi_drv_video.so
#13 0x00007fffe719fdcf in vdpau_PutSurface () at /usr/lib64/dri/radeonsi_drv_video.so
#14 0x00007fffed5ead37 in gst_vaapi_texture_glx_put_surface (flags=0, crop_rect=0x7fffe1636d20,	surface=0x7fffdc08fd90,	base_texture=0x7fffcc1385a0) at gstvaapitexture_glx.c:315
#15 0x00007fffed5ead37 in gst_vaapi_texture_glx_put_surface (texture=0x7fffcc1385a0, surface=0x7fffdc08fd90, crop_rect=0x7fffe1636d20, flags=0) at gstvaapitexture_glx.c:378
#16 0x00007fffedf6e1b1 in gst_vaapi_texture_put_surface (texture=0x7fffcc1385a0, surface=0x7fffdc08fd90, crop_rect=<optimized out>, flags=<optimized out>) at gstvaapitexture.c:332
#17 0x00007fffe99118ec in _do_upload_with_meta (context=<optimized out>, upload=0x7fffdc029600 [GstGLUpload]) at gstglupload.c:357
#18 0x00007fffe9913198 in _run_message_sync (message=0x7fffe1e37840) at gstglwindow.c:353
#19 0x00007fffe9917a82 in _run_message (message=0x7fffd8001b60)	at gstglwindow_x11.c:598
#20 0x00007ffff73857fb in g_main_context_dispatch (context=0x7fffcc010980) at gmain.c:3111
#21 0x00007ffff73857fb in g_main_context_dispatch (context=context@entry=0x7fffcc010980) at gmain.c:3710
#22 0x00007ffff7385b98 in g_main_context_iterate (context=0x7fffcc010980, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at gmain.c:3781
#23 0x00007ffff7385ec2 in g_main_loop_run (loop=0x7fffcc010b70)	at gmain.c:3975
#24 0x00007fffe9901cf7 in gst_gl_context_create_thread (context=0x7fffdc023260 [GstGLContextGLX]) at gstglcontext.c:959
#25 0x00007ffff73ac3d5 in g_thread_proxy (data=0x7fffdc0234a0) at gthread.c:764
#26 0x00007ffff6f2352a in start_thread (arg=0x7fffe1637700) at pthread_create.c:310
#27 0x00007ffff6c5f22d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

The commands generating this case is :

# ln -s vdpau_drv_video.so /usr/lib64/dri/radeonsi_drv_video.so
$ export VDPAU_DRIVER=radeonsi
$ wget http://samples.mplayerhq.hu/MPEG-4/NeroRecodeSample-MP4/NeroRecodeSample.mp4
$ gst-launch-1.0 filesrc location=NeroRecodeSample.mp4 ! qtdemux ! mpeg4videoparse ! vaapidecode ! videoconvert ! glimagesink
 
The two malloc() occuring in libxcb in read_packet() are sufficient to make gst-launch-1.0 memory use grow out of control. It seems to occur because vl_dri2_set_drawable() is called alternatively with two different drawables, each call destroying the previous drawable.