could not reproduce.
you should remove valgrid to check memory changes under <top>.
in my test,
gst-launch-0.10 filesrc location=xxx.mp4 ! qtdemux ! vaapidecode ! vaapisink sync=false
there's no obvious memory growing in top
VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
31480 8472 4736 S    5  0.1   0:00.15 gst-launch-0.10

please check <VIRT> <RES> items to see real memory status.

If only your 720p.mp4 files have this issue, please send the file to me.

720p.mp4 had been send to tinderbox.
You can get it and have a try. 

host: tinderbox
path: /var/www/html/media/Test_Container_files/720p.mp4

Reopened this bug.
I tried valgrind, and seems there really have memory leak issues.
I tracked GstVaapiDisplay, looks memory leaks here but vaapisink is ok. so the only leak should be in vaapisink.
BTW, I still didn't see any memory growing.
I'm tracking on vaapisink since some changes there may bring some bugs.

Found 2 possible reasons which has critial memory leak or brought unexpected results.
1. vaapisink memory leak on uploader alloc
2. vaapiuploader, invalid pointer out-of-bounds.

Lastest commit 73cc7dd0bdfda405cfeeb16a2d7b4cbf945b6d86,
please help verify.

gst-plugins-vaapi: (0.4)73cc7dd0bdfda405cfeeb16a2d7b4cbf945b6d86

Here is the new result:
gst_update_registry (gstregistry.c:1643)
init_post (gst.c:793)
main (gst-launch.c:984)
gst_vaapi_display_constructed (gstvaapidisplay.c:532)
gst_vaapi_ensure_display (gstvaapipluginutil.c:126)
gst_vaapisink_get_caps (gstvaapisink.c:396)
gst_base_sink_pad_getcaps (gstbasesink.c:606)
gst_pad_get_caps_unlocked (gstpad.c:2254)

Compared to the last, there is no "gst_element_change_state", but "gst_pad_get_caps_unlocked" is found.

Hi, could you post the detailed log(valgrind printed) of 'gst_pad_get_caps_unlocked' here, I didn't find mem leak in this function.
Thanks.

Created attachment 80332 [details]
Valgrind log on 5/20.

Hi Lianyue,
   I'm not sure your are tests the same version with mine. I see logs
==12431== 12 bytes in 1 blocks are possibly lost in loss record 259 of 2,171
==12431==    at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==12431==    by 0x53B8A78: g_malloc (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.3)
==12431==    by 0x53CD2DD: g_strdup (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.3)
==12431==    by 0x4E809DE: _gst_debug_category_new (gstinfo.c:1420)
==12431==    by 0xEBCB694: gst_vaapi_uploader_class_intern_init (gstvaapiuploader.c:284)
==12431==    by 0x514A626: g_type_class_ref (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.3200.3)
==12431==    by 0x5132D6C: g_object_new_valist (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.3200.3)
==12431==    by 0x51331D3: g_object_new (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.3200.3)
==12431==    by 0xEBC9723: gst_vaapisink_get_caps (gstvaapisink.c:423)
==12431==    by 0xDA62F86: gst_base_sink_pad_getcaps (gstbasesink.c:606)
==12431==    by 0x4E87945: gst_pad_get_caps_unlocked (gstpad.c:2254)
==12431==    by 0x4E8B07C: gst_pad_get_caps_reffed (gstpad.c:2338)

but "gst_vaapisink_get_caps (gstvaapisink.c:423)" in my code it's not line 423.
please use 0.4 branch with commit_id "73cc7dd0bdfda405cfeeb16a2d7b4cbf945b6d86" included

Created attachment 80452 [details]
Valgrind log on 6/7.

Hi Wind,

It's my test result just now.
Gst-plugins-vaapi used 0.4 branch with commit_id "73cc7dd0bdfda405cfeeb16a2d7b4cbf945b6d86" included.

Definitely lost of valgrind reported is as below:
gst_update_registry (gstregistry.c:1643)
init_post (gst.c:793)
main (gst-launch.c:984)
gst_vaapi_display_constructed (gstvaapidisplay.c:532)
gst_vaapi_ensure_display (gstvaapipluginutil.c:126)
gst_vaapisink_get_caps (gstvaapisink.c:396)
gst_base_sink_pad_getcaps (gstbasesink.c:606)
gst_pad_get_caps_unlocked (gstpad.c:2254)

The result is the same as that on 5/20.

Could you open gst/vaapi/gstvaapisink.c file and check line 396? I don't think <gst_vaapisink_get_caps> has this line. this function in my file starts from line 730.
I think possible reason is you have mulitple versions of gst-vaapi installed and may confused them. you can use $ lsof -p pid  to track your so files loaded.
Another reason maybe something optimized in compiling. you can disable the DEBUG with ./configure CFLAGS="-g -O0" when you compile gst-vaapi.

Thanks.

Hi Wind,

./configure gst-vaapi  with CFLAGS="-g -O0"
and no mulitple versions of gst-vaapi installed 

Test again, the last result was the same as that.

Thanks.

please make sure you find <gst_vaapisink_get_caps> in your gstvaapisink.c file with the same line your valgrind log report. please let me know this result.

==9134== 300 (60 direct, 240 indirect) bytes in 1 blocks are definitely lost in loss record 1,893 of 2,059
4471 ==9134==    at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
4472 ==9134==    by 0x59825A4: nss_parse_service_list (nsswitch.c:678)
4473 ==9134==    by 0x5983065: __nss_database_lookup (nsswitch.c:175)
4474 ==9134==    by 0xCBC92A4: ???
4475 ==9134==    by 0x593B9BC: getpwnam_r@@GLIBC_2.2.5 (getXXbyYY_r.c:256)
4476 ==9134==    by 0x53DDE32: ??? (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.3)
4477 ==9134==    by 0x53DE99C: g_get_home_dir (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.3)
4478 ==9134==    by 0x4EA5B44: gst_update_registry (gstregistry.c:1643)
4479 ==9134==    by 0x4E57E1C: init_post (gst.c:793)
4480 ==9134==    by 0x53BDF6F: g_option_context_parse (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.3)
4481 ==9134==    by 0x403505: main (gst-launch.c:984)

This issue is not our bug.