Following advice from Josep Torra:
If an appsrc sets timestamps on buffers, it has to be configured in TIME format with a:
g_object_set (data.app_source, "format", 3, NULL);
Confirmed, this is only needed when the appsrc creates timestamps though. The documentation should mention this and set the format to GST_FORMAT_TIME in the code.