Bug 14276 - missing support for video/H263 and other related
Summary: missing support for video/H263 and other related
Status: RESOLVED FIXED
Alias: None
Product: shared-mime-info
Classification: Unclassified
Component: freedesktop.org.xml (show other bugs)
Version: unspecified
Hardware: All All
: medium normal
Assignee: Shared Mime Info group
QA Contact:
URL: https://bugs.launchpad.net/bugs/502642
Whiteboard:
Keywords:
: 24690 (view as bug list)
Depends on:
Blocks:
 
Reported: 2008-01-28 05:07 UTC by Stanislav Brabec
Modified: 2010-05-21 10:13 UTC (History)
3 users (show)

See Also:
i915 platform:
i915 features:


Attachments
Adds support for video/mp2t (MPEG-2 transport streams) (1009 bytes, patch)
2010-03-03 19:48 UTC, Oliver Joos
Details | Splinter Review
Adds support for video/mp2t (MPEG-2 transport streams) + unique acronym (1.01 KB, patch)
2010-03-04 13:39 UTC, Oliver Joos
Details | Splinter Review
Patch to add support for video/mp2t (MPEG-2 transport streams) + unique acronym + unscrambled only (1.20 KB, patch)
2010-03-25 13:17 UTC, Oliver Joos
Details | Splinter Review
non-copyrighted mpeg2 transport stream recorded by a settopbox (128.70 KB, text/vnd.trolltech.linguist)
2010-04-07 17:49 UTC, Oliver Joos
Details
non-copyrighted mpeg2 transport stream recorded by a settopbox (256.48 KB, text/vnd.trolltech.linguist)
2010-04-14 16:19 UTC, Oliver Joos
Details

Description Stanislav Brabec 2008-01-28 05:07:12 UTC
IANA defines several H.263 MIME types:
http://www.iana.org/assignments/media-types/video/

http://www.rfc-editor.org/rfc/rfc3555.txt

freedesktop.org should provide at least basic support for them, at least suffix based detection.

http://www-mobile.ecs.soton.ac.uk/peter/h263/missa-10fps-QCIF-20Kbps.h263
is an example, from http://www-mobile.ecs.soton.ac.uk/peter/h263/h263.html
Comment 1 Neil Wilson 2009-11-01 02:12:14 UTC
This bug probably covers the lack of support for AVCHD HD video file extensions - as generated by many HD camcorders.

.MTS and .M2TS file extensions should resolve to mime type video/MP2T



Comment 2 Daniel Leidert 2010-01-29 12:58:18 UTC
*** Bug 24690 has been marked as a duplicate of this bug. ***
Comment 3 Daniel Leidert 2010-01-29 12:59:42 UTC
Patch for video/mp2t attached to Bug 24690.
Comment 4 Oliver Joos 2010-01-29 15:57:21 UTC
I just tested the new patch attached to http://bugs.freedesktop.org/show_bug.cgi?id=24690#c1

My .ts-Files indeed all start with 0x47. But after the patch is applied, all files that start with 0x47 are "video/mp2t", even if their filenames do not end with ".ts". I have here .ts-files that come together with meta files with same name but different suffix. Some of these meta files incidentally start with 0x47, and hence are treated as videos too.

I'd prefer that files are "video/mp2t" only if they start with 0x47 *AND* their filename ends with '.ts' (or '.mp2t' or '.mpts' or '.mpg' or other suffix that might be common for mpeg transport stream files).
Comment 5 Daniel Leidert 2010-01-29 16:06:37 UTC
Can you test with this pattern please?

<match type="big32" mask="0xff5fff1f" value="0x47400010" offset="0"/>

Regards, Daniel
Comment 6 Oliver Joos 2010-01-29 17:08:28 UTC
This does not match my .ts. They were recorded on a digital cable receiver and start like this:

00000000  47 40 1f 10 00 7f 80 24  00 00 01 00 00 00 00 27  |G@.....$.......'|
...

If I remove some more bits from your mask then it matches my .ts too:
(I did this without knowledge about mpeg stream standards!)

  <match type="big32" mask="0xff5fe01f" value="0x47400010" offset="0"/>
Comment 7 Oliver Joos 2010-03-03 19:48:02 UTC
Created attachment 33748 [details] [review]
Adds support for video/mp2t (MPEG-2 transport streams)

I have here the patch to recognize video/mp2t (MPEG-2 transport streams). It's an enhanced version of https://bugs.freedesktop.org/show_bug.cgi%3Fid%3D24690%23c1. Naming comes from http://www.rfc-editor.org/rfc/rfc3555.txt (page 38). The glob patterns come from http://en.wikipedia.org/wiki/MPEG_transport_stream and are all not yet mentioned in the mime database, except *.ts and *.m2t.

To control matching order of magic patterns one may use "magic priority=", but glob patterns match in order of their appearance in the database file, so:

*.ts is currently recognized as application/x-linguist but without any magic pattern. If we position video/mp2t AFTER it, we can control with the magic pattern if a *.ts becomes video/mp2t or not. I carefully chose the magic pattern to be exactly as strict as necessary to match all valid video/mp2t.

*.m2t is currently recognized as video/mpeg which is wrong according to Wikipedia and e.g. https://bugs.launchpad.net/bugs/89543. So I chose to position video/mp2t BEFORE video/mpeg.
Comment 8 Stanislav Brabec 2010-03-04 08:00:17 UTC
Maybe this is be better:

+    <_comment>MPEG-2 Transport Stream</_comment>
+    <acronym>MPEG-2 TS</acronym>
+    <expanded-acronym>Moving Picture Experts Group 2 Transport Stream</expanded-acronym>

It is not wise to use the same acronym for two different MIME types.
Comment 9 Oliver Joos 2010-03-04 13:39:15 UTC
Created attachment 33767 [details] [review]
Adds support for video/mp2t (MPEG-2 transport streams) + unique acronym

@Stanislav: Thanks! I don't know where these acronyms are used, but it is definitely better to keep them unique. So I replaced <acronym> and <expanded-acronym>.

The lower-case letters in <_comment> were chosen intentionally, because other mime types are like this. Probably it does not matter - this attribute is later mapped to a translated string anyway.
Comment 10 Oliver Joos 2010-03-13 07:21:27 UTC
Please verify patch.
Comment 11 Oliver Joos 2010-03-25 13:17:43 UTC
Created attachment 34447 [details] [review]
Patch to add support for video/mp2t (MPEG-2 transport streams) + unique acronym + unscrambled only

Patch has been improved. I added another 2 bits to the magic mask to only match if a transport stream is not scrambled. This is ok because scrambled streams could not be handled by common desktop apps anyway. With this mask GIMP resources don't match anymore. (They start with the 4 bytes "GIMP" which is a valid MP2T header)

If you own MP2T files (from a camcorder or a settop box), please test if they match and leave a comment! (Feel free to ask me how to apply the patch to your system)

For details about successful tests see comment 7+8 on https://bugs.launchpad.net/ubuntu/+source/shared-mime-info/+bug/502642
Comment 12 Oliver Joos 2010-04-01 07:19:47 UTC
Please add this patch to shared-mime-info. Otherwise other bugs are blocked:
https://bugzilla.gnome.org/show_bug.cgi?id=614422
http://trac.videolan.org/vlc/ticket/3485
http://bugs.xine-project.org/show_bug.cgi?id=341

The patch has been successfully applied for Ubuntu 10.04 (see URL in bug details)
Comment 13 Ville Skyttä 2010-04-04 10:19:36 UTC
Current git now has magic for linguist files, could you check if that causes any needs for changes to this patch?

A (small as sanely possible) test file would be welcome for the test suite, do you happen to have one available?
Comment 14 Oliver Joos 2010-04-06 02:01:24 UTC
I checked new magic for linguist type in current git and saw no problem. The new magic alone does not prevent video/mp2t files to match linguist type. But it does not interfere with my patch either. Together with my patch video/mp2t still are recognized reliably.

I did not check if true application/x-linguist files are affected by my patch, because I have none. But I am pretty sure that this is not the case, because the magic in my patch is quite restrictive. And to verify this I pulled the source of shared-mime-info (including all its test-files) into an Ubuntu Live-CD and then I did:

  $ sudo find /bin /boot /etc /home /lib /opt /root /sbin /tmp /usr /var \
     -xdev -type f -exec mimetype \{\} \; | grep video/mp2t

This showed no false positives.

For a small video/mp2t test file see https://bugs.launchpad.net/ubuntu/+source/shared-mime-info/+bug/502642/comments/7.
Comment 15 Ville Skyttä 2010-04-06 23:22:46 UTC
There's a linguist test file in git.  Adding this to tests/list (at end of the video files section) and the mp2t test file in place makes the test suite pass:

test-mp2t.ts video/mp2t xo

The test file seems to be taken from copyrighted material.  It's just a couple of seconds but I suppose it'd be better to have a non-copyrighted one there if someone can produce one e.g. with a camcorder.
Comment 16 Oliver Joos 2010-04-07 17:49:42 UTC
Created attachment 34779 [details]
non-copyrighted mpeg2 transport stream recorded by a settopbox

Thanks for testing. I agree that only non-copyrighted streams should be used as test files. I recorded a video/mp2t snippet with black picture and silent audio. This attachment should be ok to be included in shared-mime-info.
Comment 17 Oliver Joos 2010-04-14 16:19:42 UTC
Created attachment 35043 [details]
non-copyrighted mpeg2 transport stream recorded by a settopbox

The previous attachment was sub-optimal. It shows is black image sequence but indeed is a non-decrypted content. Now I found a test channel with a nice test pattern (color stripes without any letters or logos). Therefore I submit a new attachment which is IMHO well-suited for inclusion into the source tree of shared-mime-info.
Comment 18 Ville Skyttä 2010-05-21 10:13:34 UTC
commit e88b56dd9836c36124baf69d88ae22be8ef776b1
Author: Oliver Joos <oliver.joos@mymail.ch>
Date:   Fri May 21 20:06:31 2010 +0300

    Add video/mp2t.

    http://bugs.freedesktop.org/show_bug.cgi?id=14276


Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.