Bug 51118

Summary:	mpeg-ts files are almost never detected correctly
Product:	shared-mime-info	Reporter:	Edward Hervey <bilboed>
Component:	freedesktop.org.xml	Assignee:	Shared Mime Info group <shared_mime_info>
Status:	RESOLVED FIXED	QA Contact:
Severity:	normal
Priority:	medium	CC:	basic.master, bugzilla, daniel.leidert.spam, oliver.joos
Version:	unspecified
Hardware:	All
OS:	All
Whiteboard:
i915 platform:		i915 features:
Attachments:	Patch to fix TS/M2TS detection Fix MPEG-2 TS detection Fix MPEG-2 TS test

Description Edward Hervey 2012-06-15 06:42:11 UTC

The check for mpeg-ts files almost never identifies .ts files.

According to the mpeg-ts specifications, the 32 first bits are:

sync_byte                           :  8 bits (always 0x47)
transport_error_indication          :  1 bit
payload_unit_start_indicator        :  1 bit
transport_priority                  :  1 bit
PID                                 : 13 bits
transport_scrambling_control        :  2 bits
adaptation_field_control            :  2 bits
continuity_counter                  :  4 bits

What the current mask/value (0xff4000df/0x47400010) is checking is that:
* the first 8 bits are 0x47. No problem there.
* the payload_unit_start_indicator is set : the probability for this to be set on a random packet is very small.
* the transport_scrambling_control is set to 0x00 . This will fail for any mpeg-ts dumps where that first packet is scrambled.
* the adaptation_field_control second bit is set to 1, meaning there is a payload. So if your first packet only contains an adaptation field (like for the PCR pid), this will fail.
* the continuity_counter is 0. This is the most insane check this could be any value.

So essentially this check has got virtually no chance of working on real-world mpeg-ts streams.

The only thing you can realistically check is whether the first byte is 0x47.

Comment 1 Bastien Nocera 2012-06-15 06:55:12 UTC

Oliver and Daniel are the ones that added this magic data in bug 14276.

Comment 2 Oliver Joos 2012-06-15 15:55:16 UTC

@Edward: Please have a look at the discussion on bug 14276. My intention was to keep the magic as restrictive as possible, so that there are no false positives.

AFAIK scrambled streams cannot be played by common players like vlc, mplayer, totem or xine. So matching them is not of much practical use.
Streams can be cut at any position. But the recorded streams I saw all start with continuity_counter 0 and have payload_unit_start_indicator set.
I agree that checking the second bit of adaptation_field_control could prevent some useful matches, but I did not see such examples yet.

If you have playable mpeg-ts files that do not match with the current magic then please share them, with a comment where they are from. I'd really welcome any further improvements of mpeg-ts handling!
But please think twice before changing the current magic just because the mpeg-ts standard theoretically would allow it.

A check for false positives:  (may take hours!)
  sudo find /bin /boot /etc /home /lib /opt /root /sbin /tmp /usr /var -xdev -type f -exec mimetype \{\} \; >/tmp/mimetypes.lst
  grep video/mp2t /tmp/mimetypes.lst

Watch out for textfiles that begin with "G" (= "\x47"). Unfortunately they are quite common in systems that run *G*nome, e.g. GIMP brushes in /usr/share/gimp/2.0/brushes/  ;-)

Comment 3 Bastien Nocera 2013-02-13 10:53:11 UTC

Any more comments Edward?

Comment 4 Stefan Pöschel 2016-01-09 20:43:26 UTC

Created attachment 120922 [details] [review]
Patch to fix TS/M2TS detection

Comment 5 Stefan Pöschel 2016-01-09 20:43:47 UTC

I am as well affected by the broken TS file detection. As like Edward correctly pointed out, the initial value of the continuity counter is not defined anywhere and checking it for zero is useless. IMO the other checks also are not quite useful as they exclude certain TS files.

My proposal is to take advantage of the fixed frame size of TS files and therefore check the first five frames for the sync byte 0x47.
I added a regarding patch. It does the check for 188 byte frames of regular TS and for 192 byte frames of M2TS.

Comment 6 Stefan Pöschel 2016-07-31 16:26:08 UTC

Hmm, possibly I have to change the status...

Comment 7 Bastien Nocera 2016-08-02 11:30:52 UTC

(In reply to Stefan Pöschel from comment #6)
> Hmm, possibly I have to change the status...

Please git-format the patch (which means including a commit message, and explanation of the changes made), and, please also attach a separate patch which would contain a test case that fails before the TS/M2TS patch is committed, and fixed after.

More information about contributions are in the HACKING file, thanks.

Comment 8 Stefan Pöschel 2016-08-02 18:56:39 UTC

Created attachment 125488 [details] [review]
Fix MPEG-2 TS detection

Comment 9 Stefan Pöschel 2016-08-02 18:56:57 UTC

Created attachment 125489 [details] [review]
Fix MPEG-2 TS test

Comment 10 Stefan Pöschel 2016-08-02 18:58:57 UTC

Please find the two regarding patches added.

The TS file of the second patch is non-copyrighted (created with my MPEG-2 HW encoder TV card).

Comment 11 Bastien Nocera 2016-08-02 20:14:19 UTC

Pushed, though I fixed the commit message for the magic changes, and adding, not replacing, the MPEG-2 TS test case.

Comment 12 Stefan Pöschel 2016-08-02 21:24:51 UTC

Fine for me. Thanks!

Comment 13 Oliver Joos 2016-08-03 10:06:39 UTC

The new patch looks like an improvement over my patch in 2010!

I found that EkkehardDomning wrote a section about this new detection strategy on April 16th 2015 to https://de.wikipedia.org/wiki/MPEG-Transportstrom

@Stefan: perhaps you'd like to try my manual check for false positives:
Take a full-blown Linux system and run:
  $ sudo find /bin /boot /etc /home /lib /opt /root /sbin /tmp /usr /var \
     -xdev -type f -exec mimetype \{\} \; | grep video/mp2t
This should not show any false positives.

Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.