|Summary:||UTF-8 section symbol (0xC2A7) invokes TIS-620 decoding|
|Status:||RESOLVED MOVED||QA Contact:|
|i915 platform:||i915 features:|
|Bug Depends on:||101218|
|Attachments:||File containing a single section sign in the midst of other text|
Description pokechu022+trackers-freedesktop 2017-06-06 02:47:45 UTC
Created attachment 131730 [details] File containing a single section sign in the midst of other text A single occurrence of the section sign (§) encoded in UTF-8 causes the file to be marked as TIS-620, even if the rest of the text is English. This can be seen with the attached file (which includes a single §); curiously adding more instances of § elsewhere usually causes the file to be correctly detected as UTF-8. This may be a duplicate of bug 101218, but it's a more specific case. This was first reported at https://github.com/notepad-plus-plus/notepad-plus-plus/issues/940, but I've narrowed it down to a bug in uchardet.
Comment 1 Jehan 2017-06-06 09:32:46 UTC
This is indeed very similar to bug 101218 in the fact that technically, the text is valid in both UTF-8 and TIS-620. I can see your attachment gets a confidence of 0.567169 in TIS-620, 0.5 in WINDOWS-1252 and 0.505 in UTF-8. These are all quite low confidences and the choice is therefore made out of chance. We should really add the language detection to UTF-8.
Comment 2 GitLab Migration User 2018-10-12 21:35:09 UTC
-- GitLab Migration Automatic Message -- This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity. You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/uchardet/uchardet/issues/4.