Bug 16475 - CharSet-to-CompoundText Conversion Failed for Thai Locales
Summary: CharSet-to-CompoundText Conversion Failed for Thai Locales
Status: RESOLVED FIXED
Alias: None
Product: xorg
Classification: Unclassified
Component: Lib/Xlib (data) (show other bugs)
Version: 7.3 (2007.09)
Hardware: Other All
: medium normal
Assignee: Daniel Stone
QA Contact: Xorg Project Team
URL:
Whiteboard:
Keywords: i18n, patch
Depends on:
Blocks:
 
Reported: 2008-06-23 04:40 UTC by Theppitak Karoonboonyanan
Modified: 2009-04-13 09:30 UTC (History)
1 user (show)

See Also:
i915 platform:
i915 features:


Attachments
ct_encoding patch for Thai XLC_LOCALE's (849 bytes, patch)
2008-06-23 04:44 UTC, Theppitak Karoonboonyanan
no flags Details | Splinter Review
updated patch with ISO10646-1 added (922 bytes, patch)
2008-06-23 06:03 UTC, Theppitak Karoonboonyanan
no flags Details | Splinter Review
with ISO8859-11 fontset (1.29 KB, patch)
2009-02-12 01:59 UTC, Theppitak Karoonboonyanan
no flags Details | Splinter Review

Description Theppitak Karoonboonyanan 2008-06-23 04:40:51 UTC
This is yet another fix for SCIM on Thai locales, after bug 15719.

With bug 15719 fixed, SCIM now loads on Thai locale. But it commits nothing for Thai input.

Steps to reproduce:
1. Install a Thai-capable SCIM module, e.g. scim-tables or scim-thai
2. scim -d
3. LC_ALL=th_TH.UTF-8 XMODIFIERS=@im=SCIM xterm
4. Switch to Thai IM and type some characters

Expected result:
- Thai characters appear

What actually happens:
- Nothing appear

This is because SCIM commits string in compound text form, converted via XwcTextListToTextProperty(). But the XLC_LOCALE for th_TH and th_TH.UTF-8 declares cs1's ct_encoding as TIS620-0:GR which is commented out in src/xlibi18n/lcCT.c default_ct_data, in favor of ISO8859-11 ESC sequence.

Declaring this as ISO8859-11:GR instead just makes it work.
Comment 1 Theppitak Karoonboonyanan 2008-06-23 04:44:29 UTC
Created attachment 17321 [details] [review]
ct_encoding patch for Thai XLC_LOCALE's
Comment 2 Theppitak Karoonboonyanan 2008-06-23 04:45:53 UTC
Change component to Lib/Xlib (data). Add i18n, patch tags.
Comment 3 Daniel Stone 2008-06-23 05:17:49 UTC
Is there any plan for UTF-8 support?
Comment 4 Theppitak Karoonboonyanan 2008-06-23 05:33:51 UTC
(In reply to comment #3)
> Is there any plan for UTF-8 support?

Oh, yes. You mean adding cs2 as ISO10646-1, right?

Comment 5 Theppitak Karoonboonyanan 2008-06-23 06:03:59 UTC
Created attachment 17324 [details] [review]
updated patch with ISO10646-1 added
Comment 6 Theppitak Karoonboonyanan 2008-07-21 23:54:32 UTC
Hello. How is the patch? Is it OK for the commit?
Comment 7 Theppitak Karoonboonyanan 2008-09-26 19:01:38 UTC
This bug seems to have caused LP#273856 [1] against Ubuntu Intrepid, and a workaround concluded in a local forum is to remove scim. But I think it had better been fixed where the problem is caused.

  [1] https://bugs.launchpad.net/ubuntu/+bug/273856
Comment 8 Theppitak Karoonboonyanan 2009-02-09 18:22:27 UTC
*ping*

Note that this patch has been applied in Ubuntu Intrepid [1], and Thai Ubuntu users are happy with that.

  [1] https://bugs.launchpad.net/ubuntu/+bug/273856/comments/26

In fact, this bug is found in Debian, too. I just didn't file a Debian bug because I've already filed it here upstream. It seems to be a common practice for Debian to just forward bugs upstream (e.g. [2]). So, filing bug there appears to be redundant, once it's reported upstream.

  [2] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=443800

How about fixing this?
Comment 9 Thanomsub Noppaburana 2009-02-11 23:39:42 UTC
I found this bug in openSUSE 11.1 which is using libX11-1.1.5, too.

And everything work fine after i applied these patches:

  - Patch of bug #12759 for fix Thai XIM fails to retrieve multi-byte surrounding text on UTF-8 locale
  - Patch of bug #12517 for fix Thai XIM does not filter inputs when NumLock/CapsLock is on
  - Patch of bug #16457 (this bug) for fix CharSet-to-CompoundText Conversion Failed for Thai Locales

But when i was trying to run the simple X applications e.g. Xclock, Xmessage,
it will delay serveral seconds before the app window appeared 
and the terminal have shown the message:
"Warning: Missing charsets in String to FontSet conversion"

I guess this problem was came from the font set which assigned to TIS620-0,
so i had replaced the fs section from TIS620-0:GR to ISO8859-11:GR and try to run X apps again, then it Worked!!

In fact, openSUSE does not generates "tis620-0" encoding lines for Thai font records in neither fonts.dir nor fonts.scale, they have only iso8859-11 and tis620-2 lines.

So, How about replacing the TIS620-0:GR in the fs section to ISO8859-11:GR same as in the cs section ?
Comment 10 Theppitak Karoonboonyanan 2009-02-12 01:26:39 UTC
(In reply to comment #9)

> But when i was trying to run the simple X applications e.g. Xclock, Xmessage,
> it will delay serveral seconds before the app window appeared 
> and the terminal have shown the message:
> "Warning: Missing charsets in String to FontSet conversion"
> 
> I guess this problem was came from the font set which assigned to TIS620-0,
> so i had replaced the fs section from TIS620-0:GR to ISO8859-11:GR and try to
> run X apps again, then it Worked!!

Confirmed on Debian with all tis620-0 fonts.{dir,alias} entries removed (i.e. until "xlsfonts '*-tis620-0'" matches nothing). libx11 version used is 1.1.5.
Comment 11 Theppitak Karoonboonyanan 2009-02-12 01:43:56 UTC
(In reply to comment #9)

> In fact, openSUSE does not generates "tis620-0" encoding lines for Thai font
> records in neither fonts.dir nor fonts.scale, they have only iso8859-11 and
> tis620-2 lines.
> 
> So, How about replacing the TIS620-0:GR in the fs section to ISO8859-11:GR same
> as in the cs section ?

In fact, the tis620-0 entries provided in Debian are hard-coded by the font package maintainer (me), not by auto-generation with mkfontdir/mkfontscale. So, the absence of this problem on Debian is rather an exception.

So, if the X font utilities are made to generate iso8859-11 entries, not tis620-0, for Thai fonts, it should be reasonable to adjust either the X locale or the utilities to match each other. And, as iso8859-11 has been used in many parts of X11 for Thai, I think adjusting the X locale to iso8859-11 as proposed is more sensible.
Comment 12 Theppitak Karoonboonyanan 2009-02-12 01:59:04 UTC
Created attachment 22854 [details] [review]
with ISO8859-11 fontset
Comment 13 Thanomsub Noppaburana 2009-02-12 05:57:54 UTC
(In reply to comment #12)
> Created an attachment (id=22854) [details]
> with ISO8859-11 fontset
> 

I've tested the new patch already.
Now, everything works fine, any X apps can launch properly without tis620-0 entries in fonts.{dir,scale}.
Comment 14 Julien Cristau 2009-04-13 09:30:34 UTC
commit d108d3c706af3502820b5202564488ea19908b77
Author: Theppitak Karoonboonyanan <thep@linux.thai.net>
Date:   Thu Apr 9 11:25:25 2009 +0700

    CharSet-to-CompoundText Conversion Failed for Thai Locales
    
    SCIM fails to commit Thai input characters on Thai locales, because it commi
    string in compound text form, which was converted via
    XwcTextListToTextProperty(). But the XLC_LOCALE for th_TH and th_TH.UTF-8
    declares cs1's ct_encoding as TIS620-0:GR, which was commented out in
    src/xlibi18n/lcCT.c default_ct_data, in favor of ISO8859-11 ESC sequence.
    So, declaring cs1 as ISO8859-11:GR instead makes it work.
    
    Besides, for th_TH.UTF-8, adding cs2 class with ISO10646-1 encoding also add
    support for UTF-8 input.
    
    And, along discussion in the bug, a similar problem was found for fontset, t
    by causing delays on X apps startups on systems without tis620-0 fonts. This
    is normally the case, as mkfontdir and mkfontscale generate iso8859-11 entri
    by default for Thai X fonts. So, Thai fontset charset is also patched.
    
    X.Org Bug 16475 <http://bugs.freedesktop.org/show_bug.cgi?id=16475>
    
    Signed-off-by: Theppitak Karoonboonyanan <thep@linux.thai.net>
    Signed-off-by: Julien Cristau <jcristau@debian.org>


Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.