Bug 4939 - XCreateFontSet / XListFonts is very slow for UTF-8 locales
XCreateFontSet / XListFonts is very slow for UTF-8 locales
Status: RESOLVED DUPLICATE of bug 7832
Product: xorg
Classification: Unclassified
Component: Lib/Xfont
6.7.0
x86 (IA32) Linux (All)
: high normal
Assigned To: Xorg Project Team
Xorg Project Team
2011BRB_Reviewed
: patch
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2005-11-01 10:31 UTC by Canek Peláez
Modified: 2011-10-10 14:21 UTC (History)
5 users (show)

See Also:
i915 platform:
i915 features:


Attachments
log of the XCreateFontSet routine (5.69 KB, text/plain)
2011-01-02 13:56 UTC, Raphaël Droz
no flags Details
removes the 12 additionnal calls to get_font_name() in parse_omit_name() (683 bytes, patch)
2011-01-07 18:26 UTC, Raphaël Droz
no flags Details | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Canek Peláez 2005-11-01 10:31:36 UTC
X applications that load fonts had a *heavy* delay at startup time if using the
en_US.UTF-8 locale. The delay doesn't occur if the locale is set to
en_US.iso88591, or C.

I discovered the error with XEmacs, but it only happens with MULE. The bug also
happens in xfontsel and xman. The delay doesn't occur in xclock, for example. My
guess is because xclock doesn't use fonts. The bug is very easy to reproduce
(with LC_ALL unset):

$ LANG=en_US.UTF-8 xfontsel    # delay
$ LANG=en_US.iso88591 xfontsel # no delay
$ LANG=C xfontsel              # no delay

I'm not sure if the bug is related to Xft, fontconfig, or something else. I'm
reporting it here because I don't know better.

I'm using Gentoo, with X.org 6.8.4. Originally I reported the bug in Gentoo:

http://bugs.gentoo.org/show_bug.cgi?id=108686
Comment 1 Chris Bainbridge 2006-02-18 10:57:39 UTC
My locale is en_GB.UTF-8 setup as described at
http://www.gentoo.org/doc/en/utf-8.xml.

When starting Eterm with this locale there is a 2 minute (!) delay in the call
to XCreateFontSet. During this time the X process is consuming 100% of the cpu.
Changing the locale to C reduces the startup time to 8 seconds.

I also get the delay with xman and xfontsel.

strace shows lots of calls like:
read(3, 0xbfc68c80, 32)                 = -1 EAGAIN (Resource temporarily
unavailable)

backtrace:
0  0xffffe410 in __kernel_vsyscall ()
#1  0xb7bf521d in select () from /lib/tls/libc.so.6
#2  0xb7cbc56b in _XWaitForReadable () from /usr/lib/libX11.so.6
#3  0xb7cbc9ad in _XRead () from /usr/lib/libX11.so.6
#4  0xb7cbdbc6 in _XReply () from /usr/lib/libX11.so.6
#5  0xb7ca142f in XListFonts () from /usr/lib/libX11.so.6
#6  0xb7f23bc3 in get_font_name () from
/usr/lib/X11/locale/lib/common/xomGeneric.so.2
#7  0xb7f240cb in parse_fontdata () from
/usr/lib/X11/locale/lib/common/xomGeneric.so.2
#8  0xb7f2539c in create_oc () from /usr/lib/X11/locale/lib/common/xomGeneric.so.2
#9  0xb7cd89b1 in XCreateOC () from /usr/lib/libX11.so.6
#10 0xb7cd7c35 in XCreateFontSet () from /usr/lib/libX11.so.6
#11 0xb7f433c3 in create_fontset (font1=0x804f930 "fixed",
    font2=0xb7f964e8
"-misc-fixed-medium-r-semicondensed--13-*-75-*-c-*-iso10646-1") at command.c:1759
#12 0xb7f4444d in init_locale () at command.c:1801
#13 0xb7f46afb in init_command (argv=0xfffffdfe) at command.c:2996
#14 0xb7f82340 in eterm_bootstrap (argc=1, argv=0xbfbc5f04) at startup.c:331
#15 0x0804857e in main (argc=-514, argv=0xfffffdfe) at main.c:31
Comment 2 Chris Bainbridge 2006-02-22 06:29:54 UTC
Apparently the default glibc utf-8 locale performance just plain sucks
(http://sam.zoy.org/writings/utf8/). There are some hints in the unicode faq
http://www.cl.cam.ac.uk/~mgk25/unicode.html on optimising the code.
Comment 3 Chris Bainbridge 2006-02-28 20:41:45 UTC
Possibly a dupe of #2475?
Comment 4 Takano Akio 2006-08-10 01:40:10 UTC
See also: bug #7832.
Comment 5 Alan Coopersmith 2006-08-10 08:22:49 UTC
XCreateFontSet / XListFonts are core X11 font requests, not Xft - reassigning to
the right place so people see it.
Comment 6 Daniel Stone 2007-02-27 01:28:35 UTC
Sorry about the phenomenal bug spam, guys.  Adding xorg-team@ to the QA contact so bugs don't get lost in future.
Comment 7 Paulo Zanoni 2010-11-30 05:13:11 UTC
I think this is the same issue I've been investigating. I have a small app that might help testing.

Some libXt apps (like xmessage and xcalc) take a lot of time to start if:
- you have many fonts
- you don't have some specific fonts (see below)
- you're using utf-8.

For example, consider the following program:

===
#include <assert.h>
#include <locale.h>
#include <stdio.h>
#include <X11/Xlib.h>

int main(int argc, char *argv[]) {
    Display *d = XOpenDisplay(NULL);
    assert(d);

    char **missing_charset_list;
    int missing_charset_count;
    char *def_string;
    XFontSet f;
    int i;

    if (argc != 2) {
        fprintf(stderr, "PLEASE SPECIFY LOCALE\n");
        return 1;
    }
    setlocale(LC_ALL, argv[1]);

    f = XCreateFontSet(d, "-*-*-*-R-*-*-*-120-*-*-*-*,*", &missing_charset_list,
                       &missing_charset_count, &def_string);

    printf("missing_carset_count: %d\n", missing_charset_count);
    for (i = 0; i < missing_charset_count; i++) {
        printf("%s\n", missing_charset_list[i]);
    }

    XSync(d, False);
    return 0;
}
===

I start it with: ./cfs pt_BR.UTF-8

- if I have a lot of the "x11-font-*" packages installed
- but if I _don't_ have these fonts installed: x11-font-daewoo-misc, x11-font-isas-misc, x11-font-jis-misc, x11-font-wqy-bitmapfont,
Then this app will take like 7 seconds to start.

If I install daewoo-misc, isas-misc and jis-misc, the apps will start fast again...

This might not be the real fix for the algorithm slowness, but is a workaround =)

I'll keep investigating.
Comment 8 Raphaël Droz 2011-01-02 13:56:13 UTC
Created attachment 41576 [details]
log of the XCreateFontSet routine

with
- the code given in comment #7,
- libX11 1.4.0
- modules/om/generic/omGeneric.c with #define FONTDEBUG
- added a timer around the call to parse_omit_name() in parse_fontdata()
Comment 9 Raphaël Droz 2011-01-02 13:57:54 UTC
I found that most of the slowdown happens in:

http://cgit.freedesktop.org/xorg/lib/libX11/tree/modules/om/generic/omGeneric.c?id=f6af6dd2f76c12b56ec166bb771457b9f08fe246#n1165

This code seems to have been enhanced in the past ( ^deae12c ) but is still the main slow down for 90% of simple X apps.
With libX11 1.4.0, for the code given in comment #7, the results are :
- 250 msec: iso8859-1 (1 iteration)
- 3.5 sec : utf-8 (15 iterations + 2 iterations for NULL font_set->font_name)

And most of these 3 more seconds is spent for the NULL font names, more exactly in:
http://cgit.freedesktop.org/xorg/lib/libX11/tree/modules/om/generic/omGeneric.c?id=f6af6dd2f76c12b56ec166bb771457b9f08fe246#n971

Each call to parse_omit_name() adds approximately 1.5 seconds
(see attachment above)

As a side note, I think this problem is distinct from bug #7832 which modifies the libXfont.
Comment 10 James Cloos 2011-01-02 17:53:26 UTC
> I found that most of the slowdown happens in:

> http://cgit.freedesktop.org/xorg/lib/libX11/tree/modules/om/generic/omGeneric.c?id=f6af6dd2f76c12b56ec166bb771457b9f08fe246#n1165

All commit f6af6dd2f76c12b56ec166bb771457b9f08fe246 does is to replace
the idom:

  sprintf(r, "%s-%s", r, f[n]);

with:

  strcat(r, "-");
  strcat(r, f[n]);

(Note how r shows up as both the target and as a source for the
sprintf(3).)

If that slows down, then that suggests that there is something broken in
either your libc or the linkage.

Glibc’s C version of strcat(3) is simple and clean.  The assembly
versions, however, might be too complex for a simple invocation like
the one in omGeneraic.c.

Perhaps the whole for loop could be improved?
Comment 11 Raphaël Droz 2011-01-04 16:56:40 UTC
(In reply to comment #10)
> > I found that most of the slowdown happens in:
> 
> > http://cgit.freedesktop.org/xorg/lib/libX11/tree/modules/om/generic/omGeneric.c?id=f6af6dd2f76c12b56ec166bb771457b9f08fe246#n1165
> 
> All commit f6af6dd2f76c12b56ec166bb771457b9f08fe246 does is to replace
> the idom:
...
I'm sorry, the URL was misleading and the problem has nothing to do with this specific commit (I just wanted to point to a fixed line number)

As I attempted to show in the attached log, it happens that parse_omit_name() can takes 1.5 second before returning (called by parse_fontdata(), called by parse_fontname())

The loop l. 1165 of parse_fontname() may be rewritten (someone did in bug #7832) but the root is in parse_omit_name() which should either
- not be called innocently -especially if it is expected to return false-
- enhanced to spent less time
[ I still have to dig the insides of the slowness ]

I should also say that, obviously, removing KSC5601.1987-0 and GB2312.1980-0 from XLC_LOCALE skip them from the parse_fontname() loop and makes xcalc 4 times faster to launch.
Comment 12 Raphaël Droz 2011-01-07 18:25:08 UTC
Narrowed down :

The process so far is:

-*-*-*-R-*-*-*-120-*-*-*-* KSC5601.1987-0 : is not found
* KSC5601.1987-0 : is attempted which implies going through the part of parse_omit_name() intitled:
/* This may mot be needed anymore as XListFonts() takes care of this */

This part contains a loop calling get_font_name() for every possible wildcard prefix combinations:

*-*-GB2312.1980-0
*-*-*-GB2312.1980-0
...
*-*-*-*-*-*-*-*-*-*-*-*-*-GB2312.1980-0

And get_font_name() calls XListFonts() for each of them for a cost of 0.1 to 0.2 second for each.

The comment comes from c6349f43193b74a3c09945f3093a871b0157ba47 ("Merging XORG-CURRENT into trunk")


I didn't yet check but if I understand correctly:
XListFonts() already check every wildcard combination.
get_font_name() calls XListFonts().

get_font_name() is already called (twice) with the original font_data->name.

The 12 more times _seems_ useless.
Obvious patch, whose side effects are unknown, attached.
Comment 13 Raphaël Droz 2011-01-07 18:26:57 UTC
Created attachment 41765 [details] [review]
removes the 12 additionnal calls to get_font_name() in parse_omit_name()
Comment 14 Jeremy Huddleston 2011-10-07 18:12:14 UTC
Can you please send your patch to xorg-devel for review?
Comment 15 Peter 2011-10-08 06:04:45 UTC
Hm, as I said in bug 2475 for me in ru_RU.UTF-8 this is not an issue anymore. But probably it's worth to check in other locales as well.
Comment 16 Jeremy Huddleston 2011-10-08 10:57:54 UTC
Ok, then closing.  Thanks.
Comment 17 Raphaël Droz 2011-10-09 09:38:43 UTC
I'm still able to consistently reproduce the (huge) slowdown, with the test program from comment #7 and some X apps (like xcalc or xpdf).

Every (UTF-8) locale (generated beforehand) going through KSC5601.1987-0 and GB2312.1980-0 cases is affected.

I've not sent the patch in comment #13 to the mailing-list (see below about xpdf) but I hope there will be a consensus on reopening this bug (anyone reproducing)

# (be sure that the locale (eg: fr_FR.UTF-8) has been generated)
$ time ./cfs fr_FR.UTF-8
missing_charset_count: 2
KSC5601.1987-0
GB2312.1980-0

real	0m1.719s         # AMD 4200+ !
user	0m0.007s
sys	0m0.004s

# patch from comment #13 applied
$ time LD_LIBRARY_PATH=patched-x11-git-sources/.libs ./cfs fr_FR.UTF-8
missing_charset_count: 0

real	0m0.009s
user	0m0.005s
sys	0m0.003s


I'm quite sure there *is* a bug waiting a fix.
comment #13 is not the the right one (it makes xpdf dying with a bunch of warnings) but I'm confident it's a pointer to the right direction anyway.
Comment 18 Jeremy Huddleston 2011-10-10 14:21:20 UTC
Ok, then I think #7832 should fix this.

*** This bug has been marked as a duplicate of bug 7832 ***