Bug 3104 - Compose table cache for faster X11 application starts
Compose table cache for faster X11 application starts
Product: xorg
Classification: Unclassified
Component: Lib/Xlib
All All
: high enhancement
Assigned To: Matthias Hopf
: l10n, patch
Depends on:
  Show dependency treegraph
Reported: 2005-04-22 09:35 UTC by Matthias Hopf
Modified: 2009-01-28 02:54 UTC (History)
7 users (show)

See Also:

Pointerless compose data structure patch (25.88 KB, patch)
2005-04-22 09:37 UTC, Matthias Hopf
no flags Details | Splinter Review
Compose cache patch (17.67 KB, patch)
2005-04-22 09:40 UTC, Matthias Hopf
no flags Details | Splinter Review
Helper program for creating global cache files (5.81 KB, patch)
2005-04-22 09:46 UTC, Matthias Hopf
no flags Details | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Matthias Hopf 2005-04-22 09:35:39 UTC
As talked about on the Xorg developer conference, I implemented a compose
cache mechanism for XIM, based on the early implementation by Lubos Lunak
(llunak@suse.de). We should still think about replacing XIM in the long
term, but these patches help for application startups in the short term.

The compose cache accelerates program startup times and memory consumption
considerably, especially for UTF8 locales. If a cache file is found during
the startup of an application, the according cache is mmap()ed.

Speedup is approx. 40-200ms decreased startup time for every X application,
depending on your processor.  Approx. 240KB memory is saved per application.
This is for UTF8 locales, differences are much lower for non-UTF8 locales,
as well as for home-grown compose tables.

The patches also implement a small (single entry) cache to _XlcLocaleDirName
to deal with repeated calls.

Cache files are searched in a global cache dir (read-only) and in the
.compose-cache diretory of the user's home. If this directory does not
exist, it is *not* created, effectively disabling the possibility of
creating cache files.

The whole thing consists of three patches, one for flattening out the
internal database, one for the cache, and one for a helper program.

The patches have been applied with SuSE Linux 9.3 (already shipping), and
global cache files are shipped for all UTF8 locales as well.
No problems so far.
Comment 1 Matthias Hopf 2005-04-22 09:37:48 UTC
Created attachment 2509 [details] [review]
Pointerless compose data structure patch

This patch redefines the compose data structures to use indices instead of
pointers. This is necessary to be able to mmap() the internal data
structures instead of parsing the compose table.  Memory requirements are a
bit higher than before (I cannot realloc() for every single binding, so I 
multiply by 1.5 each time space runs out), but performance is equivalent.

Tested this patch with all locales and input methods known to me and finally
works with SCIM again as well.	Cache files are endian-safe and
typesize-safe (both encoded into filenames) and versioned.

There are a couple of issues to be thought about, though:

- I'm not exactly sure about the differences between INT32 and BITS32 types.  
  I chose these types for creating fixed sized structures.

- I defined the type of modifiers in the structure to be BITS32. It was just
  'unsigned' before. Yes, no int or anything else.

- How much memory should I allocate in the very beginning? I chose numbers 
  so that the 'C' locale bindings just fit nicely into memory.

- I have *not* tested the new XOpenFileMode function that is needed for 
  Windows only (_XOpenFileMode) due to lack of an according development

- Memory that has been allocated during the creation of the structure is not
  freed again. This could happen, when the Compose files contain errors or
  overwrites already defined keysyms.

- The Thai imput method is inherrently broken, and we should think about
  dropping it altogether.
Comment 2 Matthias Hopf 2005-04-22 09:40:52 UTC
Created attachment 2510 [details] [review]
Compose cache patch

This is the compose cache itself.

The parsed compose tables are stored in /var/X11R6/compose-cache/ (global
cache) or ~/.compose-cache/ (local cache), respectively. Files in the local
cache expire once per day. File dates are compared with the compose table
source files as well. suid and sgid programs are not allowed to use the
local cache.

Caches currently depend on the used locale (more exactly: on the encoding),
but this isn't respected in the cache file format. This has to be changed.
So at least for global cache files, only UTF8 compose files should be

For user caches this is less problematic as clashes might only occure if the
a) has created a .compose-cache
b) uses different locales at the same time
c) these locales use the same compose table 
d) parsing of the compose table differs for the used

Issues to be discussed:

- The per-user cache directory ~/.compose-cache/ is not created if it does
  not exist. In this case non-system compose tables are not cached at all.
  Should it be created automatically if it does not exist?
  I personally doubt this is really usefull.
Comment 3 Matthias Hopf 2005-04-22 09:46:55 UTC
Created attachment 2511 [details] [review]
Helper program for creating global cache files

This tool creates a cache file for a given locale and compose file.
Unfortunately, due to the way XIM is integrated into libX11, a Xserver
has to be running in order to create a cache. If somebody sees a clearer
way to deal with this you're very welcome to change it.

For distributors I also add an example script below that shows how global cache

files for the three UTF8 compose tables can created using Xvfb. This is only a
skeleton, though.

Currently paths are hardcoded. Also note that compose tables for non-UTF8
locales currently shouldn't be cached globaly, as the encoding may differ
with different locales. This is something to be addressed.

------> CUT <-------

if [ $# -gt 1 ]; then
  echo "$0 [x11-root]"
  exit 0

if [ $# -eq 1 ]; then

mkdir -p $ROOT/var/X11R6/compose-cache

tmpfile=$(mktemp /tmp/Xvfb.log.XXXXXXXXXX)
$ROOT/usr/X11R6/bin/Xvfb \
  -fp $ROOT/usr/X11R6/lib/X11/fonts/misc/ \
  -sp $ROOT/etc/X11/xserver/SecurityPolicy \
  -co $ROOT/usr/X11R6/lib/X11/rgb \
  :99 &> $tmpfile &
trap "kill $!; rm $tmpfile || true" EXIT

export DISPLAY=:99
export LD_LIBRARY_PATH=$ROOT/usr/X11R6/lib64:$ROOT/usr/X11R6/lib
export XLOCALEDIR=$ROOT/usr/X11R6/lib/X11/locale/
sleep 5
$ROOT/usr/X11R6/bin/xbiff &> /dev/null &

pushd $ROOT/

usr/X11R6/bin/mkcomposecache "en_US.UTF-8"
"usr/X11R6/lib/X11/locale/en_US.UTF-8/Compose" "var/X11R6/compose-cache"
usr/X11R6/bin/mkcomposecache "pt_BR.UTF-8"
"usr/X11R6/lib/X11/locale/pt_BR.UTF-8/Compose" "var/X11R6/compose-cache"
usr/X11R6/bin/mkcomposecache "el_GR.UTF-8"
"usr/X11R6/lib/X11/locale/el_GR.UTF-8/Compose" "var/X11R6/compose-cache"


chmod 444 $ROOT/var/X11R6/compose-cache/*
Comment 4 Matthias Hopf 2005-06-13 07:40:44 UTC
Egbert asked for it :)
Comment 5 Erik Andren 2006-04-23 06:08:40 UTC
Adding patch keyword. Any plan for merging?
Comment 6 Matthias Hopf 2006-04-24 22:35:15 UTC
We're in the middle of a release phase. After that I plan to address a couple of
minor issues I'm seeing with my patch, and merge it.
Comment 7 Daniel Stone 2006-06-03 02:50:45 UTC
7.1 is out now, so, ping?
Comment 8 Matthias Hopf 2006-06-06 03:04:43 UTC
Next thing to work on for me. Might be distracted a bit right now due to SLES 10.
The code still needs some cleanup, especially cache file names should include
Comment 9 Matthias Hopf 2006-07-04 09:40:27 UTC
Commited & released.
Comment 10 Martin Pitt 2009-01-28 02:54:40 UTC
Out of interest, why was the actual compose cache patch applied, but not the patch which adds the helper program (mkcomposecache)?