Bugzilla – Bug 3104
Compose table cache for faster X11 application starts
Last modified: 2009-01-28 02:54:40 UTC
As talked about on the Xorg developer conference, I implemented a compose
cache mechanism for XIM, based on the early implementation by Lubos Lunak
(firstname.lastname@example.org). We should still think about replacing XIM in the long
term, but these patches help for application startups in the short term.
The compose cache accelerates program startup times and memory consumption
considerably, especially for UTF8 locales. If a cache file is found during
the startup of an application, the according cache is mmap()ed.
Speedup is approx. 40-200ms decreased startup time for every X application,
depending on your processor. Approx. 240KB memory is saved per application.
This is for UTF8 locales, differences are much lower for non-UTF8 locales,
as well as for home-grown compose tables.
The patches also implement a small (single entry) cache to _XlcLocaleDirName
to deal with repeated calls.
Cache files are searched in a global cache dir (read-only) and in the
.compose-cache diretory of the user's home. If this directory does not
exist, it is *not* created, effectively disabling the possibility of
creating cache files.
The whole thing consists of three patches, one for flattening out the
internal database, one for the cache, and one for a helper program.
The patches have been applied with SuSE Linux 9.3 (already shipping), and
global cache files are shipped for all UTF8 locales as well.
No problems so far.
Created attachment 2509 [details] [review]
Pointerless compose data structure patch
This patch redefines the compose data structures to use indices instead of
pointers. This is necessary to be able to mmap() the internal data
structures instead of parsing the compose table. Memory requirements are a
bit higher than before (I cannot realloc() for every single binding, so I
multiply by 1.5 each time space runs out), but performance is equivalent.
Tested this patch with all locales and input methods known to me and finally
works with SCIM again as well. Cache files are endian-safe and
typesize-safe (both encoded into filenames) and versioned.
There are a couple of issues to be thought about, though:
- I'm not exactly sure about the differences between INT32 and BITS32 types.
I chose these types for creating fixed sized structures.
- I defined the type of modifiers in the structure to be BITS32. It was just
'unsigned' before. Yes, no int or anything else.
- How much memory should I allocate in the very beginning? I chose numbers
so that the 'C' locale bindings just fit nicely into memory.
- I have *not* tested the new XOpenFileMode function that is needed for
Windows only (_XOpenFileMode) due to lack of an according development
- Memory that has been allocated during the creation of the structure is not
freed again. This could happen, when the Compose files contain errors or
overwrites already defined keysyms.
- The Thai imput method is inherrently broken, and we should think about
dropping it altogether.
Created attachment 2510 [details] [review]
Compose cache patch
This is the compose cache itself.
The parsed compose tables are stored in /var/X11R6/compose-cache/ (global
cache) or ~/.compose-cache/ (local cache), respectively. Files in the local
cache expire once per day. File dates are compared with the compose table
source files as well. suid and sgid programs are not allowed to use the
Caches currently depend on the used locale (more exactly: on the encoding),
but this isn't respected in the cache file format. This has to be changed.
So at least for global cache files, only UTF8 compose files should be
For user caches this is less problematic as clashes might only occure if the
a) has created a .compose-cache
b) uses different locales at the same time
c) these locales use the same compose table
d) parsing of the compose table differs for the used
Issues to be discussed:
- The per-user cache directory ~/.compose-cache/ is not created if it does
not exist. In this case non-system compose tables are not cached at all.
Should it be created automatically if it does not exist?
I personally doubt this is really usefull.
Created attachment 2511 [details] [review]
Helper program for creating global cache files
This tool creates a cache file for a given locale and compose file.
Unfortunately, due to the way XIM is integrated into libX11, a Xserver
has to be running in order to create a cache. If somebody sees a clearer
way to deal with this you're very welcome to change it.
For distributors I also add an example script below that shows how global cache
files for the three UTF8 compose tables can created using Xvfb. This is only a
Currently paths are hardcoded. Also note that compose tables for non-UTF8
locales currently shouldn't be cached globaly, as the encoding may differ
with different locales. This is something to be addressed.
------> CUT <-------
if [ $# -gt 1 ]; then
echo "$0 [x11-root]"
if [ $# -eq 1 ]; then
mkdir -p $ROOT/var/X11R6/compose-cache
-fp $ROOT/usr/X11R6/lib/X11/fonts/misc/ \
-sp $ROOT/etc/X11/xserver/SecurityPolicy \
-co $ROOT/usr/X11R6/lib/X11/rgb \
:99 &> $tmpfile &
trap "kill $!; rm $tmpfile || true" EXIT
$ROOT/usr/X11R6/bin/xbiff &> /dev/null &
chmod 444 $ROOT/var/X11R6/compose-cache/*
Egbert asked for it :)
Adding patch keyword. Any plan for merging?
We're in the middle of a release phase. After that I plan to address a couple of
minor issues I'm seeing with my patch, and merge it.
7.1 is out now, so, ping?
Next thing to work on for me. Might be distracted a bit right now due to SLES 10.
The code still needs some cleanup, especially cache file names should include
Commited & released.
Out of interest, why was the actual compose cache patch applied, but not the patch which adds the helper program (mkcomposecache)?