|Summary:||Compose table cache for faster X11 application starts|
|Product:||xorg||Reporter:||Matthias Hopf <mat>|
|Component:||Lib/Xlib||Assignee:||Matthias Hopf <mat>|
|Status:||RESOLVED FIXED||QA Contact:|
|Priority:||high||CC:||dberkholz, eich, erik.andren, martin.pitt, roland.mainz, simos.bugzilla, sndirsch|
|i915 platform:||i915 features:|
Description Matthias Hopf 2005-04-22 09:35:39 UTC
As talked about on the Xorg developer conference, I implemented a compose cache mechanism for XIM, based on the early implementation by Lubos Lunak (firstname.lastname@example.org). We should still think about replacing XIM in the long term, but these patches help for application startups in the short term. The compose cache accelerates program startup times and memory consumption considerably, especially for UTF8 locales. If a cache file is found during the startup of an application, the according cache is mmap()ed. Speedup is approx. 40-200ms decreased startup time for every X application, depending on your processor. Approx. 240KB memory is saved per application. This is for UTF8 locales, differences are much lower for non-UTF8 locales, as well as for home-grown compose tables. The patches also implement a small (single entry) cache to _XlcLocaleDirName to deal with repeated calls. Cache files are searched in a global cache dir (read-only) and in the .compose-cache diretory of the user's home. If this directory does not exist, it is *not* created, effectively disabling the possibility of creating cache files. The whole thing consists of three patches, one for flattening out the internal database, one for the cache, and one for a helper program. The patches have been applied with SuSE Linux 9.3 (already shipping), and global cache files are shipped for all UTF8 locales as well. No problems so far.
Comment 1 Matthias Hopf 2005-04-22 09:37:48 UTC
Created attachment 2509 [details] [review] Pointerless compose data structure patch This patch redefines the compose data structures to use indices instead of pointers. This is necessary to be able to mmap() the internal data structures instead of parsing the compose table. Memory requirements are a bit higher than before (I cannot realloc() for every single binding, so I multiply by 1.5 each time space runs out), but performance is equivalent. Tested this patch with all locales and input methods known to me and finally works with SCIM again as well. Cache files are endian-safe and typesize-safe (both encoded into filenames) and versioned. There are a couple of issues to be thought about, though: - I'm not exactly sure about the differences between INT32 and BITS32 types. I chose these types for creating fixed sized structures. - I defined the type of modifiers in the structure to be BITS32. It was just 'unsigned' before. Yes, no int or anything else. - How much memory should I allocate in the very beginning? I chose numbers so that the 'C' locale bindings just fit nicely into memory. - I have *not* tested the new XOpenFileMode function that is needed for Windows only (_XOpenFileMode) due to lack of an according development environment. - Memory that has been allocated during the creation of the structure is not freed again. This could happen, when the Compose files contain errors or overwrites already defined keysyms. - The Thai imput method is inherrently broken, and we should think about dropping it altogether.
Comment 2 Matthias Hopf 2005-04-22 09:40:52 UTC
Created attachment 2510 [details] [review] Compose cache patch This is the compose cache itself. The parsed compose tables are stored in /var/X11R6/compose-cache/ (global cache) or ~/.compose-cache/ (local cache), respectively. Files in the local cache expire once per day. File dates are compared with the compose table source files as well. suid and sgid programs are not allowed to use the local cache. Caches currently depend on the used locale (more exactly: on the encoding), but this isn't respected in the cache file format. This has to be changed. So at least for global cache files, only UTF8 compose files should be considered. For user caches this is less problematic as clashes might only occure if the user a) has created a .compose-cache b) uses different locales at the same time c) these locales use the same compose table d) parsing of the compose table differs for the used locales Issues to be discussed: - The per-user cache directory ~/.compose-cache/ is not created if it does not exist. In this case non-system compose tables are not cached at all. Should it be created automatically if it does not exist? I personally doubt this is really usefull.
Comment 3 Matthias Hopf 2005-04-22 09:46:55 UTC
Created attachment 2511 [details] [review] Helper program for creating global cache files This tool creates a cache file for a given locale and compose file. Unfortunately, due to the way XIM is integrated into libX11, a Xserver has to be running in order to create a cache. If somebody sees a clearer way to deal with this you're very welcome to change it. For distributors I also add an example script below that shows how global cache files for the three UTF8 compose tables can created using Xvfb. This is only a skeleton, though. Currently paths are hardcoded. Also note that compose tables for non-UTF8 locales currently shouldn't be cached globaly, as the encoding may differ with different locales. This is something to be addressed. ------> CUT <------- #!/bin/sh if [ $# -gt 1 ]; then echo "$0 [x11-root]" exit 0 fi if [ $# -eq 1 ]; then ROOT=$1 fi mkdir -p $ROOT/var/X11R6/compose-cache tmpfile=$(mktemp /tmp/Xvfb.log.XXXXXXXXXX) $ROOT/usr/X11R6/bin/Xvfb \ -fp $ROOT/usr/X11R6/lib/X11/fonts/misc/ \ -sp $ROOT/etc/X11/xserver/SecurityPolicy \ -co $ROOT/usr/X11R6/lib/X11/rgb \ :99 &> $tmpfile & trap "kill $!; rm $tmpfile || true" EXIT export DISPLAY=:99 export LD_LIBRARY_PATH=$ROOT/usr/X11R6/lib64:$ROOT/usr/X11R6/lib export XLOCALEDIR=$ROOT/usr/X11R6/lib/X11/locale/ sleep 5 $ROOT/usr/X11R6/bin/xbiff &> /dev/null & pushd $ROOT/ usr/X11R6/bin/mkcomposecache "en_US.UTF-8" "usr/X11R6/lib/X11/locale/en_US.UTF-8/Compose" "var/X11R6/compose-cache" "/usr/X11R6/lib/X11/locale/en_US.UTF-8/Compose" usr/X11R6/bin/mkcomposecache "pt_BR.UTF-8" "usr/X11R6/lib/X11/locale/pt_BR.UTF-8/Compose" "var/X11R6/compose-cache" "/usr/X11R6/lib/X11/locale/pt_BR.UTF-8/Compose" usr/X11R6/bin/mkcomposecache "el_GR.UTF-8" "usr/X11R6/lib/X11/locale/el_GR.UTF-8/Compose" "var/X11R6/compose-cache" "/usr/X11R6/lib/X11/locale/el_GR.UTF-8/Compose" popd chmod 444 $ROOT/var/X11R6/compose-cache/*
Comment 4 Matthias Hopf 2005-06-13 07:40:44 UTC
Egbert asked for it :)
Comment 5 Erik Andren 2006-04-23 06:08:40 UTC
Adding patch keyword. Any plan for merging?
Comment 6 Matthias Hopf 2006-04-24 22:35:15 UTC
We're in the middle of a release phase. After that I plan to address a couple of minor issues I'm seeing with my patch, and merge it.
Comment 7 Daniel Stone 2006-06-03 02:50:45 UTC
7.1 is out now, so, ping?
Comment 8 Matthias Hopf 2006-06-06 03:04:43 UTC
Next thing to work on for me. Might be distracted a bit right now due to SLES 10. The code still needs some cleanup, especially cache file names should include encoding.
Comment 9 Matthias Hopf 2006-07-04 09:40:27 UTC
Commited & released.
Comment 10 Martin Pitt 2009-01-28 02:54:40 UTC
Out of interest, why was the actual compose cache patch applied, but not the patch which adds the helper program (mkcomposecache)?