Current fontconfig caches are supposed to be matched one-by-one to fonts. when fonts has been moved somewhere or doing bind-mounts fonts directories somewhere else would causes rebuilding caches and can't share it on them (which is the main target this time). This also affects the startup time for applications and caches should supports relocation without rebuilding.
Proposed implementation: https://cgit.freedesktop.org/~tagoh/fontconfig/log/?h=relpath-cache-2 The features of this implementation and highlight: * Writing out FC_FILE value as the fixed string (with the length of PATH_MAX) to allow rewriting it according to the currrent fonts.conf and the location of cache dirs. * Changing the cache basename from the hash to the device ID + inode number. which is required to avoid scanning fonts (and dirs) when the targeted dirs aren't yet loaded as caches. * For those changes, FC_LEAF_FILE, FC_DEV_ID, and FC_INODE properties is added to cache and bumped the cache version to 8. * Set FONTCONFIG_LOOKUP_FONTPATH to 1 to enable this feature. when loading caches through mmap(), need to re-mapping it with MAP_PRIVATE because rewrite has to be done without affecting the original cache. so we can share it with host and bind-mounted env. this is the sort of workaround to reduce the cost on env where this isn't needed. The last one is surely a disadvantage on this implementation because applications can't see the benefit of updates coming from other processes through MAP_SHARED. so they have to restart applications every time when caches is updated. Also this is only enabled for non-win32 so far because I'm not confident if this works on win32.
There shouldn't be too much affect to switch to the private-mapping. the cache is creating atomically when it needs to be updated. so any updates won't be notified through the shared-mapping. Another concern is to increase the call of stat(2) which affects the performance. need some cleanup on APIs to reduce the unnecessary calls of stat(2).
I'm concerned about this. Note that MAP_PRIVATE doesn't do what you think it does. Quoting manpage: MAP_PRIVATE Create a private copy-on-write mapping. Updates to the map‐ ping are not visible to other processes mapping the same file, and are not carried through to the underlying file. It is unspecified whether changes made to the file after the mmap() call are visible in the mapped region. The last sentence is important here. No?
(In reply to Behdad Esfahbod from comment #3) > The last sentence is important here. No? Not at all. we don't open a cache through mmap with O_RDWR. the changes won't be applied to the file. so MAP_PRIVATE simply works to modify things in the memory for the certain process.
Updated a bit. especially to reduce the call of the sort of stat(). we may need to implement something for win32 to assign a value to st_dev maybe - apparently we have a hash value to st_ino in our FcStat() implementation and always set 0 to st_dev. it may not be sufficient to make cache filenames unique.
I don't see why we have to rewrite the path as listed in the mmapped file. Can't we just allocate a new string and return it? (Tying its lifetime to that of the in-library cache.)
well, it was most easier way to do... and simply didn't have an idea to free the allocated memory. Hm, trying to think about the pre-allocating PATH_MAX size of spaces too much thing since yesterday, we could hold an address allocated somewhere and free it when a cache is closed perhaps. let me try to find much better way out to reduce allocating a memory for that.
No reference counts in the serialized FcPattern and can't figure out in it itself how long it is available, but FcCache. so when going to add something into the serialized FcPattern, it needs to be exposed to FcCache to manage the lifetime of the allocated memory. which is ugly. otherwise that may introduces the memory leaks or invalid access.
(In reply to Akira TAGOH from comment #8) > No reference counts in the serialized FcPattern Right. But we do increase ref count of the cache itself when you try reffing the FcPattern. > and can't figure out in it > itself how long it is available, but FcCache. so when going to add something > into the serialized FcPattern, it needs to be exposed to FcCache to manage > the lifetime of the allocated memory. which is ugly. otherwise that may > introduces the memory leaks or invalid access. Sure. One way or another you need to do this... When freeing the cache, walk and "free" the patterns which will free the strings. Another option is to keep the path relative, and resolve it in FcFontRenderPrepare. Maybe you should write to the list with your proposed design so we can all discuss before implementing such drastic changes? Basically what you have committed right now is taking fontconfig cache from consuming no private memory to consuming megabytes of private memory per app...
Can't you just resolve all the paths in the cache when you map it and allocate space for the rewritten version of them in a single chunk which you free with the cache.all
You probably want to change how filenames are stored in the cache (not that i know how it works now), so that filename values are indexes into an array of offsets. Then when you're rewriting filenames you can use the index with an in-memory array of (rewritten) strings instead. That would make it simple to do the rewrites too. You don't have to scan the entire cache, just rewrite everything in that table.
(In reply to Alexander Larsson from comment #10) > Can't you just resolve all the paths in the cache when you map it and > allocate space for the rewritten version of them in a single chunk which you > free with the cache.all Not doing them all together allows for lazily doing them which should result in most not needing change and as such still sharing the memory backing the mmaped cache with other processes.
Yes, we could do it lazily, true. However, I don't understand what you mean about sharing with others. Here is what i imagine: [Shared mmaped cache data] ... FontName="TheFont" FileName=offset 0 ... FontName="OtherFont" FileName=offset 1 ... FileNameTable [offset 0] [offset 12] FileNameStrings "TheFont.ttf\0OtherFont.ttf\0" This would be readonly and shared by all clients. To go from a cache element to a filename you would normally go from offset in FileNameTable to offset in FileNameStrings, all shared by everyone. However, in the case where the cache is loaded in a non-canonical location, you'd allocate an in-memory (non-shared) FileNameTable and FileNameStrings array, which would would rewrite and use instead. Such a setup could be lazy, either doing everything the first time you get a filename, or it could allocate filenames one-by-one lazily. (Although then you create a lot of small allocations instead of a single large one.
(In reply to Alexander Larsson from comment #11) > You probably want to change how filenames are stored in the cache (not that > i know how it works now), so that filename values are indexes into an array > of offsets. > Then when you're rewriting filenames you can use the index with an in-memory > array of (rewritten) strings instead. > > That would make it simple to do the rewrites too. You don't have to scan the > entire cache, just rewrite everything in that table. I was under the impression that FcPattern is public struct. I'm truly relieved to learn that I was wrong. It's FcValue that is public, but not FcPattern. So, yes, we can do all sort of things. I know what we need to do... I'll write here in a bit.
(In reply to Alexander Larsson from comment #13) > Yes, we could do it lazily, true. However, I don't understand what you mean > about sharing with others. When the mapped file is not modified, the backing memory is shared across processes. They are copy-on-write. If we can completely avoid changing is even better. I have a scheme in mind that does that. > Here is what i imagine: > > [Shared mmaped cache data] > ... > FontName="TheFont" > FileName=offset 0 > ... > FontName="OtherFont" > FileName=offset 1 > ... > FileNameTable > [offset 0] > [offset 12] > FileNameStrings > "TheFont.ttf\0OtherFont.ttf\0" > > This would be readonly and shared by all clients. > To go from a cache element to a filename you would normally > go from offset in FileNameTable to offset in FileNameStrings, all shared by > everyone. However, in the case where the cache is loaded in a non-canonical > location, you'd allocate an in-memory (non-shared) FileNameTable and > FileNameStrings array, which would would rewrite and use instead. > > Such a setup could be lazy, either doing everything the first time you get a > filename, or it could allocate filenames one-by-one lazily. (Although then > you create a lot of small allocations instead of a single large one.
(In reply to Behdad Esfahbod from comment #15) > (In reply to Alexander Larsson from comment #13) > > Yes, we could do it lazily, true. However, I don't understand what you mean > > about sharing with others. > > When the mapped file is not modified, the backing memory is shared across > processes. They are copy-on-write. If we can completely avoid changing is > even better. I have a scheme in mind that does that. > > > > Here is what i imagine: > > > > [Shared mmaped cache data] > > ... > > FontName="TheFont" > > FileName=offset 0 > > ... > > FontName="OtherFont" > > FileName=offset 1 > > ... > > FileNameTable > > [offset 0] > > [offset 12] > > FileNameStrings > > "TheFont.ttf\0OtherFont.ttf\0" > > > > This would be readonly and shared by all clients. > > To go from a cache element to a filename you would normally > > go from offset in FileNameTable to offset in FileNameStrings, all shared by > > everyone. However, in the case where the cache is loaded in a non-canonical > > location, you'd allocate an in-memory (non-shared) FileNameTable and > > FileNameStrings array, which would would rewrite and use instead. > > > > Such a setup could be lazy, either doing everything the first time you get a > > filename, or it could allocate filenames one-by-one lazily. (Although then > > you create a lot of small allocations instead of a single large one. Yes, I have something very similar to that in mind. It's doable.
As it happens, we don't even need to change FcPattern. We will store the files we currently do, that makes for zero allocations in existing setups. For relocated use-case we will allocate lazily, similar to what Alex describe. Here's the details: In FcPatternObjectGetWithBinding() if object is FC_FILE and FcRefIsConst(), then we do some extra work to see if relocation is needed. We find the cache object using FcCacheFindByAddr(). If the cache object says that there's relocation happening, then we do it. We'll just use a hash table to map the old string to the new string. No need to use indices, etc. We'll free the hash table when the cache object is being released. If avoiding small allocations is desired, we can allocate 4k at a time or something. That would be 4k per directory. But then again, this is one allocation for each font actually used, so I think we should just do the small allocation.
Akira, do you think you can do this?
Hm, just done similar implementation in this morning and the discussion ongoing. https://cgit.freedesktop.org/~tagoh/fontconfig/log/?h=relpath-cache-3 Not modifying FcPatternObjectAddWithBinding() because I'm not expecting to see any modifications more than this in the cache. only making caches writable during very limited time frame on loading.
(In reply to Akira TAGOH from comment #19) > Hm, just done similar implementation in this morning and the discussion > ongoing. > > https://cgit.freedesktop.org/~tagoh/fontconfig/log/?h=relpath-cache-3 > > Not modifying FcPatternObjectAddWithBinding() because I'm not expecting to > see any modifications more than this in the cache. only making caches > writable during very limited time frame on loading. Aren't you leaking the string? I don't like this. This is still making caches consume private memory. I think what I sketched in https://bugs.freedesktop.org/show_bug.cgi?id=101889#c17 should be implemented. I can do it myself if you don't want to.
(In reply to Behdad Esfahbod from comment #20) > Aren't you leaking the string? No, all of the allocated memory will be freed when caches is closed. > I don't like this. This is still making caches consume private memory. I > think what I sketched in > https://bugs.freedesktop.org/show_bug.cgi?id=101889#c17 should be > implemented. I can do it myself if you don't want to. Hmm, I was trying to understand and implement what you were suggesting, but there are no way to obtain the font dirs from FcConfig in FcPatternObjectGetWithBinding() because of the infinite loop of a call of FcPatternObjectGetWithBinding() through FcInitLoadConfigAndFonts() to get an instance of the initialized FcConfig...
What is the status of this? It would be nice to get this and the emoji work in a new fontconfig release.
Take 4: https://cgit.freedesktop.org/~tagoh/fontconfig/commit/?h=relpath-cache-4 This doesn't modify a cache being mmap'd anymore. but hooking up FcListAppend() for FcFontList() and FcFontRenderPrepare() for FcFontMatch() to have the correct font path in FcPattern, because FcPatterns that one can touch in their apps is a copy of the original in FcCache. so FcRefIsConst() won't usually returns true in FcPattern*Get*(). Hope that looks okay.
Hmm. Thinking more about this, i think maybe we should store the cache both at the old (pathname based) name and the new name. Because if you're using an image-based deployment mechanism, like say if the host is an ostree image, or a docker image, then you will end up loading the cache from the "correct" location, but it may have a different inode.
Maybe this is already broken though, because it seems we're storing the inode/dev in the cache file already.
Which is actually an issue for flatpak too, because we would like to be able to pre-seed the fontconfig caches in the flatpak runtimes...
Of course, those are all in the same location (same absolute pathname for all runtimes), so we can't rely on the filename in a global cache... Not sure what will work here. Needs some thinking.
Aha. good point. hmm, is there any assumptions that those images works under the read-only mode? caches needs to be updated if inode is changed as it contains the inode number. if we can't do that, we may need to have an option in fc-cache to generate caches with either of types of filename and use it differently as needed.
Ok, lets start from scratch, defining the requirements. On the host side, there is a set of system font directories, and one system font cache directory. For example /usr/share/fonts/* and /usr/lib/fontconfig/cache/ The cache is rebuilt when the font directory changed. However, this may be running on the actual host (old-school) or when building the read-only image (fedora atomic). The cache stores the time of the font directory, which can be used to validate it on use-time. In the read-only image case the font directory mtime is set to zero before the cache is generated, and when installed to disk (both atomic and flatpak does this). This works in many cases: * On the host, when rebuilding the cache locally in in post-install * On the host, in the read-only image case (due to the mtime hack) * In flatpak for the font directories and font caches that are shipped per-generated (with 0 mtime) in the runtime. However, we want to add a new case: exposing the host font directory to the flatpak (in addition to the fonts in the flatpak runtime). This is set up in the following way in the sandbox: /usr/share/fonts is from the runtime /usr/lib/fontconfig/cache/ is from the runtime /run/host/fonts is a bind-mount of the host fonts dir /run/host/fonts-cache is a bind-mount of the host font cache dir /etc/fonts/fonts.conf is from the runtime, adds all the above dirs Here we run into issues. When loading a font from /run/host/fonts, if we look up the cache by checksum("/run/host/fonts") we will not find the correct cache, even though it is visible and configured in /run/host/fonts-cache. The alternative is to look up the font by checksum(ino,dev), however, that will not work when looking up in /usr/share/fonts from the runtime, because we don't know the final dev:ino when creating the cache. One possible solution would be to make fc-scan generate both kinds of hashes (make the dev+inode cache a symlink to the real cache). Then we can specify in the config file what type of lookup to do on each cache dir. The flatpak runtime config would have: <cachedir type="devino">/run/host/fonts-cache</cachedir> This would make lookups of of fonts work both on the host and in flatpak, and additionally we would never accidentally do pathname based lookups of the host fonts from the sandbox. There is one combination where this would break though. If you are running fedora atomic on the host, then the pre-generated cache in the host /usr/lib/fontconfig/cache will not contain devinode info (or it will be wrong), and flatpak will not be able to read it. There is another approach though. Instead of using the device+inode to look up the cache we use a separate file. It would go like this: if $fontdir/.uuid exists: look up cache based on content-of($fontdir/.uuid) else look up cache based on path-of($fontdir) Then we make fc-scan create a random .uuid file if one does not exist. This would happen in the post-inst in the traditional case, but it would instead happen during image-building in the fedora atomic and flatpak runtime case. In this model we wouldn't need to set the cache dir lookup style in the config file.
Thanks. Hmm, what if one runs apps with flatpak on atomic? in that case, as you were concerned, the dev/ino approach won't works. then your "another approach" sounds good though.
Take 5: https://cgit.freedesktop.org/~tagoh/fontconfig/commit/?h=relpath-cache-5 Use uuid to determine the relationship between a font and a cache instead of dev/ino. according to this change, no changes in the cache structure anymore nor no bumps the cache version as well. also trying to keep the original place to the font in FcPattern as index 1.
I don't think the alias table works as implemented. For example, say on the host we have /usr/share/fonts. We'll generate a .uuid file for it and a cache file based on the hash of that. Then the app reads this, but in a different location, say /run/host/fonts. First we load /run/host/fonts/.uuid and use that to find the host-side cache file. But its cache->dirname is /usr/share/fonts, so we add to the alias table: /usr/share/fonts -> /run/host/fonts This means that if a fc pattern has a path of /usr/share/fonts/foo we will instead load it at /run/host/fonts/foo, which is correct if we got the pattern from the cache above. However, what if you're loading a font from the runtime, which is properly in /usr/share/fonts in the sandbox? This will *also* be mapped via the alias table. You can't use a global mapping like that, it has to depend on what cache the pattern came from. Also, why are all the hashes you added using FcStrHashIgnoreCase for the filename? You're using regular strcmp later, so its not like the hash lookup will be case insensitive anyway.
(In reply to Alexander Larsson from comment #32) > However, what if you're loading a font from the runtime, which is properly > in /usr/share/fonts in the sandbox? This will *also* be mapped via the alias > table. That shouldn't be a problem as long as they have different uuid and doesn't have the same filename of a font there. it tries to call stat() to see if it really exists before looking up the alias table. only things that it might be a problem is when the runtime has the same-name-but-different-context things though, in that case we may need to take similar cost to the creating of cache for checking. as Behdad's patch has been merged into git for speedup of creating cache, it might simply works without any trick like this, if the problem is to take too much time for startup. > Also, why are all the hashes you added using FcStrHashIgnoreCase for the > filename? You're using regular strcmp later, so its not like the hash lookup > will be case insensitive anyway. That is what fontconfig originally has and used for all of string properties.
It still looks a bit weird. I have the file ~/.fonts/DevanagariMT.ttf, and when i run fc-list in the sandbox i get: sh-4.3$ fc-list | grep DevanagariMT.ttf /run/host/user-fonts/DevanagariMT.ttf,/home/alex/.fonts/DevanagariMT.ttf: Devanagari MT:style=Regular,Régulier,Normal,Regolare,Regelmatig,Normalt,Almindelig,Vanlig,Normaali sh-4.3$ ls /run/host/user-fonts/DevanagariMT.ttf /run/host/user-fonts/DevanagariMT.ttf sh-4.3$ ls /home/alex/.fonts/DevanagariMT.ttf ls: cannot access '/home/alex/.fonts/DevanagariMT.ttf': No such file or directory i.e. fc-list lists two paths, but only one works.
Hm, I didn't get rid of the original path from FcPattern. one could see the path was substituted if FC_FILE has two entries. if no need to do, I could revert this to the other branch's behavior.
I'm in no way an expert on fontconfig so i don't know what will happen when a pattern has two paths like that. But my guess is that some apps would be confused.
Well, most apps checks first one only for FC_FILE. but sure, let's remove the old one then.
Okay, updated. FC_FILE has only one path like current behavior.
I tried this a bit, and overall it seems to work great. Initial app startup is much much faster. However, there seem to be an issue remaining. If a system font is in a subdirectory of /usr/share/fonts it will not be picked up, because the subdirs in the cache are stored by absolute pathname too. I think we need to strip cache->dir from the paths in cache->dirs[] and apply the real dir in when we add the subdirs in FcConfigAddCache().
Ah, thank you for catching it up. updated.
Created attachment 134506 [details] test app I'm testing with this. Build with flatpak-builder --repo=repo --force-clean app org.fontconfig.Test.json Then install with: flatpak --user remote add --no-gpg-verify local-fontconfig repo flatpak --user install local-fontconfig org.fontconfig.Test Then try with: rm -rf ~/.var/app/org.fontconfig.Test/cache/* flatpak run --command=sh org.fontconfig.Test I usually run fc-list in the shell to see what happens.
So, this seems a lot nicer, but there are still some fonts missing. For example. Both the runtime and the host has dejavu fonts: sh-4.3$ ls /usr/share/fonts/dejavu/ DejaVuMathTeXGyre.ttf DejaVuSansCondensed-Bold.ttf DejaVuSansMono-BoldOblique.ttf DejaVuSans-Oblique.ttf DejaVuSerifCondensed-BoldItalic.ttf DejaVuSerif-Italic.ttf DejaVuSans-BoldOblique.ttf DejaVuSansCondensed-Oblique.ttf DejaVuSansMono-Bold.ttf DejaVuSans.ttf DejaVuSerifCondensed-Bold.ttf DejaVuSerif.ttf DejaVuSans-Bold.ttf DejaVuSansCondensed.ttf DejaVuSansMono-Oblique.ttf DejaVuSerif-BoldItalic.ttf DejaVuSerifCondensed-Italic.ttf DejaVuSansCondensed-BoldOblique.ttf DejaVuSans-ExtraLight.ttf DejaVuSansMono.ttf DejaVuSerif-Bold.ttf DejaVuSerifCondensed.ttf sh-4.3$ ls /run/host/fonts/dejavu/ DejaVuLGCSans-BoldOblique.ttf DejaVuLGCSansMono-Oblique.ttf DejaVuLGCSerifCondensed.ttf DejaVuSans-ExtraLight.ttf DejaVuSerifCondensed-BoldItalic.ttf DejaVuLGCSans-Bold.ttf DejaVuLGCSansMono.ttf DejaVuLGCSerif-Italic.ttf DejaVuSansMono-BoldOblique.ttf DejaVuSerifCondensed-Bold.ttf DejaVuLGCSansCondensed-BoldOblique.ttf DejaVuLGCSans-Oblique.ttf DejaVuLGCSerif.ttf DejaVuSansMono-Bold.ttf DejaVuSerifCondensed-Italic.ttf DejaVuLGCSansCondensed-Bold.ttf DejaVuLGCSans.ttf DejaVuSans-BoldOblique.ttf DejaVuSansMono-Oblique.ttf DejaVuSerifCondensed.ttf DejaVuLGCSansCondensed-Oblique.ttf DejaVuLGCSerif-BoldItalic.ttf DejaVuSans-Bold.ttf DejaVuSansMono.ttf DejaVuSerif-Italic.ttf DejaVuLGCSansCondensed.ttf DejaVuLGCSerif-Bold.ttf DejaVuSansCondensed-BoldOblique.ttf DejaVuSans-Oblique.ttf DejaVuSerif.ttf DejaVuLGCSans-ExtraLight.ttf DejaVuLGCSerifCondensed-BoldItalic.ttf DejaVuSansCondensed-Bold.ttf DejaVuSans.ttf DejaVuLGCSansMono-BoldOblique.ttf DejaVuLGCSerifCondensed-Bold.ttf DejaVuSansCondensed-Oblique.ttf DejaVuSerif-BoldItalic.ttf DejaVuLGCSansMono-Bold.ttf DejaVuLGCSerifCondensed-Italic.ttf DejaVuSansCondensed.ttf DejaVuSerif-Bold.ttf The ones in the host have more of them though. However, fc-list in the sandbox doesn't list any of the fonts in /run/host/fonts/dejavu. It seems like when it is reading the cache for /run/host/fonts it ignores the dejavu subdirector. Maybe because it still has the pathname "/usr/share/fonts/dejavu" in it, but we already read that directory?
I guess it is because of this part: + const FcChar8 *dir = FcCacheSubdir (cache, i); + FcChar8 *s = NULL; + struct stat statb; + + if (FcStat (dir, &statb) < 0) + { In the case of dir= /usr/share/fonts/dejavu, that *does* exist in the sandbox. So we don't use the alias for that.
Okay, should be better now...
Can someone summarize the status here? Akira, should I start reviewing your branch?
(In reply to Akira TAGOH from comment #33) > (In reply to Alexander Larsson from comment #32) > > However, what if you're loading a font from the runtime, which is properly > > in /usr/share/fonts in the sandbox? This will *also* be mapped via the alias > > table. > > That shouldn't be a problem as long as they have different uuid and doesn't > have the same filename of a font there. The latter is an assumption we do NOT want to make. > it tries to call stat() to see if it > really exists before looking up the alias table. only things that it might > be a problem is when the runtime has the same-name-but-different-context > things though, It's totally possible that the two have different versions of the same font. I think both should be visible. > in that case we may need to take similar cost to the creating > of cache for checking. as Behdad's patch has been merged into git for > speedup of creating cache, it might simply works without any trick like > this, if the problem is to take too much time for startup. > > > Also, why are all the hashes you added using FcStrHashIgnoreCase for the > > filename? You're using regular strcmp later, so its not like the hash lookup > > will be case insensitive anyway. > > That is what fontconfig originally has and used for all of string properties.
(In reply to Behdad Esfahbod from comment #45) > Can someone summarize the status here? Akira, should I start reviewing your > branch? Yes, please. (In reply to Behdad Esfahbod from comment #46) > It's totally possible that the two have different versions of the same font. > I think both should be visible. Yes, but I realizezd that it won't be happened unless someone malicously copies .uuid file there. so current proposed one should works.
Got it. So, relocatability happens only if top font dir has .uuid? I like that.
I tried the latest version, and it seems better but it still has the issue behdad disliked. I.e. In my case I have on the outside (minimal example): /usr/share/fonts/dejavu/DejaVuLGCSerifCondensed.ttf /usr/share/fonts/dejavu/DejaVuSans-Bold.ttf And, in the sandbox I have: /usr/share/fonts/dejavu/DejaVuLGCSerifCondensed.ttf /run/host/fonts/dejavu/DejaVuLGCSerifCondensed.ttf /run/host/fonts/dejavu/DejaVuSans-Bold.ttf I.e. one of the sandbox-only files covers one from the host, but not all, and both host files are visible in the sandbox at a different location. If i run fc-list in the inside it will show: /usr/share/fonts/dejavu/DejaVuSans-Bold.ttf: DejaVu Sans:style=Bold /run/host/fonts/dejavu/DejaVuLGCSerifCondensed.ttf: DejaVu LGC Serif,DejaVu LGC Serif Condensed:style=Condensed,Book I.e. the "shadowed" host font is not visible in the cache. This is not really s showstopper for me, but I can see why behdad dislikes it. Anyway, I would *love* to get some review on this so we can get a release with it (so i can put it in the flatpak runtimes). The current behaviour is causing really slow first-time startups of flatpak:ed apps.
Behdad, any chance you could do some review on this?
(In reply to Alexander Larsson from comment #50) > Behdad, any chance you could do some review on this? Yes, will do in a day or two.
I reviewed. While generally looks good, and I'd be fine with this going in, my major concern is maintainability of this... 1. The copy paste hashmap implementation get annoying. We should abstract that away, 2. I think this has threadsafety issues. No? Since the lifecycle of what's put in the hashmaps is forever(ish), it's possible to make it threadsafe using pointer cmpexch(). Also, isn't the 4099 hash size a bit too large? 3. The maintainability... The whole cache code has become incomprehensible to me. I don't understand the code anymore. Some more abstractions can definitely make it much more understandable. 4. Would be great to at least have standalone tests for these. Is it possible to mount in-process without root privileges?
(In reply to Behdad Esfahbod from comment #52) > 4. Would be great to at least have standalone tests for these. Is it > possible to mount in-process without root privileges? Unfortunately no.. though it could be emulated according to the logic of this implementation if we share .uuid and have same timestamps on fonts and dirs. that said it may be different test case then. hmm.
You can't mount in-process without root, but if you use bubblewrap you can shell out to things that have a different filesystem setup as a user. For instance: bwrap --bind / / --bind /usr/bin /tmp/bin ls -l /tmp/bin/true -rwxr-xr-x. 1 nfsnobody nfsnobody 32424 18 aug 18.14 /tmp/bin/true
So, can we get a release with this?
patch should be much better now. will merge them shortly.
Did you go back to using device-id/inode? That makes it not work for anything that pre-generates an image which is deployed in some way that doesn't keep these. For instance, ostree/flatpak or docker images. In fact, this is likely to make the pregenerated font caches in e.g. the flatpak runtimes useless.
Oh, sorry looked at the wrong branch. Did you rebase all the branch versions?
ah, yes. I did to sync up to master.
Okay, merged into master. let's see how it goes.
What is the schedule for doing a release with this in it?
first devel release in this week or early next week.
Seems like 2.12.91 is out with this in it, i'll try to get that into the freedesktop platform.
Hmm, to try this I added the new fontconfig to the flatpak runtime, it seems weird. I do this at the end of the build: "touch -d @0 /usr/share/fonts /usr/share/fonts/*", "fc-cache -fs", "for i in /usr/share/fonts/.uuid /usr/share/fonts/*/.uuid; do echo $i; cat $i; echo; done", "ls -l /usr/cache/fontconfig", And I get: /usr/share/fonts/.uuid 9a4ba6ad-8a07-4b43-8e37-db03d0129f6f /usr/share/fonts/dejavu/.uuid 258f9b15-f3d5-4804-97e8-035998387d90 /usr/share/fonts/eosrei-emojione/.uuid eb4fa041-d1f0-4557-9a0b-704c39f846f5 /usr/share/fonts/gnu-free/.uuid c7aa2f19-edee-4d8c-b79c-9a5b3d531429 /usr/share/fonts/google-crosextra-caladea/.uuid cfb58672-d7b8-4433-8aff-187e9d448ad4 /usr/share/fonts/google-crosextra-carlito/.uuid 52660665-ec54-47ea-a03e-6836ce7a09f1 /usr/share/fonts/liberation-fonts/.uuid 0493b60e-fb33-4c20-9006-e73269e68cfa total 252 -rw-r--r-- 1 1000 uucp 6936 Dec 15 14:39 01783329-39dc-4438-8696-5cba1910aab3-le64.cache-7 -rw-r--r-- 3 1000 uucp 29856 Jan 1 1970 0767c642-f2f0-48f2-8628-b8f2826b1612-le64.cache-7 -rw-r--r-- 3 1000 uucp 2536 Jan 1 1970 13d7127b-2474-407c-819c-5f36cfb343a6-le64.cache-7 -rw-r--r-- 3 1000 uucp 6936 Jan 1 1970 15df92c7-d05b-432d-8819-8eab87500cfd-le64.cache-7 -rw-r--r-- 3 1000 uucp 47104 Jan 1 1970 235f148f-a3f4-43b7-9538-bb188259c060-le64.cache-7 -rw-r--r-- 3 1000 uucp 384 Jan 1 1970 2bc88d11-73a1-44c3-b628-993fd1a8eaed-le64.cache-7 -rw-r--r-- 1 1000 uucp 29856 Dec 15 14:39 32237142-aa53-4544-bf74-e3fc15e153ae-le64.cache-7 -rw-r--r-- 1 1000 uucp 19928 Dec 15 14:39 6112831d-541f-4056-a4cd-bb1701f8433d-le64.cache-7 -rw-r--r-- 3 1000 uucp 6312 Jan 1 1970 6cb9c9b4-a4bc-4790-bd82-f55fd2e6b796-le64.cache-7 -rw-r--r-- 1 1000 uucp 47104 Dec 15 14:39 728c5f2a-1d39-456f-94f1-c4dd76b0a4fa-le64.cache-7 -rw-r--r-- 3 1000 uucp 19928 Jan 1 1970 77b4858f-4364-4bd0-a9bf-704b1e74800e-le64.cache-7 -rw-r--r-- 1 1000 uucp 6312 Dec 15 14:39 7b3075e4-c99d-43e8-814e-86760686a368-le64.cache-7 -rw-r--r-- 1 1000 uucp 2536 Dec 15 14:39 7b85b0f8-9c12-49f2-b695-7b9f6e73df8e-le64.cache-7 -rw-r--r-- 1 1000 uucp 200 Dec 15 14:39 CACHEDIR.TAG -rw-r--r-- 1 1000 uucp 384 Dec 15 14:39 e478a8d1-5f27-4c2a-a0fc-b69c2550728d-le64.cache-7 It seems like none of the uuids match? But fc-cache itself created the .uuid files?
Here it is with more debug: FC_DEBUG=16 FcDirCacheCreateUUID /usr/share/fonts/.uuid: 0904e70e-804b-413c-9544-49f037570eca Created CACHEDIR.TAG at /usr/cache/fontconfig FcDirCacheWriteDir dir "/usr/share/fonts" file "/usr/cache/fontconfig/0904e70e-804b-413c-9544-49f037570eca-le64.cache-7" FcDirCacheReadUUID Unable to read /app/share/fonts/.uuid FcDirCacheReadUUID Unable to read /run/host/fonts/.uuid FcDirCacheReadUUID Unable to read /run/host/user-fonts/.uuid FcDirCacheCreateUUID /usr/share/fonts/dejavu/.uuid: 7177caba-fd81-49ff-88e2-2029207625ec FcDirCacheWriteDir dir "/usr/share/fonts/dejavu" file "/usr/cache/fontconfig/7177caba-fd81-49ff-88e2-2029207625ec-le64.cache-7" FcDirCacheCreateUUID /usr/share/fonts/eosrei-emojione/.uuid: ecde1a72-f6d0-4607-9951-1de0eba62f89 FcDirCacheWriteDir dir "/usr/share/fonts/eosrei-emojione" file "/usr/cache/fontconfig/ecde1a72-f6d0-4607-9951-1de0eba62f89-le64.cache-7" FcDirCacheCreateUUID /usr/share/fonts/gnu-free/.uuid: 56667d75-8302-4363-9149-cba919dc0e9b FcDirCacheWriteDir dir "/usr/share/fonts/gnu-free" file "/usr/cache/fontconfig/56667d75-8302-4363-9149-cba919dc0e9b-le64.cache-7" FcDirCacheCreateUUID /usr/share/fonts/google-crosextra-caladea/.uuid: e69e2609-4130-45e1-b963-fec9ebdd7ba0 FcDirCacheWriteDir dir "/usr/share/fonts/google-crosextra-caladea" file "/usr/cache/fontconfig/e69e2609-4130-45e1-b963-fec9ebdd7ba0-le64.cache-7" FcDirCacheCreateUUID /usr/share/fonts/google-crosextra-carlito/.uuid: 589a9c41-9985-449c-8097-eccb2667dee7 FcDirCacheWriteDir dir "/usr/share/fonts/google-crosextra-carlito" file "/usr/cache/fontconfig/589a9c41-9985-449c-8097-eccb2667dee7-le64.cache-7" FcDirCacheCreateUUID /usr/share/fonts/liberation-fonts/.uuid: 636dc890-8348-4b63-851e-8c36d18e3d07 FcDirCacheWriteDir dir "/usr/share/fonts/liberation-fonts" file "/usr/cache/fontconfig/636dc890-8348-4b63-851e-8c36d18e3d07-le64.cache-7" /usr/share/fonts: FcDirCacheCreateUUID /usr/share/fonts/.uuid: 9a984a81-ec45-4ba0-b22f-e18650dc1caf FcDirCacheWriteDir dir "/usr/share/fonts" file "/usr/cache/fontconfig/0904e70e-804b-413c-9544-49f037570eca-le64.cache-7" caching, new cache contents: 0 fonts, 6 dirs /usr/share/fonts/dejavu: FcDirCacheCreateUUID /usr/share/fonts/dejavu/.uuid: 20b7e832-d33c-41cf-8113-cba031e5efea FcDirCacheWriteDir dir "/usr/share/fonts/dejavu" file "/usr/cache/fontconfig/7177caba-fd81-49ff-88e2-2029207625ec-le64.cache-7" caching, new cache contents: 22 fonts, 0 dirs /usr/share/fonts/eosrei-emojione: FcDirCacheCreateUUID /usr/share/fonts/eosrei-emojione/.uuid: 7289d32a-6b07-4d10-a49d-ae217d6ec001 FcDirCacheWriteDir dir "/usr/share/fonts/eosrei-emojione" file "/usr/cache/fontconfig/ecde1a72-f6d0-4607-9951-1de0eba62f89-le64.cache-7" caching, new cache contents: 1 fonts, 0 dirs /usr/share/fonts/gnu-free: FcDirCacheCreateUUID /usr/share/fonts/gnu-free/.uuid: e9be1fec-1f3c-4a83-b94b-fe85594bf873 FcDirCacheWriteDir dir "/usr/share/fonts/gnu-free" file "/usr/cache/fontconfig/56667d75-8302-4363-9149-cba919dc0e9b-le64.cache-7" caching, new cache contents: 12 fonts, 0 dirs /usr/share/fonts/google-crosextra-caladea: FcDirCacheCreateUUID /usr/share/fonts/google-crosextra-caladea/.uuid: bcb3c312-f0ad-4a4d-bb6f-144adfac8bfd FcDirCacheWriteDir dir "/usr/share/fonts/google-crosextra-caladea" file "/usr/cache/fontconfig/e69e2609-4130-45e1-b963-fec9ebdd7ba0-le64.cache-7" caching, new cache contents: 4 fonts, 0 dirs /usr/share/fonts/google-crosextra-carlito: FcDirCacheCreateUUID /usr/share/fonts/google-crosextra-carlito/.uuid: 60ecaa41-9361-43d3-a5a9-eabfb3bb633c FcDirCacheWriteDir dir "/usr/share/fonts/google-crosextra-carlito" file "/usr/cache/fontconfig/589a9c41-9985-449c-8097-eccb2667dee7-le64.cache-7" caching, new cache contents: 4 fonts, 0 dirs /usr/share/fonts/liberation-fonts: FcDirCacheCreateUUID /usr/share/fonts/liberation-fonts/.uuid: 778b9b58-ac5a-41bc-a424-134c1cf5ae86 FcDirCacheWriteDir dir "/usr/share/fonts/liberation-fonts" file "/usr/cache/fontconfig/636dc890-8348-4b63-851e-8c36d18e3d07-le64.cache-7" caching, new cache contents: 12 fonts, 0 dirs /app/share/fonts: skipping, no such directory /run/host/fonts: skipping, no such directory /run/host/user-fonts: skipping, no such directory Created CACHEDIR.TAG at /usr/cache/fontconfig /usr/cache/fontconfig: cleaning cache directory /app/cache/fontconfig: not cleaning non-existent cache directory /run/host/fonts-cache: not cleaning non-existent cache directory /run/host/user-fonts-cache: not cleaning non-existent cache directory /usr/var/cache/fontconfig: cleaning cache directory fc-cache: succeeded /usr/share/fonts/.uuid 9a984a81-ec45-4ba0-b22f-e18650dc1caf /usr/share/fonts/dejavu/.uuid 20b7e832-d33c-41cf-8113-cba031e5efea /usr/share/fonts/eosrei-emojione/.uuid 7289d32a-6b07-4d10-a49d-ae217d6ec001 /usr/share/fonts/gnu-free/.uuid e9be1fec-1f3c-4a83-b94b-fe85594bf873 /usr/share/fonts/google-crosextra-caladea/.uuid bcb3c312-f0ad-4a4d-bb6f-144adfac8bfd /usr/share/fonts/google-crosextra-carlito/.uuid 60ecaa41-9361-43d3-a5a9-eabfb3bb633c /usr/share/fonts/liberation-fonts/.uuid 778b9b58-ac5a-41bc-a424-134c1cf5ae86 total 128 -rw-r--r-- 1 1000 uucp 384 Dec 15 14:48 0904e70e-804b-413c-9544-49f037570eca-le64.cache-7 -rw-r--r-- 1 1000 uucp 29856 Dec 15 14:48 56667d75-8302-4363-9149-cba919dc0e9b-le64.cache-7 -rw-r--r-- 1 1000 uucp 6936 Dec 15 14:48 589a9c41-9985-449c-8097-eccb2667dee7-le64.cache-7 -rw-r--r-- 1 1000 uucp 19928 Dec 15 14:48 636dc890-8348-4b63-851e-8c36d18e3d07-le64.cache-7 -rw-r--r-- 1 1000 uucp 47104 Dec 15 14:48 7177caba-fd81-49ff-88e2-2029207625ec-le64.cache-7 -rw-r--r-- 1 1000 uucp 200 Dec 15 14:48 CACHEDIR.TAG -rw-r--r-- 1 1000 uucp 6312 Dec 15 14:48 e69e2609-4130-45e1-b963-fec9ebdd7ba0-le64.cache-7 -rw-r--r-- 1 1000 uucp 2536 Dec 15 14:48 ecde1a72-f6d0-4607-9951-1de0eba62f89-le64.cache-7 Its like its creating the cache file for one uuid, but then putting a different uuid in the .uuid file??
So, i tried running fc-cache twice, and if the first run produces a .uuid A, and a cache B, then the second run produces a .uuid C and a cache A. So, it seems like it does the entire cache generation and .uuid generation, but then it overwrites the .uuid files that it just wrote with new ones?
Hm, well, the problem is: FcDirCacheReadUUID Unable to read /app/share/fonts/.uuid and fontconfig can't see the difference between missing .uuid and failing to read .uuid. how about the permissions of .uuid?
Ah, I got it. please drop -f option. that enforces to create .uuid at this moment. Hmm, any reasons to use -f to create caches?
changed the behavior to update .uuid with -r but not -f. https://cgit.freedesktop.org/fontconfig/commit/?id=dd21876e64db4eaf592297e97355ffdf87f7d2f6
I'm using -f because I need to replace all caches, because my timestamps are not reliable (all mtime 0). This is fine anyway, as it is only run once when building the image. I wouldn't mind -f updating the .uuid files for me, but it would have to do it *before* updating the caches.
Anyway, removing .uuid with -r seems like the right thing, but you still need to make sure you remove the uuid file *before* you start creating the caches, or you'll create caches that are invalid.
Yes, you're right. also fixed in: https://cgit.freedesktop.org/fontconfig/commit/?id=57eaf0ba7ea7f88510053688f3c3c4658da83596
I ran into another issue with the .uuid creation. The creation of a .uuid file modifes the mtime of the directory, so my: "touch -d @0 /usr/share/fonts /usr/share/fonts/*", "fc-cache -fs", Breaks.
Although, maybe thats just me doing weird stuff. Maybe I need to work around it by running fc-cache twice.
Maybe there could be a flag for fc-cache that just adds any missing .uuid files?
Hmm, but mtime of a directory will be still updated when .uuid is missing. it is also not what you expect right? I'll keep mtime on updating.
updated with https://cgit.freedesktop.org/fontconfig/commit/?id=182186e53a38d2c8b82d0a1785f6873f2b54316a not to modify mtime with creating .uuid file. please check.
All right, that is much better. However, I still get some issues. Here is the debug output of FC_DEBUG=144 fc-list: FC_DEBUG=144 FcDirCacheReadUUID /usr/share/fonts/.uuid -> 96736f18-0a5c-4d11-b44c-61328b57edb8 FcCacheTimeValid dir "/usr/share/fonts" cache checksum 0.0 dir checksum 0.0 FcDirCacheReadUUID Unable to read /home/alex/.var/app/org.godotengine.Godot/data/fonts/.uuid FcDirCacheReadUUID /home/alex/.fonts/.uuid -> 5a020b29-4f6b-4d42-9436-e97822a223d9 FcCacheTimeValid dir "/home/alex/.fonts" cache checksum 1505811826.289886043 dir checksum 1505811826.289886043 FcDirCacheReadUUID Unable to read /app/share/fonts/.uuid FcDirCacheReadUUID /run/host/fonts/.uuid -> b81b806a-fb12-4a31-b458-181b1be0ec23 FcCacheTimeValid dir "/usr/share/fonts" cache checksum 1506524278.401973000 dir checksum 1506524278.401973000 FcDirCacheReadUUID /run/host/user-fonts/.uuid -> 5a020b29-4f6b-4d42-9436-e97822a223d9 FcCacheTimeValid dir "/home/alex/.fonts" cache checksum 1505811826.289886043 dir checksum 1505811826.289886043 FcDirCacheReadUUID /usr/share/fonts/dejavu/.uuid -> 350521bf-704b-4720-a6fd-a2922f3c2656 FcCacheTimeValid dir "/usr/share/fonts/dejavu" cache checksum 0.0 dir checksum 0.0 FcDirCacheReadUUID /usr/share/fonts/eosrei-emojione/.uuid -> cb2d2c88-773e-40fe-8b53-990829f05129 FcCacheTimeValid dir "/usr/share/fonts/eosrei-emojione" cache checksum 0.0 dir checksum 0.0 FcDirCacheReadUUID /usr/share/fonts/gnu-free/.uuid -> f4177fc3-b370-4064-b847-e7e255d943dd FcCacheTimeValid dir "/usr/share/fonts/gnu-free" cache checksum 0.0 dir checksum 0.0 FcDirCacheReadUUID /usr/share/fonts/google-crosextra-caladea/.uuid -> 528c2085-7cac-4240-b743-2e5f688858cd FcCacheTimeValid dir "/usr/share/fonts/google-crosextra-caladea" cache checksum 0.0 dir checksum 0.0 FcDirCacheReadUUID /usr/share/fonts/google-crosextra-carlito/.uuid -> ed77fd01-5360-4fa8-a706-f9b809ca1f4d FcCacheTimeValid dir "/usr/share/fonts/google-crosextra-carlito" cache checksum 0.0 dir checksum 0.0 FcDirCacheReadUUID /usr/share/fonts/liberation-fonts/.uuid -> 9d5ca877-d2e7-420b-af6e-d5fa27d1c25e FcCacheTimeValid dir "/usr/share/fonts/liberation-fonts" cache checksum 0.0 dir checksum 0.0 FcDirCacheReadUUID /home/alex/.fonts/Fonts/.uuid -> ad7573a8-dcda-43ed-b43c-4218c5eea077 FcCacheTimeValid dir "/home/alex/.fonts/Fonts" cache checksum 1505811826.615881682 dir checksum 1505811826.615881682 FcDirCacheReadUUID /home/alex/.fonts/OSXFonts/.uuid -> c10cca52-c6ed-4111-8bf1-f67a8f2acc4b FcCacheTimeValid dir "/home/alex/.fonts/OSXFonts" cache checksum 1505811826.615881682 dir checksum 1505811826.615881682 FcDirCacheReadUUID /home/alex/.fonts/ttf-bitstream-vera-1.10/.uuid -> e93f8f93-4f7c-4eb1-aefb-48254c831c22 FcCacheTimeValid dir "/home/alex/.fonts/ttf-bitstream-vera-1.10" cache checksum 1505811826.616881668 dir checksum 1505811826.616881668 FcDirCacheReadUUID /run/host/fonts/abattis-cantarell/.uuid -> a8d13f48-0579-48bd-a840-71920e033413 FcCacheTimeValid dir "/usr/share/fonts/abattis-cantarell" cache checksum 1506524278.402973000 dir checksum 1506524278.402973000 FcDirCacheReadUUID /run/host/fonts/blender/.uuid -> 021f5368-6b52-4e90-bbd3-4bef6afce2b6 FcCacheTimeValid dir "/usr/share/fonts/blender" cache checksum 1508752685.649995000 dir checksum 1508752685.649995000 FcDirCacheReadUUID Unable to read /run/host/fonts/ccicons/.uuid FcDirCacheReadUUID /run/host/fonts/cjkuni-uming/.uuid -> f15cc3e2-6ca8-4b54-a48c-51e19b5c2f23 FcCacheTimeValid dir "/usr/share/fonts/cjkuni-uming" cache checksum 1506524278.405973000 dir checksum 1506524278.405973000 FcDirCacheReadUUID /run/host/fonts/culmus/.uuid -> d3b77353-6dcf-4d4e-9164-95b80dd1642f FcCacheTimeValid dir "/usr/share/fonts/culmus" cache checksum 1506524278.408973000 dir checksum 1506524278.408973000 FcDirCacheReadUUID /run/host/fonts/default/.uuid -> 1ab388b5-b7a2-48f0-9a26-89b74fefd9c8 FcCacheTimeValid dir "/usr/share/fonts/default" cache checksum 1506524278.426973000 dir checksum 1506524278.426973000 FcDirCacheReadUUID /run/host/fonts/dejavu/.uuid -> 07579f92-a2ce-4164-8b9f-7d1716b355b8 FcCacheTimeValid dir "/usr/share/fonts/dejavu" cache checksum 1506524278.427973000 dir checksum 1506524278.427973000 FcDirCacheReadUUID Unable to read /run/host/fonts/fira/.uuid FcDirCacheReadUUID /run/host/fonts/fontawesome/.uuid -> e74b190a-3cdd-41ee-be19-b9c09b635f22 FcCacheTimeValid dir "/usr/share/fonts/fontawesome" cache checksum 1506524278.438973000 dir checksum 1506524278.438973000 FcDirCacheReadUUID /run/host/fonts/gdouros-symbola/.uuid -> 435a85d0-c01b-4348-84ed-25f834f303ec FcCacheTimeValid dir "/usr/share/fonts/gdouros-symbola" cache checksum 1508752533.346209000 dir checksum 1508752533.346209000 FcDirCacheReadUUID /run/host/fonts/gnu-free/.uuid -> 1daf6326-1e8a-488c-aacf-b23fb0b2a592 FcCacheTimeValid dir "/usr/share/fonts/gnu-free" cache checksum 1506524278.439973000 dir checksum 1506524278.439973000 FcDirCacheReadUUID /run/host/fonts/google-crosextra-caladea/.uuid -> 7d30f7a4-3dc7-4863-b792-ec9ad950d59a FcCacheTimeValid dir "/usr/share/fonts/google-crosextra-caladea" cache checksum 1506524278.443972000 dir checksum 1506524278.443972000 FcDirCacheReadUUID /run/host/fonts/google-crosextra-carlito/.uuid -> 550feb86-4479-40d6-89ce-25d5a74a0cd0 FcCacheTimeValid dir "/usr/share/fonts/google-crosextra-carlito" cache checksum 1506524278.444972000 dir checksum 1506524278.444972000 FcDirCacheReadUUID /run/host/fonts/google-droid/.uuid -> 789c1329-2fdf-4fb2-be3d-1c1fa2721826 FcCacheTimeValid dir "/usr/share/fonts/google-droid" cache checksum 1506524278.445972000 dir checksum 1506524278.445972000 FcDirCacheReadUUID /run/host/fonts/google-noto/.uuid -> 3388692e-d1b2-46d5-8632-41497822a235 FcCacheTimeValid dir "/usr/share/fonts/google-noto" cache checksum 1506524278.447972000 dir checksum 1506524278.447972000 FcDirCacheReadUUID /run/host/fonts/google-noto-emoji/.uuid -> 496dee46-72fa-4b0e-a2de-79428a1dea86 FcCacheTimeValid dir "/usr/share/fonts/google-noto-emoji" cache checksum 1508753021.86168000 dir checksum 1508753021.86168000 FcDirCacheReadUUID Unable to read /run/host/fonts/inconsolata/.uuid FcDirCacheReadUUID /run/host/fonts/jomolhari/.uuid -> 2b4b66c8-5059-4ff6-bbf0-6f425d6817c7 FcCacheTimeValid dir "/usr/share/fonts/jomolhari" cache checksum 1506524278.448972000 dir checksum 1506524278.448972000 FcDirCacheReadUUID /run/host/fonts/jsmath-fonts/.uuid -> bcc562b3-361f-43d7-8435-70c1f64341df FcCacheTimeValid dir "/usr/share/fonts/jsmath-fonts" cache checksum 1506524278.449972000 dir checksum 1506524278.449972000 FcDirCacheReadUUID /run/host/fonts/khmeros/.uuid -> 12ab71ea-b544-4a55-9a0e-523660ce35c5 FcCacheTimeValid dir "/usr/share/fonts/khmeros" cache checksum 1506524278.450972000 dir checksum 1506524278.450972000 FcDirCacheReadUUID Unable to read /run/host/fonts/kurier/.uuid FcDirCacheReadUUID /run/host/fonts/levien-inconsolata/.uuid -> 6e1fea28-ef4d-4d54-97fa-93f01056500e FcCacheTimeValid dir "/usr/share/fonts/levien-inconsolata" cache checksum 1506524278.453972000 dir checksum 1506524278.453972000 FcDirCacheReadUUID /run/host/fonts/liberation/.uuid -> 760b20fb-992f-4475-ba45-2ed8c033256f FcCacheTimeValid dir "/usr/share/fonts/liberation" cache checksum 1506524278.453972000 dir checksum 1506524278.453972000 FcDirCacheReadUUID Unable to read /run/host/fonts/libertine/.uuid FcDirCacheReadUUID /run/host/fonts/lklug/.uuid -> c162783f-1dbe-4e86-abd0-c720c6d4da4e FcCacheTimeValid dir "/usr/share/fonts/lklug" cache checksum 1506524278.458972000 dir checksum 1506524278.458972000 FcDirCacheReadUUID Unable to read /run/host/fonts/lm/.uuid FcDirCacheReadUUID Unable to read /run/host/fonts/lm-math/.uuid FcDirCacheReadUUID /run/host/fonts/lohit-assamese/.uuid -> 5afb4093-976e-4633-ba5d-4c7950652357 FcCacheTimeValid dir "/usr/share/fonts/lohit-assamese" cache checksum 1506524278.468972000 dir checksum 1506524278.468972000 FcDirCacheReadUUID /run/host/fonts/lohit-bengali/.uuid -> 200cc04d-d485-4211-9401-a358c44bc7ca Scanning dir /run/host/fonts/lohit-bengali Scanning file /run/host/fonts/lohit-bengali/Lohit-Bengali.ttf...done charsets 1 -> 1 leaves 5 -> 5 FcDirCacheWriteDir dir "/run/host/fonts/lohit-bengali" file "/home/alex/.var/app/org.godotengine.Godot/cache/fontconfig//d2e3fac7765db9f81a2f283c64f24f81-le64.cache-7" FcDirCacheReadUUID /run/host/fonts/lohit-devanagari/.uuid -> be3ea7cb-c336-4aa7-8c04-dee2f83c92fa FcCacheTimeValid dir "/usr/share/fonts/lohit-devanagari" cache checksum 1506524278.469972000 dir checksum 1506524278.469972000 FcDirCacheReadUUID /run/host/fonts/lohit-gujarati/.uuid -> 3aec45a2-fc86-40bf-9cd6-9fd0c4bd9ab8 FcCacheTimeValid dir "/usr/share/fonts/lohit-gujarati" cache checksum 1506524278.469972000 dir checksum 1506524278.469972000 FcDirCacheReadUUID /run/host/fonts/lohit-gurmukhi/.uuid -> 5518a3c5-2bbd-4d44-ac05-2b95fb8ee6f6 FcCacheTimeValid dir "/usr/share/fonts/lohit-gurmukhi" cache checksum 1506524278.470972000 dir checksum 1506524278.470972000 FcDirCacheReadUUID /run/host/fonts/lohit-kannada/.uuid -> bad088d3-5a07-4506-a9be-1aa78893a313 Scanning dir /run/host/fonts/lohit-kannada Scanning file /run/host/fonts/lohit-kannada/Lohit-Kannada.ttf...done charsets 1 -> 1 leaves 5 -> 5 FcDirCacheWriteDir dir "/run/host/fonts/lohit-kannada" file "/home/alex/.var/app/org.godotengine.Godot/cache/fontconfig//3d0ed6e0041277cb5a06066098ef252a-le64.cache-7" FcDirCacheReadUUID /run/host/fonts/lohit-odia/.uuid -> 19cb69d9-7416-4061-9c30-9f520897de06 FcCacheTimeValid dir "/usr/share/fonts/lohit-odia" cache checksum 1506524278.471972000 dir checksum 1506524278.471972000 FcDirCacheReadUUID /run/host/fonts/lohit-tamil/.uuid -> 93c7c233-aaf3-4b4a-a61c-8103acfccab8 FcCacheTimeValid dir "/usr/share/fonts/lohit-tamil" cache checksum 1506524278.471972000 dir checksum 1506524278.471972000 FcDirCacheReadUUID /run/host/fonts/lohit-telugu/.uuid -> 19e9b256-2ffc-468e-80cb-bbfcf3e06d32 FcCacheTimeValid dir "/usr/share/fonts/lohit-telugu" cache checksum 1506524278.471972000 dir checksum 1506524278.471972000 FcDirCacheReadUUID /run/host/fonts/lyx/.uuid -> d1d2c41e-d802-4924-b464-7b04ecf9980b Scanning dir /run/host/fonts/lyx Scanning file /run/host/fonts/lyx/cmex10.ttf...done Scanning file /run/host/fonts/lyx/cmmi10.ttf...done Scanning file /run/host/fonts/lyx/cmr10.ttf...done Scanning file /run/host/fonts/lyx/cmsy10.ttf...done Scanning file /run/host/fonts/lyx/esint10.ttf...done Scanning file /run/host/fonts/lyx/eufm10.ttf...done Scanning file /run/host/fonts/lyx/msam10.ttf...done Scanning file /run/host/fonts/lyx/msbm10.ttf...done Scanning file /run/host/fonts/lyx/rsfs10.ttf...done Scanning file /run/host/fonts/lyx/stmary10.ttf...done Scanning file /run/host/fonts/lyx/wasy10.ttf...done charsets 11 -> 10 leaves 17 -> 12 FcDirCacheWriteDir dir "/run/host/fonts/lyx" file "/home/alex/.var/app/org.godotengine.Godot/cache/fontconfig//179fa86a3b998fc67511055e73c403e4-le64.cache-7" FcDirCacheReadUUID Unable to read /run/host/fonts/mnsymbol/.uuid FcDirCacheReadUUID /run/host/fonts/naver-nanum/.uuid -> b46fd656-7d83-43cd-b98a-07699f3ce4e9 FcCacheTimeValid dir "/usr/share/fonts/naver-nanum" cache checksum 1506524278.475972000 dir checksum 1506524278.475972000 FcDirCacheReadUUID /run/host/fonts/opensymbol/.uuid -> f2fa97fd-85b9-43ec-907e-14af580bb7f1 Scanning dir /run/host/fonts/opensymbol Scanning file /run/host/fonts/opensymbol/opens___.ttf...done charsets 1 -> 1 leaves 22 -> 22 FcDirCacheWriteDir dir "/run/host/fonts/opensymbol" file "/home/alex/.var/app/org.godotengine.Godot/cache/fontconfig//a2dfd3047c596329d1703b134060d368-le64.cache-7" FcDirCacheReadUUID /run/host/fonts/oxygen/.uuid -> ca8bede8-5b2d-4147-bab7-89d5c2542d76 FcCacheTimeValid dir "/usr/share/fonts/oxygen" cache checksum 1506524278.477972000 dir checksum 1506524278.477972000 FcDirCacheReadUUID /run/host/fonts/paktype-naqsh/.uuid -> c866576f-9986-4e52-960c-44c83ec93da1 FcCacheTimeValid dir "/usr/share/fonts/paktype-naqsh" cache checksum 1506524278.477972000 dir checksum 1506524278.477972000 FcDirCacheReadUUID /run/host/fonts/paktype-tehreer/.uuid -> 0a764e84-adb3-401c-ac0d-6f25bce03b75 FcCacheTimeValid dir "/usr/share/fonts/paktype-tehreer" cache checksum 1506524278.478972000 dir checksum 1506524278.478972000 FcDirCacheReadUUID /run/host/fonts/paratype-pt-sans/.uuid -> d3c5bedb-6dfa-4b58-b049-58105ed84525 Scanning dir /run/host/fonts/paratype-pt-sans Scanning file /run/host/fonts/paratype-pt-sans/PTN57F.ttf...done Scanning file /run/host/fonts/paratype-pt-sans/PTN77F.ttf...done Scanning file /run/host/fonts/paratype-pt-sans/PTS55F.ttf...done Scanning file /run/host/fonts/paratype-pt-sans/PTS56F.ttf...done Scanning file /run/host/fonts/paratype-pt-sans/PTS75F.ttf...done Scanning file /run/host/fonts/paratype-pt-sans/PTS76F.ttf...done charsets 6 -> 1 leaves 102 -> 17 FcDirCacheWriteDir dir "/run/host/fonts/paratype-pt-sans" file "/home/alex/.var/app/org.godotengine.Godot/cache/fontconfig//581a0de4c3dfdfdf9fceeefcdfba2149-le64.cache-7" FcDirCacheReadUUID /run/host/fonts/sil-abyssinica/.uuid -> c1f4cbc5-bd91-4dc7-bdee-f99fb96d05b2 FcCacheTimeValid dir "/usr/share/fonts/sil-abyssinica" cache checksum 1506524278.479972000 dir checksum 1506524278.479972000 FcDirCacheReadUUID /run/host/fonts/sil-nuosu/.uuid -> 839939e2-9988-4403-a471-ca90f62f7d17 FcCacheTimeValid dir "/usr/share/fonts/sil-nuosu" cache checksum 1506524278.479972000 dir checksum 1506524278.479972000 FcDirCacheReadUUID /run/host/fonts/sil-padauk/.uuid -> 9b18d697-b14d-4797-a7dd-9c821041fb25 FcCacheTimeValid dir "/usr/share/fonts/sil-padauk" cache checksum 1506524278.480972000 dir checksum 1506524278.480972000 FcDirCacheReadUUID /run/host/fonts/smc/.uuid -> 0a32e846-0931-4573-a3ad-f10f7362b96c FcCacheTimeValid dir "/usr/share/fonts/smc" cache checksum 1506524278.480972000 dir checksum 1506524278.480972000 FcDirCacheReadUUID Unable to read /run/host/fonts/sourcecodepro/.uuid FcDirCacheReadUUID /run/host/fonts/stix/.uuid -> 6d55280b-bc60-4fae-b02c-75b6a7e159b7 FcCacheTimeValid dir "/usr/share/fonts/stix" cache checksum 1506524278.484972000 dir checksum 1506524278.484972000 FcDirCacheReadUUID Unable to read /run/host/fonts/tex-gyre/.uuid FcDirCacheReadUUID Unable to read /run/host/fonts/tex-gyre-math/.uuid FcDirCacheReadUUID /run/host/fonts/thai-scalable/.uuid -> 339dca5b-96c9-47ef-aa93-41a774dd5b0a FcCacheTimeValid dir "/usr/share/fonts/thai-scalable" cache checksum 1506524278.490972000 dir checksum 1506524278.490972000 FcDirCacheReadUUID /run/host/fonts/un-core/.uuid -> 55c548c8-8771-4668-a052-00a3b560a28d FcCacheTimeValid dir "/usr/share/fonts/un-core" cache checksum 1506524278.491972000 dir checksum 1506524278.491972000 FcDirCacheReadUUID /run/host/fonts/vlgothic/.uuid -> ae6e15c1-8542-4b98-a981-5edb1fb03403 Scanning dir /run/host/fonts/vlgothic Scanning file /run/host/fonts/vlgothic/VL-Gothic-Regular.ttf...done Scanning file /run/host/fonts/vlgothic/VL-PGothic-Regular.ttf...done charsets 2 -> 2 leaves 248 -> 136 FcDirCacheWriteDir dir "/run/host/fonts/vlgothic" file "/home/alex/.var/app/org.godotengine.Godot/cache/fontconfig//6c21dd185dfc2469e81bb74ed5aa7a3f-le64.cache-7" FcDirCacheReadUUID /run/host/fonts/wqy-zenhei/.uuid -> e0638b0b-d55a-4b77-8919-a4779566145f FcCacheTimeValid dir "/usr/share/fonts/wqy-zenhei" cache checksum 1506524278.493972000 dir checksum 1506524278.493972000 FcDirCacheReadUUID /run/host/user-fonts/Fonts/.uuid -> ad7573a8-dcda-43ed-b43c-4218c5eea077 Scanning dir /run/host/user-fonts/Fonts Scanning file /run/host/user-fonts/Fonts/fonts.cache-1...FcDirCacheWriteDir dir "/run/host/user-fonts/Fonts" file "/home/alex/.var/app/org.godotengine.Godot/cache/fontconfig//be50a7ac08093e46a9f68ae59a1416fc-le64.cache-7" FcDirCacheReadUUID /run/host/user-fonts/OSXFonts/.uuid -> c10cca52-c6ed-4111-8bf1-f67a8f2acc4b FcCacheTimeValid dir "/home/alex/.fonts/OSXFonts" cache checksum 1505811826.615881682 dir checksum 1505811826.615881682 FcDirCacheReadUUID /run/host/user-fonts/ttf-bitstream-vera-1.10/.uuid -> e93f8f93-4f7c-4eb1-aefb-48254c831c22 FcCacheTimeValid dir "/home/alex/.fonts/ttf-bitstream-vera-1.10" cache checksum 1505811826.616881668 dir checksum 1505811826.616881668 FcDirCacheReadUUID /run/host/fonts/default/Type1/.uuid -> a618eef3-4060-4b56-875e-5d7c0bfb910c FcCacheTimeValid dir "/usr/share/fonts/default/Type1" cache checksum 1506524278.495972000 dir checksum 1506524278.495972000 FcDirCacheReadUUID /run/host/fonts/default/ghostscript/.uuid -> a494afb4-7341-46f5-9790-4f7c4cdcf757 FcCacheTimeValid dir "/usr/share/fonts/default/ghostscript" cache checksum 1506524278.518971000 dir checksum 1506524278.518971000 For whatever reason it decided that the host-side font cache was not good enough to use, so it recreated it. Take the first case: FcDirCacheReadUUID /run/host/fonts/lohit-bengali/.uuid -> 200cc04d-d485-4211-9401-a358c44bc7ca Scanning dir /run/host/fonts/lohit-bengali Scanning file /run/host/fonts/lohit-bengali/Lohit-Bengali.ttf...done charsets 1 -> 1 leaves 5 -> 5 FcDirCacheWriteDir dir "/run/host/fonts/lohit-bengali" file "/home/alex/.var/app/org.godotengine.Godot/cache/fontconfig//d2e3fac7765db9f81a2f283c64f24f81-le64.cache-7" In the sandbox this cache is accessible as: /run/host/fonts-cache/200cc04d-d485-4211-9401-a358c44bc7ca-le64.cache-7 but it seems like it is not even looking at that before scanning the directory. In fact, looking at the output from strace it seems like its falling back to the old-style cache rather than a uuid one: 28 open("/run/host/fonts/lohit-bengali/.uuid", O_RDONLY|O_CLOEXEC) = 3</run/host/fonts/lohit-bengali/.uuid> 28 read(3</run/host/fonts/lohit-bengali/.uuid>, "200cc04d-d485-4211-9401-a358c44b"..., 36) = 36 28 close(3</run/host/fonts/lohit-bengali/.uuid>) = 0 28 stat("/run/host/fonts/lohit-bengali", {st_mode=S_IFDIR|0755, st_size=44, ...}) = 0 28 open("/run/host/fonts/lohit-bengali", O_RDONLY|O_CLOEXEC) = 3</run/host/fonts/lohit-bengali> 28 fstatfs(3</run/host/fonts/lohit-bengali>, {f_type=BTRFS_SUPER_MAGIC, f_bsize=4096, f_blocks=38295040, f_bfree=12288593, f_bavail=11312579, f_files=0, f_ffree=0, f_fsid={706944059, 551825837}, f_namelen=255, f_frsize=4096, f_flags=ST_VALID|ST_RDONLY|ST_NOSUID|ST_NODEV|ST_RELATIME}) = 0 28 close(3</run/host/fonts/lohit-bengali>) = 0 28 open("/usr/cache/fontconfig//d2e3fac7765db9f81a2f283c64f24f81-le64.cache-7", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 28 open("/app/cache/fontconfig//d2e3fac7765db9f81a2f283c64f24f81-le64.cache-7", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 28 open("/home/alex/.var/app/org.godotengine.Godot/cache/fontconfig//d2e3fac7765db9f81a2f283c64f24f81-le64.cache-7", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 28 open("/run/host/fonts-cache//d2e3fac7765db9f81a2f283c64f24f81-le64.cache-7", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 28 open("/run/host/user-fonts-cache//d2e3fac7765db9f81a2f283c64f24f81-le64.cache-7", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 28 open("/usr/var/cache/fontconfig//d2e3fac7765db9f81a2f283c64f24f81-le64.cache-7", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 28 open("/home/alex/.fontconfig//d2e3fac7765db9f81a2f283c64f24f81-le64.cache-7", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
Created attachment 136250 [details] [review] Fix replace mode of FcHash
It seems to work better with this hash fix.
Fontconfig master is crashing for me now, with some font lookups but not always, in like 656: 651 while (l->value.type != FcTypeString) 652 l = FcValueListNext (l); 653 if (!l) 654 goto bail0; 655 dir = FcStrDirname (FcValueString (&l->value)); 656 if (FcHashTableFind (config->alias_table, dir, (void **) &alias)) 657 { 658 FcChar8 *base = FcStrBasename (FcValueString (&l->value)); 659 FcChar8 *s = FcStrBuildFilename (alias, base, NULL); 660 FcValue v; (gdb) Apparently config is NULL but config->alias_table accessed. I also find it bizarre that fcmatch.c has directory name manipulation code now. This should have been refactored somewhere else ideally. Thanks.
merged. thank you.
Actually, this patch is not threadsafe. You can't switch out the middle of a linked list with a single atomic compare and swap, because some other thread might be adding another item after that node in parallel, modifying b->next inbetween the bucket->next = b->next; and the cmpexch. I think for this to be safe the replace codepath has to cmpexch on b->value directly.
Actually, not even that is enough to make it threadsafe. Fundamentally, you can't do something like return the values from FcHashTableFind() while at the same time allow replacing them, because at any time after you looked up a value it may become replaced and the value freed. This is true even if you have a value_copy_func, for example in FcHashTableFind, where the value in the table could be freed by another thread doing a replace inbetween reading the bucket->value and calling value_copy_func on it. Also, if you're doing atomic replace on bucket->value, you need to pair that with reading it with get_atomic. I'm not sure why you are writing lock-less code like this, it is a *lot* of work to make it correct, especially on non-intel boxes that don't have strong ordering, and adding a bunch of atomic ops everywhere is likely to be slower than just a single lock around the entire thing unless it is actually highly contended. Anyway, given the current API i think the only solution is to make replace just add *more* values for the key, basically making FcHashBucket->value a list. That way old values are not freed, but just "deprecated".
Hm, yes, that's true. this simply worked because all of values was freed only when everything wasn't needed anymore. so need to lock or *append* a value for replacement as you said.
Created attachment 136680 [details] [review] Fix hashtable add with conflicts Turns out there was another issue with the hashtable, a simple typo when adding items to the hashtable when there was a conflict.
Oops, thanks. merged.
Anything left to do here ? Should this bug be closed ?
-- GitLab Migration Automatic Message -- This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity. You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/fontconfig/fontconfig/issues/60.
Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.