Bug 94551

Summary: Provide a means for matching and selecting a color emoji font
Product: fontconfig Reporter: Dominik Röttsches <d-r>
Component: fc-matchAssignee: fontconfig-bugs
Status: RESOLVED FIXED QA Contact: Behdad Esfahbod <freedesktop>
Severity: enhancement    
Priority: medium CC: akira, dr.khaled.hosny, freedesktop, jan.steffens
Version: unspecified   
Hardware: Other   
OS: All   
Whiteboard:
i915 platform: i915 features:

Description Dominik Röttsches 2016-03-15 09:04:42 UTC
In Chrome/Blink we need a way to explicitly select a color emoji font in order to display emoji in the correct presentation form: text presentation or emoji presentation. (as opposed to finding a non-color, outline font through regular character fallback without an "needs-color-emoji-font" hint)

Android handles this through a special locale "und-Zsye" and finding a matching font for an emoji character while specifying this locale at the same time.

This approach is somewhat in line with the proposed extensions to UTR #51:
http://www.unicode.org/reports/tr51/proposed.html#Emoji_Script
Comment 1 Behdad Esfahbod 2016-07-22 00:12:24 UTC
Or just add a new generic family named "emoji"?

Is CSS doing something like that?
Comment 2 Behdad Esfahbod 2017-07-29 16:54:48 UTC
We also need this in pango.  I suggest using the generic family name "emoji".  Not sure if CSS has done that...

https://bugzilla.gnome.org/show_bug.cgi?id=785566
Comment 3 Dominik Röttsches 2017-07-29 16:58:14 UTC
Yes, https://drafts.csswg.org/css-fonts-4/#emoji-def
Comment 4 Behdad Esfahbod 2017-07-29 17:05:48 UTC
Cool.  There's these that we don't support currently:

""""""""
 system-ui

This generic font family is intended to let text render with the default user interface font on the platform on which the UA is running. A cross-platform UA should use different fonts on its different supported platforms. The purpose of system-ui is to allow web content to integrate with the look and feel of a native app. On platforms which have a collection of system user interface fonts (e.g. for different languages), user agents may treat system-ui as a virtual font which encompasses all the platform user interface fonts. However, if this is done, the details of the virtual font must not be visible or detectable.

 emoji

This font family is intended for use with emoji characters.

 math

This font family is intended for use with mathematical expressions.

 fangsong

This font family is used for fang song typefaces in Chinese.
""""""""

Emoji and math are easy to implement.  system-ui, I don't know how to do.
Comment 5 Akira TAGOH 2017-07-30 04:53:54 UTC
If there are any way to categrize them systematically and/or mechanically, I want to integrate it into Bug#29497 and https://cgit.freedesktop.org/~tagoh/fontconfig/commit/?h=panose-sfamilyclass-support&id=a1f249360b76526a801df7213497c960a9c969fe branch. that may makes sense rather than modifying "family" property as they are classified to them.
Comment 6 Behdad Esfahbod 2017-07-31 14:39:32 UTC
(In reply to Akira TAGOH from comment #5)
> If there are any way to categrize them systematically and/or mechanically, I
> want to integrate it into Bug#29497 and
> https://cgit.freedesktop.org/~tagoh/fontconfig/commit/?h=panose-sfamilyclass-
> support&id=a1f249360b76526a801df7213497c960a9c969fe branch. that may makes
> sense rather than modifying "family" property as they are classified to them.

That's a much bigger change.  I prefer to follow our existing method of using family name for now, until we find a better way.  Plus, we should not mark non-color emoji fonts as "emoji".  Or at least, they should be lower in the priority list.
Comment 7 Akira TAGOH 2017-07-31 16:08:33 UTC
Most easier way to do this may be to add the sort of this as the configuration:

<match target="scan">
  <test name="charset">
    <name>charset</name>
    <charset>
      .....
    </charset>
  </test>
  <test name="color">
    <bool>true</bool>
  </test>
  <edit name="family" mode="append">
    <string>emoji</string>
  </edit>
</match>

we could generate the charset of "emoji" from emoji-data.txt from Unicode.org or manage by hand to have the sort of subset of emoji that may be most-used one.

FWIW requring <name> element is somewhat not intuitive... and need to shut up the multiple value in test warning.
Comment 8 Behdad Esfahbod 2017-07-31 16:19:06 UTC
Fixed:

:...skipping...
commit d7f3437ade668c60a7e31f93669b73680be6512a (HEAD -> master, origin/master)
Author: Behdad Esfahbod <behdad@behdad.org>
Date:   Mon Jul 31 17:17:16 2017 +0100

    Add generic family matching for "emoji" and "math"
    
    Fixes https://bugs.freedesktop.org/show_bug.cgi?id=94551

diff --git a/conf.d/45-generic.conf b/conf.d/45-generic.conf
new file mode 100644
index 0000000..d3f6277
--- /dev/null
+++ b/conf.d/45-generic.conf
@@ -0,0 +1,67 @@
+<?xml version="1.0"?>
+<!DOCTYPE fontconfig SYSTEM "fonts.dtd">
+<fontconfig>
+<!-- Keep in sync with 60-generic.conf -->
+
+<!-- Emoji -->
+       <alias binding="same">
+               <family>EmojiTwo</family>
+               <default><family>emoji</family></default>
+       </alias>
+       <alias binding="same">
+               <family>EmojiOne</family>
+               <default><family>emoji</family></default>
+       </alias>
+       <alias binding="same">
+               <family>Noto Color Emoji</family> <!-- Google -->
+               <default><family>emoji</family></default>
+       </alias>
+       <alias binding="same">
+               <family>Apple Color Emoji</family> <!-- Apple -->
+               <default><family>emoji</family></default>
+       </alias>
+       <alias binding="same">
+               <family>Segoe UI Emoji</family> <!-- Microsoft -->
+               <default><family>emoji</family></default>
+       </alias>
+       <alias binding="same">
+               <family>Noto Emoji</family> <!-- Google -->
+               <default><family>emoji</family></default>
+       </alias>
+       <alias binding="same">
+               <family>Android Emoji</family> <!-- Google -->
+               <default><family>emoji</family></default>
+       </alias>
+
+<!-- Math -->
+       <!-- https://en.wikipedia.org/wiki/Category:Mathematical_OpenType_typefaces -->
+       <alias binding="same">
+               <family>XITS Math</family> <!-- Khaled Hosny -->
+               <default><family>math</family></default>
+       </alias>
+       <alias binding="same">
+               <family>STIX Two Math</family> <!-- AMS -->
+               <default><family>math</family></default>
+       </alias>
+       <alias binding="same">
+               <family>Cambria Math</family> <!-- Microsoft -->
+               <default><family>math</family></default>
+       </alias>
+       <alias binding="same">
+               <family>Latin Modern Math</family> <!-- TeX -->
+               <default><family>math</family></default>
+       </alias>
+       <alias binding="same">
+               <family>Minion Math</family> <!-- Adobe -->
+               <default><family>math</family></default>
+       </alias>
+       <alias binding="same">
+               <family>Lucida Math</family> <!-- Adobe -->
+               <default><family>math</family></default>
+       </alias>
+       <alias binding="same">
+               <family>Asana Math</family>
+               <default><family>math</family></default>
+       </alias>
+
+</fontconfig>
diff --git a/conf.d/60-generic.conf b/conf.d/60-generic.conf
new file mode 100644
index 0000000..24b7466
--- /dev/null
+++ b/conf.d/60-generic.conf
@@ -0,0 +1,37 @@
+<?xml version="1.0"?>
+<!DOCTYPE fontconfig SYSTEM "fonts.dtd">
+<fontconfig>
+<!-- Keep in sync with 45-generic.conf -->
+
+<!-- Emoji -->
+       <alias binding="same">
+               <family>emoji</family>
+               <prefer>
+                       <family>EmojiTwo</family>
+                       <family>EmojiOne</family>
+                       <!-- System fonts -->
+                       <family>Noto Color Emoji</family> <!-- Google -->
+                       <family>Apple Color Emoji</family> <!-- Apple -->
+                       <family>Segoe UI Emoji</family> <!-- Microsoft -->
+                       <!-- Non-color -->
+                       <family>Noto Emoji</family> <!-- Google -->
+                       <family>Android Emoji</family> <!-- Google -->
+               </prefer>
+       </alias>
+
+<!-- Math -->
+       <alias binding="same">
+               <!-- https://en.wikipedia.org/wiki/Category:Mathematical_OpenType_typefaces -->
+               <family>math</family>
+               <prefer>
+                       <family>XITS Math</family> <!-- Khaled Hosny -->
+                       <family>STIX Two Math</family> <!-- AMS -->
+                       <family>Cambria Math</family> <!-- Microsoft -->
+                       <family>Latin Modern Math</family> <!-- TeX -->
+                       <family>Minion Math</family> <!-- Adobe -->
+                       <family>Lucida Math</family> <!-- Adobe -->
+                       <family>Asana Math</family>
+               </prefer>
+       </alias>
+
+</fontconfig>
diff --git a/conf.d/Makefile.am b/conf.d/Makefile.am
index ac68cd1..fd50a5d 100644
--- a/conf.d/Makefile.am
+++ b/conf.d/Makefile.am
@@ -32,10 +32,12 @@ CONF_LINKS = \
        30-urw-aliases.conf \
        30-metric-aliases.conf \
        40-nonlatin.conf \
+       45-generic.conf \
        45-latin.conf \
        49-sansserif.conf \
        50-user.conf \
        51-local.conf \
+       60-generic.conf \
        60-latin.conf \
        65-fonts-persian.conf \
        65-nonlatin.conf \
@@ -71,10 +73,12 @@ template_DATA =                             \
        30-urw-aliases.conf             \
        30-metric-aliases.conf          \
        40-nonlatin.conf                \
+       45-generic.conf                 \
        45-latin.conf                   \
        49-sansserif.conf               \
        50-user.conf                    \
        51-local.conf                   \
+       60-generic.conf                 \
        60-latin.conf                   \
        65-fonts-persian.conf           \
        65-khmer.conf                   \
Comment 9 Behdad Esfahbod 2017-07-31 16:19:55 UTC
(In reply to Akira TAGOH from comment #7)
> Most easier way to do this may be to add the sort of this as the
> configuration:
> 
> <match target="scan">
>   <test name="charset">
>     <name>charset</name>
>     <charset>
>       .....
>     </charset>
>   </test>
>   <test name="color">
>     <bool>true</bool>
>   </test>
>   <edit name="family" mode="append">
>     <string>emoji</string>
>   </edit>
> </match>

That's an interesting approach indeed!  I'm open to doing that.  Move the color test first though, that's faster.

 
> we could generate the charset of "emoji" from emoji-data.txt from
> Unicode.org or manage by hand to have the sort of subset of emoji that may
> be most-used one.
> 
> FWIW requring <name> element is somewhat not intuitive... and need to shut
> up the multiple value in test warning.
Comment 10 Behdad Esfahbod 2017-07-31 16:48:52 UTC
(In reply to Behdad Esfahbod from comment #9)
> (In reply to Akira TAGOH from comment #7)
> > Most easier way to do this may be to add the sort of this as the
> > configuration:
> > 
> > <match target="scan">
> >   <test name="charset">
> >     <name>charset</name>
> >     <charset>
> >       .....
> >     </charset>
> >   </test>
> >   <test name="color">
> >     <bool>true</bool>
> >   </test>
> >   <edit name="family" mode="append">
> >     <string>emoji</string>
> >   </edit>
> > </match>
> 
> That's an interesting approach indeed!  I'm open to doing that.  Move the
> color test first though, that's faster.

Going down this route I think we should just define emoji as a language and let the fc-lang facility take care of it.  Then in the config, we can prepend language emoji if family emoji is requested.  We can also add a test for color and set it to true if unspecified in the pattern, to prefer color fonts for emoji.
Comment 11 Akira TAGOH 2017-07-31 18:22:02 UTC
(In reply to Behdad Esfahbod from comment #10)
> Going down this route I think we should just define emoji as a language and
> let the fc-lang facility take care of it.  Then in the config, we can
> prepend language emoji if family emoji is requested.  We can also add a test
> for color and set it to true if unspecified in the pattern, to prefer color
> fonts for emoji.

sounds good through, then the problem comes in mind would be what codepoints should be in emoji.orth. there are no way to match things proportionally. only all or nothing method.
Comment 12 Dominik Röttsches 2017-08-01 07:46:47 UTC
In this solution, I'd prefer modifying fc_lang or other fields over modifying the name field.
Comment 13 Behdad Esfahbod 2017-08-01 09:09:33 UTC
I think I like adding language orthography files for emoji and math.  What to call them?  emoji and math?  Or is there a BCP 47 language code for those?

Then, keep the existing emoji and math config, but also prepend the language in the config.  This will make sure the curated fonts are used if present, but if not, any other emoji / math font available will be chosen.

For emoji, we will also set FC_COLOR to true if not set, to prefer color emoji fonts over non-color.

Akira, does this sound good?  Of course, I have to try it and make sure it works.
Comment 14 Akira TAGOH 2017-08-01 09:43:03 UTC
Yes, sounds good. if we follow on BCP47, we could use Zsye for emoji and Zmth for math.
Comment 15 Akira TAGOH 2017-08-01 09:54:25 UTC
https://cgit.freedesktop.org/~tagoh/fontconfig/commit/?h=emoji

proposed patch to support Emoji in lang
Comment 16 Akira TAGOH 2017-08-01 09:56:37 UTC
orth file is based on Unicode 9.0. I didn't see working with 10.0. Noto Color Emoji font seems not yet satisfying 10.0.

not testing for other emoji fonts... we may need to polish the orth much more perhaps.
Comment 17 Akira TAGOH 2017-08-01 10:09:04 UTC
BTW I don't know why we keep using the system caches first... well, containing different contents in caches is usually unlikely to happen. but we often see during development. even though no need to bump the cache version to add new orth, we could easily got confused if fontconfig prefer to use a first cache. I prefer to apply the change like this to make me happy...:

diff --git a/fonts.conf.in b/fonts.conf.in
index 7c16a70..d8de4c8 100644
--- a/fonts.conf.in
+++ b/fonts.conf.in
@@ -72,8 +72,8 @@
 
 <!-- Font cache directory list -->
 
-       <cachedir>@FC_CACHEDIR@</cachedir>
        <cachedir prefix="xdg">fontconfig</cachedir>
+       <cachedir>@FC_CACHEDIR@</cachedir>
        <!-- the following element will be removed in the future -->
        <cachedir>~/.fontconfig</cachedir>
Comment 18 Behdad Esfahbod 2017-08-01 10:34:29 UTC
(In reply to Akira TAGOH from comment #17)
> BTW I don't know why we keep using the system caches first... well,
> containing different contents in caches is usually unlikely to happen. but
> we often see during development. even though no need to bump the cache
> version to add new orth, we could easily got confused if fontconfig prefer
> to use a first cache. I prefer to apply the change like this to make me
> happy...:
> 
> diff --git a/fonts.conf.in b/fonts.conf.in
> index 7c16a70..d8de4c8 100644
> --- a/fonts.conf.in
> +++ b/fonts.conf.in
> @@ -72,8 +72,8 @@
>  
>  <!-- Font cache directory list -->
>  
> -       <cachedir>@FC_CACHEDIR@</cachedir>
>         <cachedir prefix="xdg">fontconfig</cachedir>
> +       <cachedir>@FC_CACHEDIR@</cachedir>
>         <!-- the following element will be removed in the future -->
>         <cachedir>~/.fontconfig</cachedir>

Don't know the implications.  Doesn't this make it build ~root/ caches instead of system caches when run as root?
Comment 19 Behdad Esfahbod 2017-08-01 10:42:16 UTC
(In reply to Akira TAGOH from comment #14)
> Yes, sounds good. if we follow on BCP47, we could use Zsye for emoji and
> Zmth for math.

But those are script tags.  So we'd need something like x-Zmth etc?
Comment 20 Behdad Esfahbod 2017-08-01 10:43:44 UTC
(In reply to Akira TAGOH from comment #16)
> orth file is based on Unicode 9.0. I didn't see working with 10.0. Noto
> Color Emoji font seems not yet satisfying 10.0.

I'd go with a much narrower coverage.  Maybe the original set only.  The goal is to detect emoji fonts, and the basic set does that just fine.
Comment 21 Akira TAGOH 2017-08-01 10:44:40 UTC
(In reply to Behdad Esfahbod from comment #18)
> Don't know the implications.  Doesn't this make it build ~root/ caches
> instead of system caches when run as root?

Oh, hmm, it does. need a different behavior for non-root vs root...
Comment 22 Behdad Esfahbod 2017-08-01 10:45:49 UTC
For the config, call it 10-generic.conf?  Also match family="emoji" and prepend language.  Same for math?
Comment 23 Akira TAGOH 2017-08-01 10:46:19 UTC
(In reply to Behdad Esfahbod from comment #19)
> (In reply to Akira TAGOH from comment #14)
> > Yes, sounds good. if we follow on BCP47, we could use Zsye for emoji and
> > Zmth for math.
> 
> But those are script tags.  So we'd need something like x-Zmth etc?

right. so und-zyse and und-zmth maybe?
Comment 24 Behdad Esfahbod 2017-08-01 10:48:00 UTC
(In reply to Akira TAGOH from comment #23)
> (In reply to Behdad Esfahbod from comment #19)
> > (In reply to Akira TAGOH from comment #14)
> > > Yes, sounds good. if we follow on BCP47, we could use Zsye for emoji and
> > > Zmth for math.
> > 
> > But those are script tags.  So we'd need something like x-Zmth etc?
> 
> right. so und-zyse and und-zmth maybe?

Right.  Ugly :D.  How about just emoji and math :D.
Comment 25 Akira TAGOH 2017-08-01 11:05:26 UTC
(In reply to Behdad Esfahbod from comment #22)
> For the config, call it 10-generic.conf?  Also match family="emoji" and
> prepend language.  Same for math?

doesn't it make happened more later if we want to test family="emoji" and add a language in the pattern? because you did it at 45-generic.conf so it isn't likely to see at 10-.

revised a patch; orth renamed to und_zsye.orth now. config moved to 49-generic.conf. added matching of family and prepend a lang.
Comment 26 Akira TAGOH 2017-08-01 11:08:03 UTC
(In reply to Behdad Esfahbod from comment #24)
> (In reply to Akira TAGOH from comment #23)
> > (In reply to Behdad Esfahbod from comment #19)
> > > (In reply to Akira TAGOH from comment #14)
> > > > Yes, sounds good. if we follow on BCP47, we could use Zsye for emoji and
> > > > Zmth for math.
> > > 
> > > But those are script tags.  So we'd need something like x-Zmth etc?
> > 
> > right. so und-zyse and und-zmth maybe?
> 
> Right.  Ugly :D.  How about just emoji and math :D.

I used to use those names but need to patch out to have an exception for them... particularly in FcLangNormalize(). though they are more intuitive than BCP47 one :) I don't mind either.
Comment 27 Behdad Esfahbod 2017-08-01 11:13:06 UTC
(In reply to Akira TAGOH from comment #25)
> (In reply to Behdad Esfahbod from comment #22)
> > For the config, call it 10-generic.conf?  Also match family="emoji" and
> > prepend language.  Same for math?
> 
> doesn't it make happened more later if we want to test family="emoji" and
> add a language in the pattern? because you did it at 45-generic.conf so it
> isn't likely to see at 10-.
> 
> revised a patch; orth renamed to und_zsye.orth now. config moved to
> 49-generic.conf. added matching of family and prepend a lang.

You're right.  Maybe just put it in 45-generic.conf then?  One fewer file...
Comment 28 Akira TAGOH 2017-08-01 11:20:04 UTC
Sure. moved into 45-generic.conf.
Comment 29 Akira TAGOH 2017-08-01 11:21:26 UTC
(In reply to Behdad Esfahbod from comment #20)
> (In reply to Akira TAGOH from comment #16)
> > orth file is based on Unicode 9.0. I didn't see working with 10.0. Noto
> > Color Emoji font seems not yet satisfying 10.0.
> 
> I'd go with a much narrower coverage.  Maybe the original set only.  The
> goal is to detect emoji fonts, and the basic set does that just fine.

What does the original mean here?
Comment 30 Behdad Esfahbod 2017-08-01 11:24:41 UTC
(In reply to Akira TAGOH from comment #29)
> (In reply to Behdad Esfahbod from comment #20)
> > (In reply to Akira TAGOH from comment #16)
> > > orth file is based on Unicode 9.0. I didn't see working with 10.0. Noto
> > > Color Emoji font seems not yet satisfying 10.0.
> > 
> > I'd go with a much narrower coverage.  Maybe the original set only.  The
> > goal is to detect emoji fonts, and the basic set does that just fine.
> 
> What does the original mean here?

The ones introduced in Unicode 6.0?
Comment 31 Akira TAGOH 2017-08-01 11:35:48 UTC
Okay. updated the orth.
Comment 32 Jan Alexander Steffens (heftig) 2017-08-02 11:57:18 UTC
There's also "Twitter Color Emoji" available at https://github.com/eosrei/twemoji-color-font, which supports Unicode 10 and is licensed similarly to EmojiTwo. Maybe we can prefer this one?
Comment 33 Behdad Esfahbod 2017-08-02 12:02:59 UTC
(In reply to Jan Alexander Steffens (heftig) from comment #32)
> There's also "Twitter Color Emoji" available at
> https://github.com/eosrei/twemoji-color-font, which supports Unicode 10 and
> is licensed similarly to EmojiTwo. Maybe we can prefer this one?

Thanks.  I couldn't find a font of it before.  Note that currently it's SVG-in-OpenType format only, so only works in Firefox on Linux.  I'll see if I can make a CBDT/CBLT out of it.  Probably wouldn't get to doing though.

I don't mind putting it in the list, but for the above reason, after the other major fonts.
Comment 34 Behdad Esfahbod 2017-08-02 12:05:34 UTC
(In reply to Akira TAGOH from comment #31)
> Okay. updated the orth.

Thanks.  Got orth for math as well?  Don't know what should go there. :)
Comment 35 Jan Alexander Steffens (heftig) 2017-08-02 12:10:57 UTC
Speaking of Firefox, they're shipping a font named "EmojiOne Mozilla" which I think is in COLR/CPAL format. Is that format also unsupported outside of Firefox?

Would it make sense to put the font name into the list?
Comment 36 Jan Alexander Steffens (heftig) 2017-08-02 12:16:25 UTC
Also the family name for the EmojiOne CBDT/CBLC font is "Emoji One", not "EmojiOne". Not sure where to get a copy of the EmojiTwo font...
Comment 37 Behdad Esfahbod 2017-08-02 12:33:57 UTC
(In reply to Jan Alexander Steffens (heftig) from comment #35)
> Speaking of Firefox, they're shipping a font named "EmojiOne Mozilla" which
> I think is in COLR/CPAL format. Is that format also unsupported outside of
> Firefox?

Not yet.

> Would it make sense to put the font name into the list?

Doing so now.
Comment 38 Behdad Esfahbod 2017-08-02 12:35:19 UTC
(In reply to Jan Alexander Steffens (heftig) from comment #36)
> Also the family name for the EmojiOne CBDT/CBLC font is "Emoji One", not
> "EmojiOne". Not sure where to get a copy of the EmojiTwo font...

Fixed.  Although the matcher doesn't care about space.
Comment 39 Akira TAGOH 2017-08-02 13:01:03 UTC
(In reply to Behdad Esfahbod from comment #38)
> (In reply to Jan Alexander Steffens (heftig) from comment #36)
> > Also the family name for the EmojiOne CBDT/CBLC font is "Emoji One", not
> > "EmojiOne". Not sure where to get a copy of the EmojiTwo font...
> 
> Fixed.  Although the matcher doesn't care about space.

Hmm? it does I suppose. if one wants to consider both values is same, they need to do <test ignore-blanks="yes">.
Comment 40 Behdad Esfahbod 2017-08-02 14:14:30 UTC
(In reply to Akira TAGOH from comment #39)
> (In reply to Behdad Esfahbod from comment #38)
> > (In reply to Jan Alexander Steffens (heftig) from comment #36)
> > > Also the family name for the EmojiOne CBDT/CBLC font is "Emoji One", not
> > > "EmojiOne". Not sure where to get a copy of the EmojiTwo font...
> > 
> > Fixed.  Although the matcher doesn't care about space.
> 
> Hmm? it does I suppose. if one wants to consider both values is same, they
> need to do <test ignore-blanks="yes">.

Humm.  I thought that's the default behavior.  You are probably right.  I was more thinking about the font matcher, not config matcher, so you have a point.  Anyway, fixed already.
Comment 41 Akira TAGOH 2017-08-02 14:24:39 UTC
(In reply to Behdad Esfahbod from comment #34)
> (In reply to Akira TAGOH from comment #31)
> > Okay. updated the orth.
> 
> Thanks.  Got orth for math as well?  Don't know what should go there. :)

Hmm, me neither.. we can get some from http://www.unicode.org/Public/math/revision-15/MathClass-15.txt say, I guess extra glyphs may be included in even a classification.
Comment 42 Behdad Esfahbod 2017-08-02 14:34:30 UTC
Akira, I pushed your patch out.  Will modify the config while testing.  Re the orth file, can you comment out the default-text-presentation codepoints as well?  Ie. just including Emoji_Presentation codepoints.
Comment 43 Behdad Esfahbod 2017-08-02 14:35:38 UTC
(In reply to Akira TAGOH from comment #41)
> (In reply to Behdad Esfahbod from comment #34)
> > (In reply to Akira TAGOH from comment #31)
> > > Okay. updated the orth.
> > 
> > Thanks.  Got orth for math as well?  Don't know what should go there. :)
> 
> Hmm, me neither.. we can get some from
> http://www.unicode.org/Public/math/revision-15/MathClass-15.txt say, I guess
> extra glyphs may be included in even a classification.

That's nice.

Also, Khaled should know.
Comment 44 Akira TAGOH 2017-08-02 14:43:54 UTC
(In reply to Behdad Esfahbod from comment #42)
> Akira, I pushed your patch out.  Will modify the config while testing.  Re
> the orth file, can you comment out the default-text-presentation codepoints
> as well?  Ie. just including Emoji_Presentation codepoints.

Sure. will do. comment all of them out? or apply proposed until 6.0 as well?
Comment 45 Behdad Esfahbod 2017-08-02 14:45:28 UTC
(In reply to Akira TAGOH from comment #44)
> (In reply to Behdad Esfahbod from comment #42)
> > Akira, I pushed your patch out.  Will modify the config while testing.  Re
> > the orth file, can you comment out the default-text-presentation codepoints
> > as well?  Ie. just including Emoji_Presentation codepoints.
> 
> Sure. will do. comment all of them out? or apply proposed until 6.0 as well?

Until 6.0.  Thanks.

As it happens, not even Noto Color Emoji is detected as emoji by language coverage; so we definitely need to loosen it up.
Comment 46 Akira TAGOH 2017-08-02 14:54:35 UTC
(In reply to Behdad Esfahbod from comment #45)
> As it happens, not even Noto Color Emoji is detected as emoji by language
> coverage; so we definitely need to loosen it up.

That is what I told you yesterday. basically adding more code points won't becomes getting more candidates because it is the sort of filter.

Anyway, this happens because fontconfig preferably uses the system cache so far which doesn't contain a bit of und-zsye. for non-root, just changing the order of cache element would helps. for root, stay tuned ;)
Comment 47 Akira TAGOH 2017-08-02 15:09:23 UTC
Ah, for root, we could run fc-cache -s. so we could update the order of <cachedir>.
Comment 48 Behdad Esfahbod 2017-08-02 15:49:35 UTC
(In reply to Akira TAGOH from comment #46)
> (In reply to Behdad Esfahbod from comment #45)
> > As it happens, not even Noto Color Emoji is detected as emoji by language
> > coverage; so we definitely need to loosen it up.
> 
> That is what I told you yesterday. basically adding more code points won't
> becomes getting more candidates because it is the sort of filter.

I didn't say add.  I said replace!  Also why I said up to Unicode 6.0 only...  Fixed.  It works for NotoColorEmoji now.  Yay.

> Anyway, this happens because fontconfig preferably uses the system cache so
> far which doesn't contain a bit of und-zsye.

No, I was testing properly.  I installed the font in my ~/.fonts/ and rebuilt the cache.

> for non-root, just changing the
> order of cache element would helps. for root, stay tuned ;)

I don't know why you want to do that.
Comment 49 Behdad Esfahbod 2017-08-02 15:55:52 UTC
And tested, and it works!
Comment 50 Behdad Esfahbod 2017-08-02 16:01:42 UTC
The test on lang for und-zsye doesn't seem to be working though.  Have not tracked down yet.
Comment 51 Akira TAGOH 2017-08-02 16:19:54 UTC
Hmm, we may be talking about different things then. though I could see the und-zsye lang in Noto Color Emoji without that change here. so I didn't see what you were trying to address.
Comment 52 Behdad Esfahbod 2017-08-02 16:42:47 UTC
Oh my bad then.
Comment 53 Behdad Esfahbod 2017-08-02 16:44:31 UTC
(In reply to Akira TAGOH from comment #47)
> Ah, for root, we could run fc-cache -s. so we could update the order of
> <cachedir>.

Right. Yes I think we should do that.
Comment 54 Akira TAGOH 2017-08-02 17:23:36 UTC
FWIW given that we are going to add und-zmth too, we may need to have some exception for fallback not to have any other available :lang=und fonts. only accept the exact match for und.
Comment 55 Behdad Esfahbod 2017-08-03 09:16:58 UTC
(In reply to Akira TAGOH from comment #54)
> FWIW given that we are going to add und-zmth too, we may need to have some
> exception for fallback not to have any other available :lang=und fonts. only
> accept the exact match for und.

Right.
Comment 56 Behdad Esfahbod 2017-08-03 09:37:19 UTC
Ok, it all seems to be working for me now.  Marking as fixed.

commit cc8442dec85e9d416436d19eeae1783f2d3008f0 (HEAD -> master, origin/master)
Author: Behdad Esfahbod <behdad@behdad.org>
Date:   Thu Aug 3 10:36:01 2017 +0100

    Adjust color emoji config some more
    
    Seems to work now.  Either asking for family emoji, or :lang=und-zsye returns
    the preferred color emoji font available, or just any color emoji font if none
    of the preferred ones was found.
Comment 57 Akira TAGOH 2017-08-03 09:47:07 UTC
shall we file another bug for und_zmth.orth?
Comment 58 Behdad Esfahbod 2017-08-03 09:55:05 UTC
(In reply to Akira TAGOH from comment #57)
> shall we file another bug for und_zmth.orth?

Yes.  And I'm working on fixing FcLangCompare() for und-*
Comment 59 Akira TAGOH 2017-08-03 09:58:52 UTC
Thanks. I can't do anything today so will check it later.
Comment 60 Behdad Esfahbod 2017-08-03 10:20:05 UTC
commit 064440d59797b1158badfb9144f3188cda11a791 (HEAD -> master, origin/master)
Author: Behdad Esfahbod <behdad@behdad.org>
Date:   Thu Aug 3 11:02:32 2017 +0100

    Ignore 'und-' prefix for in FcLangCompare
    
    See https://bugs.freedesktop.org/show_bug.cgi?id=94551#c54
    
    For example, matching for :lang=und-zsye matches emoji font, but searching
    for :lang=und-xyz wouldn't match an emoji font anymore.  Neither does :lang-und.
Comment 61 Khaled Hosny 2017-08-03 11:48:13 UTC
(In reply to Akira TAGOH from comment #41)
> (In reply to Behdad Esfahbod from comment #34)
> > (In reply to Akira TAGOH from comment #31)
> > > Okay. updated the orth.
> > 
> > Thanks.  Got orth for math as well?  Don't know what should go there. :)
> 
> Hmm, me neither.. we can get some from
> http://www.unicode.org/Public/math/revision-15/MathClass-15.txt say, I guess
> extra glyphs may be included in even a classification.

Math fonts vary greatly in their coverage and there is no clear cut what a math symbol and what is not. I don’t know if MathClass-XX.txt is used in anything in practice, but I guess we can make an orthography file out of it and test it with the existing math fonts, possibly reducing it until we get them all covering math.
Comment 62 Behdad Esfahbod 2017-08-03 13:33:08 UTC
Thanks.  Math moved here: https://bugs.freedesktop.org/show_bug.cgi?id=102026

Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.