Bug 91958

Summary: Language Match problem for Chinese, Japanese and Korean
Product: fontconfig Reporter: Guo Yunhe <guoyunhebrave>
Component: libraryAssignee: fontconfig-bugs
Status: RESOLVED NOTABUG QA Contact: Behdad Esfahbod <freedesktop>
Severity: normal    
Priority: medium CC: akira
Version: 2.11   
Hardware: Other   
OS: Linux (All)   
Whiteboard:
i915 platform: i915 features:
Attachments: fc-match debug information

Description Guo Yunhe 2015-09-10 09:38:55 UTC 
My system language is Chinese China (zh-CN).

I configured Chinese China, Chinese HongKong, Chinese Taiwan, Japanese and Korean fonts. Chinese China font is good as I configured. But Japanese, Korean, Chinese Hongkong, Chinese Taiwan fonts will use Chinese China font before their own fonts.

It seems "Chinese China" will override all "CJK" languages. This is not the right behavior.

For example I set sans-serif:

Nimbus Sans L
zh-cn: WenQuanYi Zen Hei
zh-tw: WenQuanYi Micro Hei
zh-hk: WenQuanYi Micro Hei
ja: Droid Sans Japanese
ko: Droid Sans Fallback

The fc-match results are:

sans:lang=zh-cn
n019003l.pfb: "Nimbus Sans L" "Regular"
n019004l.pfb: "Nimbus Sans L" "Bold"
n019023l.pfb: "Nimbus Sans L" "Regular Italic"
n019024l.pfb: "Nimbus Sans L" "Bold Italic"
wqy-zenhei.ttc: "WenQuanYi Zen Hei" "Regular"
...(correct)

sans:lang=zh-tw
n019003l.pfb: "Nimbus Sans L" "Regular"
n019004l.pfb: "Nimbus Sans L" "Bold"
n019023l.pfb: "Nimbus Sans L" "Regular Italic"
n019024l.pfb: "Nimbus Sans L" "Bold Italic"
wqy-zenhei.ttc: "WenQuanYi Zen Hei" "Regular"
wqy-microhei.ttc: "WenQuanYi Micro Hei" "Regular"
...(wrong, Chinese China font inserted before Chinese Taiwan font)

sans:lang=zh-hk
n019003l.pfb: "Nimbus Sans L" "Regular"
n019004l.pfb: "Nimbus Sans L" "Bold"
n019023l.pfb: "Nimbus Sans L" "Regular Italic"
n019024l.pfb: "Nimbus Sans L" "Bold Italic"
wqy-zenhei.ttc: "WenQuanYi Zen Hei" "Regular"
wqy-microhei.ttc: "WenQuanYi Micro Hei" "Regular"
...(wrong, Chinese China font inserted before Chinese Hongkong font)

sans:lang=ja
n019003l.pfb: "Nimbus Sans L" "Regular"
n019004l.pfb: "Nimbus Sans L" "Bold"
n019023l.pfb: "Nimbus Sans L" "Regular Italic"
n019024l.pfb: "Nimbus Sans L" "Bold Italic"
wqy-zenhei.ttc: "WenQuanYi Zen Hei" "Regular"
DroidSansJapanese.ttf: "Droid Sans Japanese" "Regular"
...(wrong, Chinese China font inserted before Japanese font)

sans:lang=ko
n019003l.pfb: "Nimbus Sans L" "Regular"
n019004l.pfb: "Nimbus Sans L" "Bold"
n019023l.pfb: "Nimbus Sans L" "Regular Italic"
n019024l.pfb: "Nimbus Sans L" "Bold Italic"
wqy-zenhei.ttc: "WenQuanYi Zen Hei" "Regular"
DroidSansFallbackFull.ttf: "Droid Sans Fallback" "Regular"
...(wrong, Chinese China font inserted before Korean font)

Here is a part of my fonts.conf user configuration:

--------------------------------------------------------
  <match>
    <test name="family">
      <string>sans-serif</string>
    </test>
    <edit binding="strong" mode="prepend" name="family">
      <string>Nimbus Sans L</string>
    </edit>
  </match>
  <match>
    <test name="family">
      <string>serif</string>
    </test>
    <edit binding="strong" mode="prepend" name="family">
      <string>Nimbus Roman No9 L</string>
    </edit>
  </match>
  <match>
    <test name="family">
      <string>monospace</string>
    </test>
    <edit binding="strong" mode="prepend" name="family">
      <string>DejaVu Sans Mono</string>
    </edit>
  </match>
  <match>
    <test name="family">
      <string>sans-serif</string>
    </test>
    <test compare="eq" name="lang">
      <string>zh-hk</string>
    </test>
    <edit mode="prepend" name="family" binding="strong">
      <string>WenQuanYi Micro Hei</string>
    </edit>
  </match>
  <match>
    <test name="family">
      <string>serif</string>
    </test>
    <test compare="eq" name="lang">
      <string>zh-hk</string>
    </test>
    <edit mode="prepend" name="family" binding="strong">
      <string>AR PL UKai HK</string>
    </edit>
  </match>
  <match>
    <test name="family">
      <string>monospace</string>
    </test>
    <test compare="eq" name="lang">
      <string>zh-hk</string>
    </test>
    <edit mode="prepend" name="family" binding="strong">
      <string>WenQuanYi Micro Hei Mono</string>
    </edit>
  </match>
  <match>
    <test name="family">
      <string>sans-serif</string>
    </test>
    <test compare="eq" name="lang">
      <string>zh-cn</string>
    </test>
    <edit mode="prepend" name="family" binding="strong">
      <string>WenQuanYi Zen Hei</string>
    </edit>
  </match>
  <match>
    <test name="family">
      <string>serif</string>
    </test>
    <test compare="eq" name="lang">
      <string>zh-cn</string>
    </test>
    <edit mode="prepend" name="family" binding="strong">
      <string>AR PL UMing CN</string>
    </edit>
  </match>
  <match>
    <test name="family">
      <string>monospace</string>
    </test>
    <test compare="eq" name="lang">
      <string>zh-cn</string>
    </test>
    <edit mode="prepend" name="family" binding="strong">
      <string>WenQuanYi Micro Hei Mono</string>
    </edit>
  </match>
  <match>
    <test name="family">
      <string>sans-serif</string>
    </test>
    <test compare="eq" name="lang">
      <string>zh-tw</string>
    </test>
    <edit mode="prepend" name="family" binding="strong">
      <string>WenQuanYi Micro Hei</string>
    </edit>
  </match>
  <match>
    <test name="family">
      <string>serif</string>
    </test>
    <test compare="eq" name="lang">
      <string>zh-tw</string>
    </test>
    <edit mode="prepend" name="family" binding="strong">
      <string>AR PL UMing TW</string>
    </edit>
  </match>
  <match>
    <test name="family">
      <string>monospace</string>
    </test>
    <test compare="eq" name="lang">
      <string>zh-tw</string>
    </test>
    <edit mode="prepend" name="family" binding="strong">
      <string>WenQuanYi Micro Hei Mono</string>
    </edit>
  </match>
  <match>
    <test name="family">
      <string>sans-serif</string>
    </test>
    <test compare="eq" name="lang">
      <string>ja</string>
    </test>
    <edit mode="prepend" name="family" binding="strong">
      <string>Droid Sans Japanese</string>
    </edit>
  </match>
  <match>
    <test name="family">
      <string>serif</string>
    </test>
    <test compare="eq" name="lang">
      <string>ja</string>
    </test>
    <edit mode="prepend" name="family" binding="strong">
      <string>AR PL UMing TW</string>
    </edit>
  </match>
  <match>
    <test name="family">
      <string>monospace</string>
    </test>
    <test compare="eq" name="lang">
      <string>ja</string>
    </test>
    <edit mode="prepend" name="family" binding="strong">
      <string>Droid Sans Japanese</string>
    </edit>
  </match>
  <match>
    <test name="family">
      <string>sans-serif</string>
    </test>
    <test compare="eq" name="lang">
      <string>ko</string>
    </test>
    <edit mode="prepend" name="family" binding="strong">
      <string>Droid Sans Fallback</string>
    </edit>
  </match>
  <match>
    <test name="family">
      <string>serif</string>
    </test>
    <test compare="eq" name="lang">
      <string>ko</string>
    </test>
    <edit mode="prepend" name="family" binding="strong">
      <string>AR PL UMing TW</string>
    </edit>
  </match>
  <match>
    <test name="family">
      <string>monospace</string>
    </test>
    <test compare="eq" name="lang">
      <string>ko</string>
    </test>
    <edit mode="prepend" name="family" binding="strong">
      <string>WenQuanYi Micro Hei Mono</string>
    </edit>
  </match>
--------------------------------------------------------
Comment 1 Akira TAGOH 2015-09-10 10:22:46 UTC 
it is hard to figure out the reason of that result by the part of configuration. I guess you may have other configuration that fontconfig determines that way. try FC_DEBUG=4 to see what kinda configuration you have or FC_DEBUG=3 to see how fontconfig evaluates fonts to determine the best font or FC_DEBUG=7 to see both.

in most cases, the sort of this claim isn't a bug in fontconfig but introduced by unexpected configurations.
Comment 2 Guo Yunhe 2015-09-10 10:45:21 UTC 
(In reply to Akira TAGOH from comment #1)
> it is hard to figure out the reason of that result by the part of
> configuration. I guess you may have other configuration that fontconfig
> determines that way. try FC_DEBUG=4 to see what kinda configuration you have
> or FC_DEBUG=3 to see how fontconfig evaluates fonts to determine the best
> font or FC_DEBUG=7 to see both.
> 
> in most cases, the sort of this claim isn't a bug in fontconfig but
> introduced by unexpected configurations.

How to use FC_DEBUG=7? Is it a command or paramater? Can you give me a full command line that I can try?
Comment 3 Akira TAGOH 2015-09-10 11:24:21 UTC 
it is an environment variable. try FC_DEBUG=3 fc-match blahblahblah.
Comment 4 Guo Yunhe 2015-09-10 11:33:57 UTC 
I tried FC_DEBUG=4 fc-match sans:lang=ko > debug.log

----------------------------------------------------
Prepend list before  "Nimbus Sans L"(s) "WenQuanYi Zen Hei"(s) [marker] "sans-serif"(s)
Prepend list after  "Nimbus Sans L"(s) "WenQuanYi Zen Hei"(s) "Droid Sans Fallback"(s) "sans-serif"(s)
FcConfigSubstitute editPattern has 9 elts (size 16)
	family: "Nimbus Sans L"(s) "WenQuanYi Zen Hei"(s) "Droid Sans Fallback"(s) "sans-serif"(s)
	rgba: 5(i)(w)
	lang: ko(s) "zh-CN"(w)
	lcdfilter: 0(i)(w)
	prgname: "fc-match"(s)
	force_hintstyle: "none"(w)
	force_autohint: False(w)
	force_bw: False(w)
	force_bw_monospace: False(w)
----------------------------------------------------

I found there is always a "zh_CN" following "ko". Fontconfig always prepends "WenQuanYi Zen Hei" (zh_CN font) before "Droid Sans Fallback" (ko font).

When I change Chinese China font in ~/.config/fontconfig/fonts.conf , fontconfig will prepend new Chinese China font before "Droid Sans Fallback" (Korean font). I don't have other user configuration, system configuration didn't do anything to the language.
Comment 5 Guo Yunhe 2015-09-10 11:34:52 UTC 
Created attachment 118185 [details]
fc-match debug information
Comment 6 Akira TAGOH 2015-09-10 11:44:38 UTC 
try drop binding="strong" from your configuration. that breaks the binding in the pattern and always force to make the result with the strong bindidng even if the binding of the matched value in the pattern was a weak.
Comment 7 Guo Yunhe 2015-09-10 11:46:51 UTC 
I have removed binding="strong" from zh-cn language match. Now fontconfig will not insert before ja, zh-tw, zh-hk fonts. (these language matches have  binding="strong" property).
Comment 8 Guo Yunhe 2015-09-10 11:49:11 UTC 
If I remove all  binding="strong" from all language matches, then Chinese China (zh-cn) will still insert before zh-tw, zh-hk, ja, ko. It seems Chinese China always has a higher binding than others. Any idea about this?
Comment 9 Akira TAGOH 2015-09-10 11:55:06 UTC 
I forgot to mention why you always have zh-cn. fontconfig picks up the lang from current locale with the weak binding. I guess because you're running on zh_CN locale and basically Chinese fonts has enough coverage of JK glyphs. so since fontconfig tries to evaluate the configuration at first, Chinese fonts has better priority for them.

The solution for that is, run apps with the locale other than zh_CN or move zh-cn configuration later than others.
Comment 10 Guo Yunhe 2015-09-10 12:11:48 UTC 
Yes, this is the answer for me!

I am configuring fontconfig for Firefox. When I visit Japanese Wikipedia, the page use zh_CN fonts, though it has lang="ja" property.

I write a small GUI tool to configure fontconfig (user configuration files). https://github.com/guoyunhe/fontweak

I have another question: If my system language is Chinese Hongkong, or Korean, will it also cover other languages?
Comment 11 Akira TAGOH 2015-09-11 02:00:08 UTC 
(In reply to Guo Yunhe from comment #10)
> Yes, this is the answer for me!
> 
> I am configuring fontconfig for Firefox. When I visit Japanese Wikipedia,
> the page use zh_CN fonts, though it has lang="ja" property.
> 
> I write a small GUI tool to configure fontconfig (user configuration files).
> https://github.com/guoyunhe/fontweak
> 
> I have another question: If my system language is Chinese Hongkong, or
> Korean, will it also cover other languages?

as long as you configure your fontconfig files properly and applications requests fonts with proper queries, yes. of course assuming renderers has enough functionality to do so.

Anyway, I'm closing this so that it isn't a bug in fontconfig.

Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.