Two mixed fonts when rendering Chinese in KDE/QT apps with Droid Sans fonts

Thanks for your report. I'm not sure about the affected packages, so let's wait with changing that.

The first thing I would like you to do to sort things out is to get a pure German locale. I'm not a Kubuntu user, and can't tell how you would do it via the GUI, but probably you can fix it by editing the /etc/default/locale file manually. Can you please edit that file so it only contains the line

LANG=de_DE.UTF-8

Then relogin and run the locale command. If the Chinese entries are now gone, please let us know if it made a difference as regards the mixed fonts.

I experimented a bit more yesterday. I didn't find a option for the sub-locales in KDE, but installed language-selector-gnome, where it is possible to change.

To be on the safe side, I just also directly edited /etc/default/locale as suggested by you.

After reboot:

"cat /etc/default/locale
LANG="de_DE.UTF-8""

"locale
LANG=de_DE.UTF-8
LANGUAGE=de:en:zh:en
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC=de_DE.UTF-8
LC_TIME=de_DE.UTF-8
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY=de_DE.UTF-8
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER=de_DE.UTF-8
LC_NAME=de_DE.UTF-8
LC_ADDRESS=de_DE.UTF-8
LC_TELEPHONE=de_DE.UTF-8
LC_MEASUREMENT=de_DE.UTF-8
LC_IDENTIFICATION=de_DE.UTF-8
LC_ALL="

The problem still exists.

Additionally I tried some more programmes. My impression now is that the problem only exists with KDE, e.g. QT programmes and things like all window titles. In Gnome applications (I tried Geany and Pidgin) Chinese text is displayed normally, in Thunderbird too (not sure what toolkit thunderbird uses).

I deinstalled "fonts-arphic-ukai" which seemed to be the "italic" font used. Now, text is still displayed using two different fonts, but the "italic" font changed. (I am using "italic" and "normal" to describe how the fonts look, but it is not equal to a text's attribute in Latin script. The difference between the two fonts could perhaps be compared to serif and sans serif in Latin script)

In the font settings, everything is set to Ubuntu, except for the monospace font which is set to Monospace.

Also, all characters in the attached text are very common, so it shouldn't be the case that some characters are displayed by a fall back font because they don't exist in the first choice.

Thanks for the additional information. It indicates that qt based applications don't play well with fontconfig.

However:

On 2014-06-27 07:50, Lukas Bunsen wrote:
> In the font settings, everything is set to Ubuntu, except for the
> monospace font which is set to Monospace.

Which font settings are those? Is it possibly something KDE specific that overrides fontconfig for KDE apps? If so, is there possibly a way to change them to 'neutral' values?

In 14.04 the default font for rendering Chinese was changed from WenQuanYi Zen Hei to Droid Sans. This might have something to do with it.

It's still not clear to me which packages are affected, but I added a few candidates so this bug report gets more attention.

I only have the problem since updating to 14.04, so it really might be connected to the font change.

It is the standard font setting dialog in the KDE control center. I could try to change the font with a different tool to see if it would change anything. Do you know what package I would have to install to get the Gnome font settings? Or is there a more low level way to change the default font? KDE also has a dialog to choose a specific font for Gnome applications, but there the font is set to Ubuntu, too.

I played a bit with the KDE font settings and tried different fonts. With some of them, Chinese text is displayed normally:
*Serif
*AR PL UKai CN
*WenQuanYi Micro Hei
The with the last two somehow make sense, as they are genue Chinese fonts. I am not sure about "Serif"?

I also tried various standard fonts, they all displayed the problem:

*Sans Serif
*Arial
*Courier
*Droid Sans Mono
*Droid Serif

Also, I noticed that I have three different Droid Mono fonts in my system font settings. Is that supposed to be this way or might it be connected to the problem? The listed Droid fonts are:

*Droid Sans [monotype]
*Droid Sans [unknown]
*Droid Sans Mono
*Droid Serif

In KDE's font manager however, Droid Sans is only displayed once. I attached screenshots for both dialogs.

Anything else I could/should try?

On 2014-06-30 08:59, Lukas Bunsen wrote:
> I only have the problem since updating to 14.04, so it really might
> be connected to the font change.

When you say that you updated, I take it that you don't have a clean 14.04 install, but upgraded from 13.10. And if that's the case, the package carrying the previous default font for Chinese may still be installed.

So, can you please check if the fonts-wqy-zenhei package is installed. If it is, can you uninstall it and let us know if it makes a difference.

I ask this because I now fear that the fontconfig config files for fonts-wqy-zenhei and fonts-droid might conflict. This is merely a guess so far, but possibly it explains the strange mix you describe.

> It is the standard font setting dialog in the KDE control center.

I see. There is no similar GUI in standard Ubuntu, and I don't know exactly what it does, but for now I suggest that you keep testing with "Ubuntu" selected. After all, you have a German locale.

OTOH, the intention is that Chinese contents should be rendered in a decent way by default even if the locale is not Chinese. This was dealt with when fixing bug 1227034.

Sorry for the late response, I must have managed to corrupt a config file, KDE currently crashes when I try to login using my main user, circling back to the login manager. Still trying to find out why, but that's another battlefield.

As side effect, I created a new user, which also shows the mixed-font problem, so it shouldn't be connected to user level files.

Yes, I upgraded, originally my system was installed using 13.04, and I have been upgrading twice since then.

fonts-wqy-zenhei was not installed. Instead, fonts-wqy-microhei was installed, which I guessed is the new replacement? Just to try, I deinstalled fonts-wqy-microhei, but the problem persists.

Thanks for the additional info. Then we can leave that theory behind.

It was not quite correct when I said that WenQuanYi Zen Hei was the previous default font for Chinese. It was in 13.10 in case of a Chinese locale, but that does not apply to your situation.

The previous package for rendering Chinese, that was previously installed for all users, was fonts-wqy-microhei, and the new replacement is fonts-droid.

http://bazaar.launchpad.net/~kubuntu-dev/ubuntu-seeds/kubuntu.trusty/revision/1265

So let's try to verify that the issue is related to this change: Can you please

1) reinstall fonts-wqy-microhei and

2) uninstall fonts-droid

and let us know if the mixed font problem disappears that way.

The problem does not exist if fonts-wqy-microhei is installed, and fonts-wqy-zenhei and fonts-droid are both uninstalled.

I could also try a 14.04 LiveCD today to check if a "fresh system" has the same problem. Also, would it make sense to reinstall fonts-droid and try if I have the problem with a Chinese language locale?

Thanks, Lukas. I changed the title of this bug report to reflect our findings so far.

Yes, it would be valuable if you could confirm the issue on a fresh install.

On 2014-07-08 04:14, Lukas Bunsen wrote:
> Also, would it make sense to reinstall fonts-droid and try if I have
> the problem with a Chinese language locale?

Yes, that would indeed make sense too. However, there is a pending fix (bug 1335482) which might affect such a test, so to accomplish it reliably I ask you to install the current Utopic version of the package language-selector-common.

http://launchpadlibrarian.net/179082601/language-selector-common_0.132_all.deb

Please download that file and install it from a terminal window with

sudo dpkg -i language-selector-common_0.132_all.deb

I tried both.
If I change the locale in my existing installation to simplified Chinese, all Chinese text is displayed correctly. (fonts-droid installed and language-selector-common updated to the Utopic version.)

I also installed Kubuntu 14.04 with German locale in a virtual machine and applied all updates. In the fresh installation the problem is slightly different: In Chinese text, some characters are not displayed at all, instead the standard squares that are used when a font doesn't include the required characters are displayed.

On 2014-07-14 09:25, Lukas Bunsen wrote:
> If I change the locale in my existing installation to simplified
> Chinese, all Chinese text is displayed correctly. (fonts-droid
> installed and language-selector-common updated to the Utopic
> version.)

Ok, with that we know that the fontconfig settings make a difference also for KDE/QT apps.

> I also installed Kubuntu 14.04 with German locale in a virtual
> machine and applied all updates. In the fresh installation the
> problem is slightly different: In Chinese text, some characters are
> not displayed at all, instead the standard squares that are used when
> a font doesn't include the required characters are displayed.

Hmm.. I have a test installation of Kubuntu which I upgraded to try to reproduce the problem. I couldn't.

Sure, if I uninstalled fonts-droid, those squares showed up when viewing a piece of Chinese text with Kate, but with fonts-droid the text seemed to be rendered normally. (Please see attachment.) So I can't help wondering if fonts-droid was installed when you tested on your VM.

Another thing struck me: The list of Droid fonts at
https://launchpadlibrarian.net/178980889/select%20system%20font.png
is much shorter than the list I saw in the font select window (see attachment). Especially I miss Droid Sans Fallback, which is essential for CJK languages.

Can you please ensure that the fonts-droid package is completely installed on your existing installation by running

sudo apt-get install --reinstall fonts-droid

Can you also post the output of the command

fc-match -a 'sans-serif'

Hi Lukas, any chance that you can address the questions in comment #14?

Also, language-selector in Trusty has been updated now, so if you haven't already done so, you should downgrade language-selector-common to version 0.129.2 so you are not shut out from possible future updates.

Hi Gunnar,

sorry for the long delay. A recent update caused a problems on my Kubuntu installation. I wasn't careful enough in my attempts to repair and ended up with a broken system.

On the positive side, this means that I can now check if the bug exists in a fresh installation of Kubuntu 14.04.1...
It does.
The local was mixed again, my hypothesis is that the Kubuntu installer sets some locale values depending on the timezone you choose? I manually reset all values to "de_DE.UTF-8" in /etc/default/locale.

I attached three screenshots. The first two are the example text in Kate and the available fonts in a fresh installation with all updates installed. The text shows _three_ different "things": First, most characters are displayed in the normal font. Some few characters, for example the first one, are displayed in a different, slighty italic font. Some characters are completely missing, and squares displayed in their place.

The third screenshot was taken after installing Chinese language support. I first added Chinese in the KDE language settings, which caused some packages being installed, for example ibus. But afterwards, the text was still displayed the same way. I then installed language-selector-gnome. When I s tarted it, gnome-language-selector was prompting that some language support packages where missing and offered to install it, among others, fonts-arphic-uming and fonts-arphic-ukai where installed. I addionally executed "apt-get install --reinstall fonts-droid".
You see the result after a reboot on the third screenshot. The text looks like on my old system, the characters only displayed as squares before now use a italic font to. The italic font changed, I assume it is arphic-ukai. As with the old installation, the font settings dialog only shows the same few fonts, and not all of the fonts you have.

I also attached the output of "fc-match -a 'sans-serif'"

This issues also happens on Ubuntu Touch, (if removed the default fonts-arphic-ukai)

Thanks for your latest info, Lukas!

On 2014-07-28 07:59, Lukas Bunsen wrote:
> The local was mixed again, my hypothesis is that the Kubuntu
> installer sets some locale values depending on the timezone you
> choose?

Yes, that's what happens. The installer guesses the locale categories which control various regional format aspects (as opposed to the display language) such as date/time and number formats.

> As with the old installation, the font settings dialog only shows the
> same few fonts, and not all of the fonts you have.

That detail is odd; can't tell if it has anything to do with the actual problem.

In the related bug #1346766 Cheng-Chia Tseng wrote in comment #7:

"Qt does not support fontconfig configuration well, the workaround is
to specify CJK font as the first candidate font instead of western font."

That statement fits well with the fact that it worked for you when you changed the display language to Chinese. It would be valuable to know if we can make it work by doing something similar even if the display language is German.

As an experiment I wrote the attached fontconfig config file. Can you please store it as

~/.config/fontconfig/conf.d/65-droid-sans-first.conf

It ought to result in a different output from the command

fc-match -a 'sans-serif'

with a bunch of Droid Sans fonts at the top of the list. The question is if it changes the rendering in qt apps of Chinese content.

I added the fontconfig file (also had to create the directory conf.d), but the bug still exists after a reboot.

However, the default font apparently changed. (At least in the KDE terminal/Konsole: As when the locale was Chinese, the same annoying phenomena exists: a huge whitespace is in front of the cursor, and even moves through the text when I use the arrow keys to move horizontally in the command entered)

Might it be possible that the problem is more connected to the "missing" Droid fonts? I think the "normal" characters stem from the Droid font. If nothing changes even if Droid is the default font, that would mean that on my installation Droid does not have enough glyphs to cover all characters, and instead the glyphs of Ukai, which has a way lower priority, are used?

Also, in the output of fc-match, I see several Droid fonts for non-latin languages, like Hebrew or Japanese, but not for Chinese. I am not sure if it might be possible that the Droid Japanese font is used to display Chinese characters, and non-covered characters are displayed with Ukai?

Hi Lukas, and thanks again for your efforts. The name of the Chinese font file is DroidSansFallbackFull.ttf, and that file is included somewhere in the fc-match output, right? So the font is there, it's just that the system doesn't see it...

Since we have several concurrent font-droid related issues, I filed bug #1351092 in an attempt to sort things out. It would be great if you as a test could uninstall fonts-droid and install fonts-droid-cjk and language-selector-common from the PPA mentioned there and let us know the result. I took the liberty of subscribing you to the other bug.

This bug was fixed in the package fonts-android - 1:4.3-3ubuntu2

---------------
fonts-android (1:4.3-3ubuntu2) utopic; urgency=medium

  [ Gunnar Hjalmarsson ]
  * debian/local/65-droid-sans-fonts.conf:
    - Dropped. We want to cherry pick Droid Sans Fallback for rendering
      Chinese content, and 65-droid-sans-fonts.conf has defeated that
      purpose and caused confusion (LP: #1334495, LP: #1351092).

  [ Iain Lane ]
  * Add fonts-droid.maintscript to clean up the now obsolete conffile on
    upgrade.
 -- Gunnar Hjalmarsson <email address hidden> Mon, 15 Sep 2014 14:56:33 +0100

This bug was fixed in the package language-selector - 0.133

---------------
language-selector (0.133) utopic; urgency=low

  * fontconfig/69-language-selector-zh-??.conf:
    Reverse changes in version 0.132 (LP: #1334495, LP: #1351092).
 -- Gunnar Hjalmarsson <email address hidden> Mon, 15 Sep 2014 16:23:00 +0200

Hello Lukas, or anyone else affected,

Accepted language-selector into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/language-selector/0.129.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Lukas, or anyone else affected,

Accepted fonts-android into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/fonts-android/1:4.3-3ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

As regards the verification, please see https://bugs.launchpad.net/ubuntu/+source/fonts-android/+bug/1351092/comments/18

This bug was fixed in the package language-selector - 0.129.3

---------------
language-selector (0.129.3) trusty-proposed; urgency=low

  * fontconfig/69-language-selector-zh-??.conf:
    Reverse changes in version 0.129.2 (LP: #1334495, LP: #1351092).
 -- Gunnar Hjalmarsson <email address hidden> Mon, 15 Sep 2014 16:34:00 +0200

The verification of the Stable Release Update for language-selector has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package fonts-android - 1:4.3-3ubuntu1.1

---------------
fonts-android (1:4.3-3ubuntu1.1) trusty; urgency=medium

  [ Gunnar Hjalmarsson ]
  * debian/local/65-droid-sans-fonts.conf:
    - Dropped. We want to cherry pick Droid Sans Fallback for rendering
      Chinese content, and 65-droid-sans-fonts.conf has defeated that
      purpose and caused confusion (LP: #1334495, LP: #1351092).

  [ Iain Lane ]
  * Add fonts-droid.maintscript to clean up the now obsolete conffile on
    upgrade.
 -- Gunnar Hjalmarsson <email address hidden> Mon, 15 Sep 2014 14:56:33 +0100