snapd generates incomplete fontconfig caches, result in emoji rendering issue in Chromium (and non-snap browsers too)

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

Status changed to 'Confirmed' because the bug affects multiple users.

(cross-posting my comment on the forum thread)

I’m seeing the same in a clean eoan VM (the chromium snap version doesn’t matter, all channels exhibit the problem). Interestingly my work laptop with the chromium snap from the stable channel doesn’t exhibit the problem, emojis are rendered correctly.
In both cases I’m seeing several instances of this error message:

[ERROR:child_process_sandbox_support_impl_linux.cc(79)] FontService unique font name matching request did not receive a response.

but it’s probably unrelated, given that emojis are rendered correctly and coloured on my laptop.

I was affected as well, solved by removing and re-installing the `fonts-noto-color-emoji` pacakge

The workaround worked for me too. I used the simplified version

    sudo apt install --reinstall fonts-noto-color-emoji

Unfortunately the workaround is temporary. I rebooted and the emoji are gone again.

I tired to figure out what exactly happens when I reinstall fonts-noto-color-emoji. According to dpkg's output, fontconfig's triggers run. fontconfig's postinst runs

  # Force regeneration of all fontconfig cache files.
  mkdir -p /var/cache/fontconfig
  fc-cache -s -v 1>/var/log/fontconfig.log 2>&1 || printf "fc-cache failed.\nSee /var/log/fontconfig.log for more information.\n"

when triggered. I looked at /var/log/fontconfig.log, and I see a lot of "skipped" messages for various directories (because "existing cache is valid" or "looped directory detected"), and then I see

  /var/cache/fontconfig: cleaning cache directory
  /var/cache/fontconfig: invalid cache file: 0bd3dc0958fa2205aaaa8ebb13e2872b-le64.cache-7
  fc-cache: succeeded

Now I'm wondering how a fontconfig cache file becomes invalid, and how do other applications cope.

Yesterday I saw a notification about chromium being auto-refreshed, restarted it, and today I noticed that color emoji are gone again.

I suspect the two facts are related.

Another experiment: while emoji are broken, I can do sudo fc-cache -s -v, and it doesn't find any invalid cache files, and when I restart chromium, I don't get color emoji.

But then I do sudo apt install --reinstall fonts-noto-color-emoji, and I see a /var/log/fontconfig.log with an up-to-date timestamp, and it says, again,

    /var/cache/fontconfig: invalid cache file: 0bd3dc0958fa2205aaaa8ebb13e2872b-le64.cache-7

and I restart chromium again, and now I have working color emoji. (Until the next snap refresh at least).

I don't understand anything.

Thanks for your findings Marius. I can reliably reproduce the problem in a clean and up-to-date focal VM with the following steps:

 - snap install chromium
 - snap run chromium # color emojis are rendered correctly
 - snap refresh chromium --beta
 - snap run chromium # color emojis are gone
 - sudo apt reinstall fonts-noto-color-emoji
 - snap run chromium # color emojis are back

FWIW rebooting also makes the problem reappear. snap info shows that I last refreshed chromium 13 days ago, but I rebooted today to install a kernel update, and color emoji disappeared.

This time I successfully resurrected color emoji by manually removing the file that fc-cache used to complain about. Or, rather, I moved it aside, for study:

sudo mv /var/cache/fontconfig/0bd3dc0958fa2205aaaa8ebb13e2872b-le64.cache-7{,.bork}

Then I restarted Chromium and emoji are now colorful again.

FWIW 0bd3dc0958fa2205aaaa8ebb13e2872b is MD5 of "/usr/share/fonts/truetype/noto". fontconfig keeps one cache file per font directory.

The timestamp of the broken fontconfig cache file is May 15, 10:37. I wonder what my computer was doing at the time.

'last reboot' indicates that I rebooted on May 12, and then today, May 20. I'm pretty sure I launched Chromium right after booting and never quit it until the next reboot.

journalctl shows me what my computer was doing on May 15 at 10:36:

geg. 15 10:36:59 blynas lxd.daemon[365715]: => Stop reason is: snap refresh
geg. 15 10:36:59 blynas lxd.daemon[365715]: => Stopping LXD
geg. 15 10:36:59 blynas lxd.daemon[1225]: => LXD exited cleanly

so it is snap refresh, not reboot, that creates the broken fontconfig cache file! It's just that when I snap refresh something that is not chromium, I don't restart chromium!

Curiously, fc-cat /var/cache/fontconfig/0bd3dc0958fa2205aaaa8ebb13e2872b-le64.cache-7.bork does not complain about the cache file being invalid. It shows information about a bunch of Noto fonts BUT! not the ColorEmoji one!

diff -u <(fc-cat /var/cache/fontconfig/*.bork|cut -d'"' -f2|sort) <(ls /usr/share/fonts/truetype/noto/|sort)

shows that NotoColorEmoji.ttf is the only font missing from the font cache file.

Now it's time for baseless speculation: could it be that snap refresh somehow runs fc-cache inside a container where /var/cache/fontconfig is bind-mounted from the host OS, but /usr/share/fonts/truetype/ comes from some base snap image?

Maybe not.

ls -ld /snap/*/current/usr/share/fonts/truetype/noto shows that only one snap has that directory (gnome-3-34-1804), and instead of containing all the fonts but NotoColorEmoji.ttf it's the exact opposite: it contains one font only, and that is NotoColorEmoji.ttf.

We could probably remove "[snap]" from the bug title. I experience the same bug in Chrome, which is a deb.

That's interesting information Daniel. Have you observed a pattern in how/when the problem occurs? Does reinstalling fonts-noto-color-emoji "fix" the problem (like in comment #25)?

It's completely random. Works most days but fails once or twice a week. Even though I open a fresh Chrome every day.

And yes reinstalling fonts-noto-color-emoji fixes the problem every time.

I ended up adding

if [ -n "$PS1" ]; then
    # only for interactive sessions please
    if [ -f /var/cache/fontconfig/0bd3dc0958fa2205aaaa8ebb13e2872b-le64.cache-7 ]; then
       echo "color emoji will be broken in Chromium again (LP: #1858636) unless you"
       echo "sudo rm /var/cache/fontconfig/0bd3dc0958fa2205aaaa8ebb13e2872b-le64.cache-7"
    fi
fi

to my ~/.bashrc to warn me when this problem is coming back. So far the appearance of this broken font cache file correlates 100% with some snap (any snap! refreshing _lxd_ which has nothing to do with GUI apps or fonts causes this!) getting refreshed, according to the timestamp reported by snap changes.

Yes I was thinking this sounds like a font cache problem. Perhaps unique to Ubuntu? I couldn't find anything like it reported against Chromium upstream.

Marius, that last comment is a very interesting data point. I'm adding a snapd task to the bug.

Confirmed by doing those steps on a focal system

($ sudo apt install fonts-noto-color-emoji
if you don't have it installed)

$ fc-cat /var/cache/fontconfig/* | grep NotoColorE
there is a result listed

"NotoColorEmoji.ttf" 0 "Noto Color Emoji:familylang=en:style=Regular:stylelang=en:fullname=Noto Color ...

$ sudo rm -rf /var/cache/fontconfig/
$ sudo snap refresh --edge core
(there is probably a smarter way to trigger a fontconfig cache refresh from snapd but that did it)

$ fc-cat /var/cache/fontconfig/* | grep NotoColorE
-> not NotoColorEmoji.ttf result anymore

I can confirm I see the bug with seb128's reproducer. First I will propose a spread test which demonstrates the issue so we don't see regressions on this front, but also it's very curious to me that this specific font:

fonts-noto-color-emoji

does not show up in the cache we generate but this font:

fonts-kiloji

do show up in the cache we generate. The spread test we have for this functionality was using the latter so we didn't see this bug til now.

Also just to clarify, we will regenerate fontconfig cache's on any snap refresh/install during the link-snap task.

Unfortunately we do not have a clear solution to this yet, so I have unassigned myself from the snapd task and we need more help before we can continue on this. Although, we have a few options as I detail here: https://github.com/snapcore/snapd/pull/8856#issuecomment-644784642

I'm not sure what the best solution is, but definitely this will need some guidance from the desktop team or others more knowledgable about fonts about how to proceed here. Some questions I have include:

* Can someone explain the difference between fonts-noto-color-emoji and fonts-kiloji to try and understand why one shows up and the other doesn't with our current fontcache mechanism in snapd?
* Can someone explain exactly what things the `dpkg-reconfigure fontcache` command calls to build the cache? is it sufficient to just call fc-cache from the fontconfig package or do we need to be doing something else too?
* Would it be possible for the fontcache to be run from inside confinement? I.e. what set of directories from the host would the fontcache binary from the fontconfig package need to see in order to build a fontcache?
* If we had to ship fontconfig in the base snaps, is it possible for us to just ship the fontconfig binary like we do today in the snapd snap, or do we really need the full package in order to have full compatibility with all fonts in Ubuntu?

Hey Ian, thanks for the updating the report. I don't have the answers to all your question and I'm unsure we have someone in desktop who is expert with fontconfig or fonts to reply to e.g the first one

I can reply to at least one though!

> * Can someone explain exactly what things the `dpkg-reconfigure fontcache` command calls to build the cache? is it sufficient to just call fc-cache from the fontconfig package or do we need to be doing something else too?

$ cat /var/lib/dpkg/info/fontconfig.postinst
...
  # Force regeneration of all fontconfig cache files.
  mkdir -p /var/cache/fontconfig
  printf "Regenerating fonts cache... "
  fc-cache -s -f -v 1>/var/log/fontconfig.log 2>&1 || (printf "failed.\nSee /var/log/fontconfig.log for more information.\n"; exit 1)
  printf "done.\n"

that's what the fontconfig package is doing

(closing the chromium part since it's not an issue with that component)

I've just been reading through the code used to build the bin/fc-cache-v6 and bin/fc-cache-v7 binaries in the core snap:

https://github.com/snapcore/fc-cache-static-builder

I think I understand what the problem is: while it is rebuilding copies of xenial's and bionic's fontconfig tools that statically link to libfontconfig, it's using the xenial version of libfreetype2 for both.

I suspect that this old freetype version does not understand all the features of the font (e.g. the scalable colour emoji), so the generated cache also misses them.

For reference, this PR might fix it. https://github.com/snapcore/fc-cache-static-builder/pull/2/files

@mvo built a snap from the PR, wiping the cache and installing it gave me a working cache listing the emoji font so it seems to resolve the issue

The mentioned fix has been released for some time in stable snapd now, are folks still seeing invalid caches created by snapd when any snap refreshes?

Feels fixed to me. If the bug was still present I would have seen it every week, but I haven't for months.

Feels fixed to me too!