[SRU] gstreamer fails with "cannot allocate memory in static TLS block" error on aarch64

2.32 is not a minor release compared with 2.31 =))))

it is the major glibc release.

Status changed to 'Confirmed' because the bug affects multiple users.

There are test packages with the WIP fix for Focal:
https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4017/+packages

I have given a try to the test PPA and I can confirm it fixes the bug.

For me Totem works now, but not totem-video-thumbnailer.

Alfonso, Balint, how totem-video-thumbnailer works for you now?

On Mon, Mar 15, 2021 at 11:42 AM RussianNeuroMancer <
<email address hidden>> wrote:

> For me Totem works now, but not totem-video-thumbnailer.
>
> Alfonso, Balint, how totem-video-thumbnailer works for you now?
>

It works for me, but I have just done a quick try. gstlaunch was what was
failing before and now is fine with the focal packages from the PPA.

>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1914044
>
> Title:
> [SRU] gstreamer fails with "cannot allocate memory in static TLS
> block" error on aarch64
>
> Status in glibc package in Ubuntu:
> Fix Released
> Status in glibc source package in Focal:
> Confirmed
> Status in glibc package in Debian:
> Fix Released
>
> Bug description:
> [Impact]
>
> gstreamer cannot load some plugins on 20.04, aarch64 due to TLS memory
> exhaustion, so these plugins cannot be used. See LP#1846837 and
> LP#1914030.
>
> This is solved apparently on glibc 2.32:
> https://sourceware.org/bugzilla/show_bug.cgi?id=25051
> by patch
>
> https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=ffb17e7ba3a5ba9632cee97330b325072fbe41dd
>
> [Test Case]
>
> Running totem-video-thumbnailer shows these errors:
>
> (totem-video-thumbnailer:12136): GStreamer-WARNING **: 00:43:52.740:
> Failed to load plugin '/usr/lib/aarch64-linux-
> gnu/gstreamer-1.0/libgstlibav.so': /lib/aarch64-linux-
> gnu/libgomp.so.1: cannot allocate memory in static TLS block
>
> [Where problems could occur]
>
> This is delicate code in glibc with quite regression potential. Said
> this, the code is in the next minor release (2.32, while focal has
> 2.31), which has been running groovy for a while.
>
> Also, apparently there is a 2.31 patch for groovy, before it moved to
> 2.32: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=964141
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1914044/+subscriptions
>

Tested on 20.10 (Lenovo Yoga C630 WOS) - totem-video-thumbnailer works there, but not on 20.04 (NanoPC-T4) updated from PPA.

@russianneuromancer How did the test on NanoPC fail for you? (I've updated the test instructions to make them clear.)

Hello Alfonso, or anyone else affected,

Accepted glibc into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/glibc/2.31-0ubuntu9.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Verified in focal:

ubuntu@host:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.2 LTS
Release: 20.04
Codename: focal

ubuntu@host:~$ apt policy libc6
libc6:
  Installed: 2.31-0ubuntu9.2
  Candidate: 2.31-0ubuntu9.2
  Version table:
 *** 2.31-0ubuntu9.2 500
        500 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 Packages
        100 /var/lib/dpkg/status
     2.31-0ubuntu9 500
        500 http://ports.ubuntu.com/ubuntu-ports focal/main arm64 Packages

ubuntu@host:~$ rm .cache/gstreamer-1.0/registry.aarch64.bin
ubuntu@host:~$ gst-inspect-1.0 >/dev/null
(gst-plugin-scanner:5268): GStreamer-WARNING **: 10:45:38.430: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstchromaprint.so': /lib/aarch64-linux-gnu/libgomp.so.1: cannot allocate memory in static TLS block
(gst-plugin-scanner:5268): GStreamer-WARNING **: 10:45:38.529: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstlibav.so': /lib/aarch64-linux-gnu/libgomp.so.1: cannot allocate memory in static TLS block

<Update to focal-proposed>

ubuntu@host:~$ apt policy libc6
libc6:
  Installed: 2.31-0ubuntu9.3
  Candidate: 2.31-0ubuntu9.3
  Version table:
 *** 2.31-0ubuntu9.3 500
        500 http://ports.ubuntu.com/ubuntu-ports focal-proposed/main arm64 Packages
        100 /var/lib/dpkg/status
     2.31-0ubuntu9.2 500
        500 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 Packages
     2.31-0ubuntu9 500
        500 http://ports.ubuntu.com/ubuntu-ports focal/main arm64 Packages

ubuntu@host:~$ rm .cache/gstreamer-1.0/registry.aarch64.bin
ubuntu@host:~$ gst-inspect-1.0 >/dev/null
(no errors now)

> How did the test on NanoPC fail for you?

It seems like fail on NanoPC-T4 was caused by different issue, I still investigating it.

All autopkgtests for the newly accepted glibc (2.31-0ubuntu9.3) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

libgdata/0.17.12-1 (armhf)
ruby-grib/0.4.0-4build1 (s390x)
libclang-perl/0.09-4build8 (arm64)
dealer/20190529+ds-1 (ppc64el)
t-coffee/12.00.7fb08c2-4build1 (s390x)
xpra/3.0.6+dfsg1-1build1 (amd64)
makedumpfile/1:1.6.7-1ubuntu2.1 (amd64)
libepubgen/0.1.1-1ubuntu2 (arm64)
apt/2.0.5 (armhf)
pg-qualstats/1.0.9-1 (s390x)
booth/1.0-174-gce9f821-1 (amd64)
gnustep-base/1.26.0-7 (s390x)
fclib/3.0.0+dfsg-2build1 (arm64)
asterisk/1:16.2.1~dfsg-2ubuntu1 (amd64)
golang-github-grpc-ecosystem-grpc-gateway/1.6.4-2 (ppc64el, s390x)
puma/3.12.4-1ubuntu2 (s390x)
cmark-gfm/0.29.0.gfm.0-4 (amd64)
lua-nvim/0.2.1-1-1 (armhf)
saods9/8.1+repack-1 (amd64)
ruby-gpgme/2.0.19-3build1 (s390x)
ruby-bootsnap/1.4.6-1 (arm64)
staden-io-lib/1.14.11-6 (ppc64el)
s3ql/3.3.2+dfsg-1ubuntu1 (armhf)
mercurial/5.3.1-1ubuntu1 (amd64)
libterm-readline-gnu-perl/1.36-2build1 (ppc64el)
casper/1.445.1 (amd64)
datefudge/1.23ubuntu1 (armhf, ppc64el)
source-extractor/2.25.0+ds-2 (s390x)
smcroute/2.4.2-4 (arm64)
disulfinder/1.2.11-8build1 (arm64)
autofs/5.1.6-2ubuntu0.1 (ppc64el)
debconf-kde/1.0.3-3 (ppc64el)
libfastahack/1.0.0+dfsg-5build1 (ppc64el)
gemma/0.98.1+dfsg-1 (arm64)
missidentify/1.0-10 (ppc64el)
r-cran-systemfonts/0.1.1-1build1 (s390x)
bcalm/2.2.1-2build1 (ppc64el)
ipgrab/0.9.10-4 (ppc64el)
ruby-libxml/3.1.0-2 (s390x)
r-bioc-delayedarray/0.12.2+dfsg-1 (armhf)
netplan.io/0.101-0ubuntu3~20.04.2 (arm64)
6tunnel/1:0.13-1 (armhf)
taptempo/1.4.4-1 (s390x, armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#glibc

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

This bug was fixed in the package glibc - 2.31-0ubuntu9.3

---------------
glibc (2.31-0ubuntu9.3) focal; urgency=medium

  [ Aurelien Jarno ]
  * debian/patches/any/git-surplus-tls-accounting.diff: backport TLS surplus
    accounting from upstream. (Closes: #964141) (LP: #1914044)

  [ Balint Reczey ]
  * Update debian/patches/ubuntu/local-disable-ld_audit.diff
  * Prevent rare deadlock in pthread_cond_signal (LP: #1899800)
  * Cherry-pick upstream patch to fix building with -moutline-atomics
  * Make libc6 provide libc6-lse on arm64.
    Libc6 is now compiled with -moutline-atomics thus the separate binary
    package is dropped. (LP: #1912652)
  * debian/control: Libc6 should Conflict and Replace libc6-lse

 -- Balint Reczey <email address hidden> Mon, 29 Mar 2021 22:11:32 +0200

The verification of the Stable Release Update for glibc has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This introduced a bug in the debian-installer for focal (LP: #1926223)

While trying to update libc6, debian-installer will get the following error

-/bin/sh: error loading shared libraries: __vdso_gettimeofday: invalid mode for dlopen(): invalid argument

This behavior can be seen without using debian-installer

On a system with libc6 version 2.31-0ubuntu9.2 and a folder 9.3 containing the extracted 2.31-0ubuntu9.3 deb file

$ LD_LIBRARY_PATH=./9.3/lib/x86_64-linux-gnu ls
ls: error while loading shared libraries: __vdso_gettimeofday: invalid mode for dlopen(): Invalid argument

I've copied 2.31-0ubuntu9.3 to https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4547 until the new update can be released including this fix.

This fix proved to be too intrusive to be released via focal-updates. It requires an immediate restart of services and a reboot which can't be guaranteed if the fix is included in a later update released into the focal-security pocket.
I'm preparing 2.31-0ubuntu9.4 _without_ this fix but including other fixes which are safe to release to Focal.
In addition to requiring an immediate reboot the fix also breaks snaps bundling glibc without this fix when they are run with an updated core20 snap including this fix.

So a side question here is, why are libraries built with the initial exec tls model on arm64? Building with tlsdesc would have almost the same performance in the usual case aiui.

Another question, how come the issues reported as regressions caused by this did not occur as people upgraded from focal to groovy?

So to answer my first question, most libraries are built as tlsdesc on arm64. But libgomp is not.

Not sure on the second part. I guess a series upgrade usually involves a reboot anyway. Or maybe something was imperfect in the backport attempt.