Cannot import openvdb 6.2 on Focal due to jemalloc

Status changed to 'Confirmed' because the bug affects multiple users.

AFAICT this issue was introduced in OpenVDB 6.2.0 when it started linking in jemalloc all the time, including when being built as a shared library.

My understanding is that shared libraries should absolutely not be doing this, as safely linking allocators such as jemalloc into a shared library is very tricky to do correctly, and can also make it impossible to use the shared library dynamically via `dlopen`.

In my case I ran into this via OpenImageIO which had a dependency on libopenvdb.so.6.2, and I was getting the error "/usr/lib/x86_64-linux-gnu/libjemalloc.so.2: cannot allocate memory in static TLS block" when doing a `dlopen`.

OpenVDB addresses this as of v7.1.0, though this hasn't actually been released. The changelog [1] reads:

    - jemalloc/tbbmalloc are no longer linked into library artifacts of the
      OpenVDB CMake build. The CONCURRENT_MALLOC CMake option now only applies
      to the executables.

My understanding is that this is fixed in the proposed libopenvdb package for groovy which uses OpenVDB v7, but getting that working as a dependency of OpenImageIO on a focal system might be tricky. For my situation I'm intending to build OpenImageIO from source, link it statically, and exclude OpenVDB at build time because I don't actually need it anyway.

Hopefully there's a backport of the fix removing jemalloc to the OpenVDB 6.x series that can become part of an update to focal, or perhaps focal could use the proposed package for groovy.

[1] https://github.com/AcademySoftwareFoundation/openvdb/blob/master/CHANGES

openvdb (7.0.0-1) experimental; urgency=medium
...
  * d/control: Remove dependency to jemalloc. See #951704 for details
...

https://salsa.debian.org/multimedia-team/openvdb/-/commit/ac393d95aa19d29c23a97dca1ace23061ebe5c17

Thank you for taking the time to report this bug and helping to make Ubuntu better. However, I am closing it because the bug has been fixed in the latest development version of Ubuntu.

If you need a fix for the bug in previous versions of Ubuntu, please perform as much as possible of the SRU Procedure [1] to bring the need to a developer's attention.

[1]: https://wiki.ubuntu.com/StableReleaseUpdates#Procedure

Thanks.

In my opinion this could warrant fixing in 20.04 LTS because:

- it's a regression from 18.04 LTS, so certain applications and usage patterns
  will break as users try to upgrade (which is what happened to me).
- it affects several packages, specifically: libopenimageio, libopenvdb and
  python3-openvdb. libopenimageio is widely used, and python3-openvdb is
  currently not able to be imported because of this issue. Any other libraries
  that depend on openvdb will also be affected.
- there's no need to upgrade to openvdb 7.x, and no code changes are required
  to the current 6.2.1 release either. It needs to be compiled with
  -DCONCURRENT_MALLOC=Tbballoc added to the CMake command line, and there's a
  patch available for this small change already.
- this wouldn't be a high-risk update.

I ended up working around my problem by setting the following environment
variable that causes jemalloc to be loaded at process start, so it doesn't try
to load itself later on in response to a dlopen() call.

    LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2

I've never worked directly with Ubuntu packages, but will take a look at the
SRU Procedure and see if I can put something together.

[Impact]

 * This issue causes apps to be unable to dlopen libopenvdb6.2 or any shared libraries that depend directly or indirectly on libopenvdb6.2. Notably this includes libopenimageio2.1 and anything that depends on it. The python3-openvdb package also doesn't work in focal at present because of this issue.

 * This issue is a good candidate for backporting to focal because (1) the fix is a very modest change to the build of the libopenvdb6.2 package; (2) this is a regression from the previous stable release of Ubuntu (bionic); (3) several packages are affected by it.

 * The change already in groovy fixes this issue by not building libopenvdb with jemalloc.

[Test Case]

 * `apt install python3-openvdb && python3 -c "import pyopenvdb"` should succeed, but because of this issue it currently fails with the error "cannot allocate memory in static TLS block".

 * Also, the following C program should print a non-nil value to stdout:

```c
#include <dlfcn.h>
#include <stdio.h>

void main() {
  printf("%p\n", dlopen("/usr/lib/x86_64-linux-gnu/libopenvdb.so.6.2", RTLD_NOW));
}
```

[Regression Potential]

 * Regressions are unlikely because the only change is to the choice of allocator for libopenvdb6.2 when configuring the build with CMake.

 * There is no need for a patch to libopenvdb.

 * The patch applied in groovy is https://salsa.debian.org/multimedia-team/openvdb/-/commit/ac393d95aa19d29c23a97dca1ace23061ebe5c17

[Other Info]

 * There are workarounds for this issue, but each has its own drawbacks and may not be possible in every situation where this issue could occur.

Please find attached a debdiff for focal

Please note this also affects gridengine-drmaa1.0, where libdrmaa.so is linked vs libjemalloc.so.2,
and attempts to dlopen() libdrmaa fail with 'cannot allocate memory in static TLS block'.

According to https://github.com/jemalloc/jemalloc/issues/1237, this can be fixed by configuring
jemalloc with '--disable-initial-exec-tls'.

Fixed upstream with https://github.com/AcademySoftwareFoundation/openvdb/commit/0fa1c68314dfac1822711f5775976ce0735e041e

If there's anything I can do to help move this patch towards inclusion in focal please just let me know. My original debdiff is here: https://bugs.launchpad.net/ubuntu/+source/openvdb/+bug/1882998/comments/9

I've sponsored the debdiff after changing the package version and running update-maintainer.

 $ dput openvdb_6.2.1-8ubuntu1.1_source.changes
Trying to upload package to ubuntu
Checking signature on .changes
gpg: /tmp/pkgs/focal/openvdb_6.2.1-8ubuntu1.1_source.changes: Valid signature from 1E918B66765B3E31
Checking signature on .dsc
gpg: /tmp/pkgs/focal/openvdb_6.2.1-8ubuntu1.1.dsc: Valid signature from 1E918B66765B3E31
Uploading to ubuntu (via ftp to upload.ubuntu.com):
  Uploading openvdb_6.2.1-8ubuntu1.1.dsc: done.
  Uploading openvdb_6.2.1-8ubuntu1.1.debian.tar.xz: done.
  Uploading openvdb_6.2.1-8ubuntu1.1_source.buildinfo: done.
  Uploading openvdb_6.2.1-8ubuntu1.1_source.changes: done.
Successfully uploaded packages.

AFAIK the changelog entry is not fully correct, instead of
+ * Remove dependency on jemalloc so dlopen() works. Closes #1882998.
it should be
+ * Remove dependency on jemalloc so dlopen() works. LP #1882998.

That's correct you need to use LP: to close a Launchpad bug. I'm fixing this and will reupload it.

Hello Guillaume, or anyone else affected,

Accepted openvdb into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openvdb/6.2.1-8ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Brian,
That's great news. The version openvdb 6.2.1-8ubuntu1.1 from proposed fixes the problem for me!

The verification of the Stable Release Update for openvdb has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package openvdb - 6.2.1-8ubuntu1.1

---------------
openvdb (6.2.1-8ubuntu1.1) focal; urgency=medium

  * Remove dependency on jemalloc so dlopen() works. (LP: #1882998)

 -- Richard Viney <email address hidden> Thu, 10 Sep 2020 01:12:04 +0000