lttng-modules 2.10.8-1ubuntu2~18.04.1 ADT test failure with linux-kvm 4.15.0-1059.60

Implicit function declaration error due to missing "vmalloc_sync_all()" is expected due to a recent change in the kernel ABI in Bionic:

https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/bionic/commit/?id=a37c73d3e2fb62bf486c0d2099deff8fad325f20

The patch mentioned in comment #2 appears upstream in kernel version 5.6:

commit 763802b53a427ed3cbd419dbba255c414fdd9e7c
Author: Joerg Roedel <email address hidden>
Date: Sat Mar 21 18:22:41 2020 -0700

    x86/mm: split vmalloc_sync_all()

It becomes an obvious problem for lttng in the kvm-kernel only because lttng-modules creates two variants of wrappers for it depending on having CONFIG_KALLSYMS set or not. If it is not set, then the wrapper just blindly calls vmalloc_sync_all() in the hope that it has been exported. And that causes the compile to fail.
However, even in the case of KALLSYMS being set, this will cause issues. Just not at compile time. Because then the code looks up the address of vmalloc_sync_all() at runtime and would then print a warning (on x86) about possibly triggering recursive page faults.

I played around with how this might be fixed. The problem is that this could potentially hit all releases that get the split of vmalloc_sync_all() via stable. Minimally I would suspect Bionic to Focal. So I tried to come up with a modification to the module build which checks the kernel header file directly and adapts accordingly. The attached patch is the result. It is made against the Focal version of lttng-modules but with renaming KBuild.common into Makefile.ABI.workarounds it should (actually tested the code in bionic) work in older series, too.

Ok, the first attempt did not work as cpp does not replace string contents. Hopefully correctly updated patch attached.

Looks good to me, ship it.

I guess in groovy onwards, this might be fixed by just replacing all with mappings in the code. Since there won't be need to support older kernels.

From the upstream discussion there is this proposed approach. I added ranges of Ubuntu kernels without upstream stable version support (right now tested only for bionic). This would be what will be proposed for backporting.

Proposed SRU for 20.04/Focal. Test build at: https://launchpad.net/~smb/+archive/ubuntu/focal

Deferring the fix for Groovy. This is being worked on in parallel and will include additional changes to allow compiling against a 5.7 kernel.

Proposed SRU for 19.10/Eoan. Test build at: https://launchpad.net/~smb/+archive/ubuntu/eoan

Proposed SRU for 18.04/Bionic. Test build at: https://launchpad.net/~smb/+archive/ubuntu/bionic (This is in the end the same as in Eoan due to syncing the packages).

Proposed SRU for 16.04/Xenial. Test build at: https://launchpad.net/~smb/+archive/ubuntu/xenial

Please update debdiffs a bit:
1. where you link the commit please use proper origin tags
  Origin: backport, <todo-URL-to-git>
  # OTHERWISE
  Origin: upstream, <todo-URL-to-git>

2. your version numbers are incorrect for SRUs
+lttng-modules (2.11.2-1ubuntu1) focal; urgency=medium
needs to be 2.11.2-1ubuntu0.1

+lttng-modules (2.10.8-1ubuntu3) eoan; urgency=medium
needs to be 2.10.8-1ubuntu2.1

+lttng-modules (2.10.8-1ubuntu3~18.04.1) bionic; urgency=medium
needs to be 2.10.8-1ubuntu2~18.04.2

xenial is correct

This is an updated debdiff for Focal. Are the changes to the patch headers looking better now?

Made a v3 which adjusts the dep-3 style "Origin:" into a git format-patch style format.

Thanks for fixing up the versions as needed.
While not 100% the header style the server team would have after the discussion I think nothing is left that should block this. Sponsoring into the -unapproved queues.

I also ran update-maintainer for you on Focal.
And finally as discussed I removed the hunk on lttng-modules-dkms.dkms.in

All four sponsored for the SRU team to take a look at it next

Hello Ian, or anyone else affected,

Accepted lttng-modules into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lttng-modules/2.11.2-1ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Ian, or anyone else affected,

Accepted lttng-modules into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lttng-modules/2.10.8-1ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-eoan. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Ian, or anyone else affected,

Accepted lttng-modules into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lttng-modules/2.10.8-1ubuntu2~18.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Ian, or anyone else affected,

Accepted lttng-modules into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lttng-modules/2.8.0-1ubuntu1~16.04.8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

ii lttng-modules-dkms 2.11.2-1
ii linux-image-5.4.0-37-generic 5.4.0-37.41

#> modprobe lttng-test
#> dmesg|grep LTTng
[ 1063.654333] LTTng: vmalloc_sync_all symbol lookup failed.
[ 1063.662949] LTTng: vmalloc_sync_all symbol lookup failed.
...
#> modprobe -r lttng-test
#> dpkg -i lttng-modules-dkms_2.11.2-1ubuntu0.1_all.deb
#> modprobe lttng-test
#> dmesg|grep LTTng
...
[ 1153.398908] LTTng: Unloaded modules v2.11.2 (Lafontaine)
[ 1520.041148] LTTng: Loaded modules v2.11.2 (Lafontaine)

ii lttng-modules-dkms 2.10.8-1ubuntu2
ii linux-image-5.3.0-59-generic 5.3.0-59.53

#> modprobe lttng-test
#> dmesg|grep LTTng
...
[ 430.580656] LTTng: vmalloc_sync_all symbol lookup failed.
[ 430.588644] LTTng: vmalloc_sync_all symbol lookup failed.
[ 430.596638] LTTng: Loaded modules v2.10.8 (KeKriek)
[ 430.605330] LTTng: vmalloc_sync_all symbol lookup failed.
#> modprobe -r lttng-test
#> dpkg -i lttng-modules-dkms_2.10.8-1ubuntu2.1_all.deb
#> modprobe lttng-test
#> dmesg|grep LTTng
...
[ 430.596638] LTTng: Loaded modules v2.10.8 (KeKriek)
[ 430.605330] LTTng: vmalloc_sync_all symbol lookup failed.
[ 728.965346] LTTng: Unloaded modules v2.10.8 (KeKriek)
[ 737.134796] LTTng: Loaded modules v2.10.8 (KeKriek)

ii lttng-modules-dkms 2.10.8-1ubuntu2~18.04.1
ii linux-image-4.15.0-106-generic 4.15.0-106.107

#> modprobe lttng-test
#> dmesg|grep LTTng
...
[ 998.703482] LTTng: vmalloc_sync_all symbol lookup failed.
[ 998.710340] LTTng: vmalloc_sync_all symbol lookup failed.
[ 998.717185] LTTng: Loaded modules v2.10.8 (KeKriek)
[ 998.726098] LTTng: vmalloc_sync_all symbol lookup failed.
#> modprobe -r lttng-test
#> dpkg -i lttng-modules-dkms_2.10.8-1ubuntu2~18.04.2_all.deb
#> modprobe lttng-test
#> dmesg|grep LTTng
...
[ 998.717185] LTTng: Loaded modules v2.10.8 (KeKriek)
[ 998.726098] LTTng: vmalloc_sync_all symbol lookup failed.
[ 1043.482947] LTTng: Unloaded modules v2.10.8 (KeKriek)
[ 2360.475042] LTTng: Loaded modules v2.10.8 (KeKriek)

ii lttng-modules-dkms 2.8.0-1ubuntu1~16.04.7
ii linux-image-4.4.0-184-generic 4.4.0-184.214

#> modprobe lttng-test
#> dmesg|grep LTTng
...
[ 688.029325] LTTng: vmalloc_sync_all symbol lookup failed.
[ 688.039163] LTTng: vmalloc_sync_all symbol lookup failed.
[ 735.260865] LTTng: vmalloc_sync_all symbol lookup failed.
[ 735.267774] LTTng: vmalloc_sync_all symbol lookup failed.
#> modprobe -r lttng-test
#> dpkg -i lttng-modules-dkms_2.8.0-1ubuntu1~16.04.8_all.deb
#> modprobe lttng-test
#> dmesg|grep LTTng
...
[ 735.260865] LTTng: vmalloc_sync_all symbol lookup failed.
[ 735.267774] LTTng: vmalloc_sync_all symbol lookup failed.
[ 1028.892742] LTTng: Unloaded modules v2.8.0 (Isseki Nicho)
[ 1346.490666] LTTng: Loaded modules v2.8.0 (Isseki Nicho)

This bug was fixed in the package lttng-modules - 2.8.0-1ubuntu1~16.04.8

---------------
lttng-modules (2.8.0-1ubuntu1~16.04.8) xenial; urgency=medium

  * Fix vmalloc_sync_all() removal (LP: #1874521). Changes from upstream lttng
    repo (1,3,4) with additional Ubuntu fixup (5).
    - d/p/compat-5.6/0001-Update-for-kernel-5.7-use-vmalloc_sync_mappings-on-k.patch
    - d/p/compat-5.6/0003-Fix-Use-vmalloc_sync_mappings-on-kernel-5.6-as-well.patch
    - d/p/compat-5.6/0004-Update-Use-vmalloc_sync_mappings-for-stable-kernels.patch
    - d/p/compat-5.6/0005-Update-Use-vmalloc_sync_mappings-for-ubuntu.patch

 -- Stefan Bader <email address hidden> Wed, 20 May 2020 17:18:34 +0200

The verification of the Stable Release Update for lttng-modules has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package lttng-modules - 2.10.8-1ubuntu2~18.04.2

---------------
lttng-modules (2.10.8-1ubuntu2~18.04.2) bionic; urgency=medium

  * Fix vmalloc_sync_all() removal (LP: #1874521). Changes from upstream lttng repo:
    - d/p/compat-5.6/0001-Update-for-kernel-5.7-use-vmalloc_sync_mappings-on-k.patch
    - d/p/compat-5.6/0002-Fix-missing-wrapper-rename-to-wrapper_vmalloc_sync_m.patch
    - d/p/compat-5.6/0003-Fix-Use-vmalloc_sync_mappings-on-kernel-5.6-as-well.patch
    - d/p/compat-5.6/0004-Update-Use-vmalloc_sync_mappings-for-stable-kernels.patch
    - d/p/compat-5.6/0005-Update-Use-vmalloc_sync_mappings-for-ubuntu.patch

 -- Stefan Bader <email address hidden> Mon, 08 Jun 2020 16:57:12 +0200

This bug was fixed in the package lttng-modules - 2.10.8-1ubuntu2.1

---------------
lttng-modules (2.10.8-1ubuntu2.1) eoan; urgency=medium

  * Fix vmalloc_sync_all() removal (LP: #1874521). Changes from upstream lttng repo:
    - d/p/compat-5.6/0001-Update-for-kernel-5.7-use-vmalloc_sync_mappings-on-k.patch
    - d/p/compat-5.6/0002-Fix-missing-wrapper-rename-to-wrapper_vmalloc_sync_m.patch
    - d/p/compat-5.6/0003-Fix-Use-vmalloc_sync_mappings-on-kernel-5.6-as-well.patch
    - d/p/compat-5.6/0004-Update-Use-vmalloc_sync_mappings-for-stable-kernels.patch
    - d/p/compat-5.6/0005-Update-Use-vmalloc_sync_mappings-for-ubuntu.patch

 -- Stefan Bader <email address hidden> Mon, 08 Jun 2020 16:37:26 +0200

This bug was fixed in the package lttng-modules - 2.11.2-1ubuntu0.1

---------------
lttng-modules (2.11.2-1ubuntu0.1) focal; urgency=medium

  * Fix vmalloc_sync_all() removal (LP: #1874521). Changes from upstream lttng repo:
    - d/p/compat-5.6/0001-Update-for-kernel-5.7-use-vmalloc_sync_mappings-on-k.patch
    - d/p/compat-5.6/0002-Fix-missing-wrapper-rename-to-wrapper_vmalloc_sync_m.patch
    - d/p/compat-5.6/0003-Fix-Use-vmalloc_sync_mappings-on-kernel-5.6-as-well.patch
    - d/p/compat-5.6/0004-Update-Use-vmalloc_sync_mappings-for-stable-kernels.patch
    - d/p/compat-5.6/0005-Update-Use-vmalloc_sync_mappings-for-ubuntu.patch

 -- Stefan Bader <email address hidden> Mon, 08 Jun 2020 16:27:24 +0200