ubuntu-kernel-tests

ubuntu_lttng_smoke_test failed on Azure 4.15

Bug #1778642 reported by Po-Hsu Lin
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
ubuntu-kernel-tests
Fix Released
Undecided
Unassigned
linux-azure (Ubuntu)
Invalid
Undecided
Unassigned
Xenial
Invalid
Undecided
Unassigned
Bionic
Invalid
Undecided
Unassigned
Cosmic
Invalid
Undecided
Unassigned
lttng-modules (Ubuntu)
Fix Released
Undecided
Unassigned
Xenial
Fix Released
Undecided
Unassigned
Bionic
Fix Released
Undecided
Unassigned
Cosmic
Fix Released
Undecided
Unassigned

Bug Description

SRU Request:

[Impact]

After bionic/linux-azure commit 4fbb658dd8ce96384852ceae18b0b09034de7b36 (UBUNTU: [Config] azure: CONFIG_HOTPLUG_CPU=n) the lttng_tracer module cannot find some symbols and cannot be loaded.

 kernel: [ 639.538251] lttng_tracer: Unknown symbol lttng_hp_prepare (err 0)
 kernel: [ 639.538280] lttng_tracer: Unknown symbol lttng_hp_online (err 0)

This is caused by lttng-modules incorrectly handling the case when CONFIG_HOTPLUG_CPU is not enabled.

[Test Case]

Install linux-azure >= 4.15.0-1014, install lttng-modules-dkms and run the autotest ubuntu_lttng_smoke_test testcase (or just 'sudo lttng list --kernel').

[Fix]

commit 894ce5dc825733d1ccab25b1aede9ce0be2a7c4e (Fix: do not use CONFIG_HOTPLUG_CPU for the new hotplug API)

commit 1ee63d8d1b0aaf30564d3879d8e9282d0ddfef1b (Cleanup: comment about CONFIG_HOTPLUG_CPU ifdef)

[Regression Potential]

Very low. The two patches can be cleanly cherry-picked from the upstream commits. The first one that really fixed the issue just changes a single #ifdef to compile a piece of code when CONFIG_HOTPLUG_CPU is not defined. The second one just updates some comments on the code to make them consistent with the changes.

[Original Description]

This test will pass with the older kernel, 4.15.0-1013-azure, but not the 4.15.0-1014-azure in -proposed.

It looks like the kernel module can be installed, but four tests have failed:

== lttng smoke test of session create/destroy ==
Session test-kernel-session created.
Traces will be written in /tmp/lttng-kernel-trace-14716-session
PASSED (lttng create)
Session test-kernel-session destroyed
PASSED (lttng destroy)

== lttng smoke test list kernel events ==
FAILED (lttng list --kernel)
FAILED (lttng list --kernel more output expected)

== lttng smoke test trace open/close system calls ==
Session test-kernel-session created.
Traces will be written in /tmp/lttng-kernel-trace-14716-session
PASSED (lttng create)
FAILED (lttng enable-event)
Session test-kernel-session destroyed

== lttng smoke test trace context switches ==
Session test-kernel-session created.
Traces will be written in /tmp/lttng-kernel-trace-14716-session
PASSED (lttng create)
FAILED (lttng enable-event)
Session test-kernel-session destroyed

Summary: 4 passed, 4 failed

http://10.246.72.46/4.15.0-1014.14-azure/bionic-linux-azure-azure-4.15.0-Standard_D12-ubuntu_lttng_smoke_test/ubuntu_lttng_smoke_test/results/ubuntu_lttng_smoke_test.lttng-smoke-test/debug/ubuntu_lttng_smoke_test.lttng-smoke-test.DEBUG.html

Error message from /var/log/syslog:

 kernel: [ 639.531889] PKCS#7 signature not signed with a trusted key
 kernel: [ 639.538251] lttng_tracer: Unknown symbol lttng_hp_prepare (err 0)
 kernel: [ 639.538280] lttng_tracer: Unknown symbol lttng_hp_online (err 0)
 lttng-sessiond[1097]: Error: Unable to load required module lttng-ring-buffer-client-discard
 lttng-sessiond[1097]: Warning: No kernel tracer available
 kernel: [ 639.606845] PKCS#7 signature not signed with a trusted key
 kernel: [ 639.613331] lttng_tracer: Unknown symbol lttng_hp_prepare (err 0)
 kernel: [ 639.613360] lttng_tracer: Unknown symbol lttng_hp_online (err 0)
 lttng-sessiond[1097]: Error: Unable to load required module lttng-ring-buffer-client-discard
 lttng-sessiond[1097]: Warning: No kernel tracer available
 kernel: [ 639.677124] PKCS#7 signature not signed with a trusted key
 kernel: [ 639.680857] lttng_tracer: Unknown symbol lttng_hp_prepare (err 0)
 kernel: [ 639.680886] lttng_tracer: Unknown symbol lttng_hp_online (err 0)
 lttng-sessiond[1097]: Error: Unable to load required module lttng-ring-buffer-client-discard
 lttng-sessiond[1097]: Warning: No kernel tracer available
 kernel: [ 639.737914] PKCS#7 signature not signed with a trusted key
 kernel: [ 639.753366] lttng_tracer: Unknown symbol lttng_hp_prepare (err 0)
 kernel: [ 639.753394] lttng_tracer: Unknown symbol lttng_hp_online (err 0)
 lttng-sessiond[1097]: Error: Unable to load required module lttng-ring-buffer-client-discard
 lttng-sessiond[1097]: Warning: No kernel tracer available
 kernel: [ 639.829710] PKCS#7 signature not signed with a trusted key
 kernel: [ 639.843736] lttng_tracer: Unknown symbol lttng_hp_prepare (err 0)
 kernel: [ 639.843765] lttng_tracer: Unknown symbol lttng_hp_online (err 0)
 lttng-sessiond[1097]: Error: Unable to load required module lttng-ring-buffer-client-discard
 lttng-sessiond[1097]: Warning: No kernel tracer available

See original description
Po-Hsu Lin (cypressyew)
description: updated
Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

Comment #1 is invalid, this test does not work on xenial/Azure because of the module build issue (bug 1765673)

Revision history for this message
Kleber Sacilotto de Souza (kleber-souza) wrote :

This is probably being caused by the following commit:

commit 4fbb658dd8ce96384852ceae18b0b09034de7b36
Author: Marcelo Henrique Cerri <email address hidden>
Date: Mon Jun 11 18:22:54 2018 -0300

    UBUNTU: [Config] azure: CONFIG_HOTPLUG_CPU=n

The lttng_hp_prepare and lttng_hp_online variables are declared inside a #ifdef block and are probably used somewhere else without the proper checking.

So this is a regression that needs to be fixed on the lttng dkms code.

Revision history for this message
Kleber Sacilotto de Souza (kleber-souza) wrote :

This bug seems to be resolved by the following commits from lttng-modules upstream:

===============================================
commit 1ee63d8d1b0aaf30564d3879d8e9282d0ddfef1b
Author: Michael Jeanson <email address hidden>
Date: Tue Mar 13 12:14:43 2018 -0400

    Cleanup: comment about CONFIG_HOTPLUG_CPU ifdef

    Signed-off-by: Michael Jeanson <email address hidden>
    Signed-off-by: Mathieu Desnoyers <email address hidden>

commit 894ce5dc825733d1ccab25b1aede9ce0be2a7c4e
Author: Lars Persson <email address hidden>
Date: Sun Mar 11 15:02:43 2018 +0100

    Fix: do not use CONFIG_HOTPLUG_CPU for the new hotplug API

    Kernel configurations without CONFIG_HOTPLUG_CPU throw an unknown
    symbol error when attempting to insert the lttng-trace module:
     lttng_tracer: Unknown symbol lttng_hp_prepare (err 0)
     lttng_tracer: Unknown symbol lttng_hp_online (err 0)

    This was caused by lttng-events and lttng-context-perf-counter not
    agreeing on which preprocessor condition that should guard the use of
    the hotplug API. In fact the API is available also on kernels built
    without CONFIG_HOTPLUG_CPU.

    Signed-off-by: Lars Persson <email address hidden>
    Signed-off-by: Mathieu Desnoyers <email address hidden>
===============================================

I will build a test package and attach here a debdiff.

Changed in linux-azure (Ubuntu):
status: New → Invalid
Changed in ubuntu-kernel-tests:
status: New → Invalid
Changed in linux-azure (Ubuntu Bionic):
status: New → Invalid
Changed in lttng-modules (Ubuntu Bionic):
status: New → In Progress
Changed in lttng-modules (Ubuntu):
status: New → Fix Released
Kleber Sacilotto de Souza (kleber-souza)
Changed in lttng-modules (Ubuntu Cosmic):
status: Fix Released → In Progress
Revision history for this message
Kleber Sacilotto de Souza (kleber-souza) wrote :
Revision history for this message
Kleber Sacilotto de Souza (kleber-souza) wrote :
Kleber Sacilotto de Souza (kleber-souza)
Changed in linux-azure (Ubuntu Xenial):
status: New → Invalid
Changed in lttng-modules (Ubuntu Xenial):
status: New → In Progress
Ubuntu Foundations Team Bug Bot (crichton)
tags: added: patch
Revision history for this message
Kleber Sacilotto de Souza (kleber-souza) wrote :
Kleber Sacilotto de Souza (kleber-souza)
description: updated
Kleber Sacilotto de Souza (kleber-souza)
description: updated
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package lttng-modules - 2.10.5-1ubuntu3

---------------
lttng-modules (2.10.5-1ubuntu3) cosmic; urgency=medium

  * ubuntu_lttng_smoke_test failed on Azure 4.15 (LP: #1778642)
    - debian/patches/0009-Fix-do-not-use-CONFIG_HOTPLUG_CPU-for-the-new-hotplu.patch
    - debian/patches/0010-Cleanup-comment-about-CONFIG_HOTPLUG_CPU-ifdef.patch

 -- Kleber Sacilotto de Souza <email address hidden> Thu, 19 Jul 2018 14:34:36 +0200

Changed in lttng-modules (Ubuntu Cosmic):
status: In Progress → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello Po-Hsu, or anyone else affected,

Accepted lttng-modules into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lttng-modules/2.8.0-1ubuntu1~16.04.7 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in lttng-modules (Ubuntu Xenial):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-xenial
Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

Verified with Xenial Azure cloud, it works as expected.

tags: added: verification-done verification-done-xenial
removed: verification-needed verification-needed-xenial
no longer affects: lttng-modules
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for lttng-modules has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package lttng-modules - 2.8.0-1ubuntu1~16.04.7

---------------
lttng-modules (2.8.0-1ubuntu1~16.04.7) xenial; urgency=medium

  * ubuntu_lttng_smoke_test failed on Azure 4.15 (LP: #1778642)
    - debian/patches/0041-Fix-do-not-use-CONFIG_HOTPLUG_CPU-for-the-new-hotplu.patch
    - debian/patches/0042-Cleanup-comment-about-CONFIG_HOTPLUG_CPU-ifdef.patch

 -- Kleber Sacilotto de Souza <email address hidden> Thu, 19 Jul 2018 17:47:16 +0200

Changed in lttng-modules (Ubuntu Xenial):
status: Fix Committed → Fix Released
Revision history for this message
Robie Basak (racb) wrote : Please test proposed package

Hello Po-Hsu, or anyone else affected,

Accepted lttng-modules into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lttng-modules/2.10.5-1ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in lttng-modules (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-bionic
removed: verification-done
Revision history for this message
Kleber Sacilotto de Souza (kleber-souza) wrote :

Verified the fix on Bionic with linux-image-4.15.0-1018-azure=4.15.0-1018.18 and lttng-modules-dkms=2.10.5-1ubuntu1.1. The fix works as expected and ubuntu_lttng_smoke_test testcase passes.

tags: added: verification-done-bionic
removed: verification-needed-bionic
tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package lttng-modules - 2.10.5-1ubuntu1.1

---------------
lttng-modules (2.10.5-1ubuntu1.1) bionic; urgency=medium

  * ubuntu_lttng_smoke_test failed on Azure 4.15 (LP: #1778642)
    - debian/patches/0003-Fix-do-not-use-CONFIG_HOTPLUG_CPU-for-the-new-hotplu.patch
    - debian/patches/0004-Cleanup-comment-about-CONFIG_HOTPLUG_CPU-ifdef.patch

 -- Kleber Sacilotto de Souza <email address hidden> Thu, 19 Jul 2018 12:49:06 +0200

Changed in lttng-modules (Ubuntu Bionic):
status: Fix Committed → Fix Released
Po-Hsu Lin (cypressyew)
Changed in ubuntu-kernel-tests:
status: Invalid → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.