lttng-smoke-test failed on Xenial s390x

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1732651

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

This issue can be found on azure 4.11 as well, but the interesting thing is that this is not failing on all the instances. Some of them has passed the test during the same period.

Hrm, tried this on an s390x VM:

$ uname -a
Linux ckingvm2 4.4.0-101-generic #124-Ubuntu SMP Fri Nov 10 18:29:39 UTC 2017 s390x s390x s390x GNU/Linux

dpkg -l | grep lttng
ii liblttng-ctl0:s390x 2.7.1-2ubuntu1 s390x LTTng control and utility library
ii liblttng-ust-ctl2:s390x 2.7.1-1 s390x LTTng 2.0 Userspace Tracer (trace control library)
ii liblttng-ust-dev:s390x 2.7.1-1 s390x LTTng 2.0 Userspace Tracer (development files)
ii liblttng-ust-python-agent0:s390x 2.7.1-1 s390x LTTng 2.0 Userspace Tracer (Python agent native library)
ii liblttng-ust0:s390x 2.7.1-1 s390x LTTng 2.0 Userspace Tracer (tracing libraries)
ii lttng-modules-dkms 2.8.0-1ubuntu1~16.04.3 all Linux Trace Toolkit (LTTng) kernel modules (DKMS)
ii lttng-tools 2.7.1-2ubuntu1 s390x LTTng control and utility programs

ubuntu@ckingvm2:~$ sudo autotest/client/autotest-local autotest/client/tests/ubuntu_lttng_smoke_test/control
11:29:35 INFO | Writing results to /home/ubuntu/autotest/client/results/default
11:29:35 INFO | START ---- ---- timestamp=1510918175 localtime=Nov 17 11:29:35
11:29:35 INFO | START ubuntu_lttng_smoke_test.lttng-smoke-test ubuntu_lttng_smoke_test.lttng-smoke-test timestamp=1510918175 localtime=Nov 17 11:29:35
11:29:40 ERROR| [stderr] [warning] Tracer discarded 1340 events between [11:29:38.065061128] and [11:29:38.150240071] in trace UUID 6614a11b486b459bce703191741816, at path: "/tmp/lttng-kernel-trace-1494-session/kernel", within stream id 0, at relative path: "channel0_0". You should consider recording a new trace with larger buffers or with fewer events enabled.
11:29:40 INFO | == lttng smoke test of session create/destroy ==
11:29:40 INFO | Session test-kernel-session created.
11:29:40 INFO | Traces will be written in /tmp/lttng-kernel-trace-1494-session
11:29:40 INFO | PASSED (lttng create)
11:29:40 INFO | Session test-kernel-session destroyed
11:29:40 INFO | PASSED (lttng destroy)
11:29:40 INFO |
11:29:40 INFO | == lttng smoke test list kernel events ==
11:29:40 INFO | PASSED (lttng list --kernel)
11:29:40 INFO |
11:29:40 INFO | lttng smoke test trace open/close system calls SKIPPED for s390x
11:29:40 INFO |
11:29:40 INFO | == lttng smoke test trace context switches ==
11:29:40 INFO | Session test-kernel-session created.
11:29:40 INFO | Traces will be written in /tmp/lttng-kernel-trace-1494-session
11:29:40 INFO | PASSED (lttng create)
11:29:40 INFO | Kernel event sched_switch created in channel channel0
11:29:40 INFO | PASSED (lttng enable-event)
11:29:40 INFO | Tracing started for session test-kernel-session
11:29:40 INFO | PASSED (lttng start)
11:29:40 INFO | Waiting for data availability.
11:29:40 INFO | Tracing stopped for session test-kernel-session
11:29:40 INFO | PASSED (lttng stop)
11:29:40 INFO | Session test-kernel-session destroyed
11:29:40 INFO | PASSED (lttng destroy)
11:29:40 INFO | ...

I tried the machine where this had failed to pass and observed that the DKMS modules for lttng had not been built by DKMS for some reason.

Looking at /var/log/apt/history.log the following happened:

1. lttng 2.8.0-1ubuntu1~16.04.2 was installed
2. new kernel 4.4.0-101 was installed
3. possibly failed to build lttng for 4.4.0-101
4. update, pulled in lttng 2.8.0-1ubuntu1~16.04.3
5. this didn't build for 4.4.0-101
6. lttng test fail because there were no lttng built modules

I just ran sudo apt-get install --reinstall lttng-modules-dkms to ensure it got rebuild and then I was able to run the tests perfectly well.

Looking at https://wiki.kubuntu.org/Kernel/Dev/DKMSPackaging it states:

"Limitation of DKMS

1. DKMS may not work well after a large change of kernel ABI - Using DKMS on a stable release kernel will be a better idea.
2. Make sure the we have toolchains and kernel header installed.
3. Using DKMS on a netbook is not a smart idea - It takes too much time to build the kernel module after kernel upgrading.
4. DKMS modules will not be loaded if there are another ones in initrd."

I wonder if the change in ABI was the issue.

I suspect the following is necessary if a large ABI change occurs:

ls /usr/src/linux-headers-* -d | sed -e 's/.*linux-headers-//' | \
> sort -V | tac | sudo xargs -n1 /usr/lib/dkms/dkms_autoinstaller start

Thanks for debugging this and the detailed information, Colin!
I will take this is a green light for the test result.

Since we only have a few systems that won't be re-deployed, I will try to run the sed command next time when encounter this again.