Incompatible libdevmapper 1.02.136 (2016-11-05) and kernel driver (unknown version).

Bug #1655735 reported by Barry Warsaw
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Expired
High
Unassigned
lvm2 (Ubuntu)
Expired
Undecided
Unassigned
multipath-tools (Ubuntu)
Expired
Undecided
Unassigned

Bug Description

There is a new Zesty regression on armhf, related to the autopkgtest for ubuntu-image. Here's a log of a recent run:

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-zesty/zesty/armhf/u/ubuntu-image/20170111_171929_431b1@/log.gz

Scroll to the bottom and you see this:

Incompatible libdevmapper 1.02.136 (2016-11-05) and kernel driver (unknown version).
device mapper prerequisites not met
/dev/mapper/control: mknod failed: Operation not permitted
Failure to communicate with kernel device-mapper driver.
Check that device-mapper is available in the kernel.
Incompatible libdevmapper 1.02.136 (2016-11-05) and kernel driver (unknown version).
device mapper prerequisites not met

Don't be misled by the gadget.yamls which name different architectures. ubuntu-image should be able to build a disk image for any architecture on any architecture.

What this test does is build the image, then run kpartx to map the partitions in the image, and then mount the partitions. It doesn't do anything other than ensure that the partitions can be mounted, and this is where it's failing.

This test succeeds on amd64, i386, and ppc64el. It's never passed on s390x so that's not a regression. It *used* to pass on armhf even on Zesty, but the recent upload of 0.13+17.10ubuntu2 displays the regression.

I will bugtask this to libdevmapper and kpartx since it's possibly a bug in one of those packages (though I still suspect the kernel).

ProblemType: Bug
DistroRelease: Ubuntu 17.04
Package: linux-image-4.9.0-11-generic 4.9.0-11.12
ProcVersionSignature: Ubuntu 4.9.0-11.12-generic 4.9.0
Uname: Linux 4.9.0-11-generic x86_64
NonfreeKernelModules: zfs zunicode zavl zcommon znvpair nvidia_uvm nvidia_drm nvidia_modeset nvidia
ApportVersion: 2.20.4-0ubuntu1
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC1: barry 5414 F.... pulseaudio
 /dev/snd/controlC0: barry 5414 F.... pulseaudio
CurrentDesktop: Unity
Date: Wed Jan 11 13:10:30 2017
EcryptfsInUse: Yes
HibernationDevice: RESUME=UUID=8cc96f00-242f-42c6-bdb9-72db178f8864
InstallationDate: Installed on 2016-01-22 (355 days ago)
InstallationMedia: Ubuntu 16.04 LTS "Xenial Xerus" - Alpha amd64 (20160117)
MachineType: Gigabyte Technology Co., Ltd. Z170X-UD5
ProcFB:

ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.9.0-11-generic root=UUID=d03718f1-9983-4b07-a942-1b42fe8352ec ro
RelatedPackageVersions:
 linux-restricted-modules-4.9.0-11-generic N/A
 linux-backports-modules-4.9.0-11-generic N/A
 linux-firmware 1.162
RfKill:

SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 07/24/2015
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: F2
dmi.board.asset.tag: To be filled by O.E.M.
dmi.board.name: Z170X-UD5-CF
dmi.board.vendor: Gigabyte Technology Co., Ltd.
dmi.board.version: x.x
dmi.chassis.asset.tag: To Be Filled By O.E.M.
dmi.chassis.type: 3
dmi.chassis.vendor: To Be Filled By O.E.M.
dmi.chassis.version: To Be Filled By O.E.M.
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvrF2:bd07/24/2015:svnGigabyteTechnologyCo.,Ltd.:pnZ170X-UD5:pvrTobefilledbyO.E.M.:rvnGigabyteTechnologyCo.,Ltd.:rnZ170X-UD5-CF:rvrx.x:cvnToBeFilledByO.E.M.:ct3:cvrToBeFilledByO.E.M.:
dmi.product.name: Z170X-UD5
dmi.product.version: To be filled by O.E.M.
dmi.sys.vendor: Gigabyte Technology Co., Ltd.

Revision history for this message
Barry Warsaw (barry) wrote :

Removed unhelpful attachments and retagged to armhf.

tags: added: armhf
removed: amd64
Revision history for this message
Brad Figg (brad-figg) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1655735

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Revision history for this message
Barry Warsaw (barry) wrote : Re: [Bug 1655735] Missing required logs.

On Jan 11, 2017, at 06:30 PM, Brad Figg wrote:

>If, due to the nature of the issue you have encountered, you are unable
>to run this command, please add a comment stating that fact and change
>the bug status to 'Confirmed'.

This is probably a regression only on armhf, so the logs generated on amd64
won't be helpful.

Changed in linux (Ubuntu):
status: Incomplete → Confirmed
Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

Does the error go away if you boot a Xenial test kernel on this machine and re-run the test? That will tell us if it is a kernel regression.

Changed in linux (Ubuntu):
importance: Undecided → High
tags: added: kernel-key
Revision history for this message
Barry Warsaw (barry) wrote : Re: [Bug 1655735] Re: Incompatible libdevmapper 1.02.136 (2016-11-05) and kernel driver (unknown version).

On Jan 11, 2017, at 08:10 PM, Joseph Salisbury wrote:

>Does the error go away if you boot a Xenial test kernel on this machine
>and re-run the test? That will tell us if it is a kernel regression.

We're a bit hardware starved atm, but trying to test this.

Revision history for this message
Barry Warsaw (barry) wrote :
Revision history for this message
Barry Warsaw (barry) wrote :

If you run `dmsetup ls` as non-root on amd64, you get an error that's close:

% dmsetup ls
/dev/mapper/control: open failed: Permission denied
Failure to communicate with kernel device-mapper driver.
Incompatible libdevmapper 1.02.136 (2016-11-05) and kernel driver (unknown version).
Command failed

where as the failure in the log file says:

[K/dev/mapper/control: mknod failed: Operation not permitted
Failure to communicate with kernel device-mapper driver.
Check that device-mapper is available in the kernel.

If you run the same command as root on amd64, it works.

So my question is: despite having needs-root, is it somehow possible that
autopkgtest is running the armhf command as non-root, and the slightly
different error message is indicating that, or is it really not available when
run as root on armhf.

If you have an armhf device with an up-to-date Zesty, can you please check?

Revision history for this message
Barry Warsaw (barry) wrote :

LP: #1656391

And it's reproducible in an amd64 container run locally. So probably some change in the images or lxc. Maybe a new missing dependency?

Revision history for this message
Barry Warsaw (barry) wrote :

I think we know almost the entire story now. See LP: #1656391

The one piece still missing is what changed. Very clearly the armhf container tests passed for several months, and then stopped working. My best guess is that they switched from using privileged containers to unprivileged containers. In any event, I'm going to close this bug. See the above referenced bug for further discussion.

Changed in linux (Ubuntu):
status: Confirmed → Invalid
Changed in lvm2 (Ubuntu):
status: New → Invalid
Changed in linux (Ubuntu):
status: Invalid → Incomplete
Changed in lvm2 (Ubuntu):
status: Invalid → Incomplete
Changed in multipath-tools (Ubuntu):
status: New → Incomplete
tags: added: kernel-da-key
removed: kernel-key
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]

Changed in linux (Ubuntu):
status: Incomplete → Expired
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for lvm2 (Ubuntu) because there has been no activity for 60 days.]

Changed in lvm2 (Ubuntu):
status: Incomplete → Expired
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for multipath-tools (Ubuntu) because there has been no activity for 60 days.]

Changed in multipath-tools (Ubuntu):
status: Incomplete → Expired
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.