Missing /boot/vmcoreinfo-{version} file is breaking kdump

Bug #988512 reported by Louis Bouchard
22
This bug affects 3 people
Affects Status Importance Assigned to Milestone
kexec-tools (Ubuntu)
Fix Released
High
Louis Bouchard
Precise
Fix Released
High
Louis Bouchard
Quantal
Fix Released
High
Louis Bouchard

Bug Description

[Impact] The kernel package no longer delivers the vmcoreinfo file so the kdump functionality is now broken.

When using linux-image-3.2.0-6-generic and testing kernel dump functionality, the kdump process fails.

This is caused by the following test in /etc/init.d/kdump :

KERNEL_IMAGE="/boot/vmlinuz-$KVER"
INITRD="/boot/initrd.img-$KVER"
VMCORE="/boot/vmcoreinfo-$KVER"

# Without makedumpfile, there will be no vmcore on crash, so no point
test -x /usr/bin/makedumpfile || exit 0
if [ ! -e "$VMCORE" -o ! -e "$KERNEL_IMAGE" -o ! -e "$INITRD" ]; then
        exit 0;
fi

The /boot/vmcoreinfo⁻{kernel version} is not shipped with the linux-image-3.2.0-6-generic.

[Test Case]
While running a 12.04 Precise system, install the linux-crashdump package that will enable the kdump functionality. Reboot the server then, as root, issue the following :
$ echo c > /proc/sysrq-trigger

Normally, the kernel dump should be created in /var/crash. Currently the kdump process fails silently and the system reboots.

[Regression Potential]
Upstream developer of makedumpfile which produces the vmcoreinfo file has confirmed that vmcoreinfo is no longer required (http://lists.infradead.org/pipermail/kexec/2012-June/006381.html). vmcore generated with this patch have proven to be usable with the 'crash' tool.

Package : linux-image-3.2.0-6-generic
Release : Ubuntu 12.04 LTS
---
AlsaDevices:
 total 0
 crw-rw---T 1 root audio 116, 1 Apr 25 15:19 seq
 crw-rw---T 1 root audio 116, 33 Apr 25 15:19 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.0.1-0ubuntu5
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
CurrentDmesg: [ 15.888032] eth0: no IPv6 routers present
DistroRelease: Ubuntu 12.04
HibernationDevice: RESUME=UUID=9569393b-2f51-4314-a5c9-1e684853ae97
InstallationMedia: Ubuntu-Server 12.04 LTS "Precise Pangolin" - Alpha amd64 (20111220)
IwConfig:
 lo no wireless extensions.

 eth0 no wireless extensions.
Lsusb: Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: Bochs Bochs
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 TERM=linux
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 EFI VGA
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.2.0-6-generic root=/dev/mapper/kdumpP-root ro crashkernel=384M-2G:64M,2G-:128M console=ttyS0,115200
ProcVersionSignature: Ubuntu 3.2.0-6.12-generic 3.2.0-rc6
RelatedPackageVersions:
 linux-restricted-modules-3.2.0-6-generic N/A
 linux-backports-modules-3.2.0-6-generic N/A
 linux-firmware 1.79
RfKill: Error: [Errno 2] No such file or directory
Tags: precise
Uname: Linux 3.2.0-6-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

dmi.bios.date: 01/01/2007
dmi.bios.vendor: Bochs
dmi.bios.version: Bochs
dmi.chassis.type: 1
dmi.chassis.vendor: Bochs
dmi.modalias: dmi:bvnBochs:bvrBochs:bd01/01/2007:svnBochs:pnBochs:pvr:cvnBochs:ct1:cvr:
dmi.product.name: Bochs
dmi.sys.vendor: Bochs

Related branches

Brad Figg (brad-figg)
affects: linux-meta (Ubuntu) → linux (Ubuntu)
Revision history for this message
Louis Bouchard (louis) wrote : AcpiTables.txt

apport information

tags: added: apport-collected precise
description: updated
Revision history for this message
Louis Bouchard (louis) wrote : BootDmesg.txt

apport information

Revision history for this message
Louis Bouchard (louis) wrote : Lspci.txt

apport information

Revision history for this message
Louis Bouchard (louis) wrote : ProcCpuinfo.txt

apport information

Revision history for this message
Louis Bouchard (louis) wrote : ProcInterrupts.txt

apport information

Revision history for this message
Louis Bouchard (louis) wrote : ProcModules.txt

apport information

Revision history for this message
Louis Bouchard (louis) wrote : UdevDb.txt

apport information

Revision history for this message
Louis Bouchard (louis) wrote : UdevLog.txt

apport information

Revision history for this message
Louis Bouchard (louis) wrote : WifiSyslog.txt

apport information

Brad Figg (brad-figg)
Changed in linux (Ubuntu):
status: New → Confirmed
Revision history for this message
Tim Gardner (timg-tpi) wrote :
Changed in linux (Ubuntu Precise):
assignee: nobody → Tim Gardner (timg-tpi)
status: Confirmed → In Progress
Changed in linux (Ubuntu Quantal):
status: New → In Progress
Changed in linux (Ubuntu Precise):
importance: Undecided → Medium
Changed in linux (Ubuntu Quantal):
importance: Undecided → Medium
Revision history for this message
Tim Gardner (timg-tpi) wrote :

Louis - the upstream for makedumpfile seems to behind the curve. The last commit before version 1.4.3 was released is:

Author: Atsushi Kumagai <email address hidden>
Date: Wed Feb 22 13:36:36 2012 +0900

    [PATCH] Support newer kernels.

    A new makedumpfile supports newer kernels.
    Now, I tested on specific kernel configuration as:

      - 2.6.37 to 3.0 (x86 FLATMEM)
      - 2.6.37 to 3.0 (x86_64 SPARSEMEM)
      - 3.2 (PPC32 FLATMEM)

    I will support more kernels on x86 in next version.

    Signed-off-by: Atsushi Kumagai <email address hidden>

Perhaps he'll eventually add support for 3.2 kernels. Keep a watch on git://makedumpfile.git.sourceforge.net/gitroot/makedumpfile/makedumpfile

Changed in linux (Ubuntu Precise):
status: In Progress → Won't Fix
tags: added: patch
Revision history for this message
Louis Bouchard (louis) wrote :

@tim,

I contacted upstream about this and here is his reply :

Le 27/04/2012 00:24, Atsushi Kumagai a écrit :
> Hello Louis,
>
> On Thu, 26 Apr 2012 16:27:31 -0400
> Bouchard Louis <email address hidden> wrote:
>
>> Hello Kumagai-san,
>>
>> Do you have an idea of the availability of makedumpfile for 3.2 flavor
>> kernels ?
>>
>> Ubuntu 12.04 Precise have just been released today and, since it is
>> based on a 3.2 kernel, makedumpfile no longer works for it.
>
> Sorry, I haven't tested makedumpfile for 3.2 kernel yet, but I think it
> will work fine. If makedumpfile doesn't work in your environmental, then
> would you report it ?
>
> For your information, Suzuki reported that makedumpfile works for 3.2 kernel
> on ppc32.
>
> http://lists.infradead.org/pipermail/kexec/2012-February/006022.html
>
>
> Thanks
> Atsushi Kumagai

If you can indicate where the vmcoreinfo is being produced by makedumpfile, I can test it myself.

...Louis

Revision history for this message
Louis Bouchard (louis) wrote :

@tim

ok, I found how to test, using the kernel source package :

$ cd ./linux-3.0.0/debian/linux-image-3.0.0-17-server/boot
$ makedumpfile -g vmcoreinfo-3.0.0-17-server -x ../build/build-server/vmlinux

The kernel version is not supported.
The created dumpfile may be incomplete.

The vmcoreinfo is saved to ./boot/vmcoreinfo-3.0.0-17-server.

makedumpfile Completed.

This is caused by the fact that the latest version defined in makedumpfile.h is :
#define LATEST_VERSION KERNEL_VERSION(2, 6, 36)/* linux-2.6.36 */

I'll check with upstream what can be done about that

Revision history for this message
Tim Gardner (timg-tpi) wrote :

Louis - the patch to enable building vmcoreinfo is attached in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/988512/comments/10. Apply it and build for amd64 as you normally would, but be sure to keep a log, e.g.,

fakeroot debian/rules clean; echo "dpkg-buildpackage -B -us -uc 2>&1 |tee log.txt"|schroot -c precise-amd64

Revision history for this message
Louis Bouchard (louis) wrote :

@tim

looking more closely at the example above, turns out that it also fails for 3.0.0 (i.e. Oneiric) kernels and the vmcoreinfo were generated for those. Any reason why it changed ?

Revision history for this message
Tim Gardner (timg-tpi) wrote :

The Oneiric version complains a little but still produces a vmcoreinfo. Precise, on the other hand, fails completely, even with the latest upstream version of makedumpfile (which I built by hand).

(precise-amd64)rtg@salmon:~/ukb/precise/amd64/master-next$ dpkg -l|grep makedumpfile
ii makedumpfile 1.4.3-1ubuntu1 VMcore extraction tool
(precise-amd64)rtg@salmon:~/ukb/precise/amd64/master-next$ makedumpfile -g vmcoreinfo -x ./ubuntu-precise/debian/build/build-generic/vmlinux
generate_vmcoreinfo: Can't find the memory type.

makedumpfile Failed.

Revision history for this message
Louis Bouchard (louis) wrote :

@tim

makedumpfile 1.4.4 was release a few days ago and according to upstream, it should support 3.2 kernels. I'm building a new 1.4.4 package and will test on the latest ubuntu kernel.

Revision history for this message
Louis Bouchard (louis) wrote :

@tim

After testing with the latest 1.4.4, the problem persist. Upstream has acknowledged the problem and will work on a fix :

http://lists.infradead.org/pipermail/kexec/2012-June/006381.html

But he also indicates that the use of vmcoreinfo is no longer required :

"BTW, do you need vmcoreinfo file ? If your purpose is only to generate dumpfile,
you can execute makedumpfile without -g option.
Because the vmcore generated by kernel 2.6.24 or later include the debug
information which makedumpfile need."

Maybe the best solution would be to get rid of the vmcoreinfo file alltogether and adapt the kexec-tools script accordingly.

Revision history for this message
Louis Bouchard (louis) wrote :

@tim

According to upstream, the vmcoreinfo file is no longer required to get a valid core file. This mean that, while the current status of the Precise kernel is fine, the toolset needs to be adapted accordingly.

This would mean that this bug should either :
 - be reassigned to kexec-tools where adaptation need to happen
 - Wait for upstream to fix the vmcoreinfo creation bug (still not working in makedumpfile 1.4.4)

I would vote for the first option, since I already tested a modified version of the kexec-tools and could propose a patch.

Any thought ?

Revision history for this message
Tim Gardner (timg-tpi) wrote :

Changing the package and ownership according to comment #19.

affects: linux (Ubuntu Quantal) → kexec-tools (Ubuntu Quantal)
Changed in kexec-tools (Ubuntu Quantal):
assignee: nobody → Louis Bouchard (louis-bouchard)
Changed in kexec-tools (Ubuntu Precise):
assignee: Tim Gardner (timg-tpi) → Louis Bouchard (louis-bouchard)
status: Won't Fix → In Progress
Revision history for this message
Stefan Bader (smb) wrote :

So while trying to make thinks work as well (without having noticed the bug being open) I took the 1.4.3 version in quantal as a starting point (as that was the latest in debian). The following patch (against makedumpfile) makes at least the vmcoreinfo generation work again in Precise for me. Maybe we want to push that forward even without it being necessary.

Though I still have problems with kexec'ing in general (which maybe needs to become its own bug) as it seems no new kernel gets started even when using kexec in the simplest form to boot via shutdown.

Revision history for this message
Louis Bouchard (louis) wrote :

@stefan

I am surprized at your statement. I'm able to get all currently supported versions of Ubuntu to trigger a kernel dump (with appropriate patches installed).

One single issue I have noticed and still need to address is the required reserved memory needed (minimum of ~110Mb reserved by crashkernel= ) but this should be treated separately. There is a long standing bug on this : LP: #785394

Can you verify with a crashkernel=128M on Precise ?

Revision history for this message
Stefan Bader (smb) wrote :

Ah, it seems that changing the memory to 128M does indeed help. The current settings are 64M up to 2G of main memory and 128M above that. Not sure how much that has to say, but on booting with crashkernel=128M, there was a short text corruption visible. But I seem to have a crash file now.

Revision history for this message
Stefan Bader (smb) wrote :

I hope this does not spam this report too much. But I thought this would be the best place for this patch. It changes the files of an installed kexec-tools package to not require the vmcoreinfo file. We still need to work on updating the kexec-tools package. This is more of a band aid for those who want to quickly enable kdump while the package is not fixed, yet.

Revision history for this message
Louis Bouchard (louis) wrote :

@Stephan,

Not that it spams the bug, but this Merge request has been sitting in the queue for quite a while :

https://code.launchpad.net/~louis-bouchard/ubuntu/precise/kexec-tools/lp988512-no-vmcoreinfo

With what seems to be the identical patch. Should I have done things differently ?

Revision history for this message
Stefan Bader (smb) wrote :

@Louis, no yours is just the version applying to the source package while I added my version as something that someone can apply to the installed system (as a work-around until the merge is done). Unfortunately I can not help as a sponsor as I got no upload rights beyond kernel.

Louis Bouchard (louis)
description: updated
Steve Langasek (vorlon)
Changed in kexec-tools (Ubuntu Precise):
importance: Medium → High
Changed in kexec-tools (Ubuntu Quantal):
importance: Medium → High
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package kexec-tools - 1:2.0.3-1ubuntu2

---------------
kexec-tools (1:2.0.3-1ubuntu2) quantal; urgency=low

  [ Louis Bouchard ]
  * Removed dependency to vmcoreinfo which is no longer required
    and no longer part of the kernel package. LP: #988512.
 -- Steve Langasek <email address hidden> Tue, 14 Aug 2012 12:48:22 -0700

Changed in kexec-tools (Ubuntu Quantal):
status: In Progress → Fix Released
Revision history for this message
Louis Bouchard (louis) wrote :

Hello,

Here is the debdiff for the Precise package for which the SRU is being requested

Revision history for this message
Steve Langasek (vorlon) wrote : Re: [Bug 988512] Re: Missing /boot/vmcoreinfo-{version} file is breaking kdump

Hi Louis,

On Tue, Aug 21, 2012 at 02:10:39PM -0000, Louis Bouchard wrote:
> Here is the debdiff for the Precise package for which the SRU is being
> requested

FWIW I've already uploaded a kexec-tools SRU to the queue, it's just waiting
SRU team review now.

Revision history for this message
Adam Conrad (adconrad) wrote : Please test proposed package

Hello Louis, or anyone else affected,

Accepted kexec-tools into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/kexec-tools/1:2.0.2-3ubuntu4.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in kexec-tools (Ubuntu Precise):
status: In Progress → Fix Committed
tags: added: verification-needed
Revision history for this message
Louis Bouchard (louis) wrote :

Hi,

I verified the package in -proposed in a fresh new Precise VM and it does produce the adequate linux-image-{version}.crash file. I can also confirmed that this file can correctly be used with the crash tool.

tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package kexec-tools - 1:2.0.2-3ubuntu4.1

---------------
kexec-tools (1:2.0.2-3ubuntu4.1) precise-proposed; urgency=low

  * Removed dependency on vmcoreinfo which is no longer required
    and no longer part of the kernel package. LP: #988512.
 -- Louis Bouchard <email address hidden> Tue, 14 Aug 2012 17:14:30 +0200

Changed in kexec-tools (Ubuntu Precise):
status: Fix Committed → Fix Released
Revision history for this message
Colin Watson (cjwatson) wrote : Update Released

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.