Drive not found since 3.2.0-21... kernel panic, unable to mount root fs

Thank you for taking the time to file a bug report on this issue.

However, given the number of bugs that the Kernel Team receives during any development cycle it is impossible for us to review them all. Therefore, we occasionally resort to using automated bots to request further testing. This is such a request.

We have noted that there is a newer version of the development kernel than the one you last tested when this issue was found. Please test again with the newer kernel and indicate in the bug if this issue still exists or not.

You can update to the latest development kernel by simply running the following commands in a terminal window:

    sudo apt-get update
    sudo apt-get dist-upgrade

If the bug still exists, change the bug status from Incomplete to Confirmed. If the bug no longer exists, change the bug status from Incomplete to Fix Released.

If you want this bot to quit automatically requesting kernel tests, add a tag named: bot-stop-nagging.

 Thank you for your help, we really do appreciate it.

James Shackleford, thank you for reporting this and helping make Ubuntu better. Could you please provide the results following https://wiki.ubuntu.com/DebuggingKernelBoot ? As well, if you could also please test the latest upstream kernel available that would be great. It will allow additional upstream developers to examine the issue. Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Once you've tested the upstream kernel, please remove the 'needs-upstream-testing' tag. This can be done by clicking on the yellow pencil icon next to the tag located at the bottom of the bug description and deleting the 'needs-upstream-testing' text. Please let us know your results.

Thanks in advance.

Here are the results of the "debugging kernel boot." Due to the nature of the issue, the best I could do is camera shots.

If there is a trick in this situation for obtaining a full boot log, I would be interested to hear.

-------------------------------
Case 1: (default as per https://wiki.ubuntu.com/DebuggingKernelBoot)

recordfail
gfxmode text
insmod gzio
insmod part_msdos
insmod ext2
set root='(hd0,msdos3)'
search --no-floppy --fs-uuid --set=root 862bcf01-fc44-4c29-8d08-f8be78763567
linux /boot/vmlinuz-3.2.0-22-generic root=/dev/sda3 ro

http://www.tshack.net/tmp/debug.jpg [158 KB]

-------------------------------
Case 2: (case 1 with raid=noautodetect)

recordfail
gfxmode text
insmod gzio
insmod part_msdos
insmod ext2
set root='(hd0,msdos3)'
search --no-floppy --fs-uuid --set=root 862bcf01-fc44-4c29-8d08-f8be78763567
linux /boot/vmlinuz-3.2.0-22-generic raid=noautodetect root=/dev/sda3 ro

http://www.tshack.net/tmp/debug_noraidauto.jpg [187 KB]
-------------------------------

Note that using root=UUID=378367a0-4aac-4e08-b078-e708b24ac4d9 does not work either. For kernel 3.2.0-20, update-grub still generates root= using UUID on the kernel line. For 3.2.0.21+ update, root= is more explicitly specified using /dev/xxxyy entries for some reason. Neither specification works for kernels >= 3.2.0.21 on my system.

I will perform an upstream kernel build as soon as time permits.

Sorry, the note at the bottom of #4 accidentally included the wrong UUID, but you get the idea.

Corrected post reads as follows:

Note that using root=UUID=862bcf01-fc44-4c29-8d08-f8be78763567 does not work either. For kernel 3.2.0-20, update-grub still generates root= using UUID on the kernel line. For 3.2.0.21+ update, root= is more explicitly specified using /dev/xxxyy entries for some reason. Neither specification works for kernels >= 3.2.0.21 on my system.

Tried the pre-compiled mainline daily for April 6th 3.4.0 :
(from: http://kernel.ubuntu.com/~kernel-ppa/mainline/daily/current/)
 * linux-headers-3.4.0-999_3.4.0-999.201204060408_all.deb
 * linux-headers-3.4.0-999-generic_3.4.0-999.201204060408_amd64.deb
 * linux-image-3.4.0-999-generic_3.4.0-999.201204060408_amd64.deb

The results are similar. (See attachment)

James Shackleford, thank you for providing the pictures and testing the mainline kernel. The next step is to perform a kernel bisect to identify the offending commit. Could you please do this following https://wiki.ubuntu.com/Kernel/KernelBisection ?

Hi Christopher. No problem. However, I am currently traveling so I will not be able to perform the kernel bisection until I return on Tuesday. I apologize for this delay.

I have performed some more work towards this issue and I have found that it is an issue not with the kernel but with the generation of initrd.img.

Ubuntu-3.2.0-20-generic is the kernel I received upon initial install via CD-ROM. This kernel came with the proper initrd.img-3.2.0-20-generic image. update-grub detects both kernel and ramdisk image correctly, and the system boots fine.

Ubuntu-3.2.0-21-generic and Ubuntu-3.2.0-22-generic will installed via repository (normal system update). initrd.img does not exist for either of these kernels. Consequently, update-grub does not generate initrd lines for these kernels and they fail to mount the system partition due to lack of necessary modules.

I built Ubuntu-3.2.0-23 from the Ubuntu kernel git repository via:
$ fakeroot debian/rules clean
$ fakeroot debian/rules binary-headers binary-generic

and installed the resulting debs:
 * linux-headers-3.2.0-23_3.2.0-23.36_all.deb
 * linux-headers-3.2.0-23-generic_3.2.0-23.36_amd64.deb
 * linux-image-3.2.0-23-generic_3.2.0-23.36_amd64.deb

Again, initrd.img-3.2.0-23-generic was never generated for some reason. So, naturally, booting this also failed.

I was able to get Ubuntu-3.2.0-23-generic to boot by:
 1) making a copy of my working initrd.img-3.2.0-20-generic
 2) unpacking it
 3) rename lib/modules/3.2.0-20-generic to lib/modules/3.2.0-23-generic
 4) repacking
 5) moving to /boot/initrd.img-3.2.0-23-generic

I know this is a terrible hack, but I am not sure how to generate a proper initrd.img by hand.

In short, initrd.img is not being generated after kernel installation by initramfs-tools on my system for some reason. I am not familiar enough with the generation of initrd.img to know where the failure could be. Help is, of course, appreciated in tracking the source of the error.

Regardless, this is no longer really a kernel bug and should probably be reassigned or closed/re-opened as appropriate.

Yes, I can confirm that on my system the problem is with update-initramfs.

After kernel installation, update-initramfs is invoked via /etc/kernel/postinst.d/initramfs-tools. However, initrd.img is not created. In fact, update-initramfs seems to do nothing.

Running update-initramfs manually via:
$ sudo update-initramfs -c -t -k 3.2.0-21-generic -v

Produces no output.

Running:
$ update-initramfs

also produces no output. Not even the usage as in 10.04 LTS.

However, I am able to use /usr/sbin/update-initramfs from my working 10.04 LTS partition to generate the initrd.img images. So, it looks like a problem with update-initramfs

I tried:
$ sudo apt-get clean
$ sudo apt-get autoclean
$ sudo apt-get --reinstall install initramfs-tools

but this did not seem to change anything.

Final update.

After further investigation, it seems that update-initramfs on my 12.04 LTS partition was merely a symlink to /bin/true

The "real" update-initramfs was diverted to /usr/sbin/update-initramfs.distrib

I am guessing there was just a hiccup in ubiquity during the install, perhaps?

Final-Final update.

No guessing. It was ubiquity:

$ sudo dpkg-divert --list
. . .
diversion of /usr/sbin/update-initramfs to /usr/sbin/update-initramfs.distrib by ubiquity
. . .

I've just done a clean install of 12.04 Beta 2 and got this problem after apt-get dist-upgrade.

I see exactly the same symptoms as James describes with the messed-up grub entries and kernel panic. My /usr/sbin/update-initramfs is also linked to /bin/true.

I've worked around it by choosing the 3.2.0-20 kernel from the grub menu, then I removed the 3.2.0-23 series with Synaptic and ran update-grub.

My laptop has kernel 3.2.0-22 and all looks OK:

richard@calder:/boot$ ls -l /boot
total 45204
-rw-r--r-- 1 root root 789425 Mar 27 18:22 abi-3.2.0-20-generic
-rw-r--r-- 1 root root 791023 Apr 3 20:13 abi-3.2.0-22-generic
-rw-r--r-- 1 root root 140211 Mar 27 18:22 config-3.2.0-20-generic
-rw-r--r-- 1 root root 140279 Apr 3 20:13 config-3.2.0-22-generic
drwxr-xr-x 3 root root 12288 Apr 10 12:23 grub
-rw-r--r-- 1 root root 14159604 Apr 6 02:13 initrd.img-3.2.0-20-generic
-rw-r--r-- 1 root root 14168869 Apr 11 17:12 initrd.img-3.2.0-22-generic
-rw-r--r-- 1 root root 176764 Nov 27 10:00 memtest86+.bin
-rw-r--r-- 1 root root 178944 Nov 27 10:00 memtest86+_multiboot.bin
-rw------- 1 root root 2884032 Mar 27 18:22 System.map-3.2.0-20-generic
-rw------- 1 root root 2884270 Apr 3 20:13 System.map-3.2.0-22-generic
-rw------- 1 root root 4965584 Mar 27 18:22 vmlinuz-3.2.0-20-generic
-rw------- 1 root root 4966000 Apr 3 20:13 vmlinuz-3.2.0-22-generic
richard@calder:/boot$ ls -l /usr/sbin/update-initramfs
-rwxr-xr-x 1 root root 9164 Sep 19 2011 /usr/sbin/update-initramfs

So, given that my laptop is OK and fully up to date, the fact I got the problem on another machine by doing dist-upgrade suggests the problem is in this update:

The following packages have been kept back:
  linux-generic linux-headers-generic linux-image-generic
0 upgraded, 0 newly installed, 0 to remove and 3 not upgraded.

For reference, I will post my solution as well.

First, I undid the divert performed by ubiquity via:
$ sudo rm /usr/sbin/update-initramfs
$ sudo dpkg-divert --rename --remove /usr/sbin/update-initramfs

Then I was able to generate initrd images for all kernels via:
$ sudo update-initramfs -c -t -k all

And then updated grub as usual:
$ sudo update-grub

I have since pulled down a kernel via "Update Manager" and update-initramfs is now being triggered as intended by /etc/kernel/postinst.d/initramfs-tools, which now generates working boot configurations.

I've followed these two steps of James's solution

$ sudo rm /usr/sbin/update-initramfs
$ sudo dpkg-divert --rename --remove /usr/sbin/update-initramfs

and can confirm that my problem machine also boots correctly with the 3.2.0-23 kernel reinstated by reinstalling linux-generic, linux-headers-generic and linux-image-generic.

I am removing the 'needs-bisect' tag.

It should be noted that in Post #10 I only mentioned building 3.2.0-23, but I also built 3.2.0-9, 3.2.0-20, 3.2.0-21, and 3.2.0-22. All of these build failed to boot due to lack of initrd.img. Naturally, the bisection failed because I could not find a "good" version since the problem was related to update-initranfs and initrd.img generation... not the kernel.

Somebody more knowledgeable than myself should probably move this bug somewhere more appropriate (ubiquity, perhaps?) since it has been shown that this is not a kernel issue, despite being kernel related.

Agree with James that this looks like an installer issue. On my fresh install everything was OK. It was only when I triggered an update that introduced a new kernel that the problem with update-initramfs became apparent.

Switched to ubiquity (Ubuntu)

I just downloaded a fresh version of the Precise Beta 2 ISO and re-installed on my problem machine. I did the same dist-upgrade that I did yesterday and all is well. No issues with update-initramfs.

I'd suggest that this is most likely a problem with one of the old ISOs and this bug should be marked invalid if James agrees?

Richard, thank you for your contributions. Good to know I was not the only person with this bug.

I agree. Setting this bug back to invalid.

I just got hit by this bug after installing the release. I haven't been using any betas at all.

To clarify: I used the Xubuntu installer, after the prompted upgrade crashed half way through, and that's when I encountered this problem.

This release of Ubuntu is no longer receiving maintenance updates. If this is still an issue on a maintained version of Ubuntu please let us know.

[Expired for ubiquity (Ubuntu) because there has been no activity for 60 days.]