in-target: mkinitramfs: failed to determine device for /

Using a virtual machine I'm able to reproduce the differing UUIDs but not able to reproduce the failure of the initrd generation.

The code that fires the error is /usr/share/initramfs-tools/hook-functions::dep_add_modules_mount()

I've now successfully reproduced the issue in the virtual machine.

Firstly, at the installer boot menu, choose Expert Mode.

This allows selection of each installer step from a menu and causes debconf to ask many more questions.

At "Install the System" choose a "normal" installation and then for "Drivers to include in the initrd:" choose "targeted: only include drivers needed for this system".

syslog shows:

May 17 22:55:29 base-installer: info: Using kernel 'linux-generic'
May 17 22:55:29 base-installer: warning: Failed to get debconf answer 'base-installer/kernel/linux/initrd'.
May 17 22:55:29 base-installer: info: Setting do_initrd='yes'.
May 17 22:55:29 base-installer: info: Setting link_in_boot='no'.
May 17 22:55:29 in-target: Reading package lists...
...
May 17 22:56:21 in-target: Running depmod.^M
May 17 22:56:22 in-target: update-initramfs: deferring update (hook will be called later)^M
May 17 22:56:22 in-target: Examining /etc/kernel/postinst.d.^M
May 17 22:56:22 in-target: run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 4.4.0-21-generic /boot/vmlinuz-4.4.0-21-generic^M
May 17 22:56:22 in-target: run-parts: executing /etc/kernel/postinst.d/initramfs-tools 4.4.0-21-generic /boot/vmlinuz-4.4.0-21-generic^M
May 17 22:56:22 in-target: update-initramfs: Generating /boot/initrd.img-4.4.0-21-generic^M
May 17 22:56:22 in-target: mkinitramfs: failed to determine device for /^M
May 17 22:56:22 in-target: mkinitramfs: workaround is MODULES=most, check:^M
May 17 22:56:22 in-target: grep -r MODULES /etc/initramfs-tools/^M
May 17 22:56:22 in-target: ^M
May 17 22:56:22 in-target: Error please report bug on initramfs-tools
May 17 22:56:22 in-target: ^M
May 17 22:56:22 in-target: Include the output of 'mount' and 'cat /proc/mounts'^M

This is triggered by the debconf setting in base-installer's library.sh:install_kernel_linux():

Where it writes out the alternate module policy for initramfs-tools:

root@tmpstorage:/# cat /etc/initramfs-tools/conf.d/driver-policy
+ cat /etc/initramfs-tools/conf.d/driver-policy
# Driver inclusion policy selected during installation
# Note: this setting overrides the value set in the file
# /etc/initramfs-tools/initramfs.conf
MODULES=dep

I added "set -x" to the start of /usr/share/initramfs-tools/hook-functions::dep_add_modules_mount() and after it failed I grabbed the output from /var/log/apt/term.log:

The link /initrd.img is a dangling linkto /boot/initrd.img-4.4.0-21-generic
vmlinuz(/boot/vmlinuz-4.4.0-21-generic
) points to /boot/vmlinuz-4.4.0-21-generic
 (/boot/vmlinuz-4.4.0-21-generic) -- doing nothing at /var/lib/dpkg/info/linux-image-4.4.0-21-generic.postinst line 491.
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 4.4.0-21-generic /boot/vmlinuz-4.4.0-21-generic
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 4.4.0-21-generic /boot/vmlinuz-4.4.0-21-generic
update-initramfs: Generating /boot/initrd.img-4.4.0-21-generic
+ local dir dev_node FSTYPE
+ local modules=
+ dir=/
+ [ ! -d /sys/devices/ ]
+ read dev mp fs opts rest
+ [ / = / ]
+ [ ext4 != rootfs ]
+ printf dev_node=/dev/sda3\nFSTYPE=ext4
+ break
+ eval dev_node=/dev/sda3
FSTYPE=ext4
+ dev_node=/dev/sda3
+ FSTYPE=ext4
+ [ / != / ]
+ [ ext4 = ubifs ]
+ [ / = / ]
+ [ /dev/sda3 = /dev/root ]
+ [ -z /dev/sda3 ]
+ readlink -f /dev/sda3
+ dev_node=/dev/sda3
+ [ -b /dev/sda3 ]
+ echo mkinitramfs: failed to determine device for /
mkinitramfs: failed to determine device for /
+ echo mkinitramfs: workaround is MODULES=most, check:
mkinitramfs: workaround is MODULES=most, check:
+ echo grep -r MODULES /etc/initramfs-tools/
grep -r MODULES /etc/initramfs-tools/
+ echo

+ echo Error please report bug on initramfs-tools
Error please report bug on initramfs-tools
+ echo Include the output of 'mount' and 'cat /proc/mounts'
Include the output of 'mount' and 'cat /proc/mounts'
+ exit 1
update-initramfs: failed for /boot/initrd.img-4.4.0-21-generic with 1.
run-parts: /etc/kernel/postinst.d/initramfs-tools exited with return code 1
Failed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/linux-image-4.4.0-21-generic.postinst line 1052.

The bug looks to be a missing $ symbol to denote the dev_node variable in the second test:

    # recheck device
    if [ -z "$dev_node" ] || ! dev_node="$(readlink -f ${dev_node})" \
        || ! [ -b "$dev_node" ]; then

! dev_node="$(readlink -f ${dev_node})"

should, I think, be:

! $dev_node="$(readlink -f ${dev_node})"

I mis-read the code in the previous comment; the condition is fine. To figure it out I added some lines that do each test independently so as to identify which of the three fails. That reveals the problem is the block-device test:

Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 4.4.0-21-generic /boot/vmlinuz-4.4.0-21-generic
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 4.4.0-21-generic /boot/vmlinuz-4.4.0-21-generic
update-initramfs: Generating /boot/initrd.img-4.4.0-21-generic
+ local dir dev_node FSTYPE
+ local modules=
+ dir=/
+ [ ! -d /sys/devices/ ]
+ read dev mp fs opts rest
+ [ / = / ]
+ [ ext4 != rootfs ]
+ printf dev_node=/dev/sda3\nFSTYPE=ext4
+ break
+ eval dev_node=/dev/sda3
FSTYPE=ext4
+ dev_node=/dev/sda3
+ FSTYPE=ext4
+ [ / != / ]
+ [ ext4 = ubifs ]
+ [ / = / ]
+ [ /dev/sda3 = /dev/root ]
+ [ -z /dev/sda3 ]
+ echo good dev_node
good dev_node
+ readlink -f /dev/sda3
+ dev_node2=/dev/sda3
+ echo readlink returned 0 dev_node2=/dev/sda3
readlink returned 0 dev_node2=/dev/sda3
+ [ -b /dev/sda3 ]
+ echo not a block device: /dev/sda3
not a block device: /dev/sda3
+ [ -z /dev/sda3 ]
+ readlink -f /dev/sda3
+ dev_node=/dev/sda3
+ [ -b /dev/sda3 ]
+ echo mkinitramfs: failed to determine device for /
mkinitramfs: failed to determine device for /
+ echo mkinitramfs: workaround is MODULES=most, check:
mkinitramfs: workaround is MODULES=most, check:
+ echo grep -r MODULES /etc/initramfs-tools/
grep -r MODULES /etc/initramfs-tools/
+ echo

+ echo Error please report bug on initramfs-tools

The problem is the /dev/sda* nodes do not exist in the /target, presumably because the host hasn't mounted devtmpfs to /target/dev.

That should be done by a call to hook-functions::setup_dev_linux() before dep_add_modules_mount()

There's a waypoint for it in debian/bootstrap-base.postinst before the install_kernel stage.

In expert mode the kernel command-line carries "priority=low" which is a debconf configuration parameter. It suggests that it affects the order of functions called such that "setup_dev" isn't being called before "install_kernel", which would make sense because in tests after a manual "update-initramfs -c -t ..." it was not possible to reproduce the issue with "dpkg-reconfigure linux-image-4.4.0-21-generic".

After noticing /var/lib/dpkg/info/live-installer.postinst and seeing it has the crucial step commented out I think the source of the issue is now identified.

live-installer debian/live-installer.postinst:

waypoint 1 check_target
waypoint 1 get_mirror_info
waypoint 100 install_live_system
waypoint 1 pre_install_hooks
#waypoint 1 setup_dev
waypoint 1 configure_apt_preferences
waypoint 1 configure_apt
waypoint 3 apt_update
waypoint 5 post_install_hooks
waypoint 1 pick_kernel_if_missing
waypoint 20 install_kernel

The question is, is live-installer in control here, or base-installer? The syslog messages indicate base-installer but there's no sign of its postinst script in the running installer.

After ensuring the "normal" option (not "live") was selecting in the "Install the System" step I checked which processes were running when it paused to ask which kernel should be installed.

This confirms the live-installer.postinst script is running:

$ grep -Ev '\]$' ~/installer-ps.log
  PID USER VSZ STAT COMMAND
    1 root 4696 S /bin/busybox init
  117 root 16648 S /lib/systemd/systemd-udevd --daemon --resolve-names=never
  161 root 4700 S /sbin/syslogd -m 0 -O /var/log/syslog -S
  163 root 4696 S /sbin/klogd -c 2
  222 root 4696 S {debian-installe} /bin/sh /sbin/debian-installer
  224 root 4696 S -/bin/sh
  225 root 4696 S /bin/busybox init
  226 root 4696 S /usr/bin/tail -f /var/log/syslog
  248 root 8696 S /usr/bin/bterm -f /lib/unifont.bgf -l C.UTF-8 /lib/debian-installer/menu
  249 root 56140 S debconf -o d-i /usr/bin/main-menu
  255 root 8660 S /usr/bin/main-menu
19469 root 6604 S udpkg --configure --force-configure live-installer
19470 root 4828 S {live-installer.} /bin/sh /var/lib/dpkg/info/live-installer.postinst con
20828 root 4700 R ps

Should 'normal' have caused base-installer to be installed into the live host before initiating the target install?

Status changed to 'Confirmed' because the bug affects multiple users.

nicely researched and documented bug. Why not fixed three years later? I hit this (or seeminly this same bug) doing an install of server 18.04.2 yesterday. I needed to enable ssh for remote install and lowered debconf priority. This gave me some good clues I'll now try some work-arounds.

This bug has a lot of troubleshooting comments, but lacks a good summary about what is going on. I don't expect the xenial installer to be changed anymore, short of critical bugs affecting it.

@tcstone, you say you hit something similar with 18.04.2. Current bionic release is 18.04.3, would you mind to try again, and if you still hit a bug, file a new one please? Please note that the default server install in bionic is subiquity ("live-server"), and bugs against that should be filed via https://bugs.launchpad.net/subiquity/+filebug.

I'm marking this as incomplete for 18.04.2 and wont fix for xenial based on @ahasenack's last input.

Thank you for taking the time to file a bug report.

Since there is not enough information in your report to begin triage or to
differentiate between a local configuration problem and a bug in Ubuntu, I
am marking this bug as "Incomplete". We would be grateful if you would:
provide a more complete description of the problem, explain why you
believe this is a bug in Ubuntu rather than a problem specific to your
system, and then change the bug status back to "New".

For local configuration issues, you can find assistance here:
http://www.ubuntu.com/support/community

Hi,

I came across this old bug while doing some clearance.

Xenial reached EOSS (End of Standard Support) so I'm afraid it can't be fixed. It is unfortunate that we were unable to resolve this defect, however, there appears to be no further action possible at this time because it's not a security bug. We are sorry that we do not always have the capacity to look at all reported bugs in a timely manner. There have been many changes in Ubuntu since the time you reported the bug and your problem may have been fixed with some of the updates.

As far as I could read in the thread, after a lot of research work (great @tj, thankyou!) and after some doubt as to whether it was the base installer or the live-installer that was the culprit here, the last suggestion was about live-installer. As it's already marked as "Incomplete" at the time I'm writing this, I'm going to mark "Incomplete" as well with respect to base-installer and initramfs-tools. If you consider it's happening still on Bionic or Focal (since Focal is lat time *-installer are present on the series) and you have more information that can shed light on this issue, please feel free to comment and to add it and mark again this bug as "New".

[Expired for base-installer (Ubuntu Bionic) because there has been no activity for 60 days.]

[Expired for live-installer (Ubuntu) because there has been no activity for 60 days.]

[Expired for live-installer (Ubuntu Bionic) because there has been no activity for 60 days.]

[Expired for initramfs-tools (Ubuntu Bionic) because there has been no activity for 60 days.]