system doesn't properly boot as expected if /usr is on its own LV
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| initramfs-tools (Ubuntu) |
Won't Fix
|
High
|
Unassigned | ||
| Xenial |
Won't Fix
|
Undecided
|
Unassigned | ||
| Bionic |
Won't Fix
|
Undecided
|
Unassigned | ||
| Disco |
Won't Fix
|
Undecided
|
Unassigned | ||
| lvm2 (Ubuntu) |
Fix Released
|
Undecided
|
Unassigned | ||
| Xenial |
Fix Released
|
Undecided
|
Unassigned | ||
| Bionic |
Fix Released
|
Medium
|
Eric Desrochers | ||
| Disco |
Fix Released
|
Medium
|
Eric Desrochers | ||
Bug Description
[Impact]
In summary, the problem is a LVM 'auto-activation' issue. Basically the LVM activation doesn't occurs during the boot process (except for LVM root partition) and since /usr is an important component before the 'pivot'[0], the init inside the initramfs space tries to mount /usr. If /usr is a LVM volume then it will try to mount it while its backend device is not activated yet (due to the actual auto-activation issue), thus the mounting will fails and bring the user to the initramfs prompt for debugging.
Workaround:
At the initramfs prompt: lvm vgchange -ay and then 'ctrl-d' to resume the boot process.
[Test Case]
- Install Bionic or Disco
- Create a VG with /usr on its on LV
- Complete the installation
- At reboot the system will be stuck at mounting /usr file system
Extra notes:
Since this problem is ONLY related to LVM activation. If one create a non-LVM /usr separate partition, everything will work as expected.
Other separate LVM (e.g. /home, /var/, /srv) if on its own separate LVM volume will not be activate either, but since they don't need to be mounted in the initramfs space, it's not a problem, they can be activated later on (after the pivot) by systemd.
So the only combination possible is when /usr is on its separate LV only.
[Regression potential]
I don't foresee any regression, the fix will instruct udev rule to pvscan activate volume based on their major/minor number if LVM is scanned.
+RUN+="
The current situation doesn't auto-activate at all.
One downside, for users who will see this fixed, will still experience the situation in standard ISO, since I'm afraid Disco won't produces new ISO. (it is not recommended to do new disco installs anyway since it's going EOL shortly) One would need to rely on the mini.iso for fetching the latest lvm2 package in the archive.
IIRC Bionic will have a new point release somewhere in Feb 2020 but until then one would need to rely on the mini.iso for fetching the latest lvm2 package in the archive as well.
or use the workaround in the [Impact] section and then apt-get dist-upgrade in order to get the latest lvm2.
[Other information]
This problem only exhibit in Bionic and Disco.
Xenial and Eoan and late didn't exhibit the situation.
https:/
https:/
[Original Description]
Only the lv for root volume get activated, because of the grub parameter that specifies/enforce it "root=/
http://
'root=...'
This argument tells the kernel what device is to be used as
the root filesystem while booting.
If one add a separate LV for /usr, the system will go straight to initramfs prompt failling to mount /usr.
At initramfs prompt, we notice that 'lv-usr' status is 'NOT available'.
Performing 'lvm vgchange -ay' at initramfs prompt workaround the problem and allow one to successfully boot.
Adding 'debug' parameter, we clearly we see /root being detected and mounted:
initramfs.debug:
=> + mount -r -t ext4 /dev/mapper/
=> + mountroot_status=0
+ [ ]
+ log_end_msg
+ _log_msg done.\n
+ [ n = y ]
+ printf done.\n
done.
+ read_fstab_entry /usr
+ found=1
+ [ -f /root/etc/fstab ]
+ read MNT_FSNAME MNT_DIR MNT_TYPE MNT_OPTS MNT_FREQ MNT_PASS MNT_JUNK
+ [ / = /usr ]
+ read MNT_FSNAME MNT_DIR MNT_TYPE MNT_OPTS MNT_FREQ MNT_PASS MNT_JUNK
+ [ /usr = /usr ]
+ [ -n ]
+ found=0
+ break 2
+ return 0
+ log_begin_msg Mounting /usr file system
+ _log_msg Begin: Mounting /usr file system ...
then the code read /etc/fstab and specifically search for /usr (most likely because of the /usr binary merged) and try to mount if if found.
initramfs-
271 maybe_break mount
272 log_begin_msg "Mounting root file system"
273 # Always load local and nfs (since these might be needed for /etc or
274 # /usr, irrespective of the boot script used to mount the rootfs).
275 . /scripts/local
276 . /scripts/nfs
277 . /scripts/${BOOT}
278 parse_numeric "${ROOT}"
279 maybe_break mountroot
280 mount_top
281 mount_premount
282 mountroot
283 log_end_msg
284
=> 285 if read_fstab_entry /usr; then
=> 286 log_begin_msg "Mounting /usr file system"
=> 287 mountfs /usr
=> 288 log_end_msg
=> 289 fi
In this case, /usr is present in /etc/fstab but the logical volume is not available, so it is mounting a filesystem without his backend device, thus goes straight to the initramfs prompt because /usr couldn't be mounted.
It clearly seems to be an 'auto-activation' issue at boot.
For other such as /home, /var, it's not a big deal cause they can be activated later on in the process and they are (I haven't check but I guess systemd or systemd unit/generator is taking care of it at some point), but /usr is a important piece to have mounted before the pivot since it contains most of the crucial binary, especially that nowadays /sbin, /bin, and /lib are pointing to /usr:
lrwxrwxrwx 1 root root 10 Aug 26 13:51 libx32 -> usr/libx32
lrwxrwxrwx 1 root root 9 Aug 26 13:51 lib64 -> usr/lib64
lrwxrwxrwx 1 root root 9 Aug 26 13:51 lib32 -> usr/lib32
lrwxrwxrwx 1 root root 7 Aug 26 13:51 lib -> usr/lib
lrwxrwxrwx 1 root root 7 Aug 26 13:51 bin -> usr/bin
lrwxrwxrwx 1 root root 8 Aug 26 13:51 sbin -> usr/sbin
NOTE:
* That doesn't affect /usr found in /etc/fstab on its separate partition when no LVM involve (e.g. /dev/vdb /usr ext4 ....). It only cause issue when /usr is in a LVM context.
* I was able to reproduce on Bionic and Disco so far.
Eoan doesn't seem to exhibit the situation so far in my testing.
* While certain release such as Bionic, Xenial doesn't come implicitly with the /usr merge approach. One can install package 'usrmerge' and convert their system /usr merged. I don't think removing the read_fstable_entry for /usr is an option here, as some user could potentially decide to convert their system with 'usrmerge' pkg.
| tags: | added: sts |
| Eric Desrochers (slashd) wrote | #1 |
| Eric Desrochers (slashd) wrote | #2 |
| description: | updated |
| Changed in lvm2 (Ubuntu): | |
| importance: | Undecided → High |
| Changed in initramfs-tools (Ubuntu): | |
| importance: | Undecided → High |
| description: | updated |
| description: | updated |
| description: | updated |
| description: | updated |
| Eric Desrochers (slashd) wrote | #5 |
| Changed in initramfs-tools (Ubuntu Disco): | |
| status: | New → Won't Fix |
| Changed in initramfs-tools (Ubuntu Bionic): | |
| status: | New → Won't Fix |
| Changed in initramfs-tools (Ubuntu Xenial): | |
| status: | New → Won't Fix |
| Changed in initramfs-tools (Ubuntu): | |
| status: | New → Won't Fix |
| Changed in lvm2 (Ubuntu): | |
| status: | New → Fix Released |
| Changed in lvm2 (Ubuntu Disco): | |
| status: | New → Confirmed |
| Changed in lvm2 (Ubuntu Bionic): | |
| status: | New → Confirmed |
| importance: | Undecided → Medium |
| Changed in lvm2 (Ubuntu Disco): | |
| importance: | Undecided → Medium |
| Changed in lvm2 (Ubuntu): | |
| importance: | High → Undecided |
| Changed in lvm2 (Ubuntu Xenial): | |
| status: | New → Won't Fix |
| description: | updated |
| description: | updated |
| description: | updated |
| description: | updated |
| Changed in lvm2 (Ubuntu Bionic): | |
| status: | Confirmed → In Progress |
| Changed in lvm2 (Ubuntu Disco): | |
| status: | Confirmed → In Progress |
| assignee: | nobody → Eric Desrochers (slashd) |
| Changed in lvm2 (Ubuntu Bionic): | |
| assignee: | nobody → Eric Desrochers (slashd) |
| Eric Desrochers (slashd) wrote | #6 |
| description: | updated |
| Changed in lvm2 (Ubuntu Xenial): | |
| status: | Won't Fix → Fix Released |
| Łukasz Zemczak (sil2100) wrote Please test proposed package | #7 |
| Changed in lvm2 (Ubuntu Disco): | |
| status: | In Progress → Fix Committed |
| tags: | added: verification-needed verification-needed-disco |
| Changed in lvm2 (Ubuntu Bionic): | |
| status: | In Progress → Fix Committed |
| tags: | added: verification-needed-bionic |
| Łukasz Zemczak (sil2100) wrote | #8 |
| description: | updated |
| description: | updated |
| Ubuntu SRU Bot (ubuntu-sru-bot) wrote Autopkgtest regression report (lvm2/2.02.176-4.1ubuntu4.2) | #9 |
| Ubuntu SRU Bot (ubuntu-sru-bot) wrote Autopkgtest regression report (lvm2/2.02.176-4.1ubuntu3.18.04.2) | #10 |
| Eric Desrochers (slashd) wrote | #11 |
| tags: |
added: verification-done-bionic removed: verification-needed-bionic |
| Eric Desrochers (slashd) wrote | #13 |
| Eric Desrochers (slashd) wrote | #14 |
| tags: |
added: verification-done-disco removed: verification-needed-disco |
| Łukasz Zemczak (sil2100) wrote Update Released | #15 |
| Launchpad Janitor (janitor) wrote | #16 |
| Changed in lvm2 (Ubuntu Bionic): | |
| status: | Fix Committed → Fix Released |
| Launchpad Janitor (janitor) wrote | #17 |
| Changed in lvm2 (Ubuntu Disco): | |
| status: | Fix Committed → Fix Released |
For context about the /usr merge:
https:/
https:/