update-grub-legacy-ec2 fails to detect xen kernel

Bug #1379080 reported by evan2645
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
cloud-init (Ubuntu)
Fix Released
Medium
Unassigned
Trusty
Fix Released
Undecided
Unassigned
Xenial
Fix Released
Medium
Unassigned
Yakkety
Fix Released
Medium
Unassigned
Zesty
Fix Released
Medium
Unassigned

Bug Description

The update-grub-legacy-ec2 script (which ships with grub-legacy-ec2, which is presumably included under cloud-init umbrella) includes a check to determine whether a kernel is Xen-capable or not. It uses this check in a feature designed to ignore non-Xen kernels on Xen guests, presumably as a safety mechanism. The way in which the check is executed is flawed - it matches against the kernel name. This means that if you are loading a kernel that has an 'unexpected' name, update-grub-legacy-ec2 will ignore the new kernel, regardless of whether or not it is actually Xen-capable. Output looks like this:

Searching for GRUB installation directory ... found: /boot/grub
Searching for default file ... found: /boot/grub/default
Testing for an existing GRUB menu.lst file ... found: /boot/grub/menu.lst
Searching for splash image ... none found, skipping ...
Ignoring non-Xen Kernel on Xen domU host: vmlinuz-3.13.0-36-custom
Found kernel: /boot/vmlinuz-3.13.0-36-generic
Updating /boot/grub/menu.lst ... done

This has been fixed on grub-legacy. Instead of matching the kernel name, it checks the config for CONFIG_XEN=y. In addition, Linux kernels come with CONFIG_XEN=y by default for some time now, and it is perfectly safe to run a Xen-capable kernel on non-Xen hosts.

Steps to reproduce:
1. Compile and generate debs for custom kernel with version string not including '-generic' or '-virtual'
2. Install debs on ec2 system (ami-a94e0c99)
3. Reboot

Expected results:
System comes up with new kernel

Actual results:
System comes up with old kernel, and new kernel is not present in /boot/grub/menu.lst

Impact:
Running ami-a94e0c99, and likely many others, a user can not install a custom kernel without manually editing files. Furthermore, those writes will likely be overwritten by administrative operations in the future.

Recommendation:
Update detection logic to search for CONFIG_XEN=y in the kernel config, and remove version string detection.

Please let me know how I can help this along. I can submit a proposed fix, but am not sure where to send it.

Related bugs:
 * bug 1655934: update-grub-legacy-ec2 should detect -aws kernels as appropriate for EC2

Related branches

Robie Basak (racb)
Changed in cloud-init (Ubuntu):
importance: Undecided → Medium
Revision history for this message
Scott Moser (smoser) wrote :

fwiw, if you put an executable program 'is_xen_kernel' anywhere in PATH when update-grub-legacy-ec2 is running, it will be used instead of the is_xen_kernel inside.

So, to make this work without any protection is just like:

  ln -s /usr/bin/is_xen_kernel /bin/true

Scott Moser (smoser)
Changed in cloud-init (Ubuntu Zesty):
status: New → Fix Committed
Changed in cloud-init (Ubuntu Yakkety):
status: New → Confirmed
Changed in cloud-init (Ubuntu Xenial):
status: New → Confirmed
Changed in cloud-init (Ubuntu Yakkety):
importance: Undecided → Medium
Changed in cloud-init (Ubuntu Xenial):
importance: Undecided → Medium
description: updated
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello evan2645, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in cloud-init (Ubuntu Xenial):
status: Confirmed → Fix Committed
tags: added: verification-needed
Revision history for this message
Brian Murray (brian-murray) wrote :

Hello evan2645, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-0ubuntu1~16.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Revision history for this message
Scott Moser (smoser) wrote :

## launched a xenial instance
$ cat /etc/cloud/build.info
build_name: server
serial: 20170207

## Enable proposed
$ rel=$(lsb_release -sc)
$ line=$(awk '$1 == "deb" && $2 ~ /ubuntu.com/ {
  printf("%s %s %s-proposed main universe\n", $1, $2, rel); exit(0) };
  ' "rel=$rel" /etc/apt/sources.list)
$ echo "$line" | sudo tee /etc/apt/sources.list.d/proposed.list
$ sudo apt-get update -qy
$ sudo apt-get install -qy cloud-init grub-legacy-ec2

## copy -generic to -ec2
$ sudo sh -c 'for i in /boot/*-generic; do n=${i%-generic}-ec2; mv -v $i $n; done'
'/boot/System.map-4.4.0-62-generic' -> '/boot/System.map-4.4.0-62-ec2'
'/boot/abi-4.4.0-62-generic' -> '/boot/abi-4.4.0-62-ec2'
'/boot/config-4.4.0-62-generic' -> '/boot/config-4.4.0-62-ec2'
'/boot/initrd.img-4.4.0-62-generic' -> '/boot/initrd.img-4.4.0-62-ec2'
'/boot/vmlinuz-4.4.0-62-generic' -> '/boot/vmlinuz-4.4.0-62-ec2'

## run update-legacy-grub-ec2
$ sudo update-grub-legacy-ec2

Searching for GRUB installation directory ... found: /boot/grub
Searching for default file ... found: /boot/grub/default
Testing for an existing GRUB menu.lst file ... found: /boot/grub/menu.lst
Searching for splash image ... none found, skipping ...
Found kernel: /boot/vmlinuz-4.4.0-62-generic
Replacing config file /run/grub/menu.lst with new version
Found kernel: /boot/vmlinuz-4.4.0-62-ec2
Replacing config file /run/grub/menu.lst with new version
Updating /boot/grub/menu.lst ... done

tags: added: verification-done
removed: verification-needed
Changed in cloud-init (Ubuntu Zesty):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Hello evan2645, or anyone else affected,

Accepted cloud-init into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-0ubuntu1~16.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in cloud-init (Ubuntu Yakkety):
status: Confirmed → Fix Committed
tags: removed: verification-done
tags: added: verification-needed
Jon Grimm (jgrimm)
tags: added: verification-done-xenial verification-needed-yakkety
removed: verification-needed
Revision history for this message
Scott Moser (smoser) wrote :

## launched a yakkety instance
$ cat /etc/cloud/build.info
build_name: server
serial: 20170207

$ lsb_release -sc
yakkety

## Enable proposed
$ rel=$(lsb_release -sc)
$ line=$(awk '$1 == "deb" && $2 ~ /ubuntu.com/ {
    printf("%s %s %s-proposed main universe\n", $1, $2, rel); exit(0) };
    ' "rel=$rel" /etc/apt/sources.list)
$ echo "$line" | sudo tee /etc/apt/sources.list.d/proposed.list
deb http://nova.clouds.archive.ubuntu.com/ubuntu/ yakkety-proposed main universe

$ sudo apt-get update -qy
$ sudo apt-get install -qy grub-legacy-ec2

$ dpkg-query show grub-legacy-ec2
grub-legacy-ec2 0.7.9-0ubuntu1~16.10.1

## copy -generic to -ec2
$ sudo sh -c 'for i in /boot/*-generic; do n=${i%-generic}-ec2; mv -v $i $n; done'
'/boot/System.map-4.8.0-37-generic' -> '/boot/System.map-4.8.0-37-ec2'
'/boot/abi-4.8.0-37-generic' -> '/boot/abi-4.8.0-37-ec2'
'/boot/config-4.8.0-37-generic' -> '/boot/config-4.8.0-37-ec2'
'/boot/initrd.img-4.8.0-37-generic' -> '/boot/initrd.img-4.8.0-37-ec2'
'/boot/vmlinuz-4.8.0-37-generic' -> '/boot/vmlinuz-4.8.0-37-ec2'

## run update-legacy-grub-ec2
## Note that we see the '-ec2' kernels, before this change they
## wouldn't be considered.
$ sudo update-grub-legacy-ec2
Searching for GRUB installation directory ... found: /boot/grub
Searching for default file ... found: /boot/grub/default
Testing for an existing GRUB menu.lst file ... found: /boot/grub/menu.lst
Searching for splash image ... none found, skipping ...
Found kernel: /boot/vmlinuz-4.8.0-37-generic
Replacing config file /run/grub/menu.lst with new version
Found kernel: /boot/vmlinuz-4.8.0-37-ec2
Replacing config file /run/grub/menu.lst with new version
Updating /boot/grub/menu.lst ... done

tags: added: verification-done-yakkety
removed: verification-needed-yakkety
Revision history for this message
evan2645 (evan2645) wrote :

Hey all - great to see this getting addressed!

Unfortunately, I'm no longer in a position to test the fix :(. Apologies. Reading through everything though, it seems that it should be effective.

Thanks for all your work!

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 0.7.9-0ubuntu1~16.10.1

---------------
cloud-init (0.7.9-0ubuntu1~16.10.1) yakkety; urgency=medium

  * debian/copyright: update License field to include Apache-2.0
  * debian/update-grub-legacy-ec2: fix to include kernels whose config
    has CONFIG_XEN=y (LP: #1379080).
  * debian/update-grub-legacy-ec2: detect kernels ending in -aws as
    ec2 bootable (LP: #1655934).
  * New upstream release.
    - doc: adjust headers in tests documentation for consistency.
    - pep8: fix issue found in zesty build with pycodestyle.
    - integration test: initial commit of integration test framework
      [Wesley Wiedenmeier]
    - LICENSE: Allow dual licensing GPL-3 or Apache 2.0 [Jon Grimm]
    - Fix config order of precedence, putting kernel command line over system.
      [Wesley Wiedenmeier] (LP: #1582323)
    - pep8: whitespace fix
    - Update the list of valid ssh keys. [Michael Felt]
    - network: add ENI unit test for statically rendered routes.
    - set_hostname: avoid erroneously appending domain to fqdn
      [Lars Kellogg-Stedman] (LP: #1647910)
    - doc: change 'nobootwait' to 'nofail' in docs [Anhad Jai Singh]
    - Replace an expired bit.ly link in code comment. [Joshua Harlow]

 -- Scott Moser <email address hidden> Tue, 31 Jan 2017 21:02:28 -0500

Changed in cloud-init (Ubuntu Yakkety):
status: Fix Committed → Fix Released
Revision history for this message
Chris Halse Rogers (raof) wrote : Update Released

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 0.7.9-0ubuntu1~16.04.2

---------------
cloud-init (0.7.9-0ubuntu1~16.04.2) xenial-proposed; urgency=medium

  * debian/update-grub-legacy-ec2: fix shell syntax error. (LP: #1662221)

cloud-init (0.7.9-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  * debian/copyright: update License field to include Apache.
  * debian/update-grub-legacy-ec2: fix to include kernels whose config
    has CONFIG_XEN=y (LP: #1379080).
  * debian/patches/azure-use-walinux-agent.patch: continue relying on
    walinux agent in stable release.
  * New upstream release.
    - doc: adjust headers in tests documentation for consistency.
    - pep8: fix issue found in zesty build with pycodestyle.
    - integration test: initial commit of integration test framework
      [Wesley Wiedenmeier]
    - LICENSE: Allow dual licensing GPL-3 or Apache 2.0 [Jon Grimm]
    - Fix config order of precedence, putting kernel command line over system.
      [Wesley Wiedenmeier] (LP: #1582323)
    - pep8: whitespace fix [Scott Moser]
    - Update the list of valid ssh keys. [Michael Felt]
    - network: add ENI unit test for statically rendered routes.
    - set_hostname: avoid erroneously appending domain to fqdn
      [Lars Kellogg-Stedman] (LP: #1647910)
    - doc: change 'nobootwait' to 'nofail' in docs [Anhad Jai Singh]
    - Replace an expired bit.ly link in code comment. [Joshua Harlow]
    - user-groups: fix bug when groups was provided as string and had spaces
      [Scott Moser] (LP: #1354694)
    - when adding a user, strip whitespace from group list
      [Lars Kellogg-Stedman] (LP: #1354694)
    - fix decoding of utf-8 chars in yaml test
    - Replace usage of sys_netdev_info with read_sys_net
      [Joshua Harlow] (LP: #1625766)
    - fix problems found in python2.6 test. [Joshua Harlow]
    - Just use file logging by default [Joshua Harlow] (LP: #1643990)
    - Improve formatting for ProcessExecutionError [Wesley Wiedenmeier]
    - flake8: fix trailing white space
    - Doc: various documentation fixes [Sean Bright]
    - cloudinit/config/cc_rh_subscription.py: Remove repos before adding
      [Brent Baude]
    - packages/redhat: fix rpm spec file.
    - main: set TZ in environment if not already set. [Ryan Harper]

 -- Scott Moser <email address hidden> Mon, 06 Feb 2017 16:18:28 -0500

Changed in cloud-init (Ubuntu Xenial):
status: Fix Committed → Fix Released
Revision history for this message
Robie Basak (racb) wrote : Please test proposed package

Hello evan2645, or anyone else affected,

Accepted cloud-init into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.5-0ubuntu1.22 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in cloud-init (Ubuntu Trusty):
status: New → Fix Committed
tags: added: verification-needed
Revision history for this message
Kamal Mostafa (kamalmostafa) wrote :

I have verified that cloud-init (0.7.5-0ubuntu1.22) in trusty -proposed does fix the problem.

tags: added: verification-done-trusty
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 0.7.5-0ubuntu1.22

---------------
cloud-init (0.7.5-0ubuntu1.22) trusty; urgency=medium

  * debian/update-grub-legacy-ec2:
    - Correctly detect kernels ending in -aws as kernels that can boot on EC2
      (LP: #1655934)
    - Consider kernels with CONFIG_XEN=y bootable on aws. (LP: #1379080)

 -- Scott Moser <email address hidden> Thu, 08 Jun 2017 19:25:41 -0400

Changed in cloud-init (Ubuntu Trusty):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.