root partition is not being grown

Bug #906722 reported by Scott Moser
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
cloud-initramfs-tools (Ubuntu)
Fix Released
High
Scott Moser
util-linux (Ubuntu)
Invalid
High
Canonical Foundations Team

Bug Description

Upon booting an openstack instance of precise-i386-20111219-failed-console.log, I found that the root filesystem had not been resized as it normally would have been.
This occurred on both i386 and amd64.

Kernel output contains something like:
Begin: Running /scripts/local-bottom ... [ 1.852884] FDC 0 is a S82078B
[ 1.862945] 8139cp 0000:00:03.0: eth0: RTL-8139C+ at 0xf840a000, 02:16:3e:21:8a:47, IRQ 11
[ 1.877189] 8139too: 8139too Fast Ethernet driver 0.9.28
***** WARNING: Resize failed, attempting to revert ******
BLKRRPART: Device or resource busy
The command to re-read the partition table failed.
Run partprobe(8), kpartx(8) or reboot your system now,
before using mkfs
***** Appears to have gone OK ****
GROWROOT: FAILED: failed to resize
Re-reading the partition table ...
done.
done.
Begin: Running /scripts/init-bottom ... done.

It should show something like:

Begin: Running /scripts/local-bottom ... [ 1.314676] 8139cp 0000:00:03.0: eth0: RTL-8139C+ at 0xf840a000, 02:16:3e:3e:94:65, IRQ 11
[ 1.320377] FDC 0 is a S82078B
[ 1.335493] 8139too: 8139too Fast Ethernet driver 0.9.28
GROWROOT: CHANGED: partition=1 start=16065 old: size=4176900 end=4192965 new: size=20948760,end=20964825
[ 1.372287] vda: vda1
[ 1.398773] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null)
done.
done.
Begin: Running /scripts/init-bottom ... done.

This worked as recently as ubuntu-precise-daily-i386-server-20111213.

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: cloud-initramfs-growroot 0.2ubuntu1
ProcVersionSignature: User Name 3.2.0-5.11-virtual 3.2.0-rc5
Uname: Linux 3.2.0-5-virtual x86_64
ApportVersion: 1.90-0ubuntu1
Architecture: amd64
Date: Tue Dec 20 05:25:48 2011
Ec2AMI: ami-0000011f
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.small
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
PackageArchitecture: all
ProcEnviron:
 PATH=(custom, user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: cloud-initramfs-tools
UpgradeStatus: No upgrade log present (probably fresh install)

Revision history for this message
Scott Moser (smoser) wrote :
Revision history for this message
Scott Moser (smoser) wrote :
Revision history for this message
Scott Moser (smoser) wrote :
Revision history for this message
Scott Moser (smoser) wrote :
Revision history for this message
Scott Moser (smoser) wrote :
Revision history for this message
Scott Moser (smoser) wrote :

I just verified that 'sfdisk -d' of "good" matches identically to that reported by 'sudo growpart /dev/vda 1 --dry-run'

So it doesn't really look like its related to bad math or parsing.

Revision history for this message
Scott Moser (smoser) wrote :

modifying growpart to not capture sfdisk output, I saw:
Checking that no-one is using this disk right now ...
BLKRRPART: Device or resource busy

This disk is currently in use - repartitioning is probably a bad idea.
Umount all file systems, and swapoff all swap partitions on this disk.
Use the --no-reread flag to suppress this check.
BLKRRPART: Device or resource busy

So, somehow the partition or disk is seen as busy.

Revision history for this message
Scott Moser (smoser) wrote :

I've not yet confirmed, but looking at git commits to util-linux from 2.19.1 to 2.20.1 reveals this one:
https://github.com/karelzak/util-linux/commit/e82810c6cf3fe94d2cf70d485a59c411ccb601b3#fdisk/sfdisk.c

The disk is definitely busy when we try to rewrite the parttion table in the initramfs.
The reason it is not unmounted is get out in the common (other-than-first-boot) path when the disk is not going to be updated.

2 things to look at:
 - --no-reread flag
 - make growroot support writing "what would you do" output, which growroot in cloud-initramfs-tools would then use.
   if it said "nothing to do", then nothing done. Otherwise it would unmount disk, do what would have been done and then continue on.

Changed in util-linux (Ubuntu):
assignee: nobody → Canonical Foundations Team (canonical-foundations)
importance: Undecided → High
Revision history for this message
Steve Langasek (vorlon) wrote :

Doesn't look to me like any changes to util-linux are called for here. I think cloud-init should just have a versioned dependency on util-linux (>= 2.20) and call sfdisk with the --no-reread option, since that's what you want. (You know that the disk is always going to be mounted in this case, and that you always want to resize if necessary, so this check doesn't seem to serve any purpose here.)

If that doesn't work for whatever reason, feel free to reopen against util-linux of course.

Changed in util-linux (Ubuntu):
status: New → Invalid
Revision history for this message
Scott Moser (smoser) wrote :

I'll fix this in cloud-initramfs-tools because its just cleaner to do there. I was calling growpart while the disk was mounted simply to optimize the case where there was a change to be made (which happens only the first boot, really).

Instead, I'm just going to call growpart once with '--debug' and if it would not make a change, then exit as before.
If it *would* make a change, then unmount / and call it again to make the change.

 but there are some issues with sfdisk, and the update made it worse.
 * --no-reread option exists in older versions of util-linux (at least in 2.19)
 * --no-reread and --force with a mounted filesystem still exits failure in 2.20, it did not in 2.19

Partition is mounted and disk not resized (system booted) at this point.

$ growpart /dev/vda 1 --dry-run 2>out
CHANGE: partition=1 start=16065 old: size=4176900 end=4192965 new: size=20948760,end=20964825
$ sed -n '/new sfdisk/,$p' err > new
$ cat new
# === new sfdisk -d ===
# partition table of /dev/vda
unit: sectors

/dev/vda1 : start= 16065, size= 20948760, Id=83, bootable
/dev/vda2 : start= 0, size= 0, Id= 0
/dev/vda3 : start= 0, size= 0, Id= 0
/dev/vda4 : start= 0, size= 0, Id= 0

$ sudo sfdisk /dev/vda --force --no-reread < new; echo $?
....
Successfully wrote the new partition table
Re-reading the partition table ...
BLKRRPART: Device or resource busy
...
1

Previously if we did --force it would exit success (i guess in all cases).

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-initramfs-tools - 0.3ubuntu1

---------------
cloud-initramfs-tools (0.3ubuntu1) precise; urgency=low

  * growroot: unmount root before attempting to grow the root
    partition (LP: #906722).
 -- Scott Moser <email address hidden> Tue, 20 Dec 2011 13:24:45 -0500

Changed in cloud-initramfs-tools (Ubuntu):
status: Confirmed → Fix Released
Revision history for this message
Steve Langasek (vorlon) wrote :

Right, the change in behavior is due to commit e82810c6cf3fe94d2cf70d485a59c411ccb601b3 upstream; prior to this, sfdisk evidently was *never* noticing a failure of the partition rereading. And '--no-reread' is only defined to suppress checking if the disk is in use, it says nothing about suppressing the attempt to force a reread of the partition table afterwards.

Likewise, --force is defined as disabling consistency checking, allowing you to override problems that sfdisk would normally complain about and continue with an operation that it considers unsafe. In this case, the failed ioctl doesn't prevent sfdisk from making any changes, it *only* impacts the error code from sfdisk at the end.

So I think the behavior of 2.20 is consistent and should not be changed, even though the behavior differed in the past.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.