Ubuntu
cloud-utils package

[yakkety regression] does not grow partition any more

Bug #1587188 reported by Martin Pitt on 2016-05-30

This bug report is a duplicate of: Bug #1587971: recent versions of util-linux do not like how growpart calls partx. Edit Remove

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	cloud-utils (Ubuntu)	Confirmed	Undecided	Unassigned

Bug Description

When booting the current yakkety cloud images (I tested http://cloud-images.ubuntu.com/yakkety/20160529/) in QEMU, resizing the file system now fails. The root file system is only 2 GB, which causes ENOSPC errors fairly quickly:

/dev/vda1 2.0G 1.1G 902M 55% /

but "fdisk -l /dev/vda" actually shows a proper ~ 22 GB partition, as intended:

Disk /dev/vda: 22.2 GiB, 23836229632 bytes, 46555136 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 74C0F76C-A422-4619-A9C7-78A51EEB1F91

Device Start End Sectors Size Type
/dev/vda1 227328 46555102 46327775 22.1G Linux filesystem
/dev/vda14 2048 10239 8192 4M BIOS boot
/dev/vda15 10240 227327 217088 106M EFI System

Relevant log entry from cloud-init-output.log:

2016-05-30 21:52:57,488 - util.py[WARNING]: Failed: growpart /dev/vda 1

Relevant log entries from cloud-init.log (full log attached):

May 30 21:52:57 ubuntu [CLOUDINIT] stages.py[DEBUG]: Running module growpart (<module 'cloudinit.config.cc_growpart' from '/usr/lib/pyth
on3/dist-packages/cloudinit/config/cc_growpart.py'>) with frequency always
May 30 21:52:57 ubuntu [CLOUDINIT] handlers.py[DEBUG]: start: init-network/config-growpart: running config-growpart with frequency always
May 30 21:52:57 ubuntu [CLOUDINIT] helpers.py[DEBUG]: Running config-growpart using lock (<cloudinit.helpers.DummyLock object at 0x7f9dbdb422e8>)
May 30 21:52:57 ubuntu [CLOUDINIT] cc_growpart.py[DEBUG]: No 'growpart' entry in cfg. Using default: {'ignore_growroot_disabled': False, 'devices': ['/'], 'mode': 'auto'}
May 30 21:52:57 ubuntu [CLOUDINIT] util.py[DEBUG]: Running command ['growpart', '--help'] with allowed return codes [0] (shell=False, capture=True)
May 30 21:52:57 ubuntu [CLOUDINIT] util.py[DEBUG]: Reading from /proc/959/mountinfo (quiet=False)
May 30 21:52:57 ubuntu [CLOUDINIT] util.py[DEBUG]: Read 4334 bytes from /proc/959/mountinfo
May 30 21:52:57 ubuntu [CLOUDINIT] util.py[DEBUG]: Reading from /sys/class/block/vda1/partition (quiet=False)
May 30 21:52:57 ubuntu [CLOUDINIT] util.py[DEBUG]: Read 2 bytes from /sys/class/block/vda1/partition
May 30 21:52:57 ubuntu [CLOUDINIT] util.py[DEBUG]: Reading from /sys/devices/pci0000:00/0000:00:04.0/virtio1/block/vda/dev (quiet=False)
May 30 21:52:57 ubuntu [CLOUDINIT] util.py[DEBUG]: Read 6 bytes from /sys/devices/pci0000:00/0000:00:04.0/virtio1/block/vda/dev
May 30 21:52:57 ubuntu [CLOUDINIT] util.py[DEBUG]: Running command ['growpart', '--dry-run', '/dev/vda', '1'] with allowed return codes [0] (shell=False, capture=True)
May 30 21:52:57 ubuntu [CLOUDINIT] util.py[DEBUG]: Running command ['growpart', '/dev/vda', '1'] with allowed return codes [0] (shell=False, capture=True)
May 30 21:52:57 ubuntu [CLOUDINIT] util.py[WARNING]: Failed: growpart /dev/vda 1
May 30 21:52:57 ubuntu [CLOUDINIT] util.py[DEBUG]: Failed: growpart /dev/vda 1#012Traceback (most recent call last):#012 File "/usr/lib/python3/dist-packages/cloudinit/config/cc_growpart.py", line 109, in resize#012 util.subp(["growpart", diskdev, partnum])#012 File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1704, in subp#012 cmd=args)#012cloudinit.util.ProcessExecutionError: Unexpected error while running command.#012Command: ['growpart', '/dev/vda', '1']#012Exit code: 2#012Reason: -#012Stdout: 'FAILED: pt_resize failed\n'#012Stderr: 'failed [pt_update:1] pt_update /dev/vda 1\npartx: partition and disk name do not match\n'
May 30 21:52:57 ubuntu [CLOUDINIT] util.py[DEBUG]: resize_devices took 0.084 seconds
May 30 21:52:57 ubuntu [CLOUDINIT] cc_growpart.py[DEBUG]: '/' FAILED: failed to resize: disk=/dev/vda, ptnum=1: Unexpected error while running command.#012Command: ['growpart', '/dev/vda', '1']#012Exit code: 2#012Reason: -#012Stdout: 'FAILED: pt_resize failed\n'#012Stderr: 'failed [pt_update:1] pt_update /dev/vda 1\npartx: partition and disk name do not match\n'
May 30 21:52:57 ubuntu [CLOUDINIT] handlers.py[DEBUG]: finish: init-network/config-growpart: SUCCESS: config-growpart ran successfully
May 30 21:52:57 ubuntu [CLOUDINIT] stages.py[DEBUG]: Running module resizefs (<module 'cloudinit.config.cc_resizefs' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_resizefs.py'>) with frequency always
May 30 21:52:57 ubuntu [CLOUDINIT] handlers.py[DEBUG]: start: init-network/config-resizefs: running config-resizefs with frequency always
May 30 21:52:57 ubuntu [CLOUDINIT] helpers.py[DEBUG]: Running config-resizefs using lock (<cloudinit.helpers.DummyLock object at 0x7f9dbdb42e48>)
May 30 21:52:57 ubuntu [CLOUDINIT] util.py[DEBUG]: Reading from /proc/959/mountinfo (quiet=False)
May 30 21:52:57 ubuntu [CLOUDINIT] util.py[DEBUG]: Read 4334 bytes from /proc/959/mountinfo
May 30 21:52:57 ubuntu [CLOUDINIT] cc_resizefs.py[DEBUG]: resize_info: dev=/dev/vda1 mnt_point=/ path=/
May 30 21:52:57 ubuntu [CLOUDINIT] util.py[DEBUG]: Running command ['running-in-container'] with allowed return codes [0] (shell=False, capture=True)
May 30 21:52:57 ubuntu [CLOUDINIT] util.py[DEBUG]: Running command ['lxc-is-container'] with allowed return codes [0] (shell=False, capture=True)
May 30 21:52:57 ubuntu [CLOUDINIT] util.py[DEBUG]: Reading from /proc/1/environ (quiet=False)
May 30 21:52:57 ubuntu [CLOUDINIT] util.py[DEBUG]: Read 152 bytes from /proc/1/environ
May 30 21:52:57 ubuntu [CLOUDINIT] util.py[DEBUG]: Reading from /proc/self/status (quiet=False)
May 30 21:52:57 ubuntu [CLOUDINIT] util.py[DEBUG]: Read 889 bytes from /proc/self/status
May 30 21:52:57 ubuntu [CLOUDINIT] cc_resizefs.py[DEBUG]: Resizing / (ext4) using resize2fs /dev/vda1
May 30 21:52:57 ubuntu [CLOUDINIT] util.py[DEBUG]: Running command ('resize2fs', '/dev/vda1') with allowed return codes [0] (shell=False, capture=True)
May 30 21:52:57 ubuntu [CLOUDINIT] util.py[DEBUG]: Resizing took 0.012 seconds
May 30 21:52:57 ubuntu [CLOUDINIT] cc_resizefs.py[DEBUG]: Resized root filesystem (type=ext4, val=True)
May 30 21:52:57 ubuntu [CLOUDINIT] handlers.py[DEBUG]: finish: init-network/config-resizefs: SUCCESS: config-resizefs ran successfully

ProblemType: Bug
DistroRelease: Ubuntu 16.10
Package: cloud-init (not installed)
ProcVersionSignature: Ubuntu 4.4.0-23.41-generic 4.4.10
Uname: Linux 4.4.0-23-generic x86_64
ApportVersion: 2.20.1-0ubuntu4
Architecture: amd64
CurrentDesktop: i3
Date: Mon May 30 23:58:57 2016
EcryptfsInUse: Yes
SourcePackage: cloud-init
UpgradeStatus: No upgrade log present (probably fresh install)

Tags:

Related branches

lp:~paelzer/cloud-utils/bug-1587188-fix-partx-failing-without-nr

Merged into lp:cloud-utils

Registry Administrators: Pending requested 2016-06-01

Revision history for this message

Martin Pitt (pitti) wrote on 2016-05-30:

cloud-init.log Edit (100.9 KiB, text/plain)
JournalErrors.txt Edit (156.6 KiB, text/plain; charset="utf-8")
ProcEnviron.txt Edit (112 bytes, text/plain; charset="utf-8")

tags:

added: regression-release

Revision history for this message

Martin Pitt (pitti) wrote on 2016-06-01:

To be precise, this can be reproduced easily with "adt-buildvm-ubuntu-cloud -r yakkety". This downloads the image, calls "qemu-img resize <image> +20G" (20G is the default value for --disk-size), and boots it.

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2016-06-01:

I can confirm the behavior, here the relevant log again - a bit formatted

Running command ['growpart', '--dry-run', '/dev/vda', '1']
Running command ['growpart', '/dev/vda', '1']

util.py[WARNING]: Failed: growpart /dev/vda 1
util.py[DEBUG]: Failed: growpart /dev/vda 1#012
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_growpart.py", line 109, in resize
    util.subp(["growpart", diskdev, partnum])
  File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1704, in subp
        cmd=args)
    cloudinit.util.ProcessExecutionError: Unexpected error while running command.
    Command: ['growpart', '/dev/vda', '1']
  Exit code: 2
  Reason: -
  Stdout: 'FAILED: pt_resize failed\n'
  Stderr: 'failed [pt_update:1] pt_update /dev/vda 1
           partx: partition and disk name do not match\n'
  util.py[DEBUG]: resize_devices took 0.084 seconds

Although it seems the actual partition growing worked as reported above I can also see a grown partition but no grown fs.

Changed in cloud-init (Ubuntu):
status:	New → Confirmed

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2016-06-01:

The bug is not cloud-init specific.
I was able to create a yakkety VM with an extra partition not consuming the full size.

Running growpart on it behaves just the same.
It actually resizes the disk, but there seems to be a new output/behavior in partx that lets it assume there is an error which causes the bad rc.

=> dry run looks good
CHANGE: partition=2 start=11718656 old: size=1048576 end=12767232 new: size=5058527,end=16777183
# === old sfdisk -d ===
label: dos
label-id: 0x6e682957
device: /dev/vda
unit: sectors

/dev/vda1 : start= 2048, size= 11716608, type=83, bootable
/dev/vda2 : start= 11718656, size= 1048576, type=83
# === new sfdisk -d ===
label: dos
label-id: 0x6e682957
device: /dev/vda
unit: sectors

/dev/vda1 : start= 2048, size= 11716608, type=83, bootable
/dev/vda2 : start= 11718656, size= 5058527, type=83

=> then without --dry-run
failed [pt_update:1] pt_update /dev/vda 2
partx: partition and disk name do not match
FAILED: pt_resize failed

=> but disk is actually updated
Disk /dev/vda: 8 GiB, 8589934592 bytes, 16777216 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x6e682957

Device Boot Start End Sectors Size Id Type
/dev/vda1 * 2048 11718655 11716608 5.6G 83 Linux
/dev/vda2 11718656 16777182 5058527 2.4G 83 Linux

affects:

cloud-init (Ubuntu) → cloud-utils (Ubuntu)

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2016-06-01:

Call is:
partx --update "$part" "$dev"
This would work
partx --update --nr "$part" "$dev"

I have to check how backward compatible (or not) that is ...

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2016-06-01:

It is existing for a long time, longer than the appearance of --update which the tool already checks for - so the fix can be rather simple.
(I guess/hope) that the remaining portions of cloud init would then just go on and work.

@smoser - do we have "partx --update" in other places we would have to think about?

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2016-06-01:

created a MP to review with the fix to flow in whatever the next release will be.

Scott Moser (smoser) on 2016-06-06

no longer affects:

cloud-images

Report a bug

This report contains Public information

Everyone can see this information.

Duplicate of bug #1587971 Remove

You are

Subscribing...

Edit bug mail

Other bug subscribers

Bug attachments

Add attachment

Remote bug watches

Bug watches keep track of this bug in other bug trackers.

Ubuntucloud-utils package

[yakkety regression] does not grow partition any more

Bug Description

Related branches

Other bug subscribers

Bug attachments

Remote bug watches

Ubuntu
cloud-utils package