curtin

attempting to preserve disk curtin expects PTTYPE in blkid output

Bug #1524031 reported by Ryan Harper
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
curtin
Fix Released
Medium
Unassigned
curtin (Ubuntu)
Fix Released
Undecided
Unassigned
Xenial
Fix Released
Undecided
Unassigned

Bug Description

[Impact]

 * On a Trusty host, if users enable storage disk preserve feature,
   the use of blkid expects PTTYPE value in the output, however, this is
   not present in Trusty blkid output. This prevents Trusty hosts from
   preserving existing partitions.

   Curtin has been updated to fallback to using parted to determine how to
   preserve existing partitions on disks when directed.

[Test Case]

 * Install proposed curtin package and deploy in a Trusty ephemeral
   environment with the following storage configuration:

   storage:
    version: 1
    config:
      - id: vdg
        model: Unknown Model
        preserve: true
        ptable: gpt
        serial: 64bddb21-61f3-4869-8
        type: disk
        wipe: superblock

  PASS: Successfully deploy image while preserving the original partition
        table.

  FAIL: Deployment fails with the following error:

 line 530, in disk_handler
  out.splitlines()))[0].split("=")[-1]
 IndexError: list index out of range
 list index out of range

[Regression Potential]

 * No known other failures when using parted as replacement for blkid
   partition data.

[Original Description]
On a trusty host, this storage configuration breaks:

storage:
  version: 1
  config:
    - id: vdg
      model: Unknown Model
      preserve: true
      ptable: gpt
      serial: 64bddb21-61f3-4869-8
      type: disk
      wipe: superblock

like this:

Running command ['partprobe', '/dev/vdg'] with allowed return codes [0, 1] (shell=False, capture=False)
Running command ['udevadm', 'settle'] with allowed return codes [0] (shell=False, capture=False)
Running command ['blkid', '-o', 'export', '/dev/vdg'] with allowed return codes [0] (shell=False, capture=True)
An error occured handling 'vdg': IndexError - list index out of range
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/curtin/commands/main.py", line 208, in main
    ret = args.func(args)
  File "/usr/lib/python3/dist-packages/curtin/commands/block_meta.py", line 63, in block_meta
    meta_custom(args)
  File "/usr/lib/python3/dist-packages/curtin/commands/block_meta.py", line 1208, in meta_custom
    handler(command, storage_config_dict)
  File "/usr/lib/python3/dist-packages/curtin/commands/block_meta.py", line 530, in disk_handler
    out.splitlines()))[0].split("=")[-1]
IndexError: list index out of range
list index out of range

From this code here:

        # Check state of current ptable
        try:
            (out, _err) = util.subp(["blkid", "-o", "export", disk],
                                    capture=True)
        except util.ProcessExecutionError:
            raise ValueError("disk '%s' has no readable partition table or \
                cannot be accessed, but preserve is set to true, so cannot \
                continue")
        current_ptable = list(filter(lambda x: "PTTYPE" in x,
                                     out.splitlines()))[0].split("=")[-1]

And the output of blkid -o export /dev/vdg on trusty instance is:

# blkid -o export /dev/vdg
UUID=ef30955b-8aa0-453f-a37d-d631a09a6f7c
UUID_SUB=9a21ae23-6f14-405d-9e9a-8955855132d5
TYPE=btrfs

See original description

Related branches

lp:~wesley-wiedenmeier/curtin/trusty-preserve
Server Team CI bot: Approve (continuous-integration)
Ryan Harper (community): Needs Fixing
Diff: 51 lines (+24/-14)
1 file modified
curtin/commands/block_meta.py (+24/-14)
lp:~curtin-dev/curtin/trunk
lp:~smoser/ubuntu/yakkety/curtin/pkg
lp:~raharper/ubuntu/xenial/curtin/pkg-sru-revno425
Scott Moser: Pending requested
Diff: 14513 lines (+10324/-1893)
94 files modified
Makefile (+3/-1)
curtin/__init__.py (+4/-0)
curtin/block/__init__.py (+249/-61)
curtin/block/clear_holders.py (+387/-0)
curtin/block/lvm.py (+96/-0)
curtin/block/mdadm.py (+18/-5)
curtin/block/mkfs.py (+10/-5)
curtin/commands/apply_net.py (+156/-1)
curtin/commands/apt_config.py (+668/-0)
curtin/commands/block_info.py (+75/-0)
curtin/commands/block_meta.py (+134/-263)
curtin/commands/block_wipe.py (+1/-2)
curtin/commands/clear_holders.py (+48/-0)
curtin/commands/curthooks.py (+61/-235)
curtin/commands/main.py (+4/-3)
curtin/config.py (+2/-3)
curtin/gpg.py (+74/-0)
curtin/net/__init__.py (+67/-30)
curtin/net/network_state.py (+45/-1)
curtin/util.py (+278/-81)
debian/changelog (+32/-2)
doc/conf.py (+21/-4)
doc/devel/README-vmtest.txt (+0/-152)
doc/devel/README.txt (+0/-55)
doc/devel/clear_holders_doc.txt (+85/-0)
doc/index.rst (+6/-0)
doc/topics/apt_source.rst (+164/-0)
doc/topics/config.rst (+551/-0)
doc/topics/development.rst (+68/-0)
doc/topics/integration-testing.rst (+245/-0)
doc/topics/networking.rst (+522/-0)
doc/topics/overview.rst (+7/-7)
doc/topics/reporting.rst (+3/-3)
doc/topics/storage.rst (+894/-0)
examples/apt-source.yaml (+267/-0)
examples/network-ipv6-bond-vlan.yaml (+56/-0)
examples/tests/apt_config_command.yaml (+85/-0)
examples/tests/apt_source_custom.yaml (+97/-0)
examples/tests/apt_source_modify.yaml (+92/-0)
examples/tests/apt_source_modify_arches.yaml (+102/-0)
examples/tests/apt_source_modify_disable_suite.yaml (+92/-0)
examples/tests/apt_source_preserve.yaml (+98/-0)
examples/tests/apt_source_search.yaml (+97/-0)
examples/tests/basic.yaml (+5/-1)
examples/tests/basic_network_static_ipv6.yaml (+22/-0)
examples/tests/basic_scsi.yaml (+1/-1)
examples/tests/network_alias.yaml (+125/-0)
examples/tests/network_mtu.yaml (+88/-0)
examples/tests/network_source_ipv6.yaml (+31/-0)
examples/tests/test_old_apt_features.yaml (+11/-0)
examples/tests/test_old_apt_features_ports.yaml (+10/-0)
examples/tests/uefi_basic.yaml (+15/-0)
examples/tests/vlan_network_ipv6.yaml (+92/-0)
setup.py (+2/-2)
tests/unittests/helpers.py (+41/-0)
tests/unittests/test_apt_custom_sources_list.py (+170/-0)
tests/unittests/test_apt_source.py (+1032/-0)
tests/unittests/test_block.py (+210/-0)
tests/unittests/test_block_lvm.py (+94/-0)
tests/unittests/test_block_mdadm.py (+28/-23)
tests/unittests/test_block_mkfs.py (+2/-2)
tests/unittests/test_clear_holders.py (+329/-0)
tests/unittests/test_make_dname.py (+200/-0)
tests/unittests/test_net.py (+54/-13)
tests/unittests/test_util.py (+180/-2)
tests/vmtests/__init__.py (+38/-38)
tests/vmtests/helpers.py (+129/-166)
tests/vmtests/test_apt_config_cmd.py (+55/-0)
tests/vmtests/test_apt_source.py (+238/-0)
tests/vmtests/test_basic.py (+21/-41)
tests/vmtests/test_bcache_basic.py (+5/-8)
tests/vmtests/test_bonding.py (+0/-204)
tests/vmtests/test_lvm.py (+2/-1)
tests/vmtests/test_mdadm_bcache.py (+21/-17)
tests/vmtests/test_multipath.py (+5/-13)
tests/vmtests/test_network.py (+205/-348)
tests/vmtests/test_network_alias.py (+40/-0)
tests/vmtests/test_network_bonding.py (+63/-0)
tests/vmtests/test_network_enisource.py (+91/-0)
tests/vmtests/test_network_ipv6.py (+53/-0)
tests/vmtests/test_network_ipv6_enisource.py (+26/-0)
tests/vmtests/test_network_ipv6_static.py (+42/-0)
tests/vmtests/test_network_ipv6_vlan.py (+34/-0)
tests/vmtests/test_network_mtu.py (+155/-0)
tests/vmtests/test_network_static.py (+44/-0)
tests/vmtests/test_network_vlan.py (+77/-0)
tests/vmtests/test_nvme.py (+2/-3)
tests/vmtests/test_old_apt_features.py (+89/-0)
tests/vmtests/test_raid5_bcache.py (+5/-8)
tests/vmtests/test_uefi_basic.py (+16/-18)
tools/jenkins-runner (+33/-7)
tools/launch (+9/-48)
tools/xkvm (+90/-2)
tox.ini (+30/-13)
Scott Moser (smoser)
Changed in curtin:
status: New → Confirmed
importance: Undecided → Medium
Wesley Wiedenmeier (wesley-wiedenmeier)
Changed in curtin:
status: Confirmed → Fix Committed
Ryan Harper (raharper)
description: updated
Revision history for this message
Andy Whitcroft (apw) wrote : Please test proposed package

Hello Ryan, or anyone else affected,

Accepted curtin into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/curtin/0.1.0~bzr425-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-needed
Andy Whitcroft (apw)
Changed in curtin (Ubuntu):
status: New → Fix Released
Changed in curtin (Ubuntu Xenial):
status: New → Fix Committed
Revision history for this message
Ryan Harper (raharper) wrote :

I've tested the proposed package on Xenial and it handles preserve partitions properly.

tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package curtin - 0.1.0~bzr425-0ubuntu1~16.04.1

---------------
curtin (0.1.0~bzr425-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  [ Scott Moser ]
  * debian/new-upstream-snapshot: add writing of debian changelog entries.

  [ Ryan Harper ]
  * New upstream snapshot.
    - unittest,tox.ini: catch and fix issue with trusty-level mock of open
    - block/mdadm: add option to ignore mdadm_assemble errors (LP: #1618429)
    - curtin/doc: overhaul curtin documentation for readthedocs.org
      (LP: #1351085)
    - curtin.util: re-add support for RunInChroot (LP: #1617375)
    - curtin/net: overhaul of eni rendering to handle mixed ipv4/ipv6 configs
    - curtin.block: refactor clear_holders logic into block.clear_holders and
      cli cmd
    - curtin.apply_net should exit non-zero upon exception. (LP: #1615780)
    - apt: fix bug in disable_suites if sources.list line is blank.
    - vmtests: disable Wily in vmtests
    - Fix the unittests for test_apt_source.
    - get CURTIN_VMTEST_PARALLEL shown correctly in jenkins-runner output
    - fix vmtest check_file_strippedline to strip lines before comparing
    - fix whitespace damage in tests/vmtests/__init__.py
    - fix dpkg-reconfigure when debconf_selections was provided.
      (LP: #1609614)
    - fix apt tests on non-intel arch
    - Add apt features to curtin. (LP: #1574113)
    - vmtest: easier use of parallel and controlling timeouts
    - mkfs.vfat: add force flag for formating whole disks (LP: #1597923)
    - block.mkfs: fix sectorsize flag (LP: #1597522)
    - block_meta: cleanup use of sys_block_path and handle cciss knames
      (LP: #1562249)
    - block.get_blockdev_sector_size: handle _lsblock multi result return
      (LP: #1598310)
    - util: add target (chroot) support to subp, add target_path helper.
    - block_meta: fallback to parted if blkid does not produce output
      (LP: #1524031)
    - commands.block_wipe: correct default wipe mode to 'superblock'
    - tox.ini: run coverage normally rather than separately
    - move uefi boot knowledge from launch and vmtest to xkvm

 -- Ryan Harper <email address hidden> Mon, 03 Oct 2016 13:43:54 -0500

Changed in curtin (Ubuntu Xenial):
status: Fix Committed → Fix Released
Revision history for this message
Martin Pitt (pitti) wrote : Update Released

The verification of the Stable Release Update for curtin has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Scott Moser (smoser) wrote : Fixed in Curtin 17.1

This bug is believed to be fixed in curtin in 17.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

Changed in curtin:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.