Libvirt driver reports incorrect error when volume-detach fails

Bug #1029463 reported by Dan Smith
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Undecided
Dan Smith
Essex
Fix Released
Undecided
Aaron Rosen
nova (Ubuntu)
Fix Released
Undecided
Unassigned
Precise
Fix Released
Undecided
Unassigned

Bug Description

Libvirt's error path during volume-detach does not properly re-raise exceptions.

Consider the scenario:

1. nova boot ... fooserver
2. nova volume-create 1
3. nova volume-attach fooserver <vol_id> /dev/vdz
4. nova volume-detach fooserver <vol_id>
5. nova volume-attach fooserver <vol_id> /dev/vdz

Step 5 currently fails with the cirrus image because ACPI hotplug is disabled, which means you can't reattach a drive at the same device. Instead of the libvirt error message, we get the dump below, because the original exception is lost during the cleanup phase and the subsequent 'raise' can't do its job.

2012-07-26 06:57:17 ERROR nova.compute.manager [req-a31302b6-eadd-4611-979b-85fc
3bb774fd demo demo] [instance: 2d4c4223-5781-44eb-b471-b305eddddc61] Failed to a
ttach volume 3759f55c-73da-4892-a598-8967e6197701 at /dev/vdz
2012-07-26 06:57:17 TRACE nova.compute.manager [instance: 2d4c4223-5781-44eb-b47
1-b305eddddc61] Traceback (most recent call last):
2012-07-26 06:57:17 TRACE nova.compute.manager [instance: 2d4c4223-5781-44eb-b47
1-b305eddddc61] File "/opt/stack/nova/nova/compute/manager.py", line 1897, in
attach_volume
2012-07-26 06:57:17 TRACE nova.compute.manager [instance: 2d4c4223-5781-44eb-b47
1-b305eddddc61] mountpoint)
2012-07-26 06:57:17 TRACE nova.compute.manager [instance: 2d4c4223-5781-44eb-b47
1-b305eddddc61] File "/opt/stack/nova/nova/exception.py", line 117, in wrapped
2012-07-26 06:57:17 TRACE nova.compute.manager [instance: 2d4c4223-5781-44eb-b47
1-b305eddddc61] temp_level, payload)
2012-07-26 06:57:17 TRACE nova.compute.manager [instance: 2d4c4223-5781-44eb-b47
1-b305eddddc61] File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2012-07-26 06:57:17 TRACE nova.compute.manager [instance: 2d4c4223-5781-44eb-b47
1-b305eddddc61] self.gen.next()
2012-07-26 06:57:17 TRACE nova.compute.manager [instance: 2d4c4223-5781-44eb-b47
1-b305eddddc61] File "/opt/stack/nova/nova/exception.py", line 92, in wrapped
2012-07-26 06:57:17 TRACE nova.compute.manager [instance: 2d4c4223-5781-44eb-b471-b305eddddc61] return f(*args, **kw)
2012-07-26 06:57:17 TRACE nova.compute.manager [instance: 2d4c4223-5781-44eb-b471-b305eddddc61] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 640, in attach_volume
2012-07-26 06:57:17 TRACE nova.compute.manager [instance: 2d4c4223-5781-44eb-b471-b305eddddc61] raise
2012-07-26 06:57:17 TRACE nova.compute.manager [instance: 2d4c4223-5781-44eb-b471-b305eddddc61] TypeError: exceptions must be old-style classes or derived from BaseException, not NoneType
2012-07-26 06:57:17 TRACE nova.compute.manager [instance: 2d4c4223-5781-44eb-b471-b305eddddc61]

Related branches

Changed in nova:
assignee: nobody → Dan Smith (danms)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/10283
Committed: http://github.com/openstack/nova/commit/4810269ef648348bbc7ef9113d4e7faf47b15c55
Submitter: Jenkins
Branch: master

commit 4810269ef648348bbc7ef9113d4e7faf47b15c55
Author: Dan Smith <email address hidden>
Date: Wed Jul 25 10:33:16 2012 -0700

    Fix exception handling in libvirt attach_volume()

    Currently, the real reason for a failure is dropped when attempting
    to unroll the volume connection (with the exception of when it is
    VIR_ERR_OPERATION_FAILED). This change uses save_and_reraise_exception()
    to correct that so that the actual reason for failure gets logged.

    Fixes bug 1029463

    Change-Id: Id47db565c4fb5a88d1a263600b41706dd3419726

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → folsom-3
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/essex)

Fix proposed to branch: stable/essex
Review: https://review.openstack.org/11809

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: stable/essex
Review: https://review.openstack.org/11833

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/essex)

Reviewed: https://review.openstack.org/11833
Committed: http://github.com/openstack/nova/commit/86fb73623f80b6549c46d5b2af35fa6390070c20
Submitter: Jenkins
Branch: stable/essex

commit 86fb73623f80b6549c46d5b2af35fa6390070c20
Author: Dan Smith <email address hidden>
Date: Wed Aug 22 13:54:35 2012 -0400

    Fix exception handling in libvirt attach_volume()

    Currently, the real reason for a failure is dropped when attempting
    to unroll the volume connection (with the exception of when it is
    VIR_ERR_OPERATION_FAILED). This change uses save_and_reraise_exception()
    to correct that so that the actual reason for failure gets logged.

    Fixes bug 1029463

    Change-Id: Id47db565c4fb5a88d1a263600b41706dd3419726
    Signed-off-by: Aaron Rosen <email address hidden>

Dave Walker (davewalker)
Changed in nova (Ubuntu):
status: New → Fix Released
Changed in nova (Ubuntu Precise):
status: New → Confirmed
Revision history for this message
Adam Gandelman (gandelman-a) wrote : Verification report.

Please find the attached test log from the Ubuntu Server Team's CI infrastructure. As part of the verification process for this bug, Nova has been deployed and configured across multiple nodes using precise-proposed as an installation source. After successful bring-up and configuration of the cluster, a number of exercises and smoke tests have be invoked to ensure the updated package did not introduce any regressions. A number of test iterations were carried out to catch any possible transient errors.

Please Note the list of installed packages at the top and bottom of the report.

For records of upstream test coverage of this update, please see the Jenkins links in the comments of the relevant upstream code-review(s):

Trunk review: https://review.openstack.org/10283
Stable review: https://review.openstack.org/11833

As per the provisional Micro Release Exception granted to this package by the Technical Board, we hope this contributes toward verification of this update.

Revision history for this message
Adam Gandelman (gandelman-a) wrote :

Test coverage log.

tags: added: verification-done
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (5.4 KiB)

This bug was fixed in the package nova - 2012.1.3+stable-20120827-4d2a4afe-0ubuntu1

---------------
nova (2012.1.3+stable-20120827-4d2a4afe-0ubuntu1) precise-proposed; urgency=low

  * New upstream snapshot, fixes FTBFS in -proposed. (LP: #1041120)
  * Resynchronize with stable/essex (4d2a4afe):
    - [5d63601] Inappropriate exception handling on kvm live/block migration
      (LP: #917615)
    - [ae280ca] Deleted floating ips can cause instance delete to fail
      (LP: #1038266)

nova (2012.1.3+stable-20120824-86fb7362-0ubuntu1) precise-proposed; urgency=low

  * New upstream snapshot. (LP: #1041120)
  * Dropped, superseded by new snapshot:
    - debian/patches/CVE-2012-3447.patch: [d9577ce]
    - debian/patches/CVE-2012-3371.patch: [25f5bd3]
    - debian/patches/CVE-2012-3360+3361.patch: [b0feaff]
  * Resynchronize with stable/essex (86fb7362):
    - [86fb736] Libvirt driver reports incorrect error when volume-detach fails
      (LP: #1029463)
    - [272b98d] nova delete lxc-instance umounts the wrong rootfs (LP: #971621)
    - [09217ab] Block storage connections are NOT restored on system reboot
      (LP: #1036902)
    - [d9577ce] CVE-2012-3361 not fully addressed (LP: #1031311)
    - [e8ef050] pycrypto is unused and the existing code is potentially insecure
      to use (LP: #1033178)
    - [3b4ac31] cannot umount guestfs (LP: #1013689)
    - [f8255f3] qpid_heartbeat setting in ineffective (LP: #1030430)
    - [413c641] Deallocation of fixed IP occurs before security group refresh
      leading to potential security issue in error / race conditions
      (LP: #1021352)
    - [219c5ca] Race condition in network/deallocate_for_instance() leads to
      security issue (LP: #1021340)
    - [f2bc403] cleanup_file_locks does not remove stale sentinel files
      (LP: #1018586)
    - [4c7d671] Deleting Flavor currently in use by instance creates error
      (LP: #994935)
    - [7e88e39] nova testsuite errors on newer versions of python-boto (e.g.
      2.5.2) (LP: #1027984)
    - [80d3026] NoMoreFloatingIps: Zero floating ips available after repeatedly
      creating and destroying instances over time (LP: #1017418)
    - [4d74631] Launching with source groups under load produces lazy load error
      (LP: #1018721)
    - [08e5128] API 'v1.1/{tenant_id}/os-hosts' does not return a list of hosts
      (LP: #1014925)
    - [801b94a] Restarting nova-compute removes ip packet filters (LP: #1027105)
    - [f6d1f55] instance live migration should create virtual_size disk image
      (LP: #977007)
    - [4b89b4f] [nova][volumes] Exceeding volumes, gigabytes and floating_ips
      quotas returns general uninformative HTTP 500 error (LP: #1021373)
    - [6e873bc] [nova][volumes] Exceeding volumes, gigabytes and floating_ips
      quotas returns general uninformative HTTP 500 error (LP: #1021373)
    - [7b215ed] Use default qemu-img cluster size in libvirt connection driver
    - [d3a87a2] Listing flavors with marker set returns 400 (LP: #956096)
    - [cf6a85a] nova-rootwrap hardcodes paths instead of using
      /sbin:/usr/sbin:/usr/bin:/bin (LP: #1013147)
    - [2efc87c] affinity filters don't work if scheduler_hints is None
      (LP: #1007573)
  ...

Read more...

Changed in nova (Ubuntu Precise):
status: Confirmed → Fix Released
Revision history for this message
Clint Byrum (clint-fewbar) wrote : Update Released

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Thierry Carrez (ttx)
Changed in nova:
milestone: folsom-3 → 2012.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.