functional tests leak cinder volumes

Bug #1613105 reported by Matt Riedemann
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
python-novaclient
Fix Released
Undecided
Takashi Natsume
Nominated for Ocata by Takashi Natsume

Bug Description

Some of the novaclient functional tests create volumes in cinder to test attaching or booting from volume with a server, but they don't cleanup the volume as part of the test so when the run is over there are volumes leftover:

stack@sulmer:/opt/stack/python-novaclient$ cinder list
+--------------------------------------+-----------+--------------------------------------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------------------------------+------+-------------+----------+-------------+
| 30c0b078-3492-4a3f-973e-c281fec8fda3 | available | b0bd8496-cdcd-4e0e-aba1-4e007cefb9fc | 1 | lvmdriver-1 | true | |
| 592b3b9a-ad78-45ee-ad21-69dd99225aa0 | available | 3878c92a-b56c-4fc9-be5e-39e03bfcc0fb | 1 | lvmdriver-1 | true | |
| ad213747-398b-4677-815c-3c404cd67ab0 | available | | 1 | lvmdriver-1 | true | |
+--------------------------------------+-----------+--------------------------------------+------+-------------+----------+-------------+

Like in the TestServersBootNovaClient tests.

It would be easier to debug which tests are leaking volumes if we named the volume after the test.

tags: added: low-hanging-fruit
Changed in python-novaclient:
status: New → Confirmed
tamil vanan (tamilhce)
Changed in python-novaclient:
assignee: nobody → tamil vanan (tamilhce)
tamil vanan (tamilhce)
Changed in python-novaclient:
assignee: tamil vanan (tamilhce) → nobody
Ali Jabbar (jabbar-ali)
Changed in python-novaclient:
assignee: nobody → JABAR ALI (jabbar-ali)
Revision history for this message
Ali Jabbar (jabbar-ali) wrote :

What are the steps for reproducing this problem ?

Revision history for this message
Ali Jabbar (jabbar-ali) wrote :

What are the steps for reproducing this issue...

Revision history for this message
Matt Riedemann (mriedem) wrote :

Run the novaclient functional tests in a local devstack and see how many volumes are leftover using 'cinder list', then trace those back to functional tests which create volumes but don't have something in place to clean them up after the test exists, like an addCleanup call to delete the volume. Some of this can be seen via just doing code inspection.

Revision history for this message
Ali Jabbar (jabbar-ali) wrote :

It is reproducible now
stack@cinv-V:/opt/stack/python-novaclient$ nova list
+----+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+----+------+--------+------------+-------------+----------+
+----+------+--------+------------+-------------+----------+
stack@cinv-V:/opt/stack/python-novaclient$
stack@cinv-V:/opt/stack/python-novaclient$ cinder list
+--------------------------------------+-----------+--------------------------------------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------------------------------+------+-------------+----------+-------------+
| 297ed42c-0a36-4b99-a159-b31c5d5a2fc1 | available | - | 1 | lvmdriver-1 | false | |
| 32df70d7-a6f7-4b82-b330-e7ad06fcbf8c | available | - | 1 | lvmdriver-1 | false | |
| 5c06942d-dfac-4fb4-ab87-8f8f8ea31178 | available | b6a9ecc0-4eb4-4dd2-b8fe-ccfdb167e305 | 1 | lvmdriver-1 | true | |
| 7046801f-b04d-444a-be7d-3665e349c5f7 | available | - | 1 | lvmdriver-1 | false | |
| 90e30704-2f40-4bfa-995b-a4fe6d568a24 | available | - | 1 | lvmdriver-1 | false | |
| e29bce32-970a-468b-b2c3-aff624a66b8d | available | 1917246f-b3c1-4620-b38f-ede7bc5755ff | 1 | lvmdriver-1 | true | |
+--------------------------------------+-----------+--------------------------------------+------+-------------+----------+-------------+
stack@cinv-V:/opt/stack/python-novaclient$

Revision history for this message
Ali Jabbar (jabbar-ali) wrote :

Strange behavior in this problem When control is in the function :wait_for_volume_status volume status shows it 'in-use' and just after the time out it is showing that - it is available, in the failure leg.

Confusing why below code in this function is not coming in available state....???/
start_time = time.time()
        while time.time() - start_time < timeout:
            volume = self.client.volumes.get(volume.id)
            if volume.status == status:
                break
            time.sleep(poll_interval)
        else:
            self.fail("Volume %s did not reach status %s after %d s"
                      % (volume.id, status, timeout))

Any clue please suggest ....

Revision history for this message
Ali Jabbar (jabbar-ali) wrote :

Got the issue - Problem was int the api call sync- after doing the modification it removing the cinder - volume.

:-)
Hope reviewer will take this fix

Revision history for this message
Ali Jabbar (jabbar-ali) wrote :

Cloud you please comment why this issue is coming --
while updating in gerrit with command git review
root@Ari:/home/ari/git/python-novaclient# git review
fatal: ICLA contributor agreement requires current contact information.

Please review your contact information:

  https://review.openstack.org/#/settings/contact

fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
root@Ari:/home/ari/git/python-novaclient# git config --list
<email address hidden>
user.name=JABAR ALI
gitreview.username=jabbar-ali
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true
remote.origin.url=https://github.com/AliJabar/python-novaclient.git
remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
branch.master.remote=origin
branch.master.merge=refs/heads/master
remote.gerrit.url=ssh://<email address hidden>:29418/openstack/python-novaclient.git
remote.gerrit.fetch=+refs/heads/*:refs/remotes/gerrit/*
root@Ari:/home/ari/git/python-novaclient#

Revision history for this message
Ali Jabbar (jabbar-ali) wrote :

i am not getting commit id ?
.git/hooks/commit-msg not found

it is saying following error:
ot@cinv-V:/home/cinv/python-novaclient# git commit --amend
.git/hooks/commit-msg: 1: .git/hooks/commit-msg: Not: not found

Any clue ??

Revision history for this message
Ali Jabbar (jabbar-ali) wrote :
Revision history for this message
Ali Jabbar (jabbar-ali) wrote :

Hi, I have made the change in the code and uploaded for the review

Please check and provide your comments

Changed in python-novaclient:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to python-novaclient (master)

Fix proposed to branch: master
Review: https://review.openstack.org/455140

Changed in python-novaclient:
assignee: Ali Jabbar (jabbar-ali) → Takashi NATSUME (natsume-takashi)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to python-novaclient (master)

Reviewed: https://review.openstack.org/455140
Committed: https://git.openstack.org/cgit/openstack/python-novaclient/commit/?id=4d956d06f8c2bd3971cd46fb7ccdab33aaa5709c
Submitter: Jenkins
Branch: master

commit 4d956d06f8c2bd3971cd46fb7ccdab33aaa5709c
Author: Takashi NATSUME <email address hidden>
Date: Mon Apr 10 16:07:04 2017 +0900

    Fix cinder volume leakage

    A cinder volume remains after
    the 'test_boot_server_with_legacy_bdm_volume_id_only'
    functional test.
    It is resource leakage. So fix it.

    Change-Id: I3cacbac6468b5fad1c9666892fc57906af24b169
    Partial-Bug: #1613105

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to python-novaclient (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/455509

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to python-novaclient (stable/ocata)

Reviewed: https://review.openstack.org/455509
Committed: https://git.openstack.org/cgit/openstack/python-novaclient/commit/?id=ad22c586a4d04197a40454f75670b498a8194c8d
Submitter: Jenkins
Branch: stable/ocata

commit ad22c586a4d04197a40454f75670b498a8194c8d
Author: Takashi NATSUME <email address hidden>
Date: Mon Apr 10 16:07:04 2017 +0900

    Fix cinder volume leakage

    A cinder volume remains after
    the 'test_boot_server_with_legacy_bdm_volume_id_only'
    functional test.
    It is resource leakage. So fix it.

    Change-Id: I3cacbac6468b5fad1c9666892fc57906af24b169
    Partial-Bug: #1613105
    (cherry-picked from commit 4d956d06f8c2bd3971cd46fb7ccdab33aaa5709c)

tags: added: in-stable-ocata
Revision history for this message
Takashi Natsume (natsume-takashi) wrote :

The bug has been fixed by the following fixes at *master*.

* Fix cinder volume leakage
https://review.openstack.org/#/c/455140/

* Microversion 2.42 - Fix tag attribute disappearing
https://review.openstack.org/#/c/429512/

NOTE: If 'image_volume_cache_enabled' is True in cinder.conf,
      an image volume cache (a volume for image cache) is left.
      But it is a cinder function and just a cache, not resource leakage.

Changed in python-novaclient:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on python-novaclient (master)

Change abandoned by Stephen Finucane (<email address hidden>) on branch: master
Review: https://review.openstack.org/382864

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.