OpenStack Compute (nova)

Broken retry mechanism for 'nova image-list'

Bug #1557584 reported by Diana Clarke
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Diana Clarke
Liberty
Fix Released
Medium
Diana Clarke
Mitaka
Fix Released
Medium
Diana Clarke

Bug Description

You can configure a list of glance API servers in nova.conf like so:

    [glance]
    api_servers=http://192.168.122.30:9292/v1,http://192.168.122.31:9292/v1
    num_retries = 5

When a call to one of the glance api servers fails, nova typically retries the call on one of the others. This is not the case for 'nova image-list'.

The retry mechanism is here:

    https://github.com/openstack/nova/blob/83261f3106a8bdde38d258a74da777add4956290/nova/image/glance.py#L249

In the case of 'nova image-list', glanceclient returns a python generator rather than an actual list of images. Because a generator is returned, an exception will never be raised there, so the retry mechanism is never executed.

    https://github.com/openstack/python-glanceclient/blob/d59e341a4cd99a8488d5cf41052d9b218379ac87/glanceclient/v1/images.py#L268

This bug was originally reported downstream: https://bugzilla.redhat.com/show_bug.cgi?id=1313254

Diana Clarke (diana-clarke)
Changed in nova:
assignee: nobody → Diana Clarke (diana-clarke)
Diana Clarke (diana-clarke)
Changed in nova:
status: New → Incomplete
status: Incomplete → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/293127

Changed in nova:
status: Confirmed → In Progress
Dan Smith (danms)
tags: added: mitaka-backport-potential
Jay Pipes (jaypipes)
Changed in nova:
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/293127
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ae6d868e2f13f90d9f97c982fdbbccdc6fb8b9c9
Submitter: Jenkins
Branch: master

commit ae6d868e2f13f90d9f97c982fdbbccdc6fb8b9c9
Author: Diana Clarke <email address hidden>
Date: Tue Mar 15 16:17:28 2016 -0400

    Fix retry mechanism for generator results

    Both v1 and v2 of the glance client return python generators in some
    cases (rather than fully fleshed out lists) which thwarts our retry
    mechanism. Convert generator results to a list, so that any potential
    exceptions get raised earlier rather than later, allowing for retries.

    Change-Id: Ibc84f1596d4eaabdef0a48f6cf4da2d1323843a8
    Closes-Bug: #1557584

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/295319

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/liberty)

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/295324

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/296081

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/mitaka)

Reviewed: https://review.openstack.org/295319
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3771c37bac826a35cf17312ef136ba50d239f262
Submitter: Jenkins
Branch: stable/mitaka

commit 3771c37bac826a35cf17312ef136ba50d239f262
Author: Diana Clarke <email address hidden>
Date: Tue Mar 15 16:17:28 2016 -0400

    Fix retry mechanism for generator results

    Both v1 and v2 of the glance client return python generators in some
    cases (rather than fully fleshed out lists) which thwarts our retry
    mechanism. Convert generator results to a list, so that any potential
    exceptions get raised earlier rather than later, allowing for retries.

    Change-Id: Ibc84f1596d4eaabdef0a48f6cf4da2d1323843a8
    Closes-Bug: #1557584
    (cherry picked from commit ae6d868e2f13f90d9f97c982fdbbccdc6fb8b9c9)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (master)

Reviewed: https://review.openstack.org/296081
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=726943f9f83c75557d6e35fc6c6b2b4f24a4c75c
Submitter: Jenkins
Branch: master

commit 726943f9f83c75557d6e35fc6c6b2b4f24a4c75c
Author: Diana Clarke <email address hidden>
Date: Tue Mar 22 17:42:19 2016 -0400

    Clean up the TestGlanceClientWrapper retry tests

    I promised to follow up the fix for Bug #1557584 with a patch to clean
    up all the TestGlanceClientWrapper retry tests rather than just the one
    I added. This patch breaks the retry tests out into a dedicated test
    class (TestGlanceClientWrapperRetries), makes sure spec is declared for
    all MagicMocks, adds a couple custom asserts and helper methods, and
    otherwise makes the group of tests more consistent.

    Change-Id: I8533f5a99d6fd7a232cd65dae7aae895a7fd00aa
    Related-Bug: #1557584

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/liberty)

Reviewed: https://review.openstack.org/295324
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=cab3d64cd92b0f2b2165e68ef370518a463b7a80
Submitter: Jenkins
Branch: stable/liberty

commit cab3d64cd92b0f2b2165e68ef370518a463b7a80
Author: Diana Clarke <email address hidden>
Date: Tue Mar 15 16:17:28 2016 -0400

    Fix retry mechanism for generator results

    Both v1 and v2 of the glance client return python generators in some
    cases (rather than fully fleshed out lists) which thwarts our retry
    mechanism. Convert generator results to a list, so that any potential
    exceptions get raised earlier rather than later, allowing for retries.

    Conflicts:

        /nova/tests/unit/image/test_glance.py

    Conflict notes: I couldn't add the following assert from master because
    client.api_servers didn't exist then. It was added in commit
    590b2c30073843f736680cdea4d6a6f8c673d725. Instead, I added one that
    follows the pattern at that time.

      self.assertEqual(str(client.api_server), "https://host2:9293")

    Change-Id: Ibc84f1596d4eaabdef0a48f6cf4da2d1323843a8
    Closes-Bug: #1557584
    (cherry picked from commit ae6d868e2f13f90d9f97c982fdbbccdc6fb8b9c9)

Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote : Fix included in openstack/nova 14.0.0.0b1

This issue was fixed in the openstack/nova 14.0.0.0b1 development milestone.

Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/nova 12.0.4

This issue was fixed in the openstack/nova 12.0.4 release.

Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/nova 13.1.0

This issue was fixed in the openstack/nova 13.1.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.