OpenStack Compute (nova)

Requests to neutron API do not use retries

Bug #1866937 reported by melanie witt
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Low
melanie witt
Queens
Fix Released
Low
s10
Rocky
Fix Released
Low
melanie witt
Stein
Fix Released
Low
melanie witt
Train
Fix Released
Low
melanie witt

Bug Description

We have a customer bug report downstream [1] where nova occasionally fails to carry out server actions requiring calls to neutron API if haproxy happens to close a connection after idle time of 10 seconds at nearly the same time as an incoming request that attempts to re-use the connection while it is being torn down. Here is an excerpt from [1]:

 The result of our investigation, the cause is follows.

 1. neutron-client in nova uses connection pool ( urllib3/requests ) for http.

 2. Sometimes, http connection is reused for different requests.

 3. Connection between neutron-client and haproxy is closed from haproxy when it is in idle for 10 seconds.

 4. If reusing connection from client side and closing connection from haproxy side are happend almost same time,
    client gets RST and end with "bad status line".

To address this problem, we can add a new config option for neutron client (similar to the existing config options we have for cinder client and glance client retries) to be more resilient during such scenarios.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1788853

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.opendev.org/712226

Changed in nova:
status: New → In Progress
OpenStack Infra (hudson-openstack)
Changed in nova:
assignee: melanie witt (melwitt) → Lee Yarwood (lyarwood)
melanie witt (melwitt)
Changed in nova:
assignee: Lee Yarwood (lyarwood) → melanie witt (melwitt)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.opendev.org/712226
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=0e34ed9733e3f23d162e3e428795807386abf1cb
Submitter: Zuul
Branch: master

commit 0e34ed9733e3f23d162e3e428795807386abf1cb
Author: melanie witt <email address hidden>
Date: Wed Mar 11 02:26:52 2020 +0000

    Add config option for neutron client retries

    Nova can occasionally fail to carry out server actions which require
    calls to neutron API if haproxy happens to close a connection after
    idle time if an incoming request attempts to re-use the connection
    while it is being torn down.

    In order to be more resilient to this scenario, we can add a config
    option for neutron client to retry requests, similar to our existing
    CONF.cinder.http_retries and CONF.glance.num_retries options.

    Because we create our neutron client [1] using a keystoneauth1 session
    [2], we can set the 'connect_retries' keyword argument to let
    keystoneauth1 handle connection retries.

    Closes-Bug: #1866937

    [1] https://github.com/openstack/nova/blob/57459c3429ce62975cebd0cd70936785bdf2f3a4/nova/network/neutron.py#L226-L237
    [2] https://docs.openstack.org/keystoneauth/latest/api/keystoneauth1.session.html#keystoneauth1.session.Session

    Change-Id: Ifb3afb13aff7e103c2e80ade817d0e63b624604a

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/715010

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/train)

Reviewed: https://review.opendev.org/715010
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=71971c206292232fff389dedbf412d780f0a557a
Submitter: Zuul
Branch: stable/train

commit 71971c206292232fff389dedbf412d780f0a557a
Author: melanie witt <email address hidden>
Date: Wed Mar 11 02:26:52 2020 +0000

    Add config option for neutron client retries

    Nova can occasionally fail to carry out server actions which require
    calls to neutron API if haproxy happens to close a connection after
    idle time if an incoming request attempts to re-use the connection
    while it is being torn down.

    In order to be more resilient to this scenario, we can add a config
    option for neutron client to retry requests, similar to our existing
    CONF.cinder.http_retries and CONF.glance.num_retries options.

    Because we create our neutron client [1] using a keystoneauth1 session
    [2], we can set the 'connect_retries' keyword argument to let
    keystoneauth1 handle connection retries.

    Closes-Bug: #1866937

    [1] https://github.com/openstack/nova/blob/57459c3429ce62975cebd0cd70936785bdf2f3a4/nova/network/neutron.py#L226-L237
    [2] https://docs.openstack.org/keystoneauth/latest/api/keystoneauth1.session.html#keystoneauth1.session.Session

    Change-Id: Ifb3afb13aff7e103c2e80ade817d0e63b624604a
    (cherry picked from commit 0e34ed9733e3f23d162e3e428795807386abf1cb)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/722077

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/stein)

Reviewed: https://review.opendev.org/722077
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a96e7ab83066bee0a13a54070aab988396bae320
Submitter: Zuul
Branch: stable/stein

commit a96e7ab83066bee0a13a54070aab988396bae320
Author: melanie witt <email address hidden>
Date: Wed Mar 11 02:26:52 2020 +0000

    Add config option for neutron client retries

    Nova can occasionally fail to carry out server actions which require
    calls to neutron API if haproxy happens to close a connection after
    idle time if an incoming request attempts to re-use the connection
    while it is being torn down.

    In order to be more resilient to this scenario, we can add a config
    option for neutron client to retry requests, similar to our existing
    CONF.cinder.http_retries and CONF.glance.num_retries options.

    Because we create our neutron client [1] using a keystoneauth1 session
    [2], we can set the 'connect_retries' keyword argument to let
    keystoneauth1 handle connection retries.

    Closes-Bug: #1866937

    [1] https://github.com/openstack/nova/blob/57459c3429ce62975cebd0cd70936785bdf2f3a4/nova/network/neutron.py#L226-L237
    [2] https://docs.openstack.org/keystoneauth/latest/api/keystoneauth1.session.html#keystoneauth1.session.Session

    Change-Id: Ifb3afb13aff7e103c2e80ade817d0e63b624604a
    (cherry picked from commit 0e34ed9733e3f23d162e3e428795807386abf1cb)
    (cherry picked from commit 71971c206292232fff389dedbf412d780f0a557a)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/722819

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/722822

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/rocky)

Reviewed: https://review.opendev.org/722819
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=6bed39cd6e77744ca4bbad9dba3f68c78cdc1ec0
Submitter: Zuul
Branch: stable/rocky

commit 6bed39cd6e77744ca4bbad9dba3f68c78cdc1ec0
Author: melanie witt <email address hidden>
Date: Wed Mar 11 02:26:52 2020 +0000

    Add config option for neutron client retries

    Nova can occasionally fail to carry out server actions which require
    calls to neutron API if haproxy happens to close a connection after
    idle time if an incoming request attempts to re-use the connection
    while it is being torn down.

    In order to be more resilient to this scenario, we can add a config
    option for neutron client to retry requests, similar to our existing
    CONF.cinder.http_retries and CONF.glance.num_retries options.

    Because we create our neutron client [1] using a keystoneauth1 session
    [2], we can set the 'connect_retries' keyword argument to let
    keystoneauth1 handle connection retries.

    Closes-Bug: #1866937

    [1] https://github.com/openstack/nova/blob/57459c3429ce62975cebd0cd70936785bdf2f3a4/nova/network/neutron.py#L226-L237
    [2] https://docs.openstack.org/keystoneauth/latest/api/keystoneauth1.session.html#keystoneauth1.session.Session

    Change-Id: Ifb3afb13aff7e103c2e80ade817d0e63b624604a
    (cherry picked from commit 0e34ed9733e3f23d162e3e428795807386abf1cb)
    (cherry picked from commit 71971c206292232fff389dedbf412d780f0a557a)
    (cherry picked from commit a96e7ab83066bee0a13a54070aab988396bae320)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/queens)

Reviewed: https://review.opendev.org/722822
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=11daac03bf4d33c77c7bc6f037bc16972ad64ea4
Submitter: Zuul
Branch: stable/queens

commit 11daac03bf4d33c77c7bc6f037bc16972ad64ea4
Author: melanie witt <email address hidden>
Date: Wed Mar 11 02:26:52 2020 +0000

    Add config option for neutron client retries

    Nova can occasionally fail to carry out server actions which require
    calls to neutron API if haproxy happens to close a connection after
    idle time if an incoming request attempts to re-use the connection
    while it is being torn down.

    In order to be more resilient to this scenario, we can add a config
    option for neutron client to retry requests, similar to our existing
    CONF.cinder.http_retries and CONF.glance.num_retries options.

    Because we create our neutron client [1] using a keystoneauth1 session
    [2], we can set the 'connect_retries' keyword argument to let
    keystoneauth1 handle connection retries.

    Conflicts:
            nova/conf/neutron.py
    NOTE(s10): conflict is due to Id7c2f0b53c8871ff47a836ec4c324c8cce430b79
    not being in Queens.

    Closes-Bug: #1866937

    [1] https://github.com/openstack/nova/blob/57459c3429ce62975cebd0cd70936785bdf2f3a4/nova/network/neutron.py#L226-L237
    [2] https://docs.openstack.org/keystoneauth/latest/api/keystoneauth1.session.html#keystoneauth1.session.Session

    Change-Id: Ifb3afb13aff7e103c2e80ade817d0e63b624604a
    (cherry picked from commit 0e34ed9733e3f23d162e3e428795807386abf1cb)
    (cherry picked from commit 71971c206292232fff389dedbf412d780f0a557a)
    (cherry picked from commit a96e7ab83066bee0a13a54070aab988396bae320)
    (cherry picked from commit 6bed39cd6e77744ca4bbad9dba3f68c78cdc1ec0)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/793512

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (master)

Reviewed: https://review.opendev.org/c/openstack/nova/+/793512
Committed: https://opendev.org/openstack/nova/commit/56eb253e9febccf721df6bca4eb851ad26cb70a6
Submitter: "Zuul (22348)"
Branch: master

commit 56eb253e9febccf721df6bca4eb851ad26cb70a6
Author: melanie witt <email address hidden>
Date: Fri May 28 00:26:04 2021 +0000

    Honor [neutron]http_retries in the manual client

    Change Ifb3afb13aff7e103c2e80ade817d0e63b624604a added a nova side
    config option for specifying neutron client retries that maps to the
    ksa connect_retries config option to provide parity with the cinder and
    glance clients that have nova side config options.

    That change missed passing CONF.neutron.http_retries to the manual
    client used for calling the port binding API. This sets the
    connect_retries attribute on the manual ksa client so http_retries
    will be honored.

    Closes-Bug: #1929886
    Related-Bug: #1866937

    Change-Id: I8296e4be9f0706fab043451b856efadbb7bd41f6

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (stable/wallaby)

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/794186

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (stable/victoria)

Related fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/795175

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (stable/ussuri)

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/nova/+/795176

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (stable/train)

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/795177

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (stable/stein)

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/c/openstack/nova/+/795138

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/nova/+/794186
Committed: https://opendev.org/openstack/nova/commit/46aa3f4ec769e948d9eb73604bf9b66f4b0230b0
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 46aa3f4ec769e948d9eb73604bf9b66f4b0230b0
Author: melanie witt <email address hidden>
Date: Fri May 28 00:26:04 2021 +0000

    Honor [neutron]http_retries in the manual client

    Change Ifb3afb13aff7e103c2e80ade817d0e63b624604a added a nova side
    config option for specifying neutron client retries that maps to the
    ksa connect_retries config option to provide parity with the cinder and
    glance clients that have nova side config options.

    That change missed passing CONF.neutron.http_retries to the manual
    client used for calling the port binding API. This sets the
    connect_retries attribute on the manual ksa client so http_retries
    will be honored.

    Closes-Bug: #1929886
    Related-Bug: #1866937

    Change-Id: I8296e4be9f0706fab043451b856efadbb7bd41f6
    (cherry picked from commit 56eb253e9febccf721df6bca4eb851ad26cb70a6)

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (stable/victoria)

Reviewed: https://review.opendev.org/c/openstack/nova/+/795175
Committed: https://opendev.org/openstack/nova/commit/f20346bc00a30c914cbefb48009db776f8e00b09
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit f20346bc00a30c914cbefb48009db776f8e00b09
Author: melanie witt <email address hidden>
Date: Fri May 28 00:26:04 2021 +0000

    Honor [neutron]http_retries in the manual client

    Change Ifb3afb13aff7e103c2e80ade817d0e63b624604a added a nova side
    config option for specifying neutron client retries that maps to the
    ksa connect_retries config option to provide parity with the cinder and
    glance clients that have nova side config options.

    That change missed passing CONF.neutron.http_retries to the manual
    client used for calling the port binding API. This sets the
    connect_retries attribute on the manual ksa client so http_retries
    will be honored.

    Closes-Bug: #1929886
    Related-Bug: #1866937

    Change-Id: I8296e4be9f0706fab043451b856efadbb7bd41f6
    (cherry picked from commit 56eb253e9febccf721df6bca4eb851ad26cb70a6)
    (cherry picked from commit 46aa3f4ec769e948d9eb73604bf9b66f4b0230b0)

tags: added: in-stable-victoria
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (stable/ussuri)

Reviewed: https://review.opendev.org/c/openstack/nova/+/795176
Committed: https://opendev.org/openstack/nova/commit/b96f93ed8633d6fd0725c4a11366ffbf38056e65
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit b96f93ed8633d6fd0725c4a11366ffbf38056e65
Author: melanie witt <email address hidden>
Date: Fri May 28 00:26:04 2021 +0000

    Honor [neutron]http_retries in the manual client

    Change Ifb3afb13aff7e103c2e80ade817d0e63b624604a added a nova side
    config option for specifying neutron client retries that maps to the
    ksa connect_retries config option to provide parity with the cinder and
    glance clients that have nova side config options.

    That change missed passing CONF.neutron.http_retries to the manual
    client used for calling the port binding API. This sets the
    connect_retries attribute on the manual ksa client so http_retries
    will be honored.

    Closes-Bug: #1929886
    Related-Bug: #1866937

    Change-Id: I8296e4be9f0706fab043451b856efadbb7bd41f6
    (cherry picked from commit 56eb253e9febccf721df6bca4eb851ad26cb70a6)
    (cherry picked from commit 46aa3f4ec769e948d9eb73604bf9b66f4b0230b0)
    (cherry picked from commit f20346bc00a30c914cbefb48009db776f8e00b09)

tags: added: in-stable-ussuri
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova queens-eol

This issue was fixed in the openstack/nova queens-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova rocky-eol

This issue was fixed in the openstack/nova rocky-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (stable/stein)

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/stein
Review: https://review.opendev.org/c/openstack/nova/+/795138
Reason: This branch transitioned to End of Life for this project, open patches needs to be closed to be able to delete the branch.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (stable/train)

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/795177
Reason: stable/train branch of nova projects' have been tagged as End of Life. All open patches have to be abandoned in order to be able to delete the branch.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.