instance.request_spec not updated upon resize

Bug #1941005 reported by Artom Lifshitz
22
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Unassigned

Bug Description

Description
===========
We don't update instance.request_spec when we resize to a new flavor. If the flavor's extra specs change, this is not reflected in instance.request_spec.

This goes generally unnoticed when *adding* stuff to the instance - like PCI devices, for example. The scheduler will still find a host based on the old request spec, but the resource claim on the destination uses the flavor and image data directly, not the request spec, so the claim will (correctly) fail if the destination does not have the new things that the instance wants.

When *removing* stuff form the instance this becomes problematic. The scheduler will attempt to schedule based on the old request spec, which could end up with NoValidHost if no hosts exist that have the *old* stuff the instance wanted, despite the new flavor not wanting any of that stuff.

Steps to reproduce
==================

Let's use PCI devices as an example.

1. Have only two hosts, one with a single PCI device and one without.
2. Boot an instance with a flavor-based PCI device.
3. Resize it to a flavor without PCI devices.

Expected result
===============
The resize works - after all, we have one free host for the now-PCIless instance.

Actual result
=============
The resize fails with NoValidHost, the PCIPassthroughFilter having removed all the hosts (because it's using the original request spec with the PCI device).

Environment
===========
Reproduced on master with a functional test, also reported on OSP 16.1 aka stable/train: https://bugzilla.redhat.com/show_bug.cgi?id=1976707

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/805882

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.opendev.org/c/openstack/nova/+/805882
Committed: https://opendev.org/openstack/nova/commit/38603a4d09a77882298423a8b20c67b283b99ad0
Submitter: "Zuul (22348)"
Branch: master

commit 38603a4d09a77882298423a8b20c67b283b99ad0
Author: Artom Lifshitz <email address hidden>
Date: Tue Aug 24 15:52:02 2021 -0400

    Reproduce bug 1941005

    We don't update instance.request_spec when we resize to a new flavor.
    If the flavor's extra specs change, this is not reflected in
    instance.request_spec.

    This generally goes unnoticed when *adding* stuff to an instance. The
    scheduler will still find a host based on the old request spec. The
    resource claim on the destination uses the flavor and image data
    directly (not the request spec) so the claim will correctly fail if
    the destination does not have the new things that the instance wants.

    When *removing* stuff form an instance this becomes problematic. The
    scheduler will attempt to schedule based on the old request spec,
    which could end up with NoValidHost if no host exists that has the
    *old* stuff the instance wanted, despite the new flavor not wanting
    any of that stuff.

    Related bug: 1941005

    Change-Id: I8c052a6526819834d79d2b5027050f1bdf36638f

Revision history for this message
Balazs Gibizer (balazs-gibizer) wrote :

As the reproduction patch showed this is real. So I confirm it.

Changed in nova:
status: New → Confirmed
assignee: nobody → Artom Lifshitz (notartom)
importance: Undecided → Medium
tags: added: resize
Revision history for this message
李亚冲 (879228763-6) wrote :

so did I . in this bugs , I think it should be modified in the conductor service, progressing

Revision history for this message
李亚冲 (879228763-6) wrote (last edit ):

HI everybody , I read the code, find in the resize workflow ,no method to update pci_requests, so I add new code_line in method of resize, in /nova/compute/api.py.

```
    @reject_vdpa_instances(instance_actions.RESIZE)
    @block_accelerators()
    @check_instance_lock
    @check_instance_state(vm_state=[vm_states.ACTIVE, vm_states.STOPPED])
    @check_instance_host(check_is_up=True)
    def resize(self, context, instance, flavor_id=None, clean_shutdown=True,
               host_name=None, auto_disk_config=None):
        ...

        filter_properties = {'ignore_hosts': []}
        if not self._allow_resize_to_same_host(same_instance_type, instance):
            filter_properties['ignore_hosts'].append(instance.host)

        request_spec = objects.RequestSpec.get_by_instance_uuid(
            context, instance.uuid)
        request_spec.ignore_hosts = filter_properties['ignore_hosts']

        # add new code
        if new_instance_type != current_instance_type:
            request_spec.pci_requests = pci_request.get_pci_requests_from_flavor(new_instance_type)
        ...
```

It work , at resize finished , the DB of placement will be update

Changed in nova:
assignee: Artom Lifshitz (notartom) → 李亚冲 (879228763-6)
assignee: 李亚冲 (879228763-6) → nobody
李亚冲 (879228763-6)
Changed in nova:
status: Confirmed → Opinion
status: Opinion → Confirmed
Changed in nova:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/852296

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by "Artom Lifshitz <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/nova/+/806049

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.opendev.org/c/openstack/nova/+/852296
Committed: https://opendev.org/openstack/nova/commit/a93092e0d5c1483f9ad48276708ee35c54ce44fe
Submitter: "Zuul (22348)"
Branch: master

commit a93092e0d5c1483f9ad48276708ee35c54ce44fe
Author: Balazs Gibizer <email address hidden>
Date: Sat Aug 6 16:09:54 2022 +0200

    Update RequestSpec.pci_request for resize

    Nova uses the RequestSpec.pci_request in the PciPassthroughFilter to
    decide if the PCI devicesm, requested via the pci_alias in the flavor
    extra_spec, are available on a potential target host. During resize the
    new flavor might contain different pci_alias request than the old flavor
    of the instance. In this case Nova should use the pci_alias from the new
    flavor to scheduler the destination host of the resize. However this
    logic was missing and Nova used the old pci_request value based on the
    old flavor. This patch adds the missing logic.

    Closes-Bug: #1983753
    Closes-Bug: #1941005
    Change-Id: I73c9ae27e9c42ee211a53bed3d849650b65f08be

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/853265

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/yoga)

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/nova/+/853267

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (stable/yoga)

Related fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/nova/+/853268

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/xena)

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/nova/+/853270

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (stable/xena)

Related fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/nova/+/853271

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/853274

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (stable/wallaby)

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/853275

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (master)

Reviewed: https://review.opendev.org/c/openstack/nova/+/853265
Committed: https://opendev.org/openstack/nova/commit/82cdfa23c7a0e269ab038e241bb7428b7f1391aa
Submitter: "Zuul (22348)"
Branch: master

commit 82cdfa23c7a0e269ab038e241bb7428b7f1391aa
Author: Balazs Gibizer <email address hidden>
Date: Tue Aug 16 13:19:03 2022 +0200

    Add reno for fixing bug 1941005

    Related-Bug: #1941005
    Related-Bug: #1983753
    Change-Id: I16ed1143ead3779c87698aa29bac005678db2993

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 26.0.0.0rc1

This issue was fixed in the openstack/nova 26.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/nova/+/853267
Committed: https://opendev.org/openstack/nova/commit/b6c7ee07ad2a6503b8aad0921cca70e61e60259d
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit b6c7ee07ad2a6503b8aad0921cca70e61e60259d
Author: Balazs Gibizer <email address hidden>
Date: Sat Aug 6 16:09:54 2022 +0200

    Update RequestSpec.pci_request for resize

    Nova uses the RequestSpec.pci_request in the PciPassthroughFilter to
    decide if the PCI devicesm, requested via the pci_alias in the flavor
    extra_spec, are available on a potential target host. During resize the
    new flavor might contain different pci_alias request than the old flavor
    of the instance. In this case Nova should use the pci_alias from the new
    flavor to scheduler the destination host of the resize. However this
    logic was missing and Nova used the old pci_request value based on the
    old flavor. This patch adds the missing logic.

    Closes-Bug: #1983753
    Closes-Bug: #1941005
    Change-Id: I73c9ae27e9c42ee211a53bed3d849650b65f08be
    (cherry picked from commit a93092e0d5c1483f9ad48276708ee35c54ce44fe)

tags: added: in-stable-yoga
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/nova/+/853268
Committed: https://opendev.org/openstack/nova/commit/19bac6e9c362ee692f4be92041bb3f6c3b9b6c23
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 19bac6e9c362ee692f4be92041bb3f6c3b9b6c23
Author: Balazs Gibizer <email address hidden>
Date: Tue Aug 16 13:19:03 2022 +0200

    Add reno for fixing bug 1941005

    Related-Bug: #1941005
    Related-Bug: #1983753
    Change-Id: I16ed1143ead3779c87698aa29bac005678db2993
    (cherry picked from commit 82cdfa23c7a0e269ab038e241bb7428b7f1391aa)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/nova/+/853270
Committed: https://opendev.org/openstack/nova/commit/696d27a0e2ccf0916fc6bd3eabf73f2b007e7830
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 696d27a0e2ccf0916fc6bd3eabf73f2b007e7830
Author: Balazs Gibizer <email address hidden>
Date: Sat Aug 6 16:09:54 2022 +0200

    Update RequestSpec.pci_request for resize

    Nova uses the RequestSpec.pci_request in the PciPassthroughFilter to
    decide if the PCI devicesm, requested via the pci_alias in the flavor
    extra_spec, are available on a potential target host. During resize the
    new flavor might contain different pci_alias request than the old flavor
    of the instance. In this case Nova should use the pci_alias from the new
    flavor to scheduler the destination host of the resize. However this
    logic was missing and Nova used the old pci_request value based on the
    old flavor. This patch adds the missing logic.

    Closes-Bug: #1983753
    Closes-Bug: #1941005
    Change-Id: I73c9ae27e9c42ee211a53bed3d849650b65f08be
    (cherry picked from commit a93092e0d5c1483f9ad48276708ee35c54ce44fe)
    (cherry picked from commit b6c7ee07ad2a6503b8aad0921cca70e61e60259d)

tags: added: in-stable-xena
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/nova/+/853271
Committed: https://opendev.org/openstack/nova/commit/cbf785709968fffc0a760e28f1cdb5c38b254232
Submitter: "Zuul (22348)"
Branch: stable/xena

commit cbf785709968fffc0a760e28f1cdb5c38b254232
Author: Balazs Gibizer <email address hidden>
Date: Tue Aug 16 13:19:03 2022 +0200

    Add reno for fixing bug 1941005

    Related-Bug: #1941005
    Related-Bug: #1983753
    Change-Id: I16ed1143ead3779c87698aa29bac005678db2993
    (cherry picked from commit 82cdfa23c7a0e269ab038e241bb7428b7f1391aa)
    (cherry picked from commit 19bac6e9c362ee692f4be92041bb3f6c3b9b6c23)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/883905

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (stable/victoria)

Related fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/883906

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/nova/+/883909

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (stable/ussuri)

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/nova/+/883930

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/883933

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (stable/train)

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/883934

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (stable/train)

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/883934
Reason: stable/train branch of nova projects' have been tagged as End of Life. All open patches have to be abandoned in order to be able to delete the branch.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/883933
Reason: stable/train branch of nova projects' have been tagged as End of Life. All open patches have to be abandoned in order to be able to delete the branch.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 25.2.1

This issue was fixed in the openstack/nova 25.2.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (stable/ussuri)

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/nova/+/883909
Reason: stable/ussuri branch of openstack/nova transitioned to End of Life and is about to be deleted. To be able to do that, all open patches need to be abandoned.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/nova/+/883930
Reason: stable/ussuri branch of openstack/nova transitioned to End of Life and is about to be deleted. To be able to do that, all open patches need to be abandoned.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova xena-eom

This issue was fixed in the openstack/nova xena-eom release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (stable/victoria)

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/883905
Reason: stable/victoria branch of openstack/nova is about to be deleted. To be able to do that, all open patches need to be abandoned. Please cherry pick the patch to unmaintained/victoria if you want to further work on this patch.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/883906
Reason: stable/victoria branch of openstack/nova is about to be deleted. To be able to do that, all open patches need to be abandoned. Please cherry pick the patch to unmaintained/victoria if you want to further work on this patch.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (stable/wallaby)

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/853274
Reason: stable/wallaby branch of openstack/nova is about to be deleted. To be able to do that, all open patches need to be abandoned. Please cherry pick the patch to unmaintained/wallaby if you want to further work on this patch.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/853275
Reason: stable/wallaby branch of openstack/nova is about to be deleted. To be able to do that, all open patches need to be abandoned. Please cherry pick the patch to unmaintained/wallaby if you want to further work on this patch.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.