Neutron fails to create bandwidth providers if CONF.host is set

Fix proposed to branch: master
Review: https://review.opendev.org/696600

Fix proposed to branch: stable/train
Review: https://review.opendev.org/699174

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/699196

Reviewed: https://review.opendev.org/696600
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=258eebea71b1cac37badf429a90d5cf57e4c455c
Submitter: Zuul
Branch: master

commit 258eebea71b1cac37badf429a90d5cf57e4c455c
Author: Bence Romsics <email address hidden>
Date: Wed Nov 27 17:59:15 2019 +0100

    Locate RP-tree parent by hypervisor name

    Previously we assumed that we can look up the resource provider (created
    by nova) to be used as the parent of the agent and physical NIC resource
    provider tree by the name set in the config option DEFAULT.host. This
    assumption was wrong.

    While nova-compute's DEFAULT.host and neutron-agent's DEFAULT.host
    must match for port binding to work, the root resource provider created
    by nova does not belong to the compute host (where nova-compute runs)
    but it belongs to the compute nodes (i.e. hypervisors). Actually there
    may be multiple compute nodes managed by a single nova-compute (think
    of ironic). Plus the value of DEFAULT.host and the compute node's ID
    may be different even when nova-compute manages a hypervisor on the
    same host because of various deployment considerations. For example
    when tripleo does not manage the undercloud (so a libvirt hypervisor
    returns the plain hostname), but the same tripleo enforces it's host
    naming conventions in nova's and neutron's DEFAULT.host settings.

    This change enables neutron to use the hypervisor name to locate the
    root of the resource provider tree.

    We introduce a new configuration option for

    (1) ovs-agent: resource_provider_hypervisors, for example:

    [ovs]
    bridge_mappings = physnet0:br-physnet0,...
    resource_provider_bandwidths = br-physnet0:10000000:10000000,...
    resource_provider_hypervisors = br-physnet0:hypervisor0,...

    (2) sriov-agent: resource_provider_hypervisors, for example:

    [sriov_nic]
    bridge_mappings = physnet1:ens5,...
    resource_provider_bandwidths = ens5:10000000:10000000,...
    resource_provider_hypervisors = ens5:hypervisor1,...

    For both agents 'resource_provider_hypervisors' values default to
    socket.gethostname() for each key in resource_provider_bandwidths.

    We try to not block later developments in which one neutron
    agent may manage devices on multiple hosts. That's why we allow
    the each physdev to be associated with a different hypervisor.

    But here we do not try to solve the problem that the natural physdev
    identifiers may not be unique accross multiple hosts. We leave solving
    this problem to whoever wants to implement an agent handling devices of
    multiple hosts.

    (3) We extend report_state message's configurations field alike:

    {
    'bridge_mappings': {'physnet0': 'br-physnet0'},
    'resource_provider_bandwidths': {
        'br-physnet0': {'egress': 10000000, 'ingress': 10000000}},
    'resource_provider_hypervisors': {'br-physnet0': 'hypervisor0'},
    ...
    }

    (4) In neutron-server we use
    report_state.configurations.resource_provider_hypervisors.PHYSDEV
    when selecting parent resource provider ...

Related fix proposed to branch: master
Review: https://review.opendev.org/702389

Reviewed: https://review.opendev.org/702389
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=03ad5bf19c916d50b04dbcfb552ddbf96e243b54
Submitter: Zuul
Branch: master

commit 03ad5bf19c916d50b04dbcfb552ddbf96e243b54
Author: Bence Romsics <email address hidden>
Date: Tue Jan 14 11:11:28 2020 +0100

    Fix bug number in release note

    I mixed up the bug number and the gerrit change number in a previous
    release note.

    TrivialChange

    Change-Id: I7599be8b7459c44db8e34d0de536154dccb19134
    Related-Bug: #1853840

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/702877

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/702878

Reviewed: https://review.opendev.org/699174
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=9a6766470ef127ee5495a5b74b7156bd5a80f03c
Submitter: Zuul
Branch: stable/train

commit 9a6766470ef127ee5495a5b74b7156bd5a80f03c
Author: Bence Romsics <email address hidden>
Date: Wed Nov 27 17:59:15 2019 +0100

    Locate RP-tree parent by hypervisor name

    Previously we assumed that we can look up the resource provider (created
    by nova) to be used as the parent of the agent and physical NIC resource
    provider tree by the name set in the config option DEFAULT.host. This
    assumption was wrong.

    While nova-compute's DEFAULT.host and neutron-agent's DEFAULT.host
    must match for port binding to work, the root resource provider created
    by nova does not belong to the compute host (where nova-compute runs)
    but it belongs to the compute nodes (i.e. hypervisors). Actually there
    may be multiple compute nodes managed by a single nova-compute (think
    of ironic). Plus the value of DEFAULT.host and the compute node's ID
    may be different even when nova-compute manages a hypervisor on the
    same host because of various deployment considerations. For example
    when tripleo does not manage the undercloud (so a libvirt hypervisor
    returns the plain hostname), but the same tripleo enforces it's host
    naming conventions in nova's and neutron's DEFAULT.host settings.

    This change enables neutron to use the hypervisor name to locate the
    root of the resource provider tree.

    We introduce a new configuration option for

    (1) ovs-agent: resource_provider_hypervisors, for example:

    [ovs]
    bridge_mappings = physnet0:br-physnet0,...
    resource_provider_bandwidths = br-physnet0:10000000:10000000,...
    resource_provider_hypervisors = br-physnet0:hypervisor0,...

    (2) sriov-agent: resource_provider_hypervisors, for example:

    [sriov_nic]
    bridge_mappings = physnet1:ens5,...
    resource_provider_bandwidths = ens5:10000000:10000000,...
    resource_provider_hypervisors = ens5:hypervisor1,...

    For both agents 'resource_provider_hypervisors' values default to
    socket.gethostname() for each key in resource_provider_bandwidths.

    We try to not block later developments in which one neutron
    agent may manage devices on multiple hosts. That's why we allow
    the each physdev to be associated with a different hypervisor.

    But here we do not try to solve the problem that the natural physdev
    identifiers may not be unique accross multiple hosts. We leave solving
    this problem to whoever wants to implement an agent handling devices of
    multiple hosts.

    (3) We extend report_state message's configurations field alike:

    {
    'bridge_mappings': {'physnet0': 'br-physnet0'},
    'resource_provider_bandwidths': {
        'br-physnet0': {'egress': 10000000, 'ingress': 10000000}},
    'resource_provider_hypervisors': {'br-physnet0': 'hypervisor0'},
    ...
    }

    (4) In neutron-server we use
    report_state.configurations.resource_provider_hypervisors.PHYSDEV
    when selecting parent resource pro...

Reviewed: https://review.opendev.org/702877
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=135e37ea52aa7598a00afbed8c8184eac94da74d
Submitter: Zuul
Branch: stable/train

commit 135e37ea52aa7598a00afbed8c8184eac94da74d
Author: Bence Romsics <email address hidden>
Date: Tue Jan 14 11:11:28 2020 +0100

    Fix bug number in release note

    I mixed up the bug number and the gerrit change number in a previous
    release note.

    TrivialChange

    Change-Id: I7599be8b7459c44db8e34d0de536154dccb19134
    Related-Bug: #1853840
    (cherry picked from commit 03ad5bf19c916d50b04dbcfb552ddbf96e243b54)

Reviewed: https://review.opendev.org/699196
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=b99dee2df6045e5fe0896acefc591798953d375d
Submitter: Zuul
Branch: stable/stein

commit b99dee2df6045e5fe0896acefc591798953d375d
Author: Bence Romsics <email address hidden>
Date: Wed Nov 27 17:59:15 2019 +0100

    Locate RP-tree parent by hypervisor name

    Previously we assumed that we can look up the resource provider (created
    by nova) to be used as the parent of the agent and physical NIC resource
    provider tree by the name set in the config option DEFAULT.host. This
    assumption was wrong.

    While nova-compute's DEFAULT.host and neutron-agent's DEFAULT.host
    must match for port binding to work, the root resource provider created
    by nova does not belong to the compute host (where nova-compute runs)
    but it belongs to the compute nodes (i.e. hypervisors). Actually there
    may be multiple compute nodes managed by a single nova-compute (think
    of ironic). Plus the value of DEFAULT.host and the compute node's ID
    may be different even when nova-compute manages a hypervisor on the
    same host because of various deployment considerations. For example
    when tripleo does not manage the undercloud (so a libvirt hypervisor
    returns the plain hostname), but the same tripleo enforces it's host
    naming conventions in nova's and neutron's DEFAULT.host settings.

    This change enables neutron to use the hypervisor name to locate the
    root of the resource provider tree.

    We introduce a new configuration option for

    (1) ovs-agent: resource_provider_hypervisors, for example:

    [ovs]
    bridge_mappings = physnet0:br-physnet0,...
    resource_provider_bandwidths = br-physnet0:10000000:10000000,...
    resource_provider_hypervisors = br-physnet0:hypervisor0,...

    (2) sriov-agent: resource_provider_hypervisors, for example:

    [sriov_nic]
    bridge_mappings = physnet1:ens5,...
    resource_provider_bandwidths = ens5:10000000:10000000,...
    resource_provider_hypervisors = ens5:hypervisor1,...

    For both agents 'resource_provider_hypervisors' values default to
    socket.gethostname() for each key in resource_provider_bandwidths.

    We try to not block later developments in which one neutron
    agent may manage devices on multiple hosts. That's why we allow
    the each physdev to be associated with a different hypervisor.

    But here we do not try to solve the problem that the natural physdev
    identifiers may not be unique accross multiple hosts. We leave solving
    this problem to whoever wants to implement an agent handling devices of
    multiple hosts.

    (3) We extend report_state message's configurations field alike:

    {
    'bridge_mappings': {'physnet0': 'br-physnet0'},
    'resource_provider_bandwidths': {
        'br-physnet0': {'egress': 10000000, 'ingress': 10000000}},
    'resource_provider_hypervisors': {'br-physnet0': 'hypervisor0'},
    ...
    }

    (4) In neutron-server we use
    report_state.configurations.resource_provider_hypervisors.PHYSDEV
    when selecting parent resource pro...

Reviewed: https://review.opendev.org/702878
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=cde3c76ec093d82c6287dbabbe64d4ba021d7a85
Submitter: Zuul
Branch: stable/stein

commit cde3c76ec093d82c6287dbabbe64d4ba021d7a85
Author: Bence Romsics <email address hidden>
Date: Tue Jan 14 11:11:28 2020 +0100

    Fix bug number in release note

    I mixed up the bug number and the gerrit change number in a previous
    release note.

    TrivialChange

    Change-Id: I7599be8b7459c44db8e34d0de536154dccb19134
    Related-Bug: #1853840
    (cherry picked from commit 03ad5bf19c916d50b04dbcfb552ddbf96e243b54)
    (cherry picked from commit 135e37ea52aa7598a00afbed8c8184eac94da74d)

This issue was fixed in the openstack/neutron 15.0.2 release.

This issue was fixed in the openstack/neutron 14.1.0 release.

This issue was fixed in the openstack/neutron 16.0.0.0b1 development milestone.

Reviewed: https://review.opendev.org/c/openstack/neutron/+/782553
Committed: https://opendev.org/openstack/neutron/commit/7f35e4e857f7c6e83c635125ce9b42df6e10a510
Submitter: "Zuul (22348)"
Branch: master

commit 7f35e4e857f7c6e83c635125ce9b42df6e10a510
Author: Bence Romsics <email address hidden>
Date: Tue Mar 23 14:07:36 2021 +0100

    Physical NIC RP should be child of agent RP

    In the fix for #1853840 I made a mistake and since then we created
    the physical NIC resource providers as a child of the hypervisor
    resource provider instead of the agent resource provider. Here:

    https://review.opendev.org/c/openstack/neutron/+/696600/3/neutron/agent/common/placement_report.py#159

    This *did not* break the minimum bandwidth aware scheduling.
    But still there are multiple problems:

    1) If you created your physical NIC RPs before the fix for #1853840
       but upgraded to after the fix for #1853840, then resource syncs
       will throw an error in neutron-server at each physical NIC RP
       update. That pollutes the logs and wastes some resources since
       the prohibited update will be forever retried.

    2) If you created your physical NIC RPs after the fix for #1853840
       then your physical NIC RPs have the wrong parent. Which again
       does not break minimum bandwidth aware scheduling. But it may pose
       problems for later features wanting to build on the originally
       planned RP tree structure.

    3) Cleanup of decommissioned RPs is a bit different than expected.
       This cleanup was always left to the admin, so it only affects a
       manual process.

    The proper RP structure was and should be the following:

    The hypervisor RP(s) must be the root(s).
    As a child of each hypervisor RP, there should be an agent RP.
    The physical NIC RPs should be the children of the agent RPs.

    Unfortunately at the moment the Placement API generically prohibits
    update of the parent resource provider id in a PUT request:

    https://docs.openstack.org/api-ref/placement/?expanded=update-resource-provider-detail#update-resource-provider

    Therefore without a later Placement change we cannot fix the RPs
    already created with the wrong parent. However we can fix the RPs
    to be created later. We do that here. We also fix a bug in the unit
    tests that allowed the wrong parent to pass unnoticed. Plus we
    add an extra log message to direct the user seeing the pollution
    in the logs to the proper bug report.

    There may be a follow up patch later, because not all RP re-parenting
    operations are problematic, therefore we are thinking of relaxing
    this blanket prohibition in Placement. When Placement allows updates
    to the parent id we can fix RPs already created with the wrong parent
    too.

    Change-Id: I7caa8827d22103600ca685a58294640fc831dbd9
    Closes-Bug: #1921150
    Co-Authored-By: "Balazs Gibizer" <email address hidden>
    Related-Bug: #1853840

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/789674

Reviewed: https://review.opendev.org/c/openstack/neutron/+/789674
Committed: https://opendev.org/openstack/neutron/commit/d3be39433cb43bcaceb36a04d2accd6ff9a3aa8b
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit d3be39433cb43bcaceb36a04d2accd6ff9a3aa8b
Author: Bence Romsics <email address hidden>
Date: Tue Mar 23 14:07:36 2021 +0100

    Physical NIC RP should be child of agent RP

    In the fix for #1853840 I made a mistake and since then we created
    the physical NIC resource providers as a child of the hypervisor
    resource provider instead of the agent resource provider. Here:

    https://review.opendev.org/c/openstack/neutron/+/696600/3/neutron/agent/common/placement_report.py#159

    This *did not* break the minimum bandwidth aware scheduling.
    But still there are multiple problems:

    1) If you created your physical NIC RPs before the fix for #1853840
       but upgraded to after the fix for #1853840, then resource syncs
       will throw an error in neutron-server at each physical NIC RP
       update. That pollutes the logs and wastes some resources since
       the prohibited update will be forever retried.

    2) If you created your physical NIC RPs after the fix for #1853840
       then your physical NIC RPs have the wrong parent. Which again
       does not break minimum bandwidth aware scheduling. But it may pose
       problems for later features wanting to build on the originally
       planned RP tree structure.

    3) Cleanup of decommissioned RPs is a bit different than expected.
       This cleanup was always left to the admin, so it only affects a
       manual process.

    The proper RP structure was and should be the following:

    The hypervisor RP(s) must be the root(s).
    As a child of each hypervisor RP, there should be an agent RP.
    The physical NIC RPs should be the children of the agent RPs.

    Unfortunately at the moment the Placement API generically prohibits
    update of the parent resource provider id in a PUT request:

    https://docs.openstack.org/api-ref/placement/?expanded=update-resource-provider-detail#update-resource-provider

    Therefore without a later Placement change we cannot fix the RPs
    already created with the wrong parent. However we can fix the RPs
    to be created later. We do that here. We also fix a bug in the unit
    tests that allowed the wrong parent to pass unnoticed. Plus we
    add an extra log message to direct the user seeing the pollution
    in the logs to the proper bug report.

    There may be a follow up patch later, because not all RP re-parenting
    operations are problematic, therefore we are thinking of relaxing
    this blanket prohibition in Placement. When Placement allows updates
    to the parent id we can fix RPs already created with the wrong parent
    too.

    Change-Id: I7caa8827d22103600ca685a58294640fc831dbd9
    Closes-Bug: #1921150
    Co-Authored-By: "Balazs Gibizer" <email address hidden>
    Related-Bug: #1853840
    (cherry picked from commit 7f35e4e857f7c6e83c635125ce9b42df6e10a510)

Related fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/790270

Reviewed: https://review.opendev.org/c/openstack/neutron/+/790270
Committed: https://opendev.org/openstack/neutron/commit/11904b20ad6ce17904f2a685438d7985e32e2cd7
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 11904b20ad6ce17904f2a685438d7985e32e2cd7
Author: Bence Romsics <email address hidden>
Date: Tue Mar 23 14:07:36 2021 +0100

    Physical NIC RP should be child of agent RP

    In the fix for #1853840 I made a mistake and since then we created
    the physical NIC resource providers as a child of the hypervisor
    resource provider instead of the agent resource provider. Here:

    https://review.opendev.org/c/openstack/neutron/+/696600/3/neutron/agent/common/placement_report.py#159

    This *did not* break the minimum bandwidth aware scheduling.
    But still there are multiple problems:

    1) If you created your physical NIC RPs before the fix for #1853840
       but upgraded to after the fix for #1853840, then resource syncs
       will throw an error in neutron-server at each physical NIC RP
       update. That pollutes the logs and wastes some resources since
       the prohibited update will be forever retried.

    2) If you created your physical NIC RPs after the fix for #1853840
       then your physical NIC RPs have the wrong parent. Which again
       does not break minimum bandwidth aware scheduling. But it may pose
       problems for later features wanting to build on the originally
       planned RP tree structure.

    3) Cleanup of decommissioned RPs is a bit different than expected.
       This cleanup was always left to the admin, so it only affects a
       manual process.

    The proper RP structure was and should be the following:

    The hypervisor RP(s) must be the root(s).
    As a child of each hypervisor RP, there should be an agent RP.
    The physical NIC RPs should be the children of the agent RPs.

    Unfortunately at the moment the Placement API generically prohibits
    update of the parent resource provider id in a PUT request:

    https://docs.openstack.org/api-ref/placement/?expanded=update-resource-provider-detail#update-resource-provider

    Therefore without a later Placement change we cannot fix the RPs
    already created with the wrong parent. However we can fix the RPs
    to be created later. We do that here. We also fix a bug in the unit
    tests that allowed the wrong parent to pass unnoticed. Plus we
    add an extra log message to direct the user seeing the pollution
    in the logs to the proper bug report.

    There may be a follow up patch later, because not all RP re-parenting
    operations are problematic, therefore we are thinking of relaxing
    this blanket prohibition in Placement. When Placement allows updates
    to the parent id we can fix RPs already created with the wrong parent
    too.

    Change-Id: I7caa8827d22103600ca685a58294640fc831dbd9
    Closes-Bug: #1921150
    Co-Authored-By: "Balazs Gibizer" <email address hidden>
    Related-Bug: #1853840
    (cherry picked from commit 7f35e4e857f7c6e83c635125ce9b42df6e10a510)
    (cherry picked from commit d3be39433cb43bcaceb36a04d2accd6ff9a...