OpenStack Compute (nova)

AntiAffinity policy problems with resize and migrate to same host

Bug #1558532 reported by John Smith
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Undecided
Charlotte Han

Bug Description

1. version
kilo 2015.1.0

2.Relevant log files:
nova-scheduler.log
（Log contains “========” is added by myself）

2016-03-17 17:54:23.092 15208 DEBUG nova.scheduler.filters.affinity_filter [req-3612987d-bef8-4727-811f-1dd668ac3328 eae626536b034af1a3635a36645dc54a 4d054614f49644c1935bdade9703c99b - - -] ================group_hosts: '[u'SBCJNailSlot3', u'sbcj-chenling-02-slot11']' host_passes /usr/lib/python2.7/site-packages/nova/scheduler/filters/affinity_filter.py:98
2016-03-17 17:54:23.093 15208 DEBUG nova.scheduler.filters.affinity_filter [req-3612987d-bef8-4727-811f-1dd668ac3328 eae626536b034af1a3635a36645dc54a 4d054614f49644c1935bdade9703c99b - - -] Group anti affinity: check if sbcj-chenling-02-slot11 not in [u'SBCJNailSlot3', u'sbcj-chenling-02-slot11'] host_passes /usr/lib/python2.7/site-packages/nova/scheduler/filters/affinity_filter.py:101
2016-03-17 17:54:23.094 15208 DEBUG nova.scheduler.filters.affinity_filter [req-3612987d-bef8-4727-811f-1dd668ac3328 eae626536b034af1a3635a36645dc54a 4d054614f49644c1935bdade9703c99b - - -] ================group_hosts: '[u'SBCJNailSlot3', u'sbcj-chenling-02-slot11']' host_passes /usr/lib/python2.7/site-packages/nova/scheduler/filters/affinity_filter.py:98
2016-03-17 17:54:23.094 15208 DEBUG nova.scheduler.filters.affinity_filter [req-3612987d-bef8-4727-811f-1dd668ac3328 eae626536b034af1a3635a36645dc54a 4d054614f49644c1935bdade9703c99b - - -] Group anti affinity: check if SBCJNailSlot3 not in [u'SBCJNailSlot3', u'sbcj-chenling-02-slot11'] host_passes /usr/lib/python2.7/site-packages/nova/scheduler/filters/affinity_filter.py:101
2016-03-17 17:54:23.094 15208 INFO nova.filters [req-3612987d-bef8-4727-811f-1dd668ac3328 eae626536b034af1a3635a36645dc54a 4d054614f49644c1935bdade9703c99b - - -] Filter ServerGroupAntiAffinityFilter returned 0 hosts for instance c48ee7d6-451e-4b96-ac39-261082f8950e
2016-03-17 17:54:23.095 15208 DEBUG nova.scheduler.filter_scheduler [req-3612987d-bef8-4727-811f-1dd668ac3328 eae626536b034af1a3635a36645dc54a 4d054614f49644c1935bdade9703c99b - - -] There are 0 hosts available but 1 instances requested to build. select_destinations /usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:77

3.Reproduce steps:
*3.1
create a anti-affinity group with policy 'anti-affinity' named "antiaffinitygroup"
boot two vms using "antiaffinitygroup",named "test_wzz1" and "test_wzz2"

results:
[root@sbcj-chenling-02-slot11 ~(keystone_admin)]# nova list
+--------------------------------------+-----------+--------+------------+-------------+-------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-----------+--------+------------+-------------+-------------------+
| c48ee7d6-451e-4b96-ac39-261082f8950e | test_wzz1 | ACTIVE | - | Running | test0301=3.3.3.60 |
| 55107de8-38bc-4b3a-8681-df17746e4f11 | test_wzz2 | ACTIVE | - | Running | test0301=3.3.3.61 |
+--------------------------------------+-----------+--------+------------+-------------+-------------------+

[root@sbcj-chenling-02-slot11 ~(keystone_admin)]# nova server-group-list
+--------------------------------------+-------------------+--------------------+------------------------------------------------------------------------------------+----------+
| Id | Name | Policies | Members | Metadata |
+--------------------------------------+-------------------+--------------------+------------------------------------------------------------------------------------+----------+
| 79ebfbfc-6e7c-4bf0-97c5-14d7fca22e96 | antiaffinitygroup | [u'anti-affinity'] | [u'55107de8-38bc-4b3a-8681-df17746e4f11', u'c48ee7d6-451e-4b96-ac39-261082f8950e'] | {} |
+--------------------------------------+-------------------+--------------------+------------------------------------------------------------------------------------+----------+

*3.2
I have two compute nodes：sbcj-chenling-02-slot11，SBCJNailSlot3.
The vm test_wzz1 in SBCJNailSlot3,and the test_wzz2 in sbcj-chenling-02-slot11,because of the anti-affinity group
 Then set allow_resize_to_same_host=true,and add ServerGroupAntiAffinityFilter in scheduler_default_filters in nova.conf,and restart services.

*3.3
 now，execute the command : nova resize test_wzz1 m1.small

Example：
[root@sbcj-chenling-02-slot11 ~(keystone_admin)]# nova resize test_wzz1 m1.small

Expected result:
vm resize to the same host successfully.

Actual result:
ERROR (BadRequest): No valid host was found. No valid host found for resize: No valid host was found. There are not enough hosts available. (HTTP 400) (Request-ID: req-3612987d-bef8-4727-811f-1dd668ac3328)
Relevant log files: See section 2

4.Bonus points for reproducing shell script / test:
Operation cold-migration has the same problem.
It seems test_wzz1 anti-affinity against to itself,so resize failed although we allow vm resize to same host.
This is unreasonable.

John Smith (wang-zengzhi)
Changed in nova:
assignee: nobody → Zengzhi Wang (wang-zengzhi)
Revision history for this message
Matt Riedemann (mriedem) wrote :

Have you reproduced this on the latest master branch code or newton rc1?

tags: added: resize scheduler
Changed in nova:
status: New → Confirmed
summary: - AntiAffinity policy problems with resize and migrate
+ AntiAffinity policy problems with resize and migrate to same host
Charlotte Han (hanrong)
Changed in nova:
assignee: Zengzhi Wang (wang-zengzhi) → Rong Han ZTE (hanrong)
Revision history for this message
Charlotte Han (hanrong) wrote :

I reproduced this in the latest master branche code.
[tecs@tecs-OpenStack-Nova devstack(keystone_admin)]$ git log -1
commit 0a2a7ae8471575d887a19b9bf8b7375ea1fd7f45
Merge: 467dbf7 8ff298a
Author: Jenkins <email address hidden>
Date: Fri Mar 18 20:24:03 2016 +0000

    Merge "Updated deprecated keystone_authtoken option"
[tecs@tecs-OpenStack-Nova devstack(keystone_admin)]$

Revision history for this message
Charlotte Han (hanrong) wrote :
Download full text (4.4 KiB)

Only one compute node.

1. create anti-affinity group
$ nova server-group-create --policy anti-affinity antiaffinitygroup
WARNING: Option "--policy" is deprecated; use positional parameters; this option will be removed in novaclient 3.3.0.
+--------------------------------------+-------------------+----------------------------------+----------------------------------+--------------------+---------+----------+
| Id | Name | Project Id | User Id | Policies | Members | Metadata |
 +--------------------------------------+-------------------+----------------------------------+----------------------------------+--------------------+---------+----------+
| 107174bc-9694-43cd-a6de-b4eb98cc8435 | antiaffinitygroup | 5c4790fed10549fcaf1f3f47a2b3ab7e | 823805f0ade648ac9a69803d7f1d8cfc | [u'anti-affinity'] | [] | {} |
 +--------------------------------------+-------------------+----------------------------------+----------------------------------+--------------------+---------+----------+

2. boot a instance in this group
$ nova boot --flavor 1 --image 425846b1-4fc8-428a-ac8a-3251a88ba7c8 --nic net-id=274d04e3-fbfc-497c-b3b2-3ce6d07c290b --hint group=107174bc-9694-43cd-a6de-b4eb98cc8435 hanrong_in_gourp1

3. nova list
+--------------------------------------+-------------------+--------+------------+-------------+--------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
 +--------------------------------------+-------------------+--------+------------+-------------+--------------------------------+
| 7a03f518-8726-4348-9caa-7ad5e8b67c92 | hanrong_in_gourp1 | ACTIVE | - | Running | public=2001:db8::5, 172.24.4.5 |
 +--------------------------------------+-------------------+--------+------------+-------------+--------------------------------+

4. $ nova flavor-create hanrong 111 512 1 2
+-----+---------+-----------+------+-----------+------+-------+-------------+-----------+
| ID | Name | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public |
 +-----+---------+-----------+------+-----------+------+-------+-------------+-----------+
| 111 | hanrong | 512 | 1 | 0 | | 2 | 1.0 | True |
 +-----+---------+-----------+------+-----------+------+-------+-------------+-----------+

5. nova resize 7a03f518-8726-4348-9caa-7ad5e8b67c92 111
ERROR (BadRequest): No valid host was found. No valid host found for resize (HTTP 400) (Request-ID: req-23bd6d72-15ea-40a9-8cb0-b87dfee9a7ac)

6. log: n-sch.log
2016-03-29 10:22:28.060 ^[[00;36mINFO nova.filters [^[[01;36mreq-23bd6d72-15ea-40a9-8cb0-b87dfee9a7ac ^[[00;36madmin admin^[[00;36m] ^[[01;35m^[[00;36mFilter ServerGroupAntiAffinityFilter returned 0 hosts^[[00m
2016-03-29 10:22:28.060 ^[[00;32mDEBUG nova.filters [^[[01;36mreq-23bd6d72-15ea-40a9-8cb0-b87dfee9a7ac ^[[00;36madmin admin^[[00;32m] ^[[01;35m^[[00;32mFiltering removed all hosts for the request with instance ID '7a03f518-8726-4348-9caa-7ad5e8b67c92'. Filter results: [('RetryFilter'...

Read more...

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/299045

Changed in nova:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/299045
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=609adcd38cdcc8e1906d8b4f93a72b7fa2032817
Submitter: Jenkins
Branch: master

commit 609adcd38cdcc8e1906d8b4f93a72b7fa2032817
Author: zte-hanrong <email address hidden>
Date: Tue Mar 29 20:19:01 2016 +0800

    Fix resize to same host failed using anti-affinity group

    ServerGroupAntiAffinityFilter selects host which is not in in the
    hosts of the members of the group. An instance does not have
    anti-affinity with itself.
    The current code has problem when select hosts, the instance's
    current host will be filtered out by ServerGroupAntiAffinityFilter.
    This can lead to failure when resize an instance with anti-affinity
    group to the same host.

    In this patch, ServerGroupAntiAffinityFite is modified.

    Change-Id: Ie016f59f5b98bb9c70b3e33556bd747f79fc77bd
    Closes-Bug:#1558532

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote : Fix included in openstack/nova 14.0.0.0b1

This issue was fixed in the openstack/nova 14.0.0.0b1 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.