OpenStack Compute (nova)

live migration job blowing up with "InvalidSharedStorage: ubuntu-xenial-2-node-osic-cloud1-s3500-8668392-565309 is not on shared storage: Live migration can not be used without shared storage except a booted from volume VM which does not have a local disk."

Bug #1687511 reported by Matt Riedemann
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Sean Dague

Bug Description

Seen here:

http://logs.openstack.org/48/461448/1/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/a3f4426/logs/screen-n-cpu.txt.gz?level=TRACE#_May_01_22_15_43_039605

May 01 22:15:43.039605 ubuntu-xenial-2-node-osic-cloud1-s3500-8668392 nova-compute[22173]: ERROR oslo_messaging.rpc.server [req-233741a8-bb50-4bf6-a9ec-6c67e6360d39 tempest-LiveBlockMigrationTestJSON-978755190 tempest-LiveBlockMigrationTestJSON-978755190] Exception during message handling
                                                                                           ERROR oslo_messaging.rpc.server Traceback (most recent call last):
                                                                                           ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 157, in _process_incoming
                                                                                           ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
                                                                                           ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
                                                                                           ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
                                                                                           ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
                                                                                           ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
                                                                                           ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/exception_wrapper.py", line 77, in wrapped
                                                                                           ERROR oslo_messaging.rpc.server function_name, call_dict, binary)
                                                                                           ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
                                                                                           ERROR oslo_messaging.rpc.server self.force_reraise()
                                                                                           ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
                                                                                           ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
                                                                                           ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/exception_wrapper.py", line 68, in wrapped
                                                                                           ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
                                                                                           ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/compute/utils.py", line 660, in decorated_function
                                                                                           ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
                                                                                           ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/compute/manager.py", line 215, in decorated_function
                                                                                           ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info())
                                                                                           ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
                                                                                           ERROR oslo_messaging.rpc.server self.force_reraise()
                                                                                           ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
                                                                                           ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
                                                                                           ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/compute/manager.py", line 203, in decorated_function
                                                                                           ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
                                                                                           ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/compute/manager.py", line 5178, in check_can_live_migrate_destination
                                                                                           ERROR oslo_messaging.rpc.server disk_over_commit)
                                                                                           ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/compute/manager.py", line 5194, in _do_check_can_live_migrate_destination
                                                                                           ERROR oslo_messaging.rpc.server dest_check_data)
                                                                                           ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/compute/rpcapi.py", line 478, in check_can_live_migrate_source
                                                                                           ERROR oslo_messaging.rpc.server dest_check_data=dest_check_data)
                                                                                           ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call
                                                                                           ERROR oslo_messaging.rpc.server retry=self.retry)
                                                                                           ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 98, in _send
                                                                                           ERROR oslo_messaging.rpc.server timeout=timeout, retry=retry)
                                                                                           ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 505, in send
                                                                                           ERROR oslo_messaging.rpc.server retry=retry)
                                                                                           ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 496, in _send
                                                                                           ERROR oslo_messaging.rpc.server raise result
                                                                                           ERROR oslo_messaging.rpc.server InvalidSharedStorage_Remote: ubuntu-xenial-2-node-osic-cloud1-s3500-8668392-565309 is not on shared storage: Live migration can not be used without shared storage except a booted from volume VM which does not have a local disk.

Looks like this just started today (5/1):

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22is%20not%20on%20shared%20storage%3A%20Live%20migration%20can%20not%20be%20used%20without%20shared%20storage%20except%20a%20booted%20from%20volume%20VM%20which%20does%20not%20have%20a%20local%20disk%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d

Revision history for this message
Matt Riedemann (mriedem) wrote :

The non-shared storage live migration test run is passing. Things fail when running against ceph, which starts the setup here:

http://logs.openstack.org/48/461448/1/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/a3f4426/console.html#_2017-05-01_22_14_21_133356

tags: added: ceph
Revision history for this message
Matt Riedemann (mriedem) wrote :

My guess is the switch to systemd in devstack yesterday broke something in the ceph job:

https://review.openstack.org/#/c/460062/

Revision history for this message
Matt Riedemann (mriedem) wrote :

Yeah it's definitely something with using systemd now, which the in-tree ceph setup scripts aren't setup to handle for the live migration job. So we might need to disable those temporarily and then fix them to work with systemd, since I think USE_SCREEN in devstack is going away, but that's another temporary alternative.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/461801

Changed in nova:
assignee: nobody → Matt Riedemann (mriedem)
status: Confirmed → In Progress
Revision history for this message
Matt Riedemann (mriedem) wrote :

https://review.openstack.org/#/c/461803/

Changed in nova:
assignee: Matt Riedemann (mriedem) → Sean Dague (sdague)
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.openstack.org/461801
Reason: https://review.openstack.org/#/c/461803/ works.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/461803
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e59eee6db5876fa86954f1cab10c667c80f35763
Submitter: Jenkins
Branch: master

commit e59eee6db5876fa86954f1cab10c667c80f35763
Author: Sean Dague <email address hidden>
Date: Tue May 2 10:16:51 2017 -0400

    Use systemctl to restart services

    The restart code for the live migration configure on the fly tests was
    really really fragile clippings of old bits of devstack code that
    barely worked (and assumed a very specific run environment).

    Given that we're going to get rid of everything other than systemd in
    devstack, just jump straight to using systemd primitives here. This
    dramatically simplifies the code, and will ensure things actually
    restart correctly.

    Closes-Bug: #1687511

    Change-Id: I914430d68f64d29932c9409d055b15e4cb384ec4

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/465766

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (stable/ocata)

Change abandoned by Matt Riedemann (<email address hidden>) on branch: stable/ocata
Review: https://review.openstack.org/465766
Reason: We don't need this in ocata since we don't run systemd in newton or ocata.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 16.0.0.0b2

This issue was fixed in the openstack/nova 16.0.0.0b2 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.