[sru] Nova-compute doesnt start

please add more info - like what release you are running,

With lxc virt type and rc1 icehouse this is being observed

Lxc/libvirt isn't supported. https://wiki.openstack.org/wiki/HypervisorSupportMatrix

also has the problem with libxl
2014-04-21 18:23:14.809 TRACE nova.openstack.common.threadgroup caps = self.get_host_capabilities()
2014-04-21 18:23:14.809 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2857, in get_host_capabilities
2014-04-21 18:23:14.809 TRACE nova.openstack.common.threadgroup libvirt.VIR_CONNECT_BASELINE_CPU_EXPAND_FEATURES)
2014-04-21 18:23:14.809 TRACE nova.openstack.common.threadgroup File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 179, in doit
2014-04-21 18:23:14.809 TRACE nova.openstack.common.threadgroup result = proxy_call(self._autowrap, f, *args, **kwargs)
2014-04-21 18:23:14.809 TRACE nova.openstack.common.threadgroup File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 139, in proxy_call
2014-04-21 18:23:14.809 TRACE nova.openstack.common.threadgroup rv = execute(f,*args,**kwargs)
2014-04-21 18:23:14.809 TRACE nova.openstack.common.threadgroup File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 77, in tworker
2014-04-21 18:23:14.809 TRACE nova.openstack.common.threadgroup rv = meth(*args,**kwargs)
2014-04-21 18:23:14.809 TRACE nova.openstack.common.threadgroup File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 3157, in baselineCPU
2014-04-21 18:23:14.809 TRACE nova.openstack.common.threadgroup if ret is None: raise libvirtError ('virConnectBaselineCPU() failed', conn=self)
2014-04-21 18:23:14.809 TRACE nova.openstack.common.threadgroup libvirtError: this function is not supported by the connection driver: virConnectBaselineCPU

Fix proposed to branch: master
Review: https://review.openstack.org/89333

Reviewed: https://review.openstack.org/82831
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5fc157e0f5245eb79be90cbc8ed545c396fd38af
Submitter: Jenkins
Branch: master

commit 5fc157e0f5245eb79be90cbc8ed545c396fd38af
Author: Chuck Short <email address hidden>
Date: Tue Mar 25 10:08:28 2014 -0400

    libvirt: Handle unsupported host capabilities

    Neither libvirt-xen nor libvirt-lxc support
    VIR_CONNECT_BASELINE_CPU_EXPAND_FEATURES so check for the proper error
    code that is returned by libvirt.

    The existing code was performing this check improperly by trying to
    catch an enum value; instead, we need to catch the `libvirtError`
    exception and then check its return code.

    Co-Authored-By: Rick Harris <email address hidden>
    Co-Authored-By: Andrew Melton <email address hidden>
    Closes-Bug: 1297962
    Change-Id: Ie8bf5c9d1f2a27c387f8b2f54a9bb729fa2f0985

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/94409

[ Impact ]

Nova 2014.1 was released a regression that prevented LXC containers and Xen domains from working when trying to start the container. When a user tries to start a container they will get the above traceback in the error log. This is due to libvirt-lxc/libvirt-xen not supporting the VIR_CONNECT_BASELINE_CPU_EXPAND_FEATURES feature in libvirt.

[ Test Case ]

1. Install nova.
2. Configure nova to run with libvirt_type=lxc
3. Start an instance, it should be able to start with this fix applied.

[ Regression Potential ]

There is no regression potential for this fix.

What are the plans for fixing this in Utopic?

Brian.

Merge proposal has already been proposed upstream. The next stable release will have this fix, in the meantime this is a stop gap measure so users can still run LXC containers.

chuck

Hello Chuck, or anyone else affected,

Accepted nova into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/nova/1:2014.1-0ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

how do I install this fix?
seems like the build failed?
https://launchpad.net/ubuntu/+source/nova/1:2014.1-0ubuntu1.1/+build/6050434

This is no longer in -proposed, and 1:2014.1-0ubuntu1.2 is in -updates.

I still have this bug after updating...

root@compute2:/var/log/nova# dpkg -l | grep nova-compute

ii nova-compute 1:2014.1-0ubuntu1.2 all OpenStack Compute - compute node base
ii nova-compute-libvirt 1:2014.1-0ubuntu1.2 all OpenStack Compute - compute node libvirt support
ii nova-compute-lxc 1:2014.1-0ubuntu1.2 all OpenStack Compute - compute node (LXC)

tail /var/log/nova/nova-compute.log

2014-06-23 11:31:32.388 3603 INFO nova.openstack.common.periodic_task [-] Skipping periodic task _periodic_update_dns because its interval is negative
2014-06-23 11:31:32.445 3603 INFO nova.virt.driver [-] Loading compute driver 'libvirt.LibvirtDriver'
2014-06-23 11:31:32.476 3603 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on controller:5672
2014-06-23 11:31:32.495 3603 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on controller:5672
2014-06-23 11:31:32.520 3603 AUDIT nova.service [-] Starting compute node (version 2014.1)
2014-06-23 11:31:32.567 3603 WARNING nova.virt.libvirt.driver [req-ff8cee55-1c7a-4720-abb6-66431ebd7081 None None] Cannot update service status on host: compute2,since it is not registered.
2014-06-23 11:31:32.569 3603 WARNING nova.virt.libvirt.driver [-] URI lxc:/// does not support events: internal error: could not initialize domain event timer
2014-06-23 11:31:32.591 3603 ERROR nova.openstack.common.threadgroup [-] this function is not supported by the connection driver: virConnectBaselineCPU
2014-06-23 11:31:32.591 3603 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
2014-06-23 11:31:32.591 3603 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/openstack/common/threadgroup.py", line 117, in wait
2014-06-23 11:31:32.591 3603 TRACE nova.openstack.common.threadgroup x.wait()
2014-06-23 11:31:32.591 3603 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/openstack/common/threadgroup.py", line 49, in wait
2014-06-23 11:31:32.591 3603 TRACE nova.openstack.common.threadgroup return self.thread.wait()
2014-06-23 11:31:32.591 3603 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 168, in wait
2014-06-23 11:31:32.591 3603 TRACE nova.openstack.common.threadgroup return self._exit_event.wait()
2014-06-23 11:31:32.591 3603 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
2014-06-23 11:31:32.591 3603 TRACE nova.openstack.common.threadgroup return hubs.get_hub().switch()
2014-06-23 11:31:32.591 3603 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch
2014-06-23 11:31:32.591 3603 TRACE nova.openstack.common.threadgroup return self.greenlet.switch()
2014-06-23 11:31:32.591 3603 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main
2014-06-23 11:31:32.591 3603 TRACE nova.openstack.common.thr...

Hello Chuck, or anyone else affected,

Accepted nova into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/nova/1:2014.1.1-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Status changed to 'Confirmed' because the bug affects multiple users.

I can now start nova-compute after updating~

root@compute2:/var/lib/nova# dpkg -l | grep nova-compute
ii nova-compute 1:2014.1.1-0ubuntu1 all OpenStack Compute - compute node base
ii nova-compute-libvirt 1:2014.1.1-0ubuntu1 all OpenStack Compute - compute node libvirt support
ii nova-compute-lxc 1:2014.1.1-0ubuntu1 all OpenStack Compute - compute node (LXC)
root@compute2:/var/lib/nova# service nova-compute status
nova-compute start/running, process 8350

but can't launch instances...
should I file a separate bug report~?

2014-06-25 04:40:57.614 8350 INFO nova.virt.libvirt.driver [req-10a54773-5c13-435e-a9d6-ea92df2e1e00 639d9bb856fe44a98b0f09cf503bc663 1bea54ab03b14b229437a95422a5c1df] [instance: 8e4146fa-ff59-4d5e-a27a-04c3c44df4eb] Creating image
2014-06-25 04:40:57.864 8350 WARNING nova.virt.disk.vfs.guestfs [req-10a54773-5c13-435e-a9d6-ea92df2e1e00 639d9bb856fe44a98b0f09cf503bc663 1bea54ab03b14b229437a95422a5c1df] Failed to close augeas aug_close: call launch before using this function
(in guestfish, don't forget to use the 'run' command)
2014-06-25 04:40:59.908 8350 ERROR nova.virt.disk.api [req-10a54773-5c13-435e-a9d6-ea92df2e1e00 639d9bb856fe44a98b0f09cf503bc663 1bea54ab03b14b229437a95422a5c1df] Failed to mount container filesystem '<nova.virt.disk.api._DiskImage object at 0x7f7a4412c390>' on '/var/lib/nova/instances/8e4146fa-ff59-4d5e-a27a-04c3c44df4eb/rootfs':
--
Failed to mount filesystem: Unexpected error while running command.
Command: sudo nova-rootwrap /etc/nova/rootwrap.conf mount /dev/nbd7 /var/lib/nova/instances/8e4146fa-ff59-4d5e-a27a-04c3c44df4eb/rootfs
Exit code: 32
Stdout: ''
Stderr: 'mount: you must specify the filesystem type\n'
2014-06-25 04:40:59.911 8350 ERROR nova.compute.manager [req-10a54773-5c13-435e-a9d6-ea92df2e1e00 639d9bb856fe44a98b0f09cf503bc663 1bea54ab03b14b229437a95422a5c1df] [instance: 8e4146fa-ff59-4d5e-a27a-04c3c44df4eb] Instance failed to spawn
2014-06-25 04:40:59.911 8350 TRACE nova.compute.manager [instance: 8e4146fa-ff59-4d5e-a27a-04c3c44df4eb] Traceback (most recent call last):
2014-06-25 04:40:59.911 8350 TRACE nova.compute.manager [instance: 8e4146fa-ff59-4d5e-a27a-04c3c44df4eb] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1714, in _spawn
2014-06-25 04:40:59.911 8350 TRACE nova.compute.manager [instance: 8e4146fa-ff59-4d5e-a27a-04c3c44df4eb] block_device_info)
2014-06-25 04:40:59.911 8350 TRACE nova.compute.manager [instance: 8e4146fa-ff59-4d5e-a27a-04c3c44df4eb] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2266, in spawn
2014-06-25 04:40:59.911 8350 TRACE nova.compute.manager [instance: 8e4146fa-ff59-4d5e-a27a-04c3c44df4eb] block_device_info)
2014-06-25 04:40:59.911 8350 TRACE nova.compute.manager [instance: 8e4146fa-ff59-4d5e-a27a-04c3c44df4eb] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3669, in _create_domain_and_network
2014-06-25 04:40:59.911 8350 TRACE nova.compute.manager [instance: 8e4146fa-ff59-4d5e-a27a-04c3c44df4eb] domain.destroy()
2014-06-25 04:40:59.911 8350 TRACE nova.compute.manager [instance: 8e4146fa-ff59-4d5e-a27a-04c3c44df4eb] UnboundLocalError: local variable 'domain' referenced before assignment

Chris,

Can you open up another bug about your issue. Your issue is not related to the original bug report. Make sure that you have uploaded your image to glance as raw image.

Thanks
chuck

The verification of the Stable Release Update for nova has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

This bug was fixed in the package nova - 1:2014.1.1-0ubuntu1

---------------
nova (1:2014.1.1-0ubuntu1) trusty; urgency=medium

  * Resynchronize with stable/icehouse (867341f) (LP: #1328134):
    - [867341f] Fix security group race condition while listing and deleting rules
    - [ffcb176] VMware: ensure rescue instance is deleted when instance is deleted
    - [fe4fe70] VMware: Log additional details of suds faults
    - [43f0437] Add info_cache as expected attribute when evacuate instance
    - [a2da9ce] VMware: uncaught exception during snapshot deletion
    - [1a45944] Catch InstanceNotFound exception if migration fails
    - [ee374f1] Do not wait for neutron event if not powering on libvirt domain
    - [705ad64] Reap child processes gracefully if greenlet thread gets killed
    - [f769bf8] Fixes arguments parsing when executing command
    - [bedb66f] Use one query instead of two for quota_usages
    - [422decd] VMWare - Check for compute node before triggering destroy
    - [6629116] Use debug level logging in unit tests, but don't save them.
    - [088b718] support local debug logging
    - [080f785] Revert "Use debug level logging during unit tests"
    - [fb03028] VMWare: add power off vm before detach disk during unrescue
    - [d93427a] Check for None or timestamp in availability zone api sample
    - [f5c3330f] Pass configured auth strategy to neutronclient
    - [74d1043] remove unneeded call to network_api on rebuild_instance
    - [f1fdb3c] Remove unnecessary call to fetch info_cache
    - [395ec82] Remove metadata's network-api dependence on the database
    - [a48d268] InvalidCPUInfo exception added to except block
    - [77392a9] Moved the registration of lifecycle event handler in init_host()
    - [40ae1ee] Fix display of server group members
    - [66c7ca1] Change errors_out_migration decorator to work with RPC
    - [e1e140b] Don't explode if we fail to unplug VIFs after a failed boot
    - [c816488] Remove unneeded call to fetch network info on shutdown
    - [7f9f3ef] Don't overwrite instance object with dict in _init_instance()
    - [2728f1e] Fix bug detach volume fails with "KeyError" in EC2
   * debian/patches/libvirt-Handle-unsupported-host-capabilities.patch: Fix exception
     when starting LXC containers. (LP: #1297962)
 -- Chuck Short <email address hidden> Tue, 24 Jun 2014 10:47:47 -0400

Reviewed: https://review.openstack.org/94409
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=8141e7aecdccaa55d92b8174633d0cfc97845412
Submitter: Jenkins
Branch: stable/icehouse

commit 8141e7aecdccaa55d92b8174633d0cfc97845412
Author: Chuck Short <email address hidden>
Date: Tue Mar 25 10:08:28 2014 -0400

    libvirt: Handle unsupported host capabilities

    Neither libvirt-xen nor libvirt-lxc support
    VIR_CONNECT_BASELINE_CPU_EXPAND_FEATURES so check for the proper error
    code that is returned by libvirt.

    The existing code was performing this check improperly by trying to
    catch an enum value; instead, we need to catch the `libvirtError`
    exception and then check its return code.

    Co-Authored-By: Rick Harris <email address hidden>
    Co-Authored-By: Andrew Melton <email address hidden>
    Closes-Bug: 1297962
    Change-Id: Ie8bf5c9d1f2a27c387f8b2f54a9bb729fa2f0985
    (cherry picked from commit 5fc157e0f5245eb79be90cbc8ed545c396fd38af)