Ubuntu
nova package

s390x: error booting instance

Bug #1564831 reported by James Page
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
libvirt (Ubuntu)
Invalid
Undecided
Unassigned
lxcfs (Ubuntu)
Fix Released
Undecided
Unassigned
nova (Ubuntu)
Invalid
Undecided
Unassigned

Bug Description

Actual error:

libvirtError: this function is not supported by the connection driver: cannot update guest CPU data for s390x architecture

2016-04-01 10:19:51.658 60051 ERROR nova.compute.manager [req-a758aabf-6eaa-4370-962d-301205a26bf9 92e0f76cbb3d4b47b70c590c488d4233 85aff3c7356e4ddd96dc90b870321748 - - -] [instance: c29ccb17-efc9-448b-bfd1-9ff18b23dfc3] Instance failed to spawn
2016-04-01 10:19:51.658 60051 ERROR nova.compute.manager [instance: c29ccb17-efc9-448b-bfd1-9ff18b23dfc3] Traceback (most recent call last):
2016-04-01 10:19:51.658 60051 ERROR nova.compute.manager [instance: c29ccb17-efc9-448b-bfd1-9ff18b23dfc3] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2218, in _build_resources
2016-04-01 10:19:51.658 60051 ERROR nova.compute.manager [instance: c29ccb17-efc9-448b-bfd1-9ff18b23dfc3] yield resources
2016-04-01 10:19:51.658 60051 ERROR nova.compute.manager [instance: c29ccb17-efc9-448b-bfd1-9ff18b23dfc3] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2064, in _build_and_run_instance
2016-04-01 10:19:51.658 60051 ERROR nova.compute.manager [instance: c29ccb17-efc9-448b-bfd1-9ff18b23dfc3] block_device_info=block_device_info)
2016-04-01 10:19:51.658 60051 ERROR nova.compute.manager [instance: c29ccb17-efc9-448b-bfd1-9ff18b23dfc3] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2768, in spawn
2016-04-01 10:19:51.658 60051 ERROR nova.compute.manager [instance: c29ccb17-efc9-448b-bfd1-9ff18b23dfc3] block_device_info=block_device_info)
2016-04-01 10:19:51.658 60051 ERROR nova.compute.manager [instance: c29ccb17-efc9-448b-bfd1-9ff18b23dfc3] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4906, in _create_domain_and_network
2016-04-01 10:19:51.658 60051 ERROR nova.compute.manager [instance: c29ccb17-efc9-448b-bfd1-9ff18b23dfc3] xml, pause=pause, power_on=power_on)
2016-04-01 10:19:51.658 60051 ERROR nova.compute.manager [instance: c29ccb17-efc9-448b-bfd1-9ff18b23dfc3] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4837, in _create_domain
2016-04-01 10:19:51.658 60051 ERROR nova.compute.manager [instance: c29ccb17-efc9-448b-bfd1-9ff18b23dfc3] guest.launch(pause=pause)
2016-04-01 10:19:51.658 60051 ERROR nova.compute.manager [instance: c29ccb17-efc9-448b-bfd1-9ff18b23dfc3] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 142, in launch
2016-04-01 10:19:51.658 60051 ERROR nova.compute.manager [instance: c29ccb17-efc9-448b-bfd1-9ff18b23dfc3] self._encoded_xml, errors='ignore')
2016-04-01 10:19:51.658 60051 ERROR nova.compute.manager [instance: c29ccb17-efc9-448b-bfd1-9ff18b23dfc3] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-04-01 10:19:51.658 60051 ERROR nova.compute.manager [instance: c29ccb17-efc9-448b-bfd1-9ff18b23dfc3] self.force_reraise()
2016-04-01 10:19:51.658 60051 ERROR nova.compute.manager [instance: c29ccb17-efc9-448b-bfd1-9ff18b23dfc3] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-04-01 10:19:51.658 60051 ERROR nova.compute.manager [instance: c29ccb17-efc9-448b-bfd1-9ff18b23dfc3] six.reraise(self.type_, self.value, self.tb)
2016-04-01 10:19:51.658 60051 ERROR nova.compute.manager [instance: c29ccb17-efc9-448b-bfd1-9ff18b23dfc3] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 137, in launch
2016-04-01 10:19:51.658 60051 ERROR nova.compute.manager [instance: c29ccb17-efc9-448b-bfd1-9ff18b23dfc3] return self._domain.createWithFlags(flags)
2016-04-01 10:19:51.658 60051 ERROR nova.compute.manager [instance: c29ccb17-efc9-448b-bfd1-9ff18b23dfc3] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
2016-04-01 10:19:51.658 60051 ERROR nova.compute.manager [instance: c29ccb17-efc9-448b-bfd1-9ff18b23dfc3] result = proxy_call(self._autowrap, f, *args, **kwargs)
2016-04-01 10:19:51.658 60051 ERROR nova.compute.manager [instance: c29ccb17-efc9-448b-bfd1-9ff18b23dfc3] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
2016-04-01 10:19:51.658 60051 ERROR nova.compute.manager [instance: c29ccb17-efc9-448b-bfd1-9ff18b23dfc3] rv = execute(f, *args, **kwargs)
2016-04-01 10:19:51.658 60051 ERROR nova.compute.manager [instance: c29ccb17-efc9-448b-bfd1-9ff18b23dfc3] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
2016-04-01 10:19:51.658 60051 ERROR nova.compute.manager [instance: c29ccb17-efc9-448b-bfd1-9ff18b23dfc3] six.reraise(c, e, tb)
2016-04-01 10:19:51.658 60051 ERROR nova.compute.manager [instance: c29ccb17-efc9-448b-bfd1-9ff18b23dfc3] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
2016-04-01 10:19:51.658 60051 ERROR nova.compute.manager [instance: c29ccb17-efc9-448b-bfd1-9ff18b23dfc3] rv = meth(*args, **kwargs)
2016-04-01 10:19:51.658 60051 ERROR nova.compute.manager [instance: c29ccb17-efc9-448b-bfd1-9ff18b23dfc3] File "/usr/lib/python2.7/dist-packages/libvirt.py", line 1065, in createWithFlags
2016-04-01 10:19:51.658 60051 ERROR nova.compute.manager [instance: c29ccb17-efc9-448b-bfd1-9ff18b23dfc3] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2016-04-01 10:19:51.658 60051 ERROR nova.compute.manager [instance: c29ccb17-efc9-448b-bfd1-9ff18b23dfc3] libvirtError: this function is not supported by the connection driver: cannot update guest CPU data for s390x architecture

domain xml:

<domain type='kvm'>
  <name>instance-00000001</name>
  <uuid>c29ccb17-efc9-448b-bfd1-9ff18b23dfc3</uuid>
  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
      <nova:package version="13.0.0"/>
      <nova:name>trusty061932</nova:name>
      <nova:creationTime>2016-04-01 10:19:49</nova:creationTime>
      <nova:flavor name="m1.small">
        <nova:memory>2048</nova:memory>
        <nova:disk>20</nova:disk>
        <nova:swap>0</nova:swap>
        <nova:ephemeral>0</nova:ephemeral>
        <nova:vcpus>1</nova:vcpus>
      </nova:flavor>
      <nova:owner>
        <nova:user uuid="92e0f76cbb3d4b47b70c590c488d4233">admin</nova:user>
        <nova:project uuid="85aff3c7356e4ddd96dc90b870321748">admin</nova:project>
      </nova:owner>
      <nova:root type="image" uuid="8167385a-4207-4c77-af5c-2c2151e4b144"/>
    </nova:instance>
  </metadata>
  <memory unit='KiB'>2097152</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <cputune>
    <shares>1024</shares>
  </cputune>
  <os>
    <type arch='s390x' machine='s390-ccw-virtio-2.5'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-model'>
    <model fallback='allow'/>
    <topology sockets='1' cores='1' threads='1'/>
  </cpu>
  <clock offset='utc'>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='rtc' tickpolicy='catchup'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-s390x</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='writethrough'/>
      <source file='/var/lib/nova/instances/c29ccb17-efc9-448b-bfd1-9ff18b23dfc3/disk'/>
      <target dev='vda' bus='virtio'/>
      <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0000'/>
    </disk>
    <interface type='bridge'>
      <mac address='fa:16:3e:f2:9a:af'/>
      <source bridge='qbr60d6a610-02'/>
      <target dev='tap60d6a610-02'/>
      <model type='virtio'/>
      <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0001'/>
    </interface>
    <console type='file'>
      <source path='/var/lib/nova/instances/c29ccb17-efc9-448b-bfd1-9ff18b23dfc3/console.log'/>
      <target type='sclplm' port='0'/>
    </console>
    <console type='pty'>
      <target type='sclp' port='1'/>
    </console>
    <memballoon model='virtio'>
      <stats period='10'/>
      <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0002'/>
    </memballoon>
  </devices>
</domain>

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: nova-compute-kvm 2:13.0.0~rc1-0ubuntu1
ProcVersionSignature: Ubuntu 4.4.0-16.32-generic 4.4.6
Uname: Linux 4.4.0-16-generic s390x
NonfreeKernelModules: xt_REDIRECT nf_nat_redirect xt_nat xt_mark ip6table_raw nf_conntrack_ipv6 ip6table_mangle xt_connmark xt_comment xt_conntrack ipt_REJECT nf_reject_ipv4 ip6table_filter iptable_raw dm_snapshot ip6_tables nbd openvswitch nf_defrag_ipv6 binfmt_misc veth ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_CHECKSUM iptable_mangle xt_tcpudp bridge iptable_filter ip_tables x_tables dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio 8021q garp mrp stp llc chsc_sch eadm_sch qeth_l2 qeth ctcm ccwgroup fsm ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi btrfs zlib_deflate raid10 raid456 async_memcpy async_raid6_recov async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 linear ghash_s390 prng aes_s390 des_s390 des_generic sha512_s390 sha256_s390 sha1_s390 sha_common zfcp qdio scsi_transport_fc dasd_eckd_mod dasd_mod
ApportVersion: 2.20.1-0ubuntu1
Architecture: s390x
Date: Fri Apr 1 10:23:51 2016
PackageArchitecture: all
ProcEnviron:
 TERM=screen-256color-bce
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: nova
UpgradeStatus: No upgrade log present (probably fresh install)

Revision history for this message
James Page (james-page) wrote :
Revision history for this message
James Page (james-page) wrote :

Some pertinent logs from libvirt as well:

Apr 01 10:07:33 juju-e58e3f3f-abe1-4393-8212-5c6936c4d369-machine-11 libvirtd[21610]: End of file while reading data: Input/output error
Apr 01 10:07:35 juju-e58e3f3f-abe1-4393-8212-5c6936c4d369-machine-11 libvirtd[21610]: this function is not supported by the connection driver: cannot compute baseline CPU of s390x architecture
Apr 01 10:08:39 juju-e58e3f3f-abe1-4393-8212-5c6936c4d369-machine-11 libvirtd[21610]: End of file while reading data: Input/output error
Apr 01 10:08:41 juju-e58e3f3f-abe1-4393-8212-5c6936c4d369-machine-11 libvirtd[21610]: this function is not supported by the connection driver: cannot compute baseline CPU of s390x architecture
Apr 01 10:19:51 juju-e58e3f3f-abe1-4393-8212-5c6936c4d369-machine-11 libvirtd[21610]: this function is not supported by the connection driver: cannot update guest CPU data for s390x architecture

James Page (james-page)
tags: added: s390
tags: removed: s390
Revision history for this message
James Page (james-page) wrote :

Should have also mentioned that this is running in a LXD container; /proc/cpuinfo is empty for some reason which I think is the root cause of this problem.

Revision history for this message
James Page (james-page) wrote :

Mapped /proc/cpuinfo into the container from the host, but still seeing the same issue...

Revision history for this message
James Page (james-page) wrote :

Also tried with the nova-compute node running directly on Ubuntu on an LPAR but still see the same problem..

I think there are two distinct bugs here

1) lxcfs support for S390x looks a little incomplete

2) nova can't boot KVM instances with mitaka + libvirt 1.3.1

Revision history for this message
Marco Pavone (pavone) wrote :

Regarding issue 2), this is most probably due to the cpu mode setting:
  <cpu mode='host-model'>
    <model fallback='allow'/>
    <topology sockets='1' cores='1' threads='1'/>
  </cpu>

The nova.conf needs the setting "cpu_mode = none" in order to not have it set to 'host-model'.
This setting has not been working on s390x - this is a statement as of Mitaka and libvirt 1.2.21 at least

Revision history for this message
Markus Zoeller (markus_z) (mzoeller) wrote :

FWIW, Devstack sets "cpu_mode = 'none'" in its default setup [1] of the libvirt driver.

References:
[1] https://github.com/openstack-dev/devstack/blob/096106184988f76bd091b131341379757e068bcd/lib/nova_plugins/hypervisor-libvirt#L42

Serge Hallyn (serge-hallyn)
affects: lxd (Ubuntu) → lxcfs (Ubuntu)
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package lxcfs - 2.0.0-0ubuntu2

---------------
lxcfs (2.0.0-0ubuntu2) xenial; urgency=medium

  * Cherrypick upstream patches to fix cpuinfo on s390x. (LP: #1564831)

 -- Serge Hallyn <email address hidden> Mon, 04 Apr 2016 11:11:27 -0500

Changed in lxcfs (Ubuntu):
status: New → Fix Released
James Page (james-page)
Changed in nova (Ubuntu):
status: New → Invalid
Changed in libvirt (Ubuntu):
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.