OpenStack Compute (nova)

"libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied" in nova-multiattach job since we started using the Queens UCA

Bug #1763382 reported by Matt Riedemann
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Matt Riedemann

Bug Description

Since the devstack patch to start using the queens UCA merged on master (rocky):

https://review.openstack.org/#/c/554314/

The nova-multiattach job has been failing with this:

http://logs.openstack.org/49/560349/1/check/nova-multiattach/ce31aed/logs/screen-n-cpu.txt.gz?level=ERROR#_Apr_12_07_23_19_791688

Apr 12 07:23:19.791688 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host [None req-fb02f74a-6ce9-4aa6-abc4-24a6d87ca6b0 None None] Connection to libvirt failed: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied: libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied
Apr 12 07:23:19.791867 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host Traceback (most recent call last):
Apr 12 07:23:19.792029 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host File "/opt/stack/new/nova/nova/virt/libvirt/host.py", line 443, in get_connection
Apr 12 07:23:19.792186 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host conn = self._get_connection()
Apr 12 07:23:19.792363 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host File "/opt/stack/new/nova/nova/virt/libvirt/host.py", line 426, in _get_connection
Apr 12 07:23:19.792521 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host {'msg': ex})
Apr 12 07:23:19.792693 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Apr 12 07:23:19.792847 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host self.force_reraise()
Apr 12 07:23:19.793001 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Apr 12 07:23:19.793171 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host six.reraise(self.type_, self.value, self.tb)
Apr 12 07:23:19.793329 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host File "/opt/stack/new/nova/nova/virt/libvirt/host.py", line 415, in _get_connection
Apr 12 07:23:19.793487 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host self._wrapped_conn = self._get_new_connection()
Apr 12 07:23:19.793664 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host File "/opt/stack/new/nova/nova/virt/libvirt/host.py", line 369, in _get_new_connection
Apr 12 07:23:19.793821 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host wrapped_conn = self._connect(self._uri, self._read_only)
Apr 12 07:23:19.794036 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host File "/opt/stack/new/nova/nova/virt/libvirt/host.py", line 225, in _connect
Apr 12 07:23:19.794191 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host libvirt.openAuth, uri, auth, flags)
Apr 12 07:23:19.794346 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
Apr 12 07:23:19.794512 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host rv = execute(f, *args, **kwargs)
Apr 12 07:23:19.794666 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
Apr 12 07:23:19.794823 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host six.reraise(c, e, tb)
Apr 12 07:23:19.794995 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
Apr 12 07:23:19.795160 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host rv = meth(*args, **kwargs)
Apr 12 07:23:19.795315 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 105, in openAuth
Apr 12 07:23:19.795470 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host if ret is None:raise libvirtError('virConnectOpenAuth() failed')
Apr 12 07:23:19.795626 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied
Apr 12 07:23:19.795788 ubuntu-xenial-inap-mtl01-0003466660 nova-compute[25984]: ERROR nova.virt.libvirt.host

There is a patch to make the nova-multiattach job using the Queens UCA, which was passing before this, but now is also hitting that issue:

https://review.openstack.org/#/c/554317/

http://logs.openstack.org/17/554317/3/check/nova-multiattach/fd35a93/logs/screen-n-cpu.txt.gz?level=ERROR

My guess would be it has something to do with the libvirt group?

https://review.openstack.org/#/c/554314/1/stackrc

Anyway, we can make the nova-multiattach job non-voting until we get this sorted out.

Revision history for this message
Matt Riedemann (mriedem) wrote :

Here the nova-multiattach job is using the "libvirt" group and adding the stack user to it:

http://logs.openstack.org/17/554317/3/check/nova-multiattach/fd35a93/logs/devstacklog.txt.gz#_2018-04-11_18_11_11_860

2018-04-11 18:11:11.855 | + lib/nova_plugins/functions-libvirt:configure_libvirt:127 : getent group libvirt
2018-04-11 18:11:11.860 | + lib/nova_plugins/functions-libvirt:configure_libvirt:128 : sudo groupadd libvirt
2018-04-11 18:11:11.884 | + lib/nova_plugins/functions-libvirt:configure_libvirt:130 : add_user_to_group stack libvirt
2018-04-11 18:11:11.887 | + functions-common:add_user_to_group:2028 : local user=stack
2018-04-11 18:11:11.889 | + functions-common:add_user_to_group:2029 : local group=libvirt
2018-04-11 18:11:11.892 | + functions-common:add_user_to_group:2031 : sudo usermod -a -G libvirt stack

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/560909

Matt Riedemann (mriedem)
summary: - "Connection to libvirt failed: Failed to connect socket to
- '/var/run/libvirt/libvirt-sock': Permission denied: libvirtError: Failed
- to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied"
- in nova-multiattach job since we started using the Queens UCA
+ libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-
+ sock': Permission denied" in nova-multiattach job since we started using
+ the Queens UCA
summary: - libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-
+ "libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-
sock': Permission denied" in nova-multiattach job since we started using
the Queens UCA
Revision history for this message
Matt Riedemann (mriedem) wrote :

Looking at the failure in https://review.openstack.org/#/c/554317/ which should make the nova-multiattach job use the Queens UCA, it looks like it's not, it's using base xenial packages:

http://logs.openstack.org/17/554317/3/check/nova-multiattach/fd35a93/logs/dpkg-l.txt.gz

ii libvirt-bin 1.3.1-1ubuntu10.19

The localrc is wrong too:

http://logs.openstack.org/17/554317/3/check/nova-multiattach/fd35a93/logs/local.conf.txt.gz

ENABLE_UBUNTU_CLOUD_ARCHIVE=False

Looking at the ARA, zuulv3 didn't remove that entry when it ran the task:

http://logs.openstack.org/17/554317/3/check/nova-multiattach/fd35a93/ara-report/result/ab8397a6-593e-43f5-a593-c2633a0e40de/

So that's the problem, and I'm assuming it's an infra problem with zuul.

Revision history for this message
Matt Riedemann (mriedem) wrote :

Related zuulv3 bug in storyboard: https://storyboard.openstack.org/#!/story/2001839

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Related fix proposed to branch: master
Review: https://review.openstack.org/560930

Changed in nova:
assignee: nobody → Matt Riedemann (mriedem)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (stable/queens)

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/560931

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (master)

Reviewed: https://review.openstack.org/560909
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=cb6c8ca1a7a5abc4d0079e285f877c18c49acaf2
Submitter: Zuul
Branch: master

commit cb6c8ca1a7a5abc4d0079e285f877c18c49acaf2
Author: Matt Riedemann <email address hidden>
Date: Thu Apr 12 09:52:42 2018 -0400

    Make the nova-multiattach job non-voting temporarily

    Since Icb971831c8d4fe5f940d9e7993d53f1c3765e30f merged
    in devstack to use the Queens UCA, the nova-multiattach
    job has been failing to start nova-compute due to permission
    errors connecting to libvirt, which probably has something
    to do with the stack user not having access to the proper
    libvirt(d) group.

    It's weird since this was working in change
    I0962474ff6dfc5fa97670c09a1af97a0f34cd54f to make the
    nova-multiattach job use the Queens UCA.

    Anyway, let's make it non-voting to stop the bleeding while
    we can figure out the actual problem.

    Change-Id: I3911c902cecfa8e66fccffd9fe0e4f462eacd588
    Related-Bug: #1763382

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/560930
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=406c9d13db80d05f6668c2eee3eae867fd91ceb9
Submitter: Zuul
Branch: master

commit 406c9d13db80d05f6668c2eee3eae867fd91ceb9
Author: Matt Riedemann <email address hidden>
Date: Thu Apr 12 10:55:48 2018 -0400

    Remove the branch specifier from the nova-multiattach job

    For in-tree job definitions, the job is branch-specific
    by design, so we don't need the stable branch exclusion
    specifier as it won't run on those branches anyway since
    it's not defined for those branches.

    Related to https://storyboard.openstack.org/#!/story/2001839
    Related-Bug: #1763382

    Change-Id: I78dead482e2ed8262745cb08c9f4ab71035adb33

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (stable/queens)

Reviewed: https://review.openstack.org/560931
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=8680781044873b4e6702dd6087aac91c65a9f5ed
Submitter: Zuul
Branch: stable/queens

commit 8680781044873b4e6702dd6087aac91c65a9f5ed
Author: Matt Riedemann <email address hidden>
Date: Thu Apr 12 10:55:48 2018 -0400

    Remove the branch specifier from the nova-multiattach job

    For in-tree job definitions, the job is branch-specific
    by design, so we don't need the stable branch exclusion
    specifier as it won't run on those branches anyway since
    it's not defined for those branches.

    Related to https://storyboard.openstack.org/#!/story/2001839
    Related-Bug: #1763382

    Change-Id: I78dead482e2ed8262745cb08c9f4ab71035adb33
    (cherry picked from commit 406c9d13db80d05f6668c2eee3eae867fd91ceb9)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/554317
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=483531ba335715168317ad8211cf466e61c44d84
Submitter: Zuul
Branch: master

commit 483531ba335715168317ad8211cf466e61c44d84
Author: Matt Riedemann <email address hidden>
Date: Mon Mar 19 14:34:04 2018 -0400

    Use Queens UCA for nova-multiattach job

    The dependent change to devstack makes devstack use the
    Queens Ubuntu Cloud Archive which has libvirt 4.0.0 so we
    can remove the flag from the job which disabled the UCA
    in devstack since the versions of qemu and libvirt in the
    Pike UCA wouldn't work for multiattach.

    This also allows us to make the job voting and gating again.

    Closes-Bug: #1763382

    Depends-On: https://review.openstack.org/554314
    Depends-On: https://review.openstack.org/560931
    Change-Id: I0962474ff6dfc5fa97670c09a1af97a0f34cd54f

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 18.0.0.0b1

This issue was fixed in the openstack/nova 18.0.0.0b1 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.