“Unable to connect to libvirt” when using virt-manager with the xl toolstack and Xen 4.2.1

Bug #1180281 reported by GizmoChicken
48
This bug affects 8 people
Affects Status Importance Assigned to Milestone
libvirt (Ubuntu)
Fix Released
Medium
Unassigned
xen (Ubuntu)
Invalid
Medium
Unassigned

Bug Description

SUMMARY: I get “Unable to connect to libvirt” when using virt-manager with the xl toolstack and Xen 4.2.1

BACKGROUND: I am testing a fully updated (as of May 30) installation of Ubuntu 13.10 that includes the following packages:

libvirt (1.0.5-0ubuntu1)
virt-manager (0.9.5-1ubuntu1)
xen-hypervisor-4.2-amd64 (4.2.1-0ubuntu3.1)

Upon setting TOOLSTACK=xm in /etc/default/xen (as well as making a few needed tweaks to /etc/xen/xend-config.sxp), libvirt and virt-manager worked great. Yippee!

I guess that I should be happy that the xm toolstack still works with Xen 4.2.1 and quit while I'm ahead, but...

According to libvirt's documentation:

"libxl: Starting with Xen 4.2, the legacy XenD/xm toolstack is deprecated in favor of libxl, also commonly called libxenlight. libvirt supports this new Xen toolstack via the libxl driver. If XenD is enabled, the legacy xen driver consisting of the above mentioned channels will be used. If XenD is disabled, the libxl driver will be used."

See http://libvirt.org/drvxen.html.

In view of the above, I attempted to use xl (instead of xm) by setting TOOLSTACK=xl in /etc/default/xen. And after doing so (both with and without, the above mentioned tweaks to /etc/xen/xend-config.sxp), I received the following error when attempting to start virt-manager:

Unable to connect to libvirt.

unable to connect to 'localhost:8000': Connection refused

Verify that:
 - A Xen host kernel was booted
 - The Xen service has been started

Libvirt URI is: xen:///

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/connection.py", line 1027, in _open_thread
    self.vmm = self._try_open()
  File "/usr/share/virt-manager/virtManager/connection.py", line 1009, in _try_open
    flags)
  File "/usr/lib/python2.7/dist-packages/libvirt.py", line 102, in openAuth
    if ret is None:raise libvirtError('virConnectOpenAuth() failed')
libvirtError: unable to connect to 'localhost:8000': Connection refused

Here the output from var/log/libvirt/libvirtd.log

2013-05-18 05:44:47.614+0000: 1304: info : libvirt version: 1.0.5
2013-05-18 05:44:47.614+0000: 1304: error : xenHypervisorDoV2Sys:1006 : Unable to issue hypervisor ioctl 3166208: Permission denied
2013-05-18 05:44:47.615+0000: 1304: error : xenHypervisorDoV2Sys:1006 : Unable to issue hypervisor ioctl 3166208: Permission denied
2013-05-18 05:44:47.617+0000: 1304: error : xenHypervisorDoV2Sys:1006 : Unable to issue hypervisor ioctl 3166208: Permission denied
2013-05-18 05:44:47.618+0000: 1304: error : xenHypervisorDoV2Sys:1006 : Unable to issue hypervisor ioctl 3166208: Permission denied
2013-05-18 05:44:47.618+0000: 1304: error : xenHypervisorDoV2Sys:1006 : Unable to issue hypervisor ioctl 3166208: Permission denied
2013-05-18 05:44:47.619+0000: 1304: error : xenHypervisorDoV2Sys:1006 : Unable to issue hypervisor ioctl 3166208: Permission denied
2013-05-18 05:44:47.635+0000: 1304: error : do_connect:103 : internal error failed to connect to xend
2013-05-18 05:44:47.636+0000: 1304: error : xenDaemonOpen_tcp:700 : unable to connect to 'localhost:8000': Connection refused
2013-05-18 05:45:13.647+0000: 1289: error : virNetSocketReadWire:1362 : Cannot recv data: Connection reset by peer

Revision history for this message
Robie Basak (racb) wrote :

Thank you for taking the time to report this bug and helping to make Ubuntu better.

Please could you check /var/log/libvirt/ and post anything relevant?

Did you get the documentation with the XenD/xm deprecation warning from somewhere tied to this version of libvirt, or from the website in general? Are you able to check that the documentation you found applies to the version of libvirt that you're using, please?

It's probably also worth checking whether this bug affects upstream libvirt or just the Ubuntu packaging.

Once done, please change the bug status back to New. Thanks!

Changed in libvirt (Ubuntu):
status: New → Incomplete
description: updated
Revision history for this message
GizmoChicken (gizmochicken) wrote :

Hi Robie,

I posted the output from var/log/libvirt/libvirtd.log in the bug report.

I'll respond to your questions/comments here:

"Did you get the documentation with the XenD/xm deprecation warning from somewhere tied to this version of libvirt, or from the website in general?"

It is well known that XenD/xm has been deprecated in Xen 4.2. But since XenD/xm is still working and in widely used by many, Xen isn't exactly "warning" anyone about the deprecation. But they are gently encouraging users to make the move as soon as practical.

"Are you able to check that the documentation you found applies to the version of libvirt that you're using, please?" and "It's probably also worth checking whether this bug affects upstream libvirt or just the Ubuntu packaging."

I posted questions relate to this bug on the "<email address hidden>" and "<email address hidden>" lists. Based on the replies that I received, I'm under the impression that libvirt 1.0.5 (which I'm using) has been made to work with the xl toolstack in a few rpm based distros, including Fedora and openSUSE. However, I can't say for certain whether it is working straight from upstream or whether those distros include their own patched versions.

I realize that this bug report is still somewhat incomplete. But I want to leave it open with the hope that others will post additional information here. Hopefully, if enough post here, we'll be able to figure out what's going wrong.

Thanks!
GizmoChicken

Changed in libvirt (Ubuntu):
status: Incomplete → New
Revision history for this message
GizmoChicken (gizmochicken) wrote :

Oh, I'll just add that, although this issue has not, to my knowledge, been reported as bug here on lauchpad, the issue has been mentioned by others in various forums, including the following: http://forums.linuxmint.com/viewtopic.php?f=42&t=112013&p=720628#p721199

Revision history for this message
Robie Basak (racb) wrote :

Thanks. Given that others have the same issue, I think it's reasonable to mark this bug "Confirmed" and leave it open. Since this relates to a non-default configuration, I think "Medium" is the highest appropriate Importance level.

Changed in libvirt (Ubuntu):
status: New → Confirmed
importance: Undecided → Medium
Revision history for this message
GizmoChicken (gizmochicken) wrote :

Thanks much. And medium sounds about right. I'll just add that the problem might actually be with libxl, which libvirt relies on to communicate with xl, rather that with libvirt itself. But I'll have that determination to someone with far more knowledge than me.

summary: - “Unable to connect to libvirt” when using xl toolstack with Xen 4.2.1
+ “Unable to connect to libvirt” when using virt-manager with the xl
+ toolstack and Xen 4.2.1
description: updated
Revision history for this message
TJ (tj) wrote :

This also affects 13.04, and it isn't limited to virt-manager.

$ virsh -V && virsh -c xen:/// version
Virsh command line tool of libvirt 1.0.2
See website at http://libvirt.org/

Compiled with support for:
 Hypervisors: QEMU/KVM LXC UML Xen OpenVZ VMWare VirtualBox Test
 Networking: Remote Network Bridging Interface netcf Nwfilter VirtualPort
 Storage: Dir Disk Filesystem SCSI Multipath iSCSI LVM RBD Sheepdog
 Miscellaneous: Daemon Nodedev AppArmor Secrets Debug Readline Modular
error: failed to connect to the hypervisor
error: unable to connect to 'localhost:8000': Connection refused

Revision history for this message
TJ (tj) wrote :

This could be related to Debian bug #706059. It seems that the Xen driver isn't being loaded.

$ LIBVIRT_DEBUG=1 virsh -c xen:/// list 2>&1 | grep driver=
2013-05-31 14:57:29.879+0000: 3386: debug : virRegisterDriver:756 : driver=0x7f8deb633f00 name=Test
2013-05-31 14:57:29.879+0000: 3386: debug : virRegisterDriver:756 : driver=0x7f8deb635540 name=OPENVZ
2013-05-31 14:57:29.879+0000: 3386: debug : virRegisterDriver:756 : driver=0x7f8deb635b00 name=VMWARE
2013-05-31 14:57:29.880+0000: 3386: debug : virRegisterDriver:756 : driver=0x7f8deb6360c0 name=VBOX
2013-05-31 14:57:29.880+0000: 3386: debug : virRegisterDriver:756 : driver=0x7f8deb6348e0 name=remote

Revision history for this message
TJ (tj) wrote :

The problem appears to be missing libxenlight.so and include files in libxen-dev. During the libvirt configure run it looks for libxenlight.so but fails to find it since it is not in libxen-dev (It is in xen-utils-4.2).

I'm currently testing a build of the xen source package with xenlight added to libxen-dev.

Revision history for this message
TJ (tj) wrote :

There seem to be several incompatible issues introduced by the Debian packaging style:

1. Debian packages libxenlight.so into "xen-utils-4.2" along with the binary executables (/usr/lib/xen-4.2/bin/xl, etc.) - other distros package it in the libxen[-4.2] package
2. libvirt's configure script expects to find libxenlight.so in the standard library locations /usr/lib, /usr/lib/x86*-linux-gnu/
3. Debian installs libxenlight.so to /usr/lib/xen-4.2/lib/
4. Debian/Ubuntu libvirt does not depend on "xen-utils" (the virtual package provided by xen-utils-4.2).
5. Debian/Ubuntu libvirt configure script checks for but does not find libxenlight.so and therefore doesn't include support

I tried adding "xen-utils" as a build-depends to libvirt but that doesn't solve the issue due to [3].

I also tried adding libxenlight.so, libxenlight.a and libxl.h to the libxen-dev package but that fails to build due to other complex library dependencies which I decided not to pursue.

Changed in xen (Ubuntu):
status: New → Confirmed
importance: Undecided → Medium
Revision history for this message
Thiago Martins (martinx) wrote :

This problem still affects Ubuntu 14.04 (devel branch).

Revision history for this message
Stefan Bader (smb) wrote :

Has this been tested in 14.04 with libvirt 1.2.1? There was a bug when merging libvirt 1.2.0 which should be resolved now.

Revision history for this message
Thiago Martins (martinx) wrote :

YAY! Working in Ubuntu 14.04 beta1! It comes with libvirt 1.2.1 as you said Stefan...

Seems to be okay now, virt-manager can at least, connect at my, PVM-only, Xen host (with XL stack) using "xen+ssh://administrative@xen-dom0/".

I'll keep testing it this weekend, first problem seems to be fixed by now.

NOTE: Next problem is now:

---
Unable to complete install: 'internal error: libxenlight failed to create new domain 'trusty-1''

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 96, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/create.py", line 1983, in do_install
    guest.start_install(False, meter=meter)
  File "/usr/lib/python2.7/dist-packages/virtinst/Guest.py", line 1246, in start_install
    noboot)
  File "/usr/lib/python2.7/dist-packages/virtinst/Guest.py", line 1314, in _create_guest
    dom = self.conn.createLinux(start_xml or final_xml, 0)
  File "/usr/lib/python2.7/dist-packages/libvirt.py", line 3199, in createLinux
    if ret is None:raise libvirtError('virDomainCreateLinux() failed', conn=self)
libvirtError: internal error: libxenlight failed to create new domain 'trusty-1'
---

But, of topic... I'll keep trying / researching...

Tks!

Revision history for this message
Stefan Bader (smb) wrote :

Closing this bug now as the connect issue should be resolved. Several other issues with libvirt have been resolved but I still need to get the latest package into the archive. Preview is in my ppa (ppa:smb/xen).

Changed in xen (Ubuntu):
status: Confirmed → Invalid
Changed in libvirt (Ubuntu):
status: Confirmed → Fix Released
Revision history for this message
mppatel (patel-mp) wrote :

I have tried with ubuntu 14.04 and libvirt 1.2.2 and getting the same error what sir/madam Thiago Martins (martinx) reported.

any body has idea to solve this error "internal error:libxenlight failed to create new domain"

this error may be because of few packages not included in libvirt for libxenlight.

Revision history for this message
Stefan Bader (smb) wrote :

The logs (/var/log/libvirt/libxl) on the host may contain more info. It should work for HVM. If you try to start a PVM you may hit bug #1326003. If your issue is a different one, please open a new bug report.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Related questions

Remote bug watches

Bug watches keep track of this bug in other bug trackers.