Ubuntu
ganeti package

Ganeti initialization fails because PyCurl is linked against GnuTLS

Bug #964989 reported by Ralf Spenneberg on 2012-03-26

108

This bug affects 21 people

Affects		Status	Importance	Assigned to	Milestone
	ganeti (Ubuntu)	Invalid	High	Unassigned
	pycurl (Ubuntu)	Invalid	Undecided	Unassigned

Bug Description

Ganeti uses PyCurl to connect to the master daemon. When PyCurl is linked against GnuTLS the connection fails:
2012-03-26 07:00:13,862: gnt-cluster init pid=5995 INFO Using PycURL libcurl/7.22.0 GnuTLS/2.12.14 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
2012-03-26 07:00:18,658: gnt-cluster init pid=5995 ERROR RPC error in version from node node1.example.com: Error 35: gnutls_handshake() failed: GnuTLS internal error.

Starting Ganeti is not possible:
# /etc/init.d/ganeti start
* Starting Ganeti cluster * ganeti-noded... [ OK ]
* ganeti-masterd... ERROR:root:RPC error in node_start_master from node node2.example.com: Error 35: gnutls_handshake() failed: GnuTLS internal error.
ERROR:root:Can't activate master IP address: Error 35: gnutls_handshake() failed: GnuTLS internal error.
[ OK ]
* ganeti-rapi... [ OK ]
* ganeti-confd... [ OK ]
# gnt-node list
Node DTotal DFree MTotal MNode MFree Pinst Sinst
node2.example.com ? ? ? ? ? 0 0

Recompiling PyCurl and linking against openssl fixes the issue:
[root]# aptitude install build-essential dpkg-dev
[root]# apt-get source python-pycurl
[root]# aptitude build-dep python-pycurl
[root]# aptitude install libcurl4-openssl-dev
[root]# cd pycurl-7.19.0
[root]# # perl -p -i -e "s/gnutls/openssl/" debian/control
[root]# dpkg-buildpackage -rfakeroot -b
[root]# cd ..
[root]# dpkg -i python-pycurl_7.19.0-4ubuntu3_amd64.deb

# /etc/init.d/ganeti restart
* Restarting Ganeti cluster
* ganeti-confd... [ OK ]
* ganeti-rapi... [ OK ]
* ganeti-masterd... [ OK ]
* ganeti-noded... [ OK ]
* ganeti-noded... [ OK ]
* ganeti-masterd... [ OK ]
* ganeti-rapi... [ OK ]
* ganeti-confd... [ OK ]
root@node1:/var/lib/ganeti# gnt-node list
Node DTotal DFree MTotal MNode MFree Pinst Sinst
node1.example.com 127.9G 127.9G 3.8G 381M 3.6G 0 0

Although during initialization still errors are logged in /var/log/ganeti/commands.log:
2012-03-26 07:18:53,879: gnt-cluster init pid=20274 INFO Using PycURL libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
2012-03-26 07:18:58,521: gnt-cluster init pid=20274 ERROR RPC error in version from node node1.example.com: Error 60: SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

But the communication is successful.

Revision history for this message

Launchpad Janitor (janitor) wrote on 2012-03-26:

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in ganeti (Ubuntu):
status:	New → Confirmed

Revision history for this message

spbike (bill-broadley) wrote on 2012-04-26:

This completely breaks ganeti2 on the Ubuntu LTS 12.04 release.

Revision history for this message

Dennis Olsson (cyberdo) wrote on 2012-06-10:

I am quite convinced this affects me as well. I get similar error messages, but your workaround does not work. There are other dependencies that link back to gnutls-dev, and it seem to be used before openssl.

My problem is that in a cluster of 3 nodes, I upgraded one of them from pre-release to <state-as-of-today>, and now that node can not communicate with the other nodes, so I can not migrate machines and continue the upgrade.

Any other hints to workarounds other than manually modifying all dependencies and their dependencies etc.?

Shouldn't there be someone maintaining this?

Revision history for this message

Lance Albertson (lance-osuosl) wrote on 2012-06-10:

There seems to be an upstream bug [1] that seems to resemble this issue but its been mostly inactive since its creation over a year ago. Hopefully some prodding will get the attention of the project. I encountered the same issue when installing ganeti from source on 12.04.

[1] http://code.google.com/p/ganeti/issues/detail?id=139

Revision history for this message

Dennis Olsson (cyberdo) wrote on 2012-06-11:

The upgrade that started things Edit (10.8 KiB, text/plain)

As far as I can tell it all came apart when I did a dist-upgrade. I have attached a snippet from apt/history.log if it might be of any help.

Issuing a gnt-node list yields this on the master (terra1):
{{{==> /var/log/ganeti/master-daemon.log <==
2012-06-11 07:17:08,015: ganeti-masterd pid=3385/ClientReq7 ERROR RPC error in node_info from node terra3: Error 35: gnutls_handshake() failed: GnuTLS internal error.}}}

and the output:
{{{Node DTotal DFree MTotal MNode MFree Pinst Sinst
terra1 1.8T 1.5T 15.6G 10.7G 6.7G 8 13
terra2 1.8T 1.6T 23.5G 9.0G 16.4G 13 8
terra3 ? ? ? ? ? 0 0}}}

No log entries on terra3

Revision history for this message

andrew bezella (abezella) wrote on 2012-07-25:

i noticed a related failure and these errors after installing a ganeti host as precise and then attempting a master-failover to it. compiling the python-pycurl package against openssl and installing it on this new precise host allows master-failover to succeed.

TJ (tj) on 2012-08-05

Changed in ganeti (Ubuntu):
status:	Confirmed → Won't Fix
importance:	Undecided → High
status:	Won't Fix → Confirmed

Revision history for this message

Svein Harald Soleim (sveinh) wrote on 2012-10-06:

I can confirm that master-failover works with python-pycurl linked against openssl

Revision history for this message

Tollef Fog Heen (tfheen) wrote on 2012-11-06:

Ditto, can confirm that linking python-pycurl against openssl works around this.

Revision history for this message

Guy K. Kloss (guy.kloss) wrote on 2013-01-15:

An easier option than rebuilding a system package with all kinds of dependencies might be to tell PyCURL what version of SSL to use, like described here:

https://bugs.launchpad.net/ubuntu/+source/pycurl/+bug/926548/comments/24

Hope that helps,

Guy

Revision history for this message

spbike (bill-broadley) wrote on 2013-02-06:

#10

Sadly this breaks ganeti2 on ubuntu 12.10 server as well. The recompiling does fix it.

Revision history for this message

Simon Déziel (sdeziel) wrote on 2013-04-26:

#11

A proper fix for GnuTLS is on it's way in https://bugs.launchpad.net/ubuntu/+source/gnutls26/+bug/1095052