Nova novncproxy doesnt work - unexpected keyword

I just confirmed:

I reverted all nova-cloud-controller lxcs back to bionic (by deploying new LXC's with series set bionic) and using the source cloud:bionic-ussuri

Console WORKS on cloud:bionic-ussuri
Console BROKEN on focal distro-proposed

Is this also the case without distro-proposed (ie. Just focal)?

@1chb1n Charm will not install without specifying openstack-origin="distro-proposed" when installing on focal. just "distro" doesnt work, it will error.

Ryan,

My bad, i was wrong, it does deploy with "distro"

But it returns the same error in the novncproxy logs

Full stacktrace:

2020-06-24 14:18:59.061 123476 DEBUG nova.console.websocketproxy [-] exception vmsg /usr/lib/python3/dist-packages/websockify/websockifyserver.py:634
2020-06-24 14:18:59.061 123476 ERROR nova.console.websocketproxy Traceback (most recent call last):
2020-06-24 14:18:59.061 123476 ERROR nova.console.websocketproxy File "/usr/lib/python3/dist-packages/websockify/websockifyserver.py", line 691, in top_new_client
2020-06-24 14:18:59.061 123476 ERROR nova.console.websocketproxy client = self.do_handshake(startsock, address)
2020-06-24 14:18:59.061 123476 ERROR nova.console.websocketproxy File "/usr/lib/python3/dist-packages/websockify/websockifyserver.py", line 585, in do_handshake
2020-06-24 14:18:59.061 123476 ERROR nova.console.websocketproxy retsock = context.wrap_socket(
2020-06-24 14:18:59.061 123476 ERROR nova.console.websocketproxy File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 438, in wrap_socket
2020-06-24 14:18:59.061 123476 ERROR nova.console.websocketproxy return GreenSSLSocket(sock, *a, _context=self, **kw)
2020-06-24 14:18:59.061 123476 ERROR nova.console.websocketproxy File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 67, in __new__
2020-06-24 14:18:59.061 123476 ERROR nova.console.websocketproxy ret = _original_wrap_socket(
2020-06-24 14:18:59.061 123476 ERROR nova.console.websocketproxy TypeError: wrap_socket() got an unexpected keyword argument '_context'
2020-06-24 14:18:59.061 123476 ERROR nova.console.websocketproxy

Honestly this is the only bug keeping me from doing full focal. Now that mysql-router is fixed i need to wait to upgrade NCC before the move to mysql-innodb

Reproducer outside of websockify:

import eventlet
import ssl
import socket

eventlet.monkey_patch()

context = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
serversocket.bind((socket.gethostname(), 8080))
socket = context.wrap_socket(serversocket, server_side=True)

any ideas?

I am working on it but eventlet's SSL monkey patching is somewhat non-trivial to follow.

Understood. Documentation is overrated, amirite?

Its more that the SSL monkey patching that eventlet does has to be disable during certain parts of the socket wrapping process.

Following it is non trivial but I think I have something we can test out now.

OK I think I have it figured out - python3-eventlet package building for focal here:

  https://launchpad.net/~james-page/+archive/ubuntu/ussuri

I'll submit the patch upstream as well for broader review.

PR upstream:

  https://github.com/eventlet/eventlet/pull/621

PR accepted and landed upstream; I've uploaded to groovy and to focal for SRU team review.

The PPA has fixed versions for both series for those blocked by this issue.

This bug was fixed in the package python-eventlet - 0.25.2-1ubuntu1

---------------
python-eventlet (0.25.2-1ubuntu1) groovy; urgency=medium

  * d/p/ssl-compat.patch: Fix compatibility with SSLContext base
    use of wrap_socket for Python >= 3.7 (LP: #1884580).

 -- James Page <email address hidden> Thu, 02 Jul 2020 07:59:52 +0100

Ok, it says fix released for groovy, is focal still affected until merge?

@qthepirate - yes focal is still affected - I've uploaded the package for the stable release updates team to review.

Debating downloading your package upload to test it before they get to it just to see if it works. But its a production stack and I'm already having issues with other items

Hello Michael, or anyone else affected,

Accepted python-eventlet into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-eventlet/0.25.1-2ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Since this was accepted upstream I guess the change is safe, as (without context) I was originally a bit afraid that some other important parameters will be missed if we stop passing **kw. But I assume this just means there are no other arguments needed there besides the context field.

Accepted upstream was my gate for starting the SRU process - it took me a while to figure out what actually needed to happen in the underlying code with regards to calling underlying un-monkey-patched code from the Python core ssl library. Passing the **kw arg its pretty pointless - the wrap_socket function does not have any other args than those called out specifically!

Verified with focal-proposed packages - able to access novnc console for an instance running on a deployed OpenStack Cloud (see attachement)

$ apt-cache policy python3-eventlet
python3-eventlet:
  Installed: 0.25.1-2ubuntu1
  Candidate: 0.25.1-2ubuntu1
  Version table:
 *** 0.25.1-2ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     0.25.1-2build1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu focal/main amd64 Packages

Beautiful! Will upgrade today for a double verification.

@qthepirate be aware there are proposed updates for nova and neutron to resolve some other upgrade issues - nova around permissions (bug 1885269) and neutron around restarts of the cleanup systemd units for ovs and linuxbridge (bug 1885264).

(you will definitely want the nova one).

@james-page

All other charms in my stack have been updated to focal-proposed (ussuri) and the last one I needed was nova-cloud-controller. Will run upgrades for updated packages though.

Confirmed fix! Deployed a new nova-cloud-controller unit, console now works.

This bug was fixed in the package python-eventlet - 0.25.1-2ubuntu1

---------------
python-eventlet (0.25.1-2ubuntu1) focal; urgency=medium

  * d/p/ssl-compat.patch: Fix compatibility with SSLContext base
    use of wrap_socket for Python >= 3.7 (LP: #1884580).

 -- James Page <email address hidden> Thu, 02 Jul 2020 08:30:30 +0100

The verification of the Stable Release Update for python-eventlet has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.