OpenStack Dashboard (Horizon)

RuntimeError: osrandom engine already registered

Bug #1601986 reported by Peter Sabaini on 2016-07-11

This bug affects 3 people

	Status	Importance	Assigned to
OpenStack Dashboard (Horizon)	Invalid	Undecided	Unassigned
Ubuntu Cloud Archive	Fix Released	High	Unassigned
Liberty	Fix Released	High	Unassigned
python-cryptography (Ubuntu)	Fix Released	High	Unassigned

Bug Description

Horizon errors with 500 Internal Server Error.

The apache error.log logs an exception "RuntimeError: osrandom engine already registered", cf. traceback below. We need to restart apache2 to recover.

This happens in a non-deterministic way, ie. Horizon will function correctly for some time after throwing this error.

Versions:

python-django-horizon 2:8.0.1-0ubuntu1~cloud0
apache2 2.4.7-1ubuntu4.10
libapache2-mod-wsgi 3.4-4ubuntu2.1.14.04.2

Traceback:

[Mon Jul 11 20:16:46.373640 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] mod_wsgi (pid=2045796): Exception occurred processing WSGI script '/usr/share/openstack-dashboard/openstack_dashboard/wsgi/django.wsgi'.
[Mon Jul 11 20:16:46.373681 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] Traceback (most recent call last):
[Mon Jul 11 20:16:46.373697 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] File "/usr/lib/python2.7/dist-packages/django/core/handlers/wsgi.py", line 168, in __call__
[Mon Jul 11 20:16:46.390398 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] self.load_middleware()
[Mon Jul 11 20:16:46.390420 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] File "/usr/lib/python2.7/dist-packages/django/core/handlers/base.py", line 46, in load_middleware
[Mon Jul 11 20:16:46.390515 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] mw_instance = mw_class()
[Mon Jul 11 20:16:46.390525 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] File "/usr/lib/python2.7/dist-packages/django/middleware/locale.py", line 23, in __init__
[Mon Jul 11 20:16:46.394033 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] for url_pattern in get_resolver(None).url_patterns:
[Mon Jul 11 20:16:46.394052 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] File "/usr/lib/python2.7/dist-packages/django/core/urlresolvers.py", line 372, in url_patterns
[Mon Jul 11 20:16:46.394500 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
[Mon Jul 11 20:16:46.394516 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] File "/usr/lib/python2.7/dist-packages/django/core/urlresolvers.py", line 366, in urlconf_module
[Mon Jul 11 20:16:46.394533 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] self._urlconf_module = import_module(self.urlconf_name)
[Mon Jul 11 20:16:46.394540 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
[Mon Jul 11 20:16:46.410602 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] __import__(name)
[Mon Jul 11 20:16:46.410618 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/urls.py", line 35, in <module>
[Mon Jul 11 20:16:46.416197 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] url(r'^api/', include('openstack_dashboard.api.rest.urls')),
[Mon Jul 11 20:16:46.416219 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] File "/usr/lib/python2.7/dist-packages/django/conf/urls/__init__.py", line 28, in include
[Mon Jul 11 20:16:46.422868 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] urlconf_module = import_module(urlconf_module)
[Mon Jul 11 20:16:46.422882 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
[Mon Jul 11 20:16:46.422899 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] __import__(name)
[Mon Jul 11 20:16:46.422905 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/__init__.py", line 36, in <module>
[Mon Jul 11 20:16:46.432789 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] from openstack_dashboard.api import cinder
[Mon Jul 11 20:16:46.432803 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/cinder.py", line 30, in <module>
[Mon Jul 11 20:16:46.440814 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] from cinderclient.v2.contrib import list_extensions as cinder_list_extensions
[Mon Jul 11 20:16:46.440829 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] File "/usr/lib/python2.7/dist-packages/cinderclient/v2/__init__.py", line 17, in <module>
[Mon Jul 11 20:16:46.453863 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] from cinderclient.v2.client import Client # noqa
[Mon Jul 11 20:16:46.453887 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] File "/usr/lib/python2.7/dist-packages/cinderclient/v2/client.py", line 16, in <module>
[Mon Jul 11 20:16:46.457850 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] from cinderclient import client
[Mon Jul 11 20:16:46.457864 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 44, in <module>
[Mon Jul 11 20:16:46.461033 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] from eventlet import sleep
[Mon Jul 11 20:16:46.461048 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] File "/usr/lib/python2.7/dist-packages/eventlet/__init__.py", line 10, in <module>
[Mon Jul 11 20:16:46.468784 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] from eventlet import convenience
[Mon Jul 11 20:16:46.468810 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] File "/usr/lib/python2.7/dist-packages/eventlet/convenience.py", line 3, in <module>
[Mon Jul 11 20:16:46.473247 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] from eventlet import greenio
[Mon Jul 11 20:16:46.473265 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] File "/usr/lib/python2.7/dist-packages/eventlet/greenio/__init__.py", line 3, in <module>
[Mon Jul 11 20:16:46.475382 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] from eventlet.greenio.base import * # noqa
[Mon Jul 11 20:16:46.475406 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 441, in <module>
[Mon Jul 11 20:16:46.476165 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] from OpenSSL import SSL
[Mon Jul 11 20:16:46.476185 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] File "/usr/lib/python2.7/dist-packages/OpenSSL/__init__.py", line 8, in <module>
[Mon Jul 11 20:16:46.482761 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] from OpenSSL import rand, crypto, SSL
[Mon Jul 11 20:16:46.482781 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] File "/usr/lib/python2.7/dist-packages/OpenSSL/rand.py", line 11, in <module>
[Mon Jul 11 20:16:46.490827 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] from OpenSSL._util import (
[Mon Jul 11 20:16:46.490842 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] File "/usr/lib/python2.7/dist-packages/OpenSSL/_util.py", line 7, in <module>
[Mon Jul 11 20:16:46.497707 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] binding = Binding()
[Mon Jul 11 20:16:46.497725 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] File "/usr/lib/python2.7/dist-packages/cryptography/hazmat/bindings/openssl/binding.py", line 63, in __init__
[Mon Jul 11 20:16:46.505876 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] self._ensure_ffi_initialized()
[Mon Jul 11 20:16:46.505897 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] File "/usr/lib/python2.7/dist-packages/cryptography/hazmat/bindings/openssl/binding.py", line 95, in _ensure_ffi_initialized
[Mon Jul 11 20:16:46.505922 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] cls._register_osrandom_engine()
[Mon Jul 11 20:16:46.505934 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] File "/usr/lib/python2.7/dist-packages/cryptography/hazmat/bindings/openssl/binding.py", line 70, in _register_osrandom_engine
[Mon Jul 11 20:16:46.505954 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] raise RuntimeError("osrandom engine already registered")
[Mon Jul 11 20:16:46.505984 2016] [:error] [pid 2045796:tid 139828791035648] [remote 172.16.4.81:33908] RuntimeError: osrandom engine already registered

Tags:

Revision history for this message

JuanJo Ciarlante (jjo) wrote on 2016-07-12:

Note this seems to come from python-cryptography, eg fixed
by fedora ca Jan/2016[0] and suse ca Oct/2015[1] -
this trusty/liberty install has:

python-cryptography (1.0.1-1ubuntu1~cloud1) trusty-liberty; urgency=medium
from Mon, 23 Nov 2015 17:40:42 +0000.

[0] https://bugzilla.redhat.com/show_bug.cgi?id=1284148
[1] https://lists.opensuse.org/opensuse-commit/2015-10/msg00881.html

Revision history for this message

Corey Bryant (corey.bryant) wrote on 2016-12-13:

I think this is fixed by the following commit to python-cryptography.

commit 9837cb15b84fea92ffce3306d14160a8c11b1c65
Author: Paul Kehrer <email address hidden>
Date: Mon Aug 24 08:00:10 2015 -0500

make engine addition idempotent

    Threading issues keep cropping up. ENGINE_add already
    acquires a lock at the C layer via CRYPTO_w_lock (provided you
    have registered the locking callbacks) so let's try to use that.

    As part of this we'll try to init the openssl locks, but of course
    there's potentially a race there as well. Clearly this isn't the real
    fix but it might improve the situation while we try to determine what to
    do.

Changed in python-cryptography (Ubuntu):
status:	New → Triaged
importance:	Undecided → High

Revision history for this message

Corey Bryant (corey.bryant) wrote on 2016-12-13:

commit 9837cb15b84fea92ffce3306d14160a8c11b1c65 is included in mitaka and above, so I'm marking this as fix released for python-cryptography in Ubuntu. We'll target a fix for the Liberty cloud-archive.

Changed in python-cryptography (Ubuntu):
status:	Triaged → Fix Released

Corey Bryant (corey.bryant) on 2016-12-13

Changed in cloud-archive:
status:	New → Fix Released
importance:	Undecided → High

Revision history for this message

Corey Bryant (corey.bryant) wrote on 2016-12-21:

A new version of python-cryptography with this fix has been uploaded to liberty-staging and is awaiting promotion to liberty-proposed.

Revision history for this message

Ryan Beisner (1chb1n) wrote on 2017-01-03: Please test proposed package

Hello Peter, or anyone else affected,

Accepted python-cryptography into liberty-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

sudo add-apt-repository cloud-archive:liberty-proposed
sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-liberty-needed to verification-liberty-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-liberty-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags:

added: verification-liberty-needed

Corey Bryant (corey.bryant) on 2017-01-05

Changed in horizon:
status:	New → Invalid

Corey Bryant (corey.bryant) on 2017-01-12

tags:

added: verification-liberty-done
removed: verification-liberty-needed

Revision history for this message

James Page (james-page) wrote on 2017-01-23: Update Released

The verification of the Stable Release Update for python-cryptography has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message

James Page (james-page) wrote on 2017-01-23:

This bug was fixed in the package python-cryptography - 1.0.1-1ubuntu1~cloud1.2
---------------

python-cryptography (1.0.1-1ubuntu1~cloud1.2) trusty-liberty; urgency=medium
.
   * d/p/fix-attribute-error-in-consume-errors.patch: Cherry-pick patch from
     upstream to fix AttributeError: 'int' object has no attribute 'ERR_GET_FUNC'
     (LP: #1601986).

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

redhat-bugs #1284148
[CLOSED ERRATA] Edit

Bug watches keep track of this bug in other bug trackers.