python 3 failures - /usr/bin/masakari-wsgi and unit tests

Bug #1815657 reported by Corey Bryant
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
masakari
Fix Released
High
Shilpa Devharakar

Bug Description

There are 2 unit tests failing with python 3.6 and 3.7. This can be recreated on Ubuntu Disco by running the py35 target which by default will use python3 which is /usr/bin/python3 which is python3.7 in Disco. python3.7 is also available in bionic so you can likely update tox.ini to specifically use python3.7. python3.6 is also available on bionic.

==============================
Failed 2 tests - output below:
==============================

masakari.tests.unit.test_hacking.HackingTestCase.test_check_doubled_words
-------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    b'Traceback (most recent call last):'
    b' File "/home/corey/pkg/stein/upstream/masakari/masakari/tests/unit/test_hacking.py", line 371, in test_check_doubled_words'
    b' expected_errors=errors)'
    b' File "/home/corey/pkg/stein/upstream/masakari/masakari/tests/unit/test_hacking.py", line 234, in _assert_has_errors'
    b' self.assertEqual(expected_errors or [], actual_errors)'
    b' File "/home/corey/pkg/stein/upstream/masakari/.tox/py35/lib/python3.7/site-packages/testtools/testcase.py", line 411, in assertEqual'
    b' self.assertThat(observed, matcher, message)'
    b' File "/home/corey/pkg/stein/upstream/masakari/.tox/py35/lib/python3.7/site-packages/testtools/testcase.py", line 498, in assertThat'
    b' raise mismatch_error'
    b"testtools.matchers._impl.MismatchError: [(1, 0, 'M325')] != []"
    b''

masakari.tests.unit.test_api_validation.PatternPropertiesTestCase.test_validate_patternProperties_fails
-------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    b'Traceback (most recent call last):'
    b' File "/home/corey/pkg/stein/upstream/masakari/masakari/api/validation/validators.py", line 181, in validate'
    b' self.validator.validate(*args, **kwargs)'
    b' File "/home/corey/pkg/stein/upstream/masakari/.tox/py35/lib/python3.7/site-packages/jsonschema/validators.py", line 129, in validate'
    b' for error in self.iter_errors(*args, **kwargs):'
    b' File "/home/corey/pkg/stein/upstream/masakari/.tox/py35/lib/python3.7/site-packages/jsonschema/validators.py", line 105, in iter_errors'
    b' for error in errors:'
    b' File "/home/corey/pkg/stein/upstream/masakari/.tox/py35/lib/python3.7/site-packages/jsonschema/_validators.py", line 14, in patternProperties'
    b' if re.search(pattern, k):'
    b' File "/home/corey/pkg/stein/upstream/masakari/.tox/py35/lib/python3.7/re.py", line 183, in search'
    b' return _compile(pattern, flags).search(string)'
    b'TypeError: expected string or bytes-like object'
    b''
    b'During handling of the above exception, another exception occurred:'
    b''
    b'Traceback (most recent call last):'
    b' File "/home/corey/pkg/stein/upstream/masakari/masakari/tests/unit/test_api_validation.py", line 100, in check_validation_error'
    b' method(body=body, req=req,)'
    b' File "/home/corey/pkg/stein/upstream/masakari/masakari/api/validation/__init__.py", line 54, in wrapper'
    b" schema_validator.validate(kwargs['body'])"
    b' File "/home/corey/pkg/stein/upstream/masakari/masakari/api/validation/validators.py", line 198, in validate'
    b' raise exception.ValidationError(detail=detail)'
    b'masakari.exception.ValidationError: expected string or bytes-like object'
    b''
    b'During handling of the above exception, another exception occurred:'
    b''
    b'Traceback (most recent call last):'
    b' File "/home/corey/pkg/stein/upstream/masakari/masakari/tests/unit/test_api_validation.py", line 278, in test_validate_patternProperties_fails'
    b' expected_detail=detail)'
    b' File "/home/corey/pkg/stein/upstream/masakari/masakari/tests/unit/test_api_validation.py", line 108, in check_validation_error'
    b" 'Exception details did not match expected')"
    b' File "/home/corey/pkg/stein/upstream/masakari/.tox/py35/lib/python3.7/site-packages/testtools/testcase.py", line 411, in assertEqual'
    b' self.assertThat(observed, matcher, message)'
    b' File "/home/corey/pkg/stein/upstream/masakari/.tox/py35/lib/python3.7/site-packages/testtools/testcase.py", line 498, in assertThat'
    b' raise mismatch_error'
    b"testtools.matchers._impl.MismatchError: 'expected string or buffer' != 'expected string or bytes-like object': Exception details did not match expected"
    b''

======
Totals
======
Ran: 717 tests in 11.7636 sec.
 - Passed: 712
 - Skipped: 3
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 2
Sum of execute time for each test: 29.1769 sec.

Revision history for this message
Corey Bryant (corey.bryant) wrote :

Looks like these are also failing for python3.6 which you can also get at with bionic.

description: updated
Revision history for this message
Corey Bryant (corey.bryant) wrote :

I'm also hitting this when trying to run the API under apache2 with mod_wsgi:

2019-02-12 22:27:04.439009 mod_wsgi (pid=6146): Exception occurred processing WSGI script '/usr/bin/masakari-wsgi'.
2019-02-12 22:27:04.439820 Traceback (most recent call last):
2019-02-12 22:27:04.439873 File "/usr/lib/python3/dist-packages/masakari/api/urlmap.py", line 288, in __call__
2019-02-12 22:27:04.439894 return app(environ, start_response)
2019-02-12 22:27:04.439915 File "/usr/lib/python3/dist-packages/masakari/api/urlmap.py", line 185, in wrap
2019-02-12 22:27:04.439924 return app(environ, start_response)
2019-02-12 22:27:04.439934 File "/usr/lib/python3/dist-packages/webob/dec.py", line 143, in __call__
2019-02-12 22:27:04.439942 return resp(environ, start_response)
2019-02-12 22:27:04.439953 File "/usr/lib/python3/dist-packages/webob/response.py", line 1305, in __call__
2019-02-12 22:27:04.439960 start_response(self.status, headerlist)
2019-02-12 22:27:04.439989 TypeError: expected unicode object, value of type bytes found

summary: - python 3.7 unit test failures
+ python 3 failures - /usr/bin/masakari-wsgi and unit tests
Revision history for this message
Corey Bryant (corey.bryant) wrote :

More details for the error in #2 from /usr/lib/python3/dist-packages/webob/response.py prior to the failure:

self.status=200 OK
headerlist=[('Content-Length', b'192'), ('Content-Type', b'application/json')]

Revision history for this message
Corey Bryant (corey.bryant) wrote :

I dropped all the utils.utf8() calls in /usr/lib/python3/dist-packages/masakari/api/openstack/wsgi.py and it seems to fix the API.

Revision history for this message
Corey Bryant (corey.bryant) wrote :

It appears that, at least with Python 3, only the output needs to be a byte type and the headers should be plain text. See my experiments with a simple mod_wsgi app below:

https://paste.ubuntu.com/p/9nNZqQGtv9/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to masakari (master)

Fix proposed to branch: master
Review: https://review.openstack.org/636749

Changed in masakari:
assignee: nobody → Corey Bryant (corey.bryant)
status: New → In Progress
Tushar Patil (tpatil)
Changed in masakari:
importance: Undecided → High
Changed in masakari:
assignee: Corey Bryant (corey.bryant) → Shilpa Devharakar (shilpasd)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to masakari (master)

Reviewed: https://review.openstack.org/636749
Committed: https://git.openstack.org/cgit/openstack/masakari/commit/?id=041af276d725456cdf6e45d09a013e38188d803d
Submitter: Zuul
Branch: master

commit 041af276d725456cdf6e45d09a013e38188d803d
Author: Corey Bryant <email address hidden>
Date: Wed Feb 13 15:04:33 2019 -0500

    Py3: Ensure wsgi headers are not bytes type

    In Python 3, running the Masakari API as a wsgi application was
    resulting in "TypeError: expected unicode object, value of type
    bytes found". It looks like headerlist should not include bytes
    type objects so let's ensure they are str objects for Python 3.

    Change-Id: Ica030c540970474cd2511ffe0ba653d7bb057849
    Partial-Bug: #1815657

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to masakari (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/641039

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to masakari (stable/rocky)

Reviewed: https://review.openstack.org/641039
Committed: https://git.openstack.org/cgit/openstack/masakari/commit/?id=250068a7deba6230387fa99a851cfe4cee56ccf0
Submitter: Zuul
Branch: stable/rocky

commit 250068a7deba6230387fa99a851cfe4cee56ccf0
Author: Corey Bryant <email address hidden>
Date: Wed Feb 13 15:04:33 2019 -0500

    Py3: Ensure wsgi headers are not bytes type

    In Python 3, running the Masakari API as a wsgi application was
    resulting in "TypeError: expected unicode object, value of type
    bytes found". It looks like headerlist should not include bytes
    type objects so let's ensure they are str objects for Python 3.

    Change-Id: Ica030c540970474cd2511ffe0ba653d7bb057849
    Partial-Bug: #1815657
    (cherry picked from commit 041af276d725456cdf6e45d09a013e38188d803d)

tags: added: in-stable-rocky
Changed in masakari:
status: In Progress → Fix Committed
Changed in masakari:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.