Serializer strips Exception kwargs

"This is resulting in the Exception kwargs being stripped from the serialized Exception."
I tried to reproduce that before the referenced oslo.serialization change the kwargs of the Exception is serialized. But I failed:

(py27) ebalgib@elxa95hld12:~/upstream/git/oslo.serialization$ pip freeze | grep seriali
oslo.serialization==2.21.1
(py27) ebalgib@elxa95hld12:~/upstream/git/oslo.serialization$ python
Python 2.7.12 (default, Dec 4 2017, 14:50:18)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> class MyException(Exception):
... def __init__(self, named_arg):
... super(MyException, self).__init__('message')
...
>>> from oslo_serialization import jsonutils
>>> print jsonutils.to_primitive(MyException(named_arg='alma'))
oslo_serialization/jsonutils.py:188: UserWarning: Cannot convert MyException('message',) to primitive, will raise ValueError instead of warning in version 3.0
  "instead of warning in version 3.0" % (value,))
message

Could you attach some logs that shows that Exception kwargs being serialized with old enough oslo serialization?

https://review.openstack.org/#/c/554607/

Change abandoned by Tyler Blakeslee (<email address hidden>) on branch: master
Review: https://review.openstack.org/554607

Balazs Gibizer - When calling jsonutils.to_primitive(), nova passes in the kwarg convert_instances=True from https://github.com/openstack/nova/blob/master/nova/rpc.py#L119.

If you do that in your personal environment, you should not see that warning message.

Thanks for the extra information. Now I see that there is different behavior. Next I will try to see how the notifications look differently with the two versions.

(py27) ebalgib@elxa95hld12:~/upstream/git/oslo.serialization$ pip freeze | grep serialization
oslo.serialization==2.21.1
(py27) ebalgib@elxa95hld12:~/upstream/git/oslo.serialization$ python
Python 2.7.12 (default, Dec 4 2017, 14:50:18)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>>
>>> class MyException(Exception):
... def __init__(self, named_arg):
... super(MyException, self).__init__('message')
...
>>> from oslo_serialization import jsonutils
>>> print jsonutils.to_primitive(MyException(named_arg='alma'))
oslo_serialization/jsonutils.py:188: UserWarning: Cannot convert MyException('message',) to primitive, will raise ValueError instead of warning in version 3.0
  "instead of warning in version 3.0" % (value,))
message
>>> print jsonutils.to_primitive(MyException(named_arg='alma'), convert_instances=True)
{}
>>>

---

(py27) ebalgib@elxa95hld12:~/upstream/git/oslo.serialization$ pip freeze | grep serialization
oslo.serialization==2.25.1.dev1
(py27) ebalgib@elxa95hld12:~/upstream/git/oslo.serialization$ python
Python 2.7.12 (default, Dec 4 2017, 14:50:18)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> class MyException(Exception):
... def __init__(self, named_arg):
... super(MyException, self).__init__('message')
...
>>>
>>> from oslo_serialization import jsonutils
>>> print jsonutils.to_primitive(MyException(named_arg='alma'))
MyException('message',)
>>> print jsonutils.to_primitive(MyException(named_arg='alma'), convert_instances=True)
MyException('message',)
>>>

---

So with oslo.serialization 2.21.1 the reason field of the compute_task.build_instances legacy notification looks like:

'reason': {'kwargs': {'code': 500, 'reason': ''},
            'message': u'No valid host was found. '},

while with newer oslo.serialization it looks like:

'reason': "NoValidHost(u'No valid host was found. ',)",

So the oslo.serialization change did affect the payload of the legacy compute_task.build_instances ERROR notification.

Please note that this notification hasn't been transformed to the new versioned framework yet. But this bug does not directly affect any already transformed notification as the Exception payload in the versioned notifications does not directly store a serialized Exception object. Instead it contains just the exception type and the exception message.

https://github.com/openstack/nova/blob/master/nova/notifications/objects/exception.py#L25
https://github.com/openstack/nova/blob/master/doc/notification_samples/instance-create-error.json

---

See the full payload below:

{'instance_id': 'c1168eba-89a1-43af-86bd-b6c164dd0f94',
 'instance_properties': {'availability_zone': None,
                         'ephemeral_gb': 0,
                         'memory_mb': 512,
                         'numa_topology': None,
                         'pci_requests': {'requests': []},
                         'project_id': u'6f70656e737461636b20342065766572',
                         'root_gb': 1,
                         'uuid': 'c1168eba-89a1-43af-86bd-b6c164dd0f94',
                         'vcpus': 1},
 'method': 'build_instances',
 'reason': {'kwargs': {'code': 500, 'reason': ''},
            'message': u'No valid host was found. '},
 'request_spec': {'image': {'container_format': u'raw',
                            'created_at': '2011-01-01T01:02:03.000000',
                            'disk_format': u'raw',
                            'id': '155d900f-4e14-4e4c-a73d-069cbf4541e6',
                            'name': u'fakeimage123456',
                            'properties': {'hw_architecture': u'x86_64'},
                            'size': 25165824,
                            'status': u'active',
                            'updated_at': '2011-01-01T01:02:03.000000'},
                  'instance_properties': {'availability_zone': None,
                                          'ephemeral_gb': 0,
                                          'memory_mb': 512,
                                          'numa_topology': None,
                                          'pci_requests': {'requests': []},
                                          'project_id': u'6f70656e737461636b20342065766572',
                                          'root_gb': 1,
                                          'uuid': 'c1168eba-89a1-43af-86bd-b6c164dd0f94',
                                          'vcpus': 1},
                  'instance_type': {'created_at': '2018-03-26T13:05:22.000000',
                                    'deleted': False,
                                    'deleted_at': None,
                                    'description': None,
                                    'disabled': False,
                         ...

Reviewed: https://review.openstack.org/555812
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=cc457dfffad7c4980e6f36364bdf190c79cf23e1
Submitter: Zuul
Branch: master

commit cc457dfffad7c4980e6f36364bdf190c79cf23e1
Author: Tyler Blakeslee <email address hidden>
Date: Fri Mar 23 08:21:27 2018 -0600

    Add __repr__ for NovaException

    Due to change c1a7079c26d27a2e46cca26963d3d9aa040bdbe8 in
    oslo.serialization, serialized exceptions in errors no longer
    have kwargs. This change fixes the problem by adding __repr__
    to NovaException which will return a string representation
    of exception.__dict__. This string contains the exception message
    along with all the kwargs. Without __repr__, the serialization
    process for exceptions just returns the exception type and the
    default message, without the kwargs.

    Change-Id: I653282a030d03362dfca0fd1026cebe920d54e37
    Closes-Bug: #1756360

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/559158

Fix proposed to branch: master
Review: https://review.openstack.org/559169

Change abandoned by Tyler Blakeslee (<email address hidden>) on branch: master
Review: https://review.openstack.org/559169
Reason: The issue this change set was set to address does not appear to exist. Since the problem can't be reproduced on master, we'll abandon this change set.

Reviewed: https://review.openstack.org/559158
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=9affdb02977e1016c7c0cca0b9b677ad4b3870d3
Submitter: Zuul
Branch: stable/queens

commit 9affdb02977e1016c7c0cca0b9b677ad4b3870d3
Author: Tyler Blakeslee <email address hidden>
Date: Fri Mar 23 08:21:27 2018 -0600

    Add __repr__ for NovaException

    Due to change c1a7079c26d27a2e46cca26963d3d9aa040bdbe8 in
    oslo.serialization, serialized exceptions in errors no longer
    have kwargs. This change fixes the problem by adding __repr__
    to NovaException which will return a string representation
    of exception.__dict__. This string contains the exception message
    along with all the kwargs. Without __repr__, the serialization
    process for exceptions just returns the exception type and the
    default message, without the kwargs.

    Change-Id: I653282a030d03362dfca0fd1026cebe920d54e37
    Closes-Bug: #1756360
    (cherry picked from commit cc457dfffad7c4980e6f36364bdf190c79cf23e1)

This issue was fixed in the openstack/nova 18.0.0.0b1 development milestone.

This issue was fixed in the openstack/nova 17.0.3 release.