many UT cases are broken due to "AttributeError: type object 'Route' has no attribute 'foreign_keys'"

Bug #1731623 reported by YAMAMOTO Takashi
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
networking-midonet
Fix Released
Critical
YAMAMOTO Takashi
neutron
Fix Released
Critical
YAMAMOTO Takashi

Bug Description

eg. http://logs.openstack.org/24/518824/1/gate/openstack-tox-py27/6ea4c06/job-output.txt.gz

2017-11-11 07:19:32.914772 | ubuntu-xenial | {2} midonet.neutron.tests.unit.test_extension_taas.TestMidonetTaasCaseML2.test_delete_tap_service_error_delete_neutron_resource [0.865560s] ... ok
2017-11-11 07:19:32.931319 | ubuntu-xenial | INFO [alembic.runtime.migration] Running upgrade 24f28869838b -> 41b509d10b5e, VPNaaS endpoint groups
2017-11-11 07:19:32.947636 | ubuntu-xenial | add_router_interface failed: No details.
2017-11-11 07:19:32.947704 | ubuntu-xenial | Traceback (most recent call last):
2017-11-11 07:19:32.947748 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/api/v2/resource.py", line 98, in resource
2017-11-11 07:19:32.947775 | ubuntu-xenial | result = method(request=request, **args)
2017-11-11 07:19:32.947812 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/db/api.py", line 92, in wrapped
2017-11-11 07:19:32.947837 | ubuntu-xenial | setattr(e, '_RETRY_EXCEEDED', True)
2017-11-11 07:19:32.947889 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/networking-midonet/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-11-11 07:19:32.947946 | ubuntu-xenial | self.force_reraise()
2017-11-11 07:19:32.948002 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/networking-midonet/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-11-11 07:19:32.948029 | ubuntu-xenial | six.reraise(self.type_, self.value, self.tb)
2017-11-11 07:19:32.948086 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/db/api.py", line 88, in wrapped
2017-11-11 07:19:32.948117 | ubuntu-xenial | return f(*args, **kwargs)
2017-11-11 07:19:32.948169 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/networking-midonet/.tox/py27/local/lib/python2.7/site-packages/oslo_db/api.py", line 150, in wrapper
2017-11-11 07:19:32.948191 | ubuntu-xenial | ectxt.value = e.inner_exc
2017-11-11 07:19:32.948259 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/networking-midonet/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-11-11 07:19:32.948291 | ubuntu-xenial | self.force_reraise()
2017-11-11 07:19:32.948363 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/networking-midonet/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-11-11 07:19:32.948393 | ubuntu-xenial | six.reraise(self.type_, self.value, self.tb)
2017-11-11 07:19:32.948444 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/networking-midonet/.tox/py27/local/lib/python2.7/site-packages/oslo_db/api.py", line 138, in wrapper
2017-11-11 07:19:32.948466 | ubuntu-xenial | return f(*args, **kwargs)
2017-11-11 07:19:32.948504 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/db/api.py", line 127, in wrapped
2017-11-11 07:19:32.948533 | ubuntu-xenial | LOG.debug("Retry wrapper got retriable exception: %s", e)
2017-11-11 07:19:32.948586 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/networking-midonet/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-11-11 07:19:32.948607 | ubuntu-xenial | self.force_reraise()
2017-11-11 07:19:32.948659 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/networking-midonet/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-11-11 07:19:32.948686 | ubuntu-xenial | six.reraise(self.type_, self.value, self.tb)
2017-11-11 07:19:32.948723 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/db/api.py", line 123, in wrapped
2017-11-11 07:19:32.948746 | ubuntu-xenial | return f(*dup_args, **dup_kwargs)
2017-11-11 07:19:32.948786 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/api/v2/base.py", line 247, in _handle_action
2017-11-11 07:19:32.948817 | ubuntu-xenial | ret_value = getattr(self._plugin, name)(*arg_list, **kwargs)
2017-11-11 07:19:32.948867 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/networking-midonet/.tox/py27/local/lib/python2.7/site-packages/oslo_log/helpers.py", line 67, in wrapper
2017-11-11 07:19:32.948890 | ubuntu-xenial | return method(*args, **kwargs)
2017-11-11 07:19:32.948927 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/db/api.py", line 162, in wrapped
2017-11-11 07:19:32.948950 | ubuntu-xenial | return method(*args, **kwargs)
2017-11-11 07:19:32.948988 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/db/api.py", line 92, in wrapped
2017-11-11 07:19:32.949012 | ubuntu-xenial | setattr(e, '_RETRY_EXCEEDED', True)
2017-11-11 07:19:32.949067 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/networking-midonet/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-11-11 07:19:32.949089 | ubuntu-xenial | self.force_reraise()
2017-11-11 07:19:32.949142 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/networking-midonet/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-11-11 07:19:32.949167 | ubuntu-xenial | six.reraise(self.type_, self.value, self.tb)
2017-11-11 07:19:32.949205 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/db/api.py", line 88, in wrapped
2017-11-11 07:19:32.949226 | ubuntu-xenial | return f(*args, **kwargs)
2017-11-11 07:19:32.949276 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/networking-midonet/.tox/py27/local/lib/python2.7/site-packages/oslo_db/api.py", line 150, in wrapper
2017-11-11 07:19:32.949297 | ubuntu-xenial | ectxt.value = e.inner_exc
2017-11-11 07:19:32.949349 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/networking-midonet/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-11-11 07:19:32.949368 | ubuntu-xenial | self.force_reraise()
2017-11-11 07:19:32.949427 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/networking-midonet/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-11-11 07:19:32.949455 | ubuntu-xenial | six.reraise(self.type_, self.value, self.tb)
2017-11-11 07:19:32.949504 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/networking-midonet/.tox/py27/local/lib/python2.7/site-packages/oslo_db/api.py", line 138, in wrapper
2017-11-11 07:19:32.949526 | ubuntu-xenial | return f(*args, **kwargs)
2017-11-11 07:19:32.949563 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/db/api.py", line 127, in wrapped
2017-11-11 07:19:32.949591 | ubuntu-xenial | LOG.debug("Retry wrapper got retriable exception: %s", e)
2017-11-11 07:19:32.949642 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/networking-midonet/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-11-11 07:19:32.949662 | ubuntu-xenial | self.force_reraise()
2017-11-11 07:19:32.949715 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/networking-midonet/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-11-11 07:19:32.949741 | ubuntu-xenial | six.reraise(self.type_, self.value, self.tb)
2017-11-11 07:19:32.949777 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/db/api.py", line 123, in wrapped
2017-11-11 07:19:32.949800 | ubuntu-xenial | return f(*dup_args, **dup_kwargs)
2017-11-11 07:19:32.949835 | ubuntu-xenial | File "midonet/neutron/services/l3/l3_midonet.py", line 181, in add_router_interface
2017-11-11 07:19:32.949859 | ubuntu-xenial | context, router_id, interface_info)
2017-11-11 07:19:32.949899 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/db/l3_db.py", line 1874, in add_router_interface
2017-11-11 07:19:32.949923 | ubuntu-xenial | context, router_id, interface_info)
2017-11-11 07:19:32.949960 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/db/api.py", line 162, in wrapped
2017-11-11 07:19:32.950283 | ubuntu-xenial | return method(*args, **kwargs)
2017-11-11 07:19:32.950337 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/db/l3_db.py", line 844, in add_router_interface
2017-11-11 07:19:32.950368 | ubuntu-xenial | context, router, interface_info['subnet_id'], device_owner)
2017-11-11 07:19:32.950423 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/db/l3_db.py", line 782, in _add_interface_by_subnet
2017-11-11 07:19:32.950463 | ubuntu-xenial | subnet['network_id'], [subnet])
2017-11-11 07:19:32.950511 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/db/l3_db.py", line 615, in _check_for_dup_router_subnets
2017-11-11 07:19:32.950532 | ubuntu-xenial | filters=id_filter)
2017-11-11 07:19:32.950569 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/db/api.py", line 162, in wrapped
2017-11-11 07:19:32.950592 | ubuntu-xenial | return method(*args, **kwargs)
2017-11-11 07:19:32.950687 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/db/db_base_plugin_v2.py", line 1023, in get_subnets
2017-11-11 07:19:32.951328 | ubuntu-xenial | marker, page_reverse)
2017-11-11 07:19:32.951383 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/db/db_base_plugin_common.py", line 282, in _get_subnets
2017-11-11 07:19:32.951403 | ubuntu-xenial | **filters)
2017-11-11 07:19:32.951444 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/objects/base.py", line 468, in get_objects
2017-11-11 07:19:32.951500 | ubuntu-xenial | return [cls._load_object(context, db_obj) for db_obj in db_objs]
2017-11-11 07:19:32.951563 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/objects/base.py", line 403, in _load_object
2017-11-11 07:19:32.951591 | ubuntu-xenial | obj.from_db_object(db_obj)
2017-11-11 07:19:32.951649 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/objects/subnet.py", line 250, in from_db_object
2017-11-11 07:19:32.951680 | ubuntu-xenial | super(Subnet, self).from_db_object(db_obj)
2017-11-11 07:19:32.951722 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/objects/base.py", line 347, in from_db_object
2017-11-11 07:19:32.951754 | ubuntu-xenial | self.load_synthetic_db_fields(db_obj)
2017-11-11 07:19:32.951838 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/objects/base.py", line 596, in load_synthetic_db_fields
2017-11-11 07:19:32.952042 | ubuntu-xenial | foreign_keys = objclass.foreign_keys.get(clsname)
2017-11-11 07:19:32.952080 | ubuntu-xenial | AttributeError: type object 'Route' has no attribute 'foreign_keys'

tags: added: gate-failure midokura-jira-tracked
Changed in networking-midonet:
importance: Undecided → Critical
Revision history for this message
YAMAMOTO Takashi (yamamoto) wrote :

the recent neutron change [1] caused the problem.

[1] I04e155cb92be65f4d5e8e9a4c21e6ca736aff82c

Changed in neutron:
assignee: nobody → YAMAMOTO Takashi (yamamoto)
status: New → In Progress
Revision history for this message
YAMAMOTO Takashi (yamamoto) wrote :
Changed in neutron:
importance: Undecided → Critical
Revision history for this message
Slawek Kaplonski (slaweq) wrote :

I think that I found what is reason of issue. I was checking one example test in midonet: midonet.neutron.tests.unit.test_midonet_plugin_ml2.TestMidonetSubnetsML2.test_create_subnet_with_none_gateway_allocation_pool

It looks that it is failing because it tries to load Route object from os_vif.objects.route.Route instead of neutron.objects.subnet.Route and that cause problem.
IMHO patch which You are trying to revert is not reason of issue - it only triggered "real" bug but it would be great is someone more experienced with OVO could look at it also.

Revision history for this message
YAMAMOTO Takashi (yamamoto) wrote :

maybe we should have our own registry?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/519017
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1470baf02beb6e9f732129edc4c8979881ddea8f
Submitter: Zuul
Branch: master

commit 1470baf02beb6e9f732129edc4c8979881ddea8f
Author: YAMAMOTO Takashi <email address hidden>
Date: Sat Nov 11 13:16:43 2017 +0000

    Revert "objects: get, update and delete converted to Subnet OVO usage"

    This reverts commit 32c757babd1622b72fa15a8f5fb1248ff9e45ddf.

    Closes-Bug: #1731623
    Change-Id: Ie5c773165948d6db55f1b7ba3b7e312a5bb1318b

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
Slawek Kaplonski (slaweq) wrote :

"maybe we should have our own registry?"

who You mean by saying "we"? Neutron or networking-midonet?

Revision history for this message
Ihar Hrachyshka (ihar-hrachyshka) wrote :

Yes, we need a separate registry for Neutron objects.

Revision history for this message
YAMAMOTO Takashi (yamamoto) wrote :

Slawek, Neutron and os-vif i meant.

Revision history for this message
Slawek Kaplonski (slaweq) wrote :

@YAMAMOTO: yes, there is patch for that already merged in Neutron: https://review.openstack.org/#/c/519622/
It looks that it solves this problem and we will be able to merge again patch which triggered this issue.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 12.0.0.0b2

This issue was fixed in the openstack/neutron 12.0.0.0b2 development milestone.

tags: added: neutron-proactive-backport-potential
Changed in networking-midonet:
assignee: nobody → YAMAMOTO Takashi (yamamoto)
milestone: none → 6.0.0
status: New → Fix Released
tags: removed: neutron-proactive-backport-potential
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.