gate-rally-dsvm-neutron-neutron-ubuntu-xenial failing 100% in NeutronTrunks.create_and_list_trunk_subports

Bug #1714901 reported by Dr. Jens Harbott
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Critical
Kevin Benton

Bug Description

This has started about two days ago:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22NeutronTrunks.create_and_list_trunk_subports%5C%22%20AND%20message%3A%5C%22FAIL%5C%22

Sample failure log:

http://logs.openstack.org/61/500261/2/check/gate-rally-dsvm-neutron-neutron-ubuntu-xenial/f5742a5/rally-plot/detailed.txt.gz

--------------------------------------------------------------------------------
Task fe68e403-0694-4016-be5d-e02f67cb04fd has 4 error(s)
--------------------------------------------------------------------------------

NotFound: The resource could not be found.
Neutron server returns request_ids: ['req-ff94c196-cb60-41d6-9e2c-42f780960d69']

Traceback (most recent call last):
  File "/opt/stack/new/rally/rally/task/runner.py", line 72, in _run_scenario_once
    getattr(scenario_inst, method_name)(**scenario_kwargs)
  File "/home/jenkins/.rally/plugins/plugins/trunk_scenario.py", line 41, in run
    trunk = self._create_trunk(trunk_payload)
  File "/opt/stack/new/rally/rally/task/atomic.py", line 85, in func_atomic_actions
    f = func(self, *args, **kwargs)
  File "/home/jenkins/.rally/plugins/plugins/trunk_scenario.py", line 53, in _create_trunk
    return self.clients("neutron").create_trunk({'trunk': trunk_payload})
  File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 2067, in create_trunk
    return self.post(self.trunks_path, body=body)
  File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 357, in post
    headers=headers, params=params)
  File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 292, in do_request
    self._handle_fault_response(status_code, replybody, resp)
  File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 268, in _handle_fault_response
    exception_handler_v20(status_code, error_body)
  File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 92, in exception_handler_v20
    request_ids=request_ids)
NotFound: The resource could not be found.
Neutron server returns request_ids: ['req-ff94c196-cb60-41d6-9e2c-42f780960d69']

Revision history for this message
Dr. Jens Harbott (j-harbott) wrote :

Error in q-svc log:

Sep 03 09:47:41.411813 ubuntu-xenial-infracloud-vanilla-10753242 neutron-server[24740]: WARNING neutron.pecan_wsgi.controllers.root [None req-ff94c196-cb60-41d6-9e2c-42f780960d69 c_rally_b1783656_tSZKaxSM c_rally_b1783656_SFIfLxli] No controller found for: trunks - returni
ng response code 404: PecanNotFound
Sep 03 09:47:41.412576 ubuntu-xenial-infracloud-vanilla-10753242 neutron-server[24740]: INFO neutron.pecan_wsgi.hooks.translation [None req-ff94c196-cb60-41d6-9e2c-42f780960d69 c_rally_b1783656_tSZKaxSM c_rally_b1783656_SFIfLxli] POST failed (client error): The resource co
uld not be found.
Sep 03 09:47:41.412970 ubuntu-xenial-infracloud-vanilla-10753242 neutron-server[24740]: DEBUG neutron.pecan_wsgi.hooks.notifier [None req-ff94c196-cb60-41d6-9e2c-42f780960d69 c_rally_b1783656_tSZKaxSM c_rally_b1783656_SFIfLxli] Skipping NotifierHook processing as there was
 no resource associated with the request {{(pid=24832) after /opt/stack/new/neutron/neutron/pecan_wsgi/hooks/notifier.py:64}}
Sep 03 09:47:41.413865 ubuntu-xenial-infracloud-vanilla-10753242 neutron-server[24740]: INFO neutron.wsgi [None req-ff94c196-cb60-41d6-9e2c-42f780960d69 c_rally_b1783656_tSZKaxSM c_rally_b1783656_SFIfLxli] 15.184.65.164 "POST /v2.0/trunks HTTP/1.1" status: 404 len: 309 ti
me: 0.0698869

Revision history for this message
Dr. Jens Harbott (j-harbott) wrote :
Revision history for this message
Kevin Benton (kevinbenton) wrote :

Yeah, I think some infra config changed quite a while ago and rally just now started doing the correct thing and failing the job on errors.

Changed in neutron:
importance: Undecided → Critical
assignee: nobody → Kevin Benton (kevinbenton)
status: New → Confirmed
Revision history for this message
Kevin Benton (kevinbenton) wrote :

Infra change to add the trunk plugin: https://review.openstack.org/#/c/500496/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/500527

Changed in neutron:
assignee: Kevin Benton (kevinbenton) → YAMAMOTO Takashi (yamamoto)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (master)

Change abandoned by YAMAMOTO Takashi (<email address hidden>) on branch: master
Review: https://review.openstack.org/500527
Reason: https://review.openstack.org/#/c/500501/ is already tested on gate

Changed in neutron:
assignee: YAMAMOTO Takashi (yamamoto) → Kevin Benton (kevinbenton)
Revision history for this message
Andriy Kurilin (andreykurilin) wrote :

There was no default SLA in Rally for a long time. It means that if SLA section is not specified in task config, Rally will not mark the workload as failed in case of errors.
This thing is changed last week. Now, if SLA section is not specified explicitly, failure rate max 100% is used by default. Such change is a root cause of your failures.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/500501
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=07ddb11685663a00ff963549ad169614d2592c30
Submitter: Jenkins
Branch: master

commit 07ddb11685663a00ff963549ad169614d2592c30
Author: Kevin Benton <email address hidden>
Date: Mon Sep 4 03:24:48 2017 -0700

    Match load_rc_for_rally logic to load_rc_hook

    This makes the rally extras loading logic compatible
    with devstack-gate and unblocks the rally job.

    Closes-Bug: #1714901
    Change-Id: I80c9a155ee2b52558109c764075a58dfabee44d4

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/500902

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/500903

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/500904

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/ocata)

Reviewed: https://review.openstack.org/500903
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=6377bb1140f739f8da2c8d223d81af4bacbf2b34
Submitter: Jenkins
Branch: stable/ocata

commit 6377bb1140f739f8da2c8d223d81af4bacbf2b34
Author: Kevin Benton <email address hidden>
Date: Mon Sep 4 03:24:48 2017 -0700

    Match load_rc_for_rally logic to load_rc_hook

    This makes the rally extras loading logic compatible
    with devstack-gate and unblocks the rally job.

    Closes-Bug: #1714901
    Change-Id: I80c9a155ee2b52558109c764075a58dfabee44d4
    (cherry picked from commit 07ddb11685663a00ff963549ad169614d2592c30)

tags: added: in-stable-ocata
tags: added: in-stable-newton
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/newton)

Reviewed: https://review.openstack.org/500904
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=042cd342ddc6d96adba15247b61a572fe790a016
Submitter: Jenkins
Branch: stable/newton

commit 042cd342ddc6d96adba15247b61a572fe790a016
Author: Kevin Benton <email address hidden>
Date: Mon Sep 4 03:24:48 2017 -0700

    Match load_rc_for_rally logic to load_rc_hook

    This makes the rally extras loading logic compatible
    with devstack-gate and unblocks the rally job.

    Closes-Bug: #1714901
    Change-Id: I80c9a155ee2b52558109c764075a58dfabee44d4
    (cherry picked from commit 07ddb11685663a00ff963549ad169614d2592c30)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/pike)

Reviewed: https://review.openstack.org/500902
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=63acf41ee83d77fab90e12fa81e9a171f4a48afe
Submitter: Jenkins
Branch: stable/pike

commit 63acf41ee83d77fab90e12fa81e9a171f4a48afe
Author: Kevin Benton <email address hidden>
Date: Mon Sep 4 03:24:48 2017 -0700

    Match load_rc_for_rally logic to load_rc_hook

    This makes the rally extras loading logic compatible
    with devstack-gate and unblocks the rally job.

    Closes-Bug: #1714901
    Change-Id: I80c9a155ee2b52558109c764075a58dfabee44d4
    (cherry picked from commit 07ddb11685663a00ff963549ad169614d2592c30)

tags: added: in-stable-pike
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 10.0.4

This issue was fixed in the openstack/neutron 10.0.4 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 11.0.1

This issue was fixed in the openstack/neutron 11.0.1 release.

tags: added: neutron-proactive-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 12.0.0.0b1

This issue was fixed in the openstack/neutron 12.0.0.0b1 development milestone.

tags: removed: neutron-proactive-backport-potential
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.