devstack

Failure in stacking devstack

Bug #1832677 reported by Aman Kumar Sinha
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
devstack
Expired
Undecided
Unassigned

Bug Description

For PowerVM-CI, the stacking devstack fails because two of the services(q-l3 and q-agt)are not getting up with the logs. This happens specially for PowerVM driver:
05:22:51 +functions-common:run_process:1559  local service=q-agt
05:22:51 +functions-common:run_process:1560  local 'command= --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini'
05:22:51 +functions-common:run_process:1561  local group=
05:22:51 +functions-common:run_process:1562  local user=
05:22:52 +functions-common:run_process:1564  local name=q-agt
05:22:52 +functions-common:run_process:1566  time_start run_process
05:22:52 +functions-common:time_start:2303  local name=run_process
05:22:52 +functions-common:time_start:2304  local start_time=
05:22:52 +functions-common:time_start:2305  [[ -n '' ]]
05:22:52 ++functions-common:time_start:2308  date +%s%3N
05:22:52 +functions-common:time_start:2308  _TIME_START[$name]=1559035372020
05:22:52 +functions-common:run_process:1567  is_service_enabled q-agt
05:22:52 +functions-common:is_service_enabled:1969  return 0
05:22:52 +functions-common:run_process:1568  _run_under_systemd q-agt ' --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini' '' ''
05:22:52 +functions-common:_run_under_systemd:1524  local service=q-agt
05:22:52 +functions-common:_run_under_systemd:1525  local 'command= --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini'
05:22:52 +functions-common:_run_under_systemd:1526  local 'cmd= --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini'
05:22:52 +functions-common:_run_under_systemd:1528  _common_systemd_pitfalls ' --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini'
05:22:52 +functions-common:_common_systemd_pitfalls:1496  local 'cmd= --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini'
05:22:52 +functions-common:_common_systemd_pitfalls:1499  [[ --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini =~ sudo ]]
05:22:52 +functions-common:_common_systemd_pitfalls:1510  [[ ! --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini =~ ^/ ]]
05:22:52 +functions-common:_common_systemd_pitfalls:1511  read -r -d '' msg
05:22:52 +functions-common:_common_systemd_pitfalls:1516  true
05:22:52 +functions-common:_common_systemd_pitfalls:1517  die 1517 'The cmd=" --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini" does not start with an absolute path. It will fail to
05:22:52 start under systemd.
05:22:52
05:22:52 Please update your run_process stanza to have an absolute path.'

And the q-l3 is failing because interface-driver is None.

'neutron.objects.qos.policy.QosPolicy'>> rbac-policy before_create 55550000 {{(pid=6627) subscribe /usr/local/lib/python2.7/dist-packages/neutron_lib/callbacks/manager.py:48}}
2019-06-07 13:32:14.296 | Jun 07 08:29:27 test3-25 neutron-l3-agent[6627]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: <bound method DeclarativeObject.validate_rbac_policy_change of <class 'neutron.objects.qos.policy.QosPolicy'>> rbac-policy before_update 55550000 {{(pid=6627) subscribe /usr/local/lib/python2.7/dist-packages/neutron_lib/callbacks/manager.py:48}}
2019-06-07 13:32:14.296 | Jun 07 08:29:27 test3-25 neutron-l3-agent[6627]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: <bound method DeclarativeObject.validate_rbac_policy_change of <class 'neutron.objects.qos.policy.QosPolicy'>> rbac-policy before_delete 55550000 {{(pid=6627) subscribe /usr/local/lib/python2.7/dist-packages/neutron_lib/callbacks/manager.py:48}}
2019-06-07 13:32:14.296 | Jun 07 08:29:27 test3-25 neutron-l3-agent[6627]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: <bound method DeclarativeObject.validate_rbac_policy_change of <class 'neutron.objects.securitygroup.SecurityGroup'>> rbac-policy before_create 55550000 {{(pid=6627) subscribe /usr/local/lib/python2.7/dist-packages/neutron_lib/callbacks/manager.py:48}}
2019-06-07 13:32:14.296 | Jun 07 08:29:27 test3-25 neutron-l3-agent[6627]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: <bound method DeclarativeObject.validate_rbac_policy_change of <class 'neutron.objects.securitygroup.SecurityGroup'>> rbac-policy before_update 55550000 {{(pid=6627) subscribe /usr/local/lib/python2.7/dist-packages/neutron_lib/callbacks/manager.py:48}}
2019-06-07 13:32:14.296 | Jun 07 08:29:27 test3-25 neutron-l3-agent[6627]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: <bound method DeclarativeObject.validate_rbac_policy_change of <class 'neutron.objects.securitygroup.SecurityGroup'>> rbac-policy before_delete 55550000 {{(pid=6627) subscribe /usr/local/lib/python2.7/dist-packages/neutron_lib/callbacks/manager.py:48}}
2019-06-07 13:32:14.296 | Jun 07 08:29:28 test3-25 neutron-l3-agent[6627]: ERROR neutron.agent.l3.agent [-] An interface driver must be specified
2019-06-07 13:32:14.296 | Jun 07 08:29:28 test3-25 systemd[1]: <email address hidden>: Main process exited, code=exited, status=1/FAILURE
2019-06-07 13:32:14.296 | Jun 07 08:29:28 test3-25 systemd[1]: <email address hidden>: Unit entered failed state.
2019-06-07 13:32:14.298 | Jun 07 08:29:28 test3-25 systemd[1]: <email address hidden>: Failed with result 'exit-code'.
2019-06-07 13:32:14.308 | +functions-common:service_check:1 ^[[m^O exit_trap
2019-06-07 13:32:14.313 | +./stack.sh:exit_trap:525 ^[[m^O local r=3
2019-06-07 13:32:14.322 | ++./stack.sh:exit_trap:526 ^[[m^O jobs -p
2019-06-07 13:32:14.330 | +./stack.sh:exit_trap:526 ^[[m^O jobs=
2019-06-07 13:32:14.337 | +./stack.sh:exit_trap:529 ^[[m^O [[ -n '' ]]
2019-06-07 13:32:14.342 | +./stack.sh:exit_trap:535 ^[[m^O '[' -f /tmp/tmp.3BET3mbD7a ']'
2019-06-07 13:32:14.359 | +./stack.sh:exit_trap:536 ^[[m^O rm /tmp/tmp.3BET3mbD7a
2019-06-07 13:32:14.364 | +./stack.sh:exit_trap:540 ^[[m^O kill_spinner
2019-06-07 13:32:14.374 | +./stack.sh:kill_spinner:435 ^[[m^O '[' '!' -z '' ']'
2019-06-07 13:32:14.386 | +./stack.sh:exit_trap:542 ^[[m^O [[ 3 -ne 0 ]]
2019-06-07 13:32:14.394 | +./stack.sh:exit_trap:543 ^[[m^O echo 'Error on exit'
2019-06-07 13:32:14.394 | Error on exit
2019-06-07 13:32:14.404 | +./stack.sh:exit_trap:545 ^[[m^O type -p generate-subunit
2019-06-07 13:32:14.417 | +./stack.sh:exit_trap:546 ^[[m^O generate-subunit 1559912916 1418 fail
2019-06-07 13:32:15.261 | +./stack.sh:exit_trap:548 ^[[m^O [[ -z /opt/stack/logs ]]
2019-06-07 13:32:15.267 | +./stack.sh:exit_trap:551 ^[[m^O /opt/stack/devstack/tools/worlddump.py -d /opt/stack/logs
2019-06-07 13:32:16.605 | neutron-dhcp-agent: no process found
2019-06-07 13:32:16.673 | neutron-metadata-agent: no process found
2019-06-07 13:32:16.737 | +./stack.sh:exit_trap:560 ^[[m^O exit 3

See original description
Revision history for this message
Aman Kumar Sinha (amansi26) wrote :

https://review.opendev.org/#/c/664797/

Changed in nova-powervm:
assignee: nobody → Aman Kumar Sinha (amansi26)
Divya K Konoor (dikonoor)
affects: nova-powervm → devstack
Revision history for this message
Divya K Konoor (dikonoor) wrote :

Aman,

the proposed changeset at https://review.opendev.org/#/c/664797/ says-

While stacking devstack in PowerVM-CI, q-agt was failing because the proper command is not getting passed. While q-l3 was failing due to no interface driver found.

I am unable to find a "no interface driver found" error anywhere in the stacktrace. Have you missed adding it? You might want to also add some information on since when this problem is been observed and does this happen with regular latest devstack installation or merely with the PowerVM drivers.

OpenStack Infra (hudson-openstack)
Changed in devstack:
status: New → In Progress
Aman Kumar Sinha (amansi26)
description: updated
Revision history for this message
Dr. Jens Harbott (j-harbott) wrote :

This is still lacking information in order to be properly triaged. Please upload a copy of the local.conf file you are using and a complete log of the stack.sh run.

Changed in devstack:
status: In Progress → Incomplete
Revision history for this message
Veda Annayappa (vedanu) wrote :

Its observe that currently the powerVM-CI is failing for glance.

Revision history for this message
Veda Annayappa (vedanu) wrote :

PFA of local.conf that was used.

Revision history for this message
Dr. Jens Harbott (j-harbott) wrote :

Looking at your local.conf file, this will no longer work:

enable_plugin neutron git://git.openstack.org/openstack/neutron

as we no longer support the git protocol. But then in your CI, you seem to be pre-cloning those repos from a correct source, so that error doesn't happen there.

But the error in your log doesn't match what you show in the bug description. Please upload a log file with an error that matches the description, otherwise this looks invalid. Do you also have a pointer to where complete results from runs of your CI are available?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on devstack (master)

Change abandoned by Jens Harbott (frickler) (<email address hidden>) on branch: master
Review: https://review.opendev.org/664797

Dr. Jens Harbott (j-harbott)
Changed in devstack:
assignee: Aman Kumar Sinha (amansi26) → nobody
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for devstack because there has been no activity for 60 days.]

Changed in devstack:
status: Incomplete → Expired
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.