L2 Agent switch to non-dvr mode on first RPC failure

Carl,

Can you please let us know why this is classified 'High'.

A restart of L2 Agent will bring it back to DVR mode (after controller is made up).

I classified as High because I didn't see a very friendly way to diagnose this problem in order to apply the L2 agent restart to fix it. My impression is that this would manifest itself externally as a general failure of DVR without much indication about where to start debugging.

If you think this would be obvious to deployers then I could downgrade the importance of this bug.

Vivek, are you actively working on a fix?

Yes, i have done some changes already.

I have moved the DVR RPCs to be now invoked from rpc_loop of neutron agent instead of init_()-constructor of the agent.

Some test cases are affected and so am fixing them. I will post patch for review next week.

Carls question:

The L2 Agent report state information carries which mode it operates on using enable_distributed_routing flag.

When the first rpc failure happens, from that point on all the report-state carry enable_distributed_routing=False in the payload. So just by doing neutron agent-show <openvswitch-agent-id of the node where problem is>, the admin can recognize that the agent is operating is non-dvr mode.

So we donot silently move to a situation of running in non-dvr mode. It is easily diagnosable.

So I request to reduce the severity.

I do see this failure right after an RPC error:

2014-09-23 13:31:58.292 10102 DEBUG oslo.messaging._drivers.impl_rabbit [req-67030e45-807c-46ea-8553-42b8b278d1f5 ] Timed out waiting for RPC response: timed out _error_callback /opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py:721
2014-09-23 13:31:58.293 10102 ERROR neutron.plugins.openvswitch.agent.ovs_dvr_neutron_agent [req-67030e45-807c-46ea-8553-42b8b278d1f5 None] DVR: Failed to obtain local DVR Mac address

But strangely I still see the distributed routing flag as True in the report messages:

2014-09-23 13:34:25.842 10102 DEBUG neutron.common.rpc [-] neutron.agent.rpc.PluginReportStateAPI method cast called with arguments (<neutron.context.ContextBase object at 0x7f7fa0299d50>, {'args': {'agent_state': {'agent_state': {'binary': 'neutron-openvswitch-agent', 'topic': 'N/A', 'host': 'overcloud-controllermgmt0-nszfnfjwm4fy', 'agent_type': 'Open vSwitch agent', 'configurations': {'arp_responder_enabled': False, 'tunneling_ip': '192.0.2.29', 'devices': 0, 'l2_population': True, 'tunnel_types': ['vxlan'], 'enable_distributed_routing': True, 'bridge_mappings': {}}}}, 'time': '2014-09-23T13:34:25.842207'}, 'namespace': None, 'method': 'report_state'}) {} wrapper /opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/common/log.py:35

It is probably still the same error, since a restart of the openvswitch agent fixes the problem.

Fix proposed to branch: master
Review: https://review.openstack.org/123911

This seems like a reasonable back port candidate for Juno once the fix lands in Kilo.

Reviewed: https://review.openstack.org/123911
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=51303b5fe4785d0cda76f095c95eb4d746d7d783
Submitter: Jenkins
Branch: master

commit 51303b5fe4785d0cda76f095c95eb4d746d7d783
Author: Brian Haley <email address hidden>
Date: Wed Sep 24 21:45:06 2014 -0400

    Make L2 DVR Agent start successfully without an active neutron server

    If the L2 Agent is started before the neutron controller
    is available, it will fail to obtain its unique DVR MAC
    address, and fall-back to operate in non-DVR mode
    permanently.

    This fix does two things:
    1. Makes the L2 Agent attempt to retry obtaining a DVR MAC
    address up to five times on initialization, which should be
    enough time for RPC to be successful. On failure, it will
    fall back to non-DVR mode, ensuring that basic switching
    continues to be functional.

    2. Correctly obtains the current operating mode of the
    L2 Agent in _report_state(), instead of only reporting
    the configured state. This operating mode is carried
    in 'in_distributed_mode' attribute of agent state, and
    is separate from the existing enable_distributed_routing
    static config that is already sent.

    Change-Id: I5fd9bf4163eafa321c5fca7ffb7901ae289f323b
    Closes-bug: #1364215

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/163349

Change abandoned by Kyle Mestery (<email address hidden>) on branch: stable/juno
Review: https://review.openstack.org/163349
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.