test_assert_pings_during_br_int_setup_not_lost fails with oslo_rootwrap.wrapper.NoFilterMatched

Bug #1490051 reported by Assaf Muller
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
neutron
Expired
Undecided
Unassigned
Kilo
Expired
Undecided
Unassigned
Changed in neutron:
assignee: nobody → Ihar Hrachyshka (ihar-hrachyshka)
Revision history for this message
Armando Migliaccio (armando-migliaccio) wrote :

10 hits in 7 days

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/220656

Changed in neutron:
milestone: none → liberty-rc1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.openstack.org/220656
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ed3c317ed9182538747f74713154ad94e9d866db
Submitter: Jenkins
Branch: master

commit ed3c317ed9182538747f74713154ad94e9d866db
Author: Ihar Hrachyshka <email address hidden>
Date: Fri Sep 4 22:21:41 2015 +0200

    Fixed functional test that validates graceful ovs agent restart

    The async_ping function returns a callable that returns True when all ping
    futures are done. Since those futures are running for 10 secs, there was no
    chance that the result of the callable was True.

    The test was bailing out without calling bridge reset even a single time,
    effectively leaving the feature untested in gate.

    Another thing to note is that for some reason the patch fixed oslo rootwrap
    errors in the test when executed locally. Since I still don't understand how
    it's possible that it fixes the issue for me, I mark the bug as related only,
    and will track logstash after it's merged to see whether it applies unknown
    magic to gate jobs too.

    Change-Id: Iaa977abddf1a0c6af7e964f1a5cd545ffb79585a
    Related-Bug: #1490051

Revision history for this message
Armando Migliaccio (armando-migliaccio) wrote :

As of today, only 1 failure (which is genuine). I am ready to close this bug in the next 24 hours.

Changed in neutron:
status: Confirmed → In Progress
Revision history for this message
Armando Migliaccio (armando-migliaccio) wrote :

Optimistically closing this one. The last occurrence is a fals positive. Nice one, Ihar.

Changed in neutron:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/221211

Revision history for this message
Armando Migliaccio (armando-migliaccio) wrote :

darn

Changed in neutron:
status: Fix Committed → In Progress
Revision history for this message
Armando Migliaccio (armando-migliaccio) wrote :

I closed it on purpose to lure out the bug in the open...that was all part of a master plan ;)

Revision history for this message
Ihar Hrachyshka (ihar-hrachyshka) wrote :

Some brain dump, I think I am onto smth.

So we saw another failure today: http://logs.openstack.org/02/216902/4/check/gate-neutron-dsvm-functional/a97df90/console.html

Since we reuse rootwrap daemons per worker, and the error looks like the daemon does not see the functional tests filters, the first thing I checked is which test started the daemon for the worker. It's:

http://logs.openstack.org/02/216902/4/check/gate-neutron-dsvm-functional/a97df90/logs/neutron.tests.functional.agent.linux.test_interface.OVSInterfaceDriverTestCase.test_plug_checks_if_bridge_exists_vsctl_.log.txt.gz

Note that it started at 05:44:47.284.

Now compare it to console log where we call to deploy_rootwrap.sh at 05:44:47.335. So it looks like the test started to execute before tox called to deploy_rootwrap, and the daemon was started before the filters were deployed.

Now, I wonder how tests could start at the time when we still have 'dsvm-functional create: /opt/stack/new/neutron/.tox/dsvm-functional' in console log. The test relies on testscenarios. Could it be an issue?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.openstack.org/221211
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1b67012794932a06ce90976f9759fc588da269b5
Submitter: Jenkins
Branch: master

commit 1b67012794932a06ce90976f9759fc588da269b5
Author: Ihar Hrachyshka <email address hidden>
Date: Tue Sep 8 11:20:10 2015 +0200

    tests: disable process monitor before managers

    Otherwise the monitor may respawn managers later, leaving them running.

    Issue spotted in:
    http://logs.openstack.org/02/216902/4/check/gate-neutron-dsvm-functional/a97df90

    Change-Id: I0e68b06c87b5770756fdf7b9201e1986cc67e07b
    Related-Bug: #1490051

Revision history for this message
Ihar Hrachyshka (ihar-hrachyshka) wrote :

One more interesting thing. When running the test locally, I can reproduce the failure in ~1/10 runs.

Here is smth interesting in the logs:

2015-09-08 15:43:16.304 27343 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'test-09ea1917-8fe1-4c5a-abc8-99b067d8f71c', 'ping', '-c', '10', '-W', '1', '10.220.181.185'] execute_rootwrap_daemon neutron/agent/linux/utils.py:102
2015-09-08 15:43:16.305 27343 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'test-09ea1917-8fe1-4c5a-abc8-99b067d8f71c', 'ping', '-c', '10', '-W', '1', '10.75.125.235'] execute_rootwrap_daemon neutron/agent/linux/utils.py:102
2015-09-08 15:43:16.307 27343 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'test-09ea1917-8fe1-4c5a-abc8-99b067d8f71c', 'ping', '-c', '10', '-W', '1', '10.247.223.213'] execute_rootwrap_daemon neutron/agent/linux/utils.py:102
...
2015-09-08 15:43:25.343 27343 DEBUG neutron.agent.linux.utils [-]
Command: ['ip', 'netns', 'exec', 'test-09ea1917-8fe1-4c5a-abc8-99b067d8f71c', 'ping', '-c', 10, '-W', 1, '10.247.223.213']
Exit code: 0

Seems like first two ping attempts didn't even get into linux.utils?..

Revision history for this message
Ihar Hrachyshka (ihar-hrachyshka) wrote :

Nah, the last comment is not that interesting: the exit code is logged only if command didn't raise, and it raised NoFilterMatched for us. And we have a single trace because f.result() bailed out due to the first exception. So apparently those ping attempts got into rootwrap code.

Revision history for this message
Ihar Hrachyshka (ihar-hrachyshka) wrote :

And no, it does not look like testscenarios since even when I converted the test to a scenario-less one, it fails.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (feature/pecan)

Related fix proposed to branch: feature/pecan
Review: https://review.openstack.org/224334

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Related fix proposed to branch: feature/pecan
Review: https://review.openstack.org/224357

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (feature/pecan)
Download full text (73.6 KiB)

Reviewed: https://review.openstack.org/224357
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=fdc3431ccd219accf6a795079d9b67b8656eed8e
Submitter: Jenkins
Branch: feature/pecan

commit fe236bdaadb949661a0bfb9b62ddbe432b4cf5f1
Author: Miguel Angel Ajo <email address hidden>
Date: Thu Sep 3 15:40:12 2015 +0200

    No network devices on network attached qos policies

    Network devices, like internal router legs, or dhcp ports
    should not be affected by bandwidth limiting rules.

    This patch disables application of network attached policies
    to network/neutron owned ports.

    Closes-bug: #1486039
    DocImpact

    Change-Id: I75d80227f1e6c4b3f5fa7762b8dc3b0c0f1abd46

commit db4a06f7caa20a4c7879b58b20e95b223ed8eeaf
Author: Ken'ichi Ohmichi <email address hidden>
Date: Wed Sep 16 10:04:32 2015 +0000

    Use tempest-lib's token_client

    Now tempest-lib provides token_client modules as library and the
    interface is stable. So neutron repogitory doesn't need to contain
    these modules.
    This patch makes neutron use tempest-lib's token_client and removes
    the own modules for the maintenance.

    Change-Id: Ieff7eb003f6e8257d83368dbc80e332aa66a156c

commit 78aed58edbe6eb8a71339c7add491fe9de9a0546
Author: Jakub Libosvar <email address hidden>
Date: Thu Aug 13 09:08:20 2015 +0000

    Fix establishing UDP connection

    Previously, in establish_connection() for UDP protocol data were sent
    but never read on peer socket. That lead to successful read on peer side
    if this connection was filtered. Having constant testing string masked
    this issue as we can't distinguish to which test of connectivity data
    belong.

    This patch makes unique data string per test_connectivity() and
    also makes establish_connection() to create an ASSURED entry in
    conntrack table. Finally, in last test after firewall filter was
    removed, connection is re-established in order to avoid troubles with
    terminated processes or TCP continuing sending packets which weren't
    successfully delivered.

    Closes-Bug: 1478847
    Change-Id: I2920d587d8df8d96dc1c752c28f48ba495f3cf0f

commit e6292fcdd6262434a7b713ad8802db6bc8a6d3dc
Author: YAMAMOTO Takashi <email address hidden>
Date: Wed Sep 16 13:20:51 2015 +0900

    ovsdb: Fix a few docstring

    Change-Id: I53e1e21655b28fe5da60e58aeeb7cbbd103ae014

commit c22949a4449d96a67caa616290cf76b67b182917
Author: fumihiko kakuma <email address hidden>
Date: Wed Sep 16 11:52:59 2015 +0900

    Remove requirements.txt for the ofagent mechanism driver

    It is no longer used.

    Related-Blueprint: core-vendor-decomposition
    https://blueprints.launchpad.net/neutron/+spec/core-vendor-decomposition

    Change-Id: Ib31fb3febf8968e50d86dd66e1e6e1ea2313f8ac

commit d1d4de19d85f961d388c91e70f31b3bafec418c5
Author: Kevin Benton <email address hidden>
Date: Thu Sep 3 20:25:57 2015 -0700

    Always return iterables in L3 get_candidates

    The caller of this function expects iterables.

    Closes-Bug: #1494996
    Change-Id: I3d103e63f4e127a77268502415c0ddb0d804b54a

commit 1ad6ac448067306...

tags: added: in-feature-pecan
Revision history for this message
Ihar Hrachyshka (ihar-hrachyshka) wrote :

No patch up for review, moving to M.

Changed in neutron:
milestone: liberty-rc1 → mitaka-1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (feature/pecan)

Change abandoned by Doug Wiegley (<email address hidden>) on branch: feature/pecan
Review: https://review.openstack.org/224334

Changed in neutron:
milestone: mitaka-1 → mitaka-2
Revision history for this message
Armando Migliaccio (armando-migliaccio) wrote :
Changed in neutron:
milestone: mitaka-2 → mitaka-3
Changed in neutron:
milestone: mitaka-3 → mitaka-rc1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/kilo)

Related fix proposed to branch: stable/kilo
Review: https://review.openstack.org/291774

Revision history for this message
Armando Migliaccio (armando-migliaccio) wrote :

0 hits in 7 days

Changed in neutron:
status: In Progress → Incomplete
assignee: Ihar Hrachyshka (ihar-hrachyshka) → nobody
milestone: mitaka-rc1 → none
Revision history for this message
Armando Migliaccio (armando-migliaccio) wrote :

@Ihar: any update on this one?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/kilo)

Reviewed: https://review.openstack.org/291774
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=40ae0d302d35814d482352527d4ffe11a7e685d5
Submitter: Jenkins
Branch: stable/kilo

commit 40ae0d302d35814d482352527d4ffe11a7e685d5
Author: Ihar Hrachyshka <email address hidden>
Date: Tue Sep 8 11:20:10 2015 +0200

    tests: disable process monitor before managers

    Otherwise the monitor may respawn managers later, leaving them running.

    Issue spotted in:
    http://logs.openstack.org/02/216902/4/check/gate-neutron-dsvm-functional/a97df90

    Change-Id: I0e68b06c87b5770756fdf7b9201e1986cc67e07b
    Related-Bug: #1490051
    (cherry picked from commit 1b67012794932a06ce90976f9759fc588da269b5)

tags: added: in-stable-kilo
Revision history for this message
Ihar Hrachyshka (ihar-hrachyshka) wrote :

We haven't seen any failures like that for a while. I guess we are safe to expire the bug.

Changed in neutron:
importance: High → Undecided
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for neutron because there has been no activity for 60 days.]

Changed in neutron:
status: Incomplete → Expired
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for neutron kilo because there has been no activity for 60 days.]

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.