icmp_seq=1 Destination unreachable: Address unreachable causing built failures on fortnebula nodes

Bug #1856760 reported by Matt Riedemann
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack-Gate
New
Undecided
Donny Davis

Bug Description

Seen here:

https://zuul.opendev.org/t/openstack/build/79107b8c3bac4fdba4d21a059311e9c3/log/logs/devstack-gate-setup-host.txt#3408

}

2019-12-17 19:07:10.885 | localhost | SUCCESS | network_sanity_check : Get NODEPOOL_MIRROR_HOST >>

mirror.regionone.fortnebula.opendev.org

2019-12-17 19:07:11.086 | 192.168.48.65 | SUCCESS | network_sanity_check : Get NODEPOOL_MIRROR_HOST >>

mirror.regionone.fortnebula.opendev.org

2019-12-17 19:07:11.459 | localhost | SUCCESS | network_sanity_check : Get NODEPOOL_PYPI_MIRROR >>

http://mirror.regionone.fortnebula.opendev.org/pypi/simple

2019-12-17 19:07:11.907 | 192.168.48.65 | SUCCESS | network_sanity_check : Get NODEPOOL_PYPI_MIRROR >>

http://mirror.regionone.fortnebula.opendev.org/pypi/simple

2019-12-17 19:07:13.366 | localhost | FAILED | network_sanity_check : Perform ping check | rc=1 >>

PING mirror.regionone.fortnebula.opendev.org(2001:470:e045:2:f816:3eff:fee6:691d (2001:470:e045:2:f816:3eff:fee6:691d)) 56 data bytes

From 2001:470:e045:8000::1 (2001:470:e045:8000::1) icmp_seq=1 Destination unreachable: Address unreachable

--

Looks like it's mostly fortnebula nodes:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22%7C%20localhost%20%7C%20FAILED%20%7C%20network_sanity_check%20%3A%20Perform%20ping%20check%20%7C%20rc%3D1%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d

The grenade jobs here are using devstack-gate. I don't know if there is something different about how the host network setup ping check works in d-g than the zuulv3 native jobs but maybe something to adjust in d-g.

Revision history for this message
Donny Davis (donny-g) wrote :

This issue on fortnebula was determined to be a bad set of static routes for test node network to mirror node network communications.

Static routes are used for more direct access from tenant to tenant. A system update forced FN to have to recreate the external gateways for each tenant router, but the static routes were never updated to reflect the new path.

Changed in openstack-gate:
assignee: nobody → Donny Davis (donny-g)
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.