Logs full with "ICMPv6 RA: ndisc_router_discovery() failed to add default route"

Bug #988183 reported by Sedat Dilek
58
This bug affects 10 people
Affects Status Importance Assigned to Milestone
NetworkManager
Fix Released
Medium
network-manager (Fedora)
Invalid
Medium
network-manager (Ubuntu)
Fix Released
High
Mathieu Trudel-Lapierre
Precise
Fix Released
High
Mathieu Trudel-Lapierre

Bug Description

[Impact]
Minor impact, this should only be affecting users of the kernel mainline packages since we're not shipping kernel 3.3.0 where the issue is presumed to occur, take care of letting these users also benefit from stable connections since they may need these kernels for new hardware support (and kernels that may be backported in the future). This only affects IPv6-enabled network where the kernel expects to setup a default route by itself and doesn't play nice with something else modifying the routes.

[Development Fix]
Fixed upstream, as http://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=77de91e5a8b1c1993ae65c54b37e0411e78e6fe6. See https://bugzilla.gnome.org/show_bug.cgi?id=673476 (upstream bug). This change only affects NetworkManager's code, doesn't require rebuild of any dependant package.

[Stable Fix]
See above "Development fix".

[Test Case]
1) Connect to an IPv6 network with a kernel 3.3.0 or higher.
2) Observe behavior in 'ip route -6' and messages in /var/log/syslog. Wireless connections should remain stable.

[Regression Potential]
Only IPv6 could be affected, no potential for regression expected for IPv4 routing or other parts of NetworkManager. Routing may be incorrect, resulting in the wrong device taking precedence for data being sent over the network when IPv6 is in use; IPv6 connections (or connections altogether, if IPv6 is set to "Required" (which is not by default)) may be torn down or unavailable due to invalid routes.

---------

Hi,

I am observing a lot of "ICMPv6 RA: ndisc_router_discovery() failed to add default route" in my logs.
After a fresh reboot:

$ dmesg | grep "ICMPv6 RA: ndisc_router_discovery() failed to add default route." | wc -l
46

First, I thought this is a kernel issue, but than I found a bug-report in the Red Hat BTS in [1].

Just FYI: I am on Linux-3.4-rc4 - with the original Ubuntu/precise kernel I haven't seen this which is linux-image-3.2.0-23-generic (3.2.0-23.36).

- Sedat -

[1] https://bugzilla.redhat.com/show_bug.cgi?id=785772

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: network-manager 0.9.4.0-0ubuntu3
Uname: Linux 3.4.0-rc4-4-generic x86_64
ApportVersion: 2.0.1-0ubuntu5
Architecture: amd64
Date: Wed Apr 25 08:40:50 2012
ExecutablePath: /usr/sbin/NetworkManager
InstallationMedia: Ubuntu 12.04 LTS "Precise Pangolin" - Beta amd64 (20120301)
IpRoute:
 default via 192.168.178.1 dev wlan0 proto static
 169.254.0.0/16 dev wlan0 scope link metric 1000
 192.168.178.0/24 dev wlan0 proto kernel scope link src 192.168.178.22 metric 2
NetworkManager.state:
 [main]
 NetworkingEnabled=true
 WirelessEnabled=true
 WWANEnabled=true
 WimaxEnabled=true
ProcEnviron:
 TERM=linux
 PATH=(custom, no user)
 LANG=de_DE.UTF-8
SourcePackage: network-manager
UpgradeStatus: No upgrade log present (probably fresh install)
nmcli-con:
 NAME UUID TYPE TIMESTAMP TIMESTAMP-REAL AUTOCONNECT READONLY DBUS-PATH
 myCastle-WLAN WPA (Wireless LAN) a74880ae-a336-4133-9454-b0376557cb6c 802-11-wireless 1335335775 Mi 25 Apr 2012 08:36:15 CEST yes no /org/freedesktop/NetworkManager/Settings/0
nmcli-dev:
 DEVICE TYPE STATE DBUS-PATH
 eth0 802-3-ethernet unmanaged /org/freedesktop/NetworkManager/Devices/1
 wlan0 802-11-wireless connected /org/freedesktop/NetworkManager/Devices/0
nmcli-nm:
 RUNNING VERSION STATE NET-ENABLED WIFI-HARDWARE WIFI WWAN-HARDWARE WWAN
 running 0.9.4.0 connected enabled enabled enabled enabled enabled

Revision history for this message
In , Yann (yann-redhat-bugs) wrote :

Created attachment 572767
/var/log/messages

It seems that, with the latest kernel, my network connection (Ethernet 100BaseT) is unstable:

The connection is "disconnected" every 10 minutes interval according to NetworkManager (with a notification on screen).

I'm using
    kernel-3.3.0-4.fc16.x86_64
    NetworkManager-0.9.2-1.fc16.x86_64.

My network interface is a
    03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 02)

My computer is connected to a FreeBox v5 ADSL router, which provides "native" IPv6 broadband access.

Revision history for this message
In , Yann (yann-redhat-bugs) wrote :

Created attachment 572778
log of a a tcpdump ip6 session

Revision history for this message
In , Neil (neil-redhat-bugs) wrote :

please see bz 785772. Upgrading to the latest NM in rawhide fixes this issue.

Revision history for this message
In , Yann (yann-redhat-bugs) wrote :

I'm using Fedora 16. Installing NetworkManager from Rawhide require too much dependencies to be installed alone safely.

Is it possible to have a test update package ?

Revision history for this message
In , Neil (neil-redhat-bugs) wrote :

http://koji.fedoraproject.org/koji/taskinfo?taskID=3936339

There you go, thats the rawhide NetworkManager built for f16.

If you don't want to install that, you can also set your NM IPv6 configuration for all your interfaces to 'ignore'. That should also fix the problem.

Revision history for this message
In , Yann (yann-redhat-bugs) wrote :

(In reply to comment #4)
> http://koji.fedoraproject.org/koji/taskinfo?taskID=3936339
>
> There you go, thats the rawhide NetworkManager built for f16.
>

Thanks a lot for the packages.

I tried to install them but updated NetworkManager-gnome is missing.

Revision history for this message
In , Yann (yann-redhat-bugs) wrote :

Created attachment 573150
NetworkManager install log

(In reply to comment #5)
>
> I tried to install them but updated NetworkManager-gnome is missing.

Here's the log if needed.

Revision history for this message
In , Neil (neil-redhat-bugs) wrote :

Thanks, not sure whats going on there. I'll have to figure that out. The spec file isn't explicitly Required:-ing that package.

While I figure it out, please just set the interface settings for the interface in questions for IPv6 to Ignore or Disable, that will provide an equally good solution.

Revision history for this message
In , Jirka (jirka-redhat-bugs) wrote :

(In reply to comment #5)
> (In reply to comment #4)
> > http://koji.fedoraproject.org/koji/taskinfo?taskID=3936339
> >
> > There you go, thats the rawhide NetworkManager built for f16.
> >
>
> Thanks a lot for the packages.
>
> I tried to install them but updated NetworkManager-gnome is missing.

Please try F16 update here (it's not in stable at the time of this posting)
http://koji.fedoraproject.org/koji/buildinfo?buildID=310135

Revision history for this message
In , Nicolas (nicolas-redhat-bugs) wrote :

I also have the same problem, but the updated NetworkManager doesn't seem to help:

# yum info NetworkManager
Installed Packages
Name : NetworkManager
Arch : x86_64
Epoch : 1
Version : 0.9.4
Release : 1.git20120328.fc16
Size : 6.7 M
Repo : installed
From repo : updates-testing

Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 02)

I have a "Freebox v6" ADSL router.

Revision history for this message
In , Yann (yann-redhat-bugs) wrote :

Created attachment 574435
/var/log/messages

After installing the provided packages, I'm still having the problem.

See this in the log.

Additionally, I'm hitting bug #799591 .

Revision history for this message
In , Neil (neil-redhat-bugs) wrote :

Did you all make sure you're configuration for ipv6 in NM hadn't reverted to automatic or dhcp?

Revision history for this message
In , Yann (yann-redhat-bugs) wrote :

(In reply to comment #11)
> Did you all make sure you're configuration for ipv6 in NM hadn't reverted to
> automatic or dhcp?

After installing the provided package here, I've checked that ipv6 configuration is in "automatic" mode. And it's still in "automatic" mode.

Revision history for this message
In , Neil (neil-redhat-bugs) wrote :

no, I'm asking if it wasn't in automatic mode previously. If its in automatic now, you need to move it to ignore or disabled mode. the rawhide NM has fixed this problem for some, but others still need to explicitly disable NM's ipv6 handling if they use SLAAC at the moment.

Revision history for this message
In , Yann (yann-redhat-bugs) wrote :

(In reply to comment #13)
> no, I'm asking if it wasn't in automatic mode previously. If its in automatic
> now, you need to move it to ignore or disabled mode. the rawhide NM has fixed
> this problem for some, but others still need to explicitly disable NM's ipv6
> handling if they use SLAAC at the moment.

If "ignore" or "disable" is still required, the updated NM packages are not fixing the problem as expected.

So I'm going to switch from "automatic" (the current and previous mode) to "ignore" which was proven to workaround the problem.

Revision history for this message
In , Neil (neil-redhat-bugs) wrote :

 As noted above some people have this problem automatically fixed when using the latest NM, others dont. I'm still trying to figure out what the difference between the two is.

Given that this thread seems to contain people in the latter group, I'll reassign this over to NM for a closer look.

Revision history for this message
In , Dan (dan-redhat-bugs) wrote :

*** This bug has been marked as a duplicate of bug 785772 ***

Revision history for this message
Sedat Dilek (dileks) wrote :
Revision history for this message
Sedat Dilek (dileks) wrote :

I found the upstream commit which points to the RH-Bug:

commit 77de91e5a8b1c1993ae65c54b37e0411e78e6fe6
core: don't fight with the kernel over the default IPv6 route

[1] http://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=77de91e5a8b1c1993ae65c54b37e0411e78e6fe6

Revision history for this message
Sedat Dilek (dileks) wrote :

I have rebuilt a new NM package which includes the upstream fix.
It fixes here WLAN (IPv6) dis-/reconnects and log-files aren't spammed anymore.

Thanks RAOF for vital help in Ubuntu's way of packaging on IRC.
Also RAOF encouraged me to add my debdiff to this BR.

Have fun!

Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "debdiff to network-manager-0.9.4.0 (0.9.4.0-0ubuntu3)" of this bug report has been identified as being a patch in the form of a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. In the event that this is in fact not a patch you can resolve this situation by removing the tag 'patch' from the bug report and editing the attachment so that it is not flagged as a patch. Additionally, if you are member of the ubuntu-sponsors team please also unsubscribe the team from this bug report.

[This is an automated message performed by a Launchpad user owned by Brian Murray. Please contact him regarding any issues with the action taken in this bug report.]

tags: added: patch
Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

In Progress/High. It's kind of annoying, if only just for the log entries. All of the connection/disconnection states depend a lot on the RA interval used, but that's still also quite annoying, and the fix is clearly safe (as it's a new route with a different metric, so we don't risk breaking stuff too much).

I'm sponsoring this to -proposed shortly, just need to apply the debdiff, quickly test that it's properly adding the routes, and we should be ok. It will need to be carefully tested to make sure the added metric doesn't clash with wireless/wired coexistence: basically, we need to make sure that on a system connected to both wired and wireless that has ipv6 on both interfaces, the wireless default v6 route added by NM needs to have a lower priority (higher metric) than the default ipv6 route added by NM for the wired interface. This will make sure ethernet remains "preferred" in the event that these networks are in fact different, and I'm a little worried that the upstream patch breaks this.

Changed in network-manager (Ubuntu):
status: New → In Progress
importance: Undecided → High
assignee: nobody → Mathieu Trudel-Lapierre (mathieu-tl)
Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

(I'm already preparing an upload with two other patches to NM)

Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

Unsubscribing sponsors and subscribing ubuntu-sru; but I'll do the steps of counter-verifying the routes set with that patch enabled before uploading to precise-proposed.

description: updated
description: updated
Changed in network-manager:
importance: Unknown → Medium
status: Unknown → Fix Released
Revision history for this message
Luke J Militello (kilahurtz) wrote :

Has this been pushed to the proposed repository yet? Just installed 12.04 and I am also having this problem. I also have said system up to date with the proposed repository and still have this issue.

Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

No, it's not, since this appears to be mostly dependent on a newer kernel version that is not yet being shipped by Ubuntu anyway.

Unless you can confirm you get these messages in dmesg or syslog on the currently shipped kernel version in Precise?

There's additional issues possibly introduced by this patch, the routes appear to be at least slightly incorrect since IIRC wired should always have a lower metric than wireless.

Revision history for this message
Luke J Militello (kilahurtz) wrote :

Yes, I can confirm...see below.

[luke@Hades ~]$ dpkg -l linux-image-*-generic
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Description
+++-======================================-======================================-============================================================================================
ii linux-image-3.2.0-23-generic 3.2.0-23.36 Linux kernel image for version 3.2.0 on 64 bit x86 SMP
ii linux-image-3.2.0-24-generic 3.2.0-24.37 Linux kernel image for version 3.2.0 on 64 bit x86 SMP
[luke@Hades ~]$ uname -a
Linux Hades 3.2.0-24-generic #37-Ubuntu SMP Wed Apr 25 08:43:22 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
[luke@Hades ~]$ dmesg | tail
[186520.943347] ICMPv6 RA: ndisc_router_discovery() failed to add default route.
[186692.340473] ICMPv6 RA: ndisc_router_discovery() failed to add default route.
[186891.648149] ICMPv6 RA: ndisc_router_discovery() failed to add default route.
[187080.672781] ICMPv6 RA: ndisc_router_discovery() failed to add default route.
[187242.565527] ICMPv6 RA: ndisc_router_discovery() failed to add default route.
[187394.152437] ICMPv6 RA: ndisc_router_discovery() failed to add default route.
[187557.177427] ICMPv6 RA: ndisc_router_discovery() failed to add default route.
[187725.112449] ICMPv6 RA: ndisc_router_discovery() failed to add default route.
[187902.836666] ICMPv6 RA: ndisc_router_discovery() failed to add default route.
[188066.652343] ICMPv6 RA: ndisc_router_discovery() failed to add default route.
[luke@Hades ~]$

Revision history for this message
Luke J Militello (kilahurtz) wrote :

Oh, I should point this out, this is a server install -- no GUI. And also is wired and not wireless.

Revision history for this message
Matthew Roy (matthew-royhousehold) wrote :

Bug #982693 is marked as a duplicate of this bug. If that's the case this is happening in the current Precise kernel: 3.2.0-24-generic with the vanilla desktop install.

Revision history for this message
Sedat Dilek (dileks) wrote :

Hmm, network-manager (0.9.4.0-0ubuntu4) seems not to have the (upstream) patch from here.

What's status? Especially wireless + wired (with 3.2.0 precise-kernel)?
( OK, Ubuntu developers are visiting UDS, so... :-) )

Revision history for this message
Sedat Dilek (dileks) wrote :

debdiff against network-manager (0.9.4.0-0ubuntu4) from precise-proposed

Revision history for this message
kolya (mar-kolya) wrote :

Just to note: this problem also affects older versions. I have 11.04 and updated kernel from ubuntu kernel PPA to have better wireless support and btrfs fixes. So with 3.3 and 3.4 kernels I had wifi connection dropped very often. Apparently setting IPV6 to 'ignore' in NM settings for this connection fixes drop outs and IPV6 still works weirdly enough.

Revision history for this message
Sedat Dilek (dileks) wrote :

Thanks, kolya.
See also thread on linux-wireless ML [1] for more details.

[1] http://marc.info/?t=133700428200001&r=1&w=2

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package network-manager - 0.9.4.0-0ubuntu5

---------------
network-manager (0.9.4.0-0ubuntu5) quantal; urgency=low

  * debian/network-manager.upstart: add "and static-network-up" to ensure the
    loopback device is really up before we start dnsmasq. (LP: #993379)
  * debian/patches/git_kernel_ipv6_default_route_77de91e.patch: avoid fighting
    with the kernel for what IPv6 default route should be set: let the kernel
    set his own, then add a new route with a different metric so that we can
    go back and remove it later. (LP: #988183)
  * debian/patches/nm-ip6-rs.patch: avoid disconnections due to RDNSS expiry,
    send a Router Sollicit to try and get new RDNSS data. (LP: #993571)
  * debian/patches/git_remove_ifpppstatsreq_6b64e4d.patch: remove the use of
    the ifpppstatsreq struct, which has been dropped in newer kernels: use
    ifreq and ppp_stats separately instead.
 -- Mathieu Trudel-Lapierre <email address hidden> Wed, 23 May 2012 15:28:36 -0400

Changed in network-manager (Ubuntu):
status: In Progress → Fix Released
Changed in network-manager (Ubuntu Precise):
status: New → In Progress
importance: Undecided → High
assignee: nobody → Mathieu Trudel-Lapierre (mathieu-tl)
Revision history for this message
Luke J Militello (kilahurtz) wrote :

This will not fix a server environment where there is no GUI present and, thus, no network manager. I think this goes deeper into the kernel source itself.

Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

I don't think so. The issue here was caused by the fact that NetworkManager replaces the default route set by the kernel with its own; which is definitely an issue in NetworkManager.

If you're running into similar issues and you're *not* using NM, then please file a separate bug report (and comment back here with the number). I'll try to reproduce it and triage the bug as necessary. However, this means we'll need to figure out what changes the routes; as in what program; so we'll need a list of processes running (ps) and the routing table (ip -6 route). Again, this would be for a separate bug report since it wouldn't be the same issue as we're trying to fix here.

Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

Sedat, if you can, please try the current development release; liveCDs for that release should be getting built and normally should pretty much work -- that will include the new version of NetworkManager with the patch, and a new enough kernel to be able to reproduce the bug.

I'll upload this to precise-proposed as well shortly.

As for older releases, I'm not sure how well this is going to be possible to provide as a stable update: it depends on whether newer kernels are supported on these releases (whether they are officially shipped). For Precise (12.04), it's special since AFAIK there are plans to backport newer kernels eventually. For other releases, I'd have to check.

Revision history for this message
Chris Halse Rogers (raof) wrote : Please test proposed package

Hello Sedat, or anyone else affected,

Accepted network-manager into precise-proposed. The package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Changed in network-manager (Ubuntu Precise):
status: In Progress → Fix Committed
tags: added: verification-needed
Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

Can someone seeing this bug please see comment #21 and follow the steps to verify that the new proposed package fixes the issue?

Thanks!

Revision history for this message
Sedat Dilek (dileks) wrote :

Hi, I have rebuilt 0.9.4.0-0ubuntu5 for quantal on my precise/amd64 system and thus I do not see the errmsgs no more. As usually I tested in an IPv6 WLAN only environment.

Revision history for this message
Stéphane Graber (stgraber) wrote :

Reproduced the issue using the 3.3.7 mainline build from the kernel PPA. Upgrading to the version from -proposed indeed suppresses the errors, connectivity is still working as expected.

tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package network-manager - 0.9.4.0-0ubuntu4.1

---------------
network-manager (0.9.4.0-0ubuntu4.1) precise-proposed; urgency=low

  * debian/network-manager.upstart: add "and static-network-up" to ensure the
    loopback device is really up before we start dnsmasq. (LP: #993379)
  * debian/patches/git_kernel_ipv6_default_route_77de91e.patch: avoid fighting
    with the kernel for what IPv6 default route should be set: let the kernel
    set his own, then add a new route with a different metric so that we can
    go back and remove it later. (LP: #988183)
  * debian/patches/nm-ip6-rs.patch: avoid disconnections due to RDNSS expiry,
    send a Router Sollicit to try and get new RDNSS data. (LP: #993571)
 -- Mathieu Trudel-Lapierre <email address hidden> Thu, 24 May 2012 20:39:12 -0400

Changed in network-manager (Ubuntu Precise):
status: Fix Committed → Fix Released
Changed in network-manager (Fedora):
importance: Unknown → Medium
status: Unknown → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.