network-manager stops connecting to Wi-Fi after 1 PB transferred on any interface

Bug #1222377 reported by Paul
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
isc-dhcp (Ubuntu)
Confirmed
Medium
Mathieu Trudel-Lapierre

Bug Description

Transfer 1 PB or more over any eth* interface (not tested with lo, impractical to test with wlan*). Use ifconfig to confirm interface byte counters.

Once a counter exceeds 1 PB, network-manager will fail to connect to any Wi-Fi access points, though existing connections will stay up.

A single 10 Gb/s link can transfer 1 PB in two weeks.

Related branches

Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

I fail to see how the counters on eth* would have any effect on the wifi devices. Regardless, tentatively Triaging; I'll take a good look at the NMDevice code to see if something stands out.

Could you paste the output of ifconfig for an affected interface, without the IP addresses (which we don't really need)?

I wonder if the issue couldn't just be that we somehow fail to parse things because of an integer overflow somewhere?

Changed in network-manager (Ubuntu):
importance: Undecided → Medium
status: New → Incomplete
assignee: nobody → Mathieu Trudel-Lapierre (mathieu-tl)
Revision history for this message
Paul (i41bktob-launchpad-net) wrote :

I have not determined if the fault occurs after 10^15 or 2^30 bytes, but the former seems more probable. Perhaps the code does something like read from /sys/class/net/eth0/statistics/{rx,tx}_bytes into a 16 byte buffer, which would fail at 10^15. But a quick look through the network-manager source didn't turn up anything like that.

I'll provide the ifconfig output once the problem happens again.

Revision history for this message
Paul (i41bktob-launchpad-net) wrote :

The bug is in isc-dhcp, file common/discover.c, function next_iface4():

        char buf[256];

This buffer is exceeded when the function performs an fgets() from /proc/net/dev and an interface has a very high (PB or greater) number of bytes, collisions, etc. as per this ifconfig output:

eth0 Link encap:Ethernet HWaddr XX:XX:XX:XX:XX:XX
          UP BROADCAST MULTICAST MTU:1500 Metric:1
          RX packets:1006693089307755 errors:6040167125781120 dropped:2013394768550100 overruns:1006697384275050 frame:5033478331440660
          TX packets:1006697384275050 errors:4026789537100200 dropped:0 overruns:1006697384275050 carrier:2013394768550100
          collisions:5033486921375250 txqueuelen:1000
          RX bytes:1006693089307755 (1.0 PB) TX bytes:1006697384275050 (1.0 PB)

I have raised this with ISC and hopefully they will fix it. If you want to patch it anyway, simply double the buffer size.

Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

Agree. This is consistent with the code that handles this in ifconfig (using buf[512]).

Reassigning to isc-dhcp.

affects: network-manager (Ubuntu) → isc-dhcp (Ubuntu)
Changed in isc-dhcp (Ubuntu):
importance: Medium → Undecided
status: Incomplete → New
status: New → Triaged
importance: Undecided → Medium
status: Triaged → In Progress
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package isc-dhcp - 4.2.4-7ubuntu8

---------------
isc-dhcp (4.2.4-7ubuntu8) saucy; urgency=low

  * debian/patches/large_interface_counters.patch: Allow for very large packet
    counts or bytes transferred without failing connections. (LP: #1222377)
 -- Mathieu Trudel-Lapierre <email address hidden> Tue, 08 Oct 2013 15:15:44 -0400

Changed in isc-dhcp (Ubuntu):
status: In Progress → Fix Released
Revision history for this message
Paul (i41bktob-launchpad-net) wrote :

Bug still exists as fix was incomplete.

Changed in isc-dhcp (Ubuntu):
status: Fix Released → Confirmed
Revision history for this message
Paul (i41bktob-launchpad-net) wrote :

This bug is still present, up to and including Ubuntu 14.04 Trusty. large_interface_counters.patch enlarges the buffer at common/discover.c:420 in begin_iface_scan(), but not the buffer at common/discover.c:493 in next_iface4(), which is where the error occurs (see comment #3).

The following buffers should be enlarged from 256 to 512 bytes:
common/discover.c:493:next_iface4()
common/discover.c:617:next_iface6()

I still haven't heard anything from ISC other than they were going to consider this, so it looks like Debian patching it is the correct solution.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.