ISST-LTE: no network after install due to interface order change

Bug #1537136 reported by bugproxy
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
debian-installer (Ubuntu)
Trusty
Fix Released
High
Martin Pitt
systemd (Ubuntu)
Trusty
Fix Released
High
Martin Pitt

Bug Description

== Comment: #0 - Chanh H. Nguyen <email address hidden> - 2015-12-10 20:32:29 ==
We use the network to install Ubuntu 14.04.4 version. The installation is successful but then there is no network after system reboot in post install. Checking the network and I saw the order of interface are changed. This is causing no network at all.

There is 2 cards (Houston 4ports & Mellanox 2 ports). We are using the Mellanox interface for this installation.
At the installation menu, we select this interface eth2 (mellanox card)

??????????????????????? [!!] Configure the network ????????????????????????
  ? ?
  ? Your system has multiple network interfaces. Choose the one to use as ?
  ? the primary network interface during the installation. If possible, ?
  ? the first connected network interface found has been selected. ?
  ? ?
  ? Primary network interface: ?
  ? ?
  ? eth0: Emulex Corporation OneConnect NIC (Lancer) ?
  ? eth1: Emulex Corporation OneConnect NIC (Lancer) ?
  ? eth2: Mellanox Technologies MT27500 Family [ConnectX-3] ?
  ? eth3: Mellanox Technologies MT27500 Family [ConnectX-3] ?
  ? eth4: Emulex Corporation OneConnect NIC (Lancer) ?
  ? eth5: Emulex Corporation OneConnect NIC (Lancer) ?
  ? ?
  ? <Go Back> ?
  ? ?
  ???????????????????????????????????????????????????????????????????????????

After install,
root@monklp3:~# ifconfig
eth2 Link encap:Ethernet HWaddr 00:00:c9:d1:be:28
          UP BROADCAST MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:65536 Metric:1
          RX packets:32 errors:0 dropped:0 overruns:0 frame:0
          TX packets:32 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:2368 (2.3 KB) TX bytes:2368 (2.3 KB)
root@monklp3:~# ifconfig -a|grep eth
eth0 Link encap:Ethernet HWaddr 00:00:c9:d1:be:26
eth1 Link encap:Ethernet HWaddr 00:00:c9:d1:be:27
eth2 Link encap:Ethernet HWaddr 00:00:c9:d1:be:28
eth3 Link encap:Ethernet HWaddr 00:02:c9:b7:60:b0
eth4 Link encap:Ethernet HWaddr 00:02:c9:b7:60:b1
eth5 Link encap:Ethernet HWaddr 00:00:c9:d1:be:29
root@monklp3:~# cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth2
iface eth2 inet dhcp
root@monklp3:~# dmesg |grep "eth"
[ 0.950725] ibmveth: IBM Power Virtual Ethernet Driver 1.05
[ 0.950735] ehea: IBM eHEA ethernet device driver (Release EHEA_0107)
[ 9.810472] mlx4_en: eth3: Link Up
[ 40.554271] be2net 0001:80:00.2 eth2: Link is Down
[ 40.560738] IPv6: ADDRCONF(NETDEV_UP): eth2: link is not ready
root@monklp3:~#

The mellanox inteface is now on eth3 but on the /etc/network/interface file, it is set on eth2.

SRU TEST CASE:
--------------
 * Do a netboot install in a VM or machine with more than one NIC. If you use a VM, take care to not use the default MAC addresses as they are blacklisted from the persistent net generator. E. g. use "-net nic,model=virtio,macaddr=A4:4E:31:12:34:XX" with QEMU.
 * In the d-i environment, switch to a console with Alt+F2 after the base system got installed.
 * With current trusty, there is no /etc/udev/rules.d/70-persistent-net.rules and thus it also does not exist in /target/.
 * With the proposed update, /etc/udev/rules.d/70-persistent-net.rules should exist and be copied to /target/etc/udev/rules.d/ as well.

Revision history for this message
bugproxy (bugproxy) wrote : /etc/udev/rules.d/70-persistent-net.rules file

Default Comment by Bridge

tags: added: architecture-ppc64le bugnameltc-134000 severity-high targetmilestone-inin14044
Changed in ubuntu:
assignee: nobody → Taco Screen team (taco-screen-team)
Kevin W. Rudd (kevinr)
affects: ubuntu → debian-installer (Ubuntu)
Steve Langasek (vorlon)
Changed in debian-installer (Ubuntu):
assignee: Taco Screen team (taco-screen-team) → Martin Pitt (pitti)
importance: Undecided → High
status: New → Triaged
Revision history for this message
Martin Pitt (pitti) wrote :

This has nothing to do with bug 1473542, as in 15.04 the network interface naming was completely changed (see https://lists.ubuntu.com/archives/ubuntu-devel/2015-May/038761.html).

Is the attached 70-persistent-net.rules right or wrong? I. e. is the problem that this assigns the wrong name to eth2, or that the file is right but it is not actually applied correctly during boot? Or, in other words, which MAC address do you actually *want* to see as eth2?

As far as I can see, the installer gives both cards (eth2 and eth3 as they were named at install time) the exact same name:

  ? eth2: Mellanox Technologies MT27500 Family [ConnectX-3] ?
  ? eth3: Mellanox Technologies MT27500 Family [ConnectX-3] ?

So it's not immediately clear which one you want. However, it should of course create a consistent 70-persistent-net.rules file and /etc/network/interfaces either way.

Are you sure that eth2 and eth3 moved around? They have completely different MAC addresses, while eth0 to eth2+eth5 and eth3/eth4 seem to be related to each other (ascending MACs), i. e. eth0/1/2/5 are the Emulex ones while eth3/4 are the Mellanox ones.

Revision history for this message
Martin Pitt (pitti) wrote :

How did you install this machine? Ubuntu-Server image or netboot? Can you please point to the installation steps for netboot (sorry, I'm not familiar with it), did you use https://help.ubuntu.com/community/Installation/Netboot or something else?

I'd like to reproduce this using a VM with multiple network cards and checking how the installer's persistent network names propagate to the installed system.

Revision history for this message
Martin Pitt (pitti) wrote :

In case you did use netboot: I tried this with a VM with two eth cards, and I see that the d-i environment does not have /lib/udev/rules.d/75-persistent-net-generator.rules; it only has biosdevname, and as that does not work with most non-Dell servers it's a no-op.

So udev-udeb does not ship the generator (for 14.04, this isn't relevant for 15.04 and up), but also there is nothing in d-i which would copy a generated 70-persistent-net.rules from the installer environment to /target/etc/udev/rules.d/. It seems to me that we need to do this both.

Revision history for this message
Martin Pitt (pitti) wrote :

Just note keeping for testing:

cjwatson | pitti: if you need more complicated initrd changes, then you can grab the debian-installer source, drop updated udebs into build/localudebs/, and run "make -C build rebuild_netboot"

Revision history for this message
Martin Pitt (pitti) wrote :

Adjusting bug tasks. This isn't an issue in wily and higher as we use ifnames there, which don't need this 70-persistent-net.rules file any more.

> there is nothing in d-i which would copy a generated 70-persistent-net.rules from the installer environment

I overlooked udev-udeb's base-installer.d/05udev script, which does that. So we should only need to ship the generator in udev-udeb, and then rebuild d-i against that.

Changed in systemd (Ubuntu):
status: New → In Progress
assignee: nobody → Martin Pitt (pitti)
Changed in debian-installer (Ubuntu):
status: Triaged → Invalid
importance: High → Undecided
assignee: Martin Pitt (pitti) → nobody
Changed in debian-installer (Ubuntu Trusty):
status: New → Triaged
assignee: nobody → Martin Pitt (pitti)
importance: Undecided → High
Changed in systemd (Ubuntu):
status: In Progress → Invalid
Changed in systemd (Ubuntu Trusty):
status: New → In Progress
assignee: nobody → Martin Pitt (pitti)
Changed in systemd (Ubuntu):
assignee: Martin Pitt (pitti) → nobody
Revision history for this message
Martin Pitt (pitti) wrote :

This is the systemd debdiff. With a local build of that, and a local rebuild of d-i, and PXE-booting the resulting netboot.tar.gz I now get a 70-persistent-net.rules copied to /target/etc/udev/rules.d/.

I can't upload this yet as bug 1371403 is the current SRU in trusty-proposed. This either needs to get verified or removed from -proposed.

Revision history for this message
Steve Langasek (vorlon) wrote :

> I can't upload this yet as bug 1371403 is the current SRU in
> trusty-proposed. This either needs to get verified or removed
> from -proposed.

The submitter has declared the SRU is "stuck", so I think it's clear the previous SRU should be removed from the queue to make room for others (such as this one).

Revision history for this message
Martin Pitt (pitti) wrote : Please test proposed package

Hello bugproxy, or anyone else affected,

Accepted systemd into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/204-5ubuntu20.18 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in systemd (Ubuntu Trusty):
status: In Progress → Fix Committed
tags: added: verification-needed
Revision history for this message
Martin Pitt (pitti) wrote :

Note: This still needs a corresponding debian-installer upload, so you can't test this just yet.

description: updated
Revision history for this message
Martin Pitt (pitti) wrote :

Hello bugproxy, or anyone else affected,

Accepted debian-installer into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/debian-installer/20101020ubuntu318.34 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in debian-installer (Ubuntu Trusty):
status: Triaged → Fix Committed
Martin Pitt (pitti)
description: updated
Revision history for this message
Martin Pitt (pitti) wrote :

I re-verified netboot with http://archive.ubuntu.com/ubuntu/dists/trusty-proposed/main/installer-amd64/current/images/netboot/netboot.tar.gz (that's the output from the above debian-installer rebuild) and confirm that I now get an /etc/udev/rules.d/70-persistent-net.rules which gets copied into the installation.

However, I don't want to self-verify this. So can you please test the trusty-proposed ppc64el netboot on your actual system? http://ports.ubuntu.com/dists/trusty-proposed/main/installer-ppc64el/current/images/netboot/netboot.tar.gz

Thanks!

Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2016-01-27 23:03 EDT-------
(In reply to comment #34)
> Our system is hitting another bug. We will verify this soon after we get
> some news on other bug. Thanks

bugproxy (bugproxy)
tags: added: verification-done
removed: verification-needed
Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2016-02-02 15:11 EDT-------
We do not want to change our whole installation environment for 14.04.4 so I try to use "apt-setup/proposed=true" at the boot command line. Is that enough ?
I still see this error after finishes install on new kernel -27.

Revision history for this message
Martin Pitt (pitti) wrote :

I haven't tested that, but that should work. I downloaded the netboot tarball anyway for putting into tftp, I don't know how to set up an indirection that would make TFTP always download/serve the latest tarball.

tags: added: verification-needed
removed: verification-done
Mathew Hodson (mhodson)
Changed in systemd (Ubuntu Trusty):
importance: Undecided → High
Mathew Hodson (mhodson)
no longer affects: debian-installer (Ubuntu)
no longer affects: systemd (Ubuntu)
Revision history for this message
Timo Aaltonen (tjaalton) wrote :

What's the status now? There's another upload that would need to get in proposed soon, but this is blocking it.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package debian-installer - 20101020ubuntu318.34

---------------
debian-installer (20101020ubuntu318.34) trusty; urgency=medium

  * No-change upload to pick up udev-udeb with persistent net generator fix.
    (LP: #1537136)

 -- Martin Pitt <email address hidden> Tue, 26 Jan 2016 11:26:33 +0100

Changed in debian-installer (Ubuntu Trusty):
status: Fix Committed → Fix Released
Revision history for this message
Timo Aaltonen (tjaalton) wrote :

sorry, I then realized that the d-i change was a simple rebuild.. so let that through now

Revision history for this message
Martin Pitt (pitti) wrote :

As the systemd update provides the changed udev-udeb that d-i picked up, I'm releasing systemd along, to avoid trusty-updates getting out of sync.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package systemd - 204-5ubuntu20.18

---------------
systemd (204-5ubuntu20.18) trusty-proposed; urgency=medium

  * debian/udev-udeb.install: Ship persistent net generator rules, so that the
    naming from the d-i environment gets copied into the installed system (via
    base-installer.d/05udev d-i hook, which already is shipped). With that a
    d-i generated /etc/network/interfaces will still be valid in the installed
    system. (LP: #1537136)

 -- Martin Pitt <email address hidden> Mon, 25 Jan 2016 15:03:51 +0100

Changed in systemd (Ubuntu Trusty):
status: Fix Committed → Fix Released
Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2016-02-15 16:34 EDT-------
Moving to verfication-done based on the previous comment:

(In reply to comment #39)
> We download the netboot tarball and set it up on our env. The network stays
> after install and it is working fine on the dedicate processor mode.
>
> The order at the installation menu:
> ?????????????????????
>
> Primary network interface: ?
> ? ?
> ? eth0: Emulex Corporation OneConnect NIC (Lancer) ?
> ? eth1: Mellanox Technologies MT27500 Family [ConnectX-3] ?
> ? eth2: Mellanox Technologies MT27500 Family [ConnectX-3] ?
> ? eth3: Emulex Corporation OneConnect NIC (Lancer) ?
> ? eth4: Emulex Corporation OneConnect NIC (Lancer) ?
> ? eth5: Emulex Corporation OneConnect NIC (Lancer) ?
> ? ?
> ? <Go Back>
>
> After install, the order stays the same...
> root@monklp3:~# ifconfig -a|grep eth
> eth0 Link encap:Ethernet HWaddr 00:00:c9:d1:be:26
> eth1 Link encap:Ethernet HWaddr 00:02:c9:b7:60:b0
> eth2 Link encap:Ethernet HWaddr 00:02:c9:b7:60:b1
> eth3 Link encap:Ethernet HWaddr 00:00:c9:d1:be:27
> eth4 Link encap:Ethernet HWaddr 00:00:c9:d1:be:28
> eth5 Link encap:Ethernet HWaddr 00:00:c9:d1:be:29
>

tags: added: verification-done
removed: verification-needed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.