ISST-KVM:R3-0:quenv:Ubuntu 14.04.03 installation using Netboot fails on PowerNV machine

SRU Justification:

[Impact]
Users of the 3.19 kernel in Trusty will be unable to netboot if using bnx2x drivers that utilize 7.10.51.0 firmware.

[Test Case]
Try netboot installing on a machine that has a bnx2x driver that uses 7.10.51.0 firmware

[Fix]
Adding 7.10.51.0 firmware to the linux-firmware udeb

--

== Comment: #0 - INDIRA P. JOGA <email address hidden> - 2015-06-25 06:09:52 ==
Classification the type of defect:
------------------------------------------

  DEFECT : Distros

Description: I tried to install ubuntu build 14.04.03 on powernv machine, quenv.isst.aus.stglabs.ibm.com via netboot it is failing with network configuration error.

> I have tried with 'configure network manually' option , that also fails.

> Tried ubuntu 14.04 installation as well, it also fails with network configuration.

> Tried ubuntu 15.10 and it installed successfully.

root@quenv:~# cat /etc/issue
Ubuntu Wily Werewolf (development branch) \n \l

FYI
---

Error seen is

   ?????????????????????? [!!] Configure the network ???????????????????????
   ?
   ? Network autoconfiguration failed ?
   ? Your network is probably not using the DHCP protocol. Alternatively, ?
   ? the DHCP server may be slow or some network hardware is not working
     properly.

                                 <Continue>

   ?????????????????????????????????????????????????????????????????????

RECENT SYSTEM CHANGES : none
-----------------------------------------
  none

== Comment: #1 - Richard M. Scheller <email address hidden> - 2015-06-25 11:10:13 ==
The network interface is using the bnx2x driver:

root@quenv:~# lspci |grep -i broadcom
0003:0e:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57800 1/10 Gigabit Ethernet (rev 10)
0003:0e:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM57800 1/10 Gigabit Ethernet (rev 10)
0003:0e:00.2 Ethernet controller: Broadcom Corporation NetXtreme II BCM57800 1/10 Gigabit Ethernet (rev 10)
0003:0e:00.3 Ethernet controller: Broadcom Corporation NetXtreme II BCM57800 1/10 Gigabit Ethernet (rev 10)

Since Ubuntu 15.10 is working, but 14.04.3 is not, can someone check to see if there are some driver fixes that need to be backported?

== Comment: #2 - David Heller <email address hidden> - 2015-06-25 19:18:01 ==
You are probably correct: there may be some patches that need to be backported. But it's not clear whether it's a bnx2x driver issue just yet. I say that because of recent bug 122410 and bug 122009 which found some things lacking in the "linux-firmware" package. Can you please check your logs to see if you have the same error messages seen there, regarding cannot load firmware?

At the point the install fails you should be able to "go back" and then "execute a shell", and then you can tail your /var/log/syslog and see if you see those messages, or whatever the issue is you should see some evidence there. You also should be able to run "lsmod", "modinfo bnx2x" and "ip link" from the installer shell so please collect that as well.

If you can get the system booted into 14.04,3, perhaps by installing over a different NIC, you could check these things and the linux-firmware package level also. When I logged into your node it was already booted into 15.10, so I went ahead and collected the info there. If you could do the same for 14.04,3 we can compare. With that, and the install log information, we should be able to see what the problem is. Thx.

== Comment: #3 - David Heller <email address hidden> - 2015-06-25 19:21:44 ==
The bnx2x driver and linux-firmware versions for Wily (working case):

root@quenv:~# uname -a
Linux quenv 3.19.0-22-generic #22-Ubuntu SMP Tue Jun 16 17:15:17 UTC 2015 ppc64le ppc64le ppc64le GNU/Linux

root@quenv:~# cat /etc/os-release
NAME="Ubuntu"
VERSION="15.10 (Wily Werewolf)"
...

root@quenv:~# lsmod | grep bnx2x
bnx2x 891548 0
mdio 5849 2 bnx2x,cxgb3
libcrc32c 1750 2 bnx2x,iw_nes

root@quenv:~# modinfo bnx2x
filename: /lib/modules/3.19.0-22-generic/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko
firmware: bnx2x/bnx2x-e2-7.10.51.0.fw
firmware: bnx2x/bnx2x-e1h-7.10.51.0.fw
firmware: bnx2x/bnx2x-e1-7.10.51.0.fw
version: 1.710.51-0
license: GPL
description: Broadcom NetXtreme II BCM57710/57711/57711E/57712/57712_MF/57800/57800_MF/57810/57810_MF/57840/57840_MF Driver
author: Eliezer Tamir
srcversion: 889DCADC858F9813A94EFE5
...

root@quenv:~# ip -o link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default \ link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enP3p14s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq portid 98be94582034 state UP mode DEFAULT group default qlen 1000\ link/ether 98:be:94:58:20:34 brd ff:ff:ff:ff:ff:ff
3: enP3p14s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop portid 98be94582035 state DOWN mode DEFAULT group default qlen 1000\ link/ether 98:be:94:58:20:35 brd ff:ff:ff:ff:ff:ff
4: enP3p14s0f2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop portid 98be94582036 state DOWN mode DEFAULT group default qlen 1000\ link/ether 98:be:94:58:20:36 brd ff:ff:ff:ff:ff:ff
5: enP3p14s0f3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop portid 98be94582037 state DOWN mode DEFAULT group default qlen 1000\ link/ether 98:be:94:58:20:37 brd ff:ff:ff:ff:ff:ff

root@quenv:~# ethtool -i enP3p14s0f0
driver: bnx2x
version: 1.710.51-0
firmware-version: bc 7.10.4
bus-info: 0003:0e:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

root@quenv:~# aptitude versions linux-firmware
Package linux-firmware:
i A 1.144 wily 500
Package linux-firmware-nonfree:
p 1.16 wily 500

root@quenv:~# aptitude changelog linux-firmware
Get: Changelog of linux-firmware
linux-firmware (1.144) wily; urgency=medium

  * Rebase against git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git 8e181320b11de501046cf75c8c30915cd09a1e39
    Add v1.4.0 firmware for ath9k_htc.
    ath9k_htc: update versions in WHENCE
    firmware: tegra: Add XHCI firmware v45.46 for T124
    iwlwifi: update -10.ucode for 3160 / 7260 / 7265 / 7265D
    Revert these to ath9k-htc 1.3.1 - to keep backwards compatibility with older kernels.
    ath9k_htc: revert versions in WHENCE to 1.3.1
    Add v1.4.0 firmware for ath9k_htc.
    ath10k: add firmware 10.2.4.45 as firmware-4.bin
    radeon: Update Kaveri MEC firmware to #396
    brcm: add firmware for BCM43242 USB device
    brcm: add firmware for BCM4356 PCIe device
    brcm: add firmware files for BCM43602 PCIe device
    brcm: add firmware for BCM43455 SDIO device
    brcm: add firmware for BCM4339 SDIO device
    brcm: add firmware for BCM43340/1 SDIO devices
    brcm: upgrade firmware for BCM4354 SDIO device
    cxgb4: update firmware to revision 1.13.32.0 for T4 and T5
    firmware: tegra: Add XHCI firmware v50.07 for T210
    firmware: WHENCE: note nvidia xusb.bin firmware versions
    ath3k: update AR3012 2.2.1 firmware with v.170
    iwlwifi: update -12.ucode for 3160 / 7260 / 7265 / 7265D
    brcm: add firmware for BCM43569 USB device
    brcm: add firmware for BCM43570 PCIe device
    brcm: replace non-working firmware for BCM4339
    brcm: add firmware for BCM4324(1) B5 chipset
    linux-firmware: Update firmware patch for Intel Bluetooth 7265 (C0/D0)
    linux-firmware: Update firmware patch for Intel Bluetooth 7260 (B5/B6)
    linux-firmware: Update firmware patch for Intel Bluetooth 7260 (B3/B4)
    linux-firmware: Fix nvidia/tegra210/xusb.bin permissions
    brcm: add brcmfmac43241b5-sdio.bin to WHENCE
    linux-firmware: Add Realtek Bluetooth HCD firmware
    wl18xx: update firmware file, remove conf file

  [ Seth Forshee ]

  * Remove DH_COMPAT definition from debian/rules
  * Add ${misc:Depends} dependency for binary package
  * Remove explicit gcc build dependency
  * Bump Standards-Version to 3.9.6

 -- Seth Forshee <email address hidden> Fri, 08 May 2015 13:20:27 -0500

linux-firmware (1.143) vivid; urgency=medium

  * Rebase against git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git cef33368c4d3425f11306496f0250f8ef1cf3c1f
    rtl_nic: update firmware for RTL8168H and RTL8107E
    linux-firmware: Fix qat_895xcc.bin permissions
    linux-firmware: Add QLogic BR Series Adapter Firmware
    linux-firmware: Fix firmware files with executable permission set

  [ Leann Ogasawara ]

  * UBUNTU: Add bnx2x firmware to linux-firmware udeb
    -LP: #1424884

 -- Seth Forshee <email address hidden> Wed, 04 Mar 2015 00:16:58 -0600

    ...

== Comment: #4 - INDIRA P. JOGA <email address hidden> - 2015-06-26 02:39:28 ==
HI David,

From /var/log/syslog file seen below messages

 # cat /var/log/syslog | grep firmware
Jun 26 05:43:41 kernel: [ 0.000000] firmware_features = 0x0000000430000000
Jun 26 05:43:41 kernel: [ 1.350067] mlx5_core 0004:01:00.0: firmware version: 10.10.4020
Jun 26 05:43:49 check-missing-firmware: /dev/.udev/firmware-missing does not exist, skipping
Jun 26 05:43:49 check-missing-firmware: /run/udev/firmware-missing does not exist, skipping
Jun 26 05:43:49 check-missing-firmware: no missing firmware in /dev/.udev/firmware-missing /run/udev/firmware-missing
Jun 26 05:43:49 kernel: [ 10.310760] bnx2x 0003:0e:00.0: Direct firmware load for bnx2x/bnx2x-e2-7.10.51.0.fw failed with error -2
Jun 26 05:43:49 kernel: [ 10.310778] bnx2x: [bnx2x_init_firmware:12972(eth0)]Can't load firmware file bnx2x/bnx2x-e2-7.10.51.0.fw
Jun 26 05:43:49 kernel: [ 10.310785] bnx2x: [bnx2x_func_hw_init:5523(eth0)]Error loading firmware
Jun 26 05:43:49 kernel: [ 10.398684] bnx2x 0003:0e:00.1: Direct firmware load for bnx2x/bnx2x-e2-7.10.51.0.fw failed with error -2
Jun 26 05:43:49 kernel: [ 10.398692] bnx2x: [bnx2x_init_firmware:12972(eth1)]Can't load firmware file bnx2x/bnx2x-e2-7.10.51.0.fw
Jun 26 05:43:49 kernel: [ 10.398696] bnx2x: [bnx2x_func_hw_init:5523(eth1)]Error loading firmware
Jun 26 05:43:49 kernel: [ 10.498654] bnx2x 0003:0e:00.2: Direct firmware load for bnx2x/bnx2x-e2-7.10.51.0.fw failed with error -2
Jun 26 05:43:49 kernel: [ 10.498658] bnx2x: [bnx2x_init_firmware:12972(eth2)]Can't load firmware file bnx2x/bnx2x-e2-7.10.51.0.fw
Jun 26 05:43:49 kernel: [ 10.498660] bnx2x: [bnx2x_func_hw_init:5523(eth2)]Error loading firmware
Jun 26 05:43:49 kernel: [ 10.626644] bnx2x 0003:0e:00.3: Direct firmware load for bnx2x/bnx2x-e2-7.10.51.0.fw failed with error -2
Jun 26 05:43:49 kernel: [ 10.626648] bnx2x: [bnx2x_init_firmware:12972(eth3)]Can't load firmware file bnx2x/bnx2x-e2-7.10.51.0.fw
Jun 26 05:43:49 kernel: [ 10.626649] bnx2x: [bnx2x_func_hw_init:5523(eth3)]Error loading firmware
Jun 26 05:43:50 check-missing-firmware: /dev/.udev/firmware-missing does not exist, skipping
Jun 26 05:43:50 check-missing-firmware: /run/udev/firmware-missing does not exist, skipping
Jun 26 05:43:50 check-missing-firmware: no missing firmware in /dev/.udev/firmware-missing /run/udev/firmware-missing
Jun 26 05:43:52 kernel: [ 13.582657] bnx2x 0003:0e:00.0: Direct firmware load for bnx2x/bnx2x-e2-7.10.51.0.fw failed with error -2
Jun 26 05:43:52 kernel: [ 13.582664] bnx2x: [bnx2x_init_firmware:12972(eth0)]Can't load firmware file bnx2x/bnx2x-e2-7.10.51.0.fw
Jun 26 05:43:52 kernel: [ 13.582666] bnx2x: [bnx2x_func_hw_init:5523(eth0)]Error loading firmware
Jun 26 05:43:58 kernel: [ 19.734657] bnx2x 0003:0e:00.0: Direct firmware load for bnx2x/bnx2x-e2-7.10.51.0.fw failed with error -2
Jun 26 05:43:58 kernel: [ 19.734664] bnx2x: [bnx2x_init_firmware:12972(eth0)]Can't load firmware file bnx2x/bnx2x-e2-7.10.51.0.fw
Jun 26 05:43:58 kernel: [ 19.734666] bnx2x: [bnx2x_func_hw_init:5523(eth0)]Error loading firmware
~ #

Collected other info from installer shell:
----------------------------------------------------

~ # lsmod | grep bnx2x
bnx2x 874586 0
libcrc32c 1758 1 bnx2x
mdio 5817 1 bnx2x

~ # modinfo bnx2x
filename: /lib/modules/3.19.0-22-

generic/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko
firmware: bnx2x/bnx2x-e2-7.10.51.0.fw
firmware: bnx2x/bnx2x-e1h-7.10.51.0.fw
firmware: bnx2x/bnx2x-e1-7.10.51.0.fw
version: 1.710.51-0
license: GPL
description: Broadcom NetXtreme II

BCM57710/57711/57711E/57712/57712_MF/57800/57800_MF/57810/57810_MF/57

840/57840_MF Driver
author: Eliezer Tamir
srcversion: 889DCADC858F9813A94EFE5
alias: pci:v000014E4d0000163Fsv*sd*bc*sc*i*
alias: pci:v000014E4d0000163Esv*sd*bc*sc*i*
alias: pci:v000014E4d0000163Dsv*sd*bc*sc*i*
alias: pci:v000014E4d000016ADsv*sd*bc*sc*i*
alias: pci:v000014E4d000016A4sv*sd*bc*sc*i*
alias: pci:v000014E4d000016ABsv*sd*bc*sc*i*
alias: pci:v000014E4d000016AFsv*sd*bc*sc*i*
alias: pci:v000014E4d000016A2sv*sd*bc*sc*i*
alias: pci:v000014E4d000016A1sv*sd*bc*sc*i*
alias: pci:v000014E4d0000168Dsv*sd*bc*sc*i*
alias: pci:v000014E4d000016AEsv*sd*bc*sc*i*
alias: pci:v000014E4d0000168Esv*sd*bc*sc*i*
alias: pci:v000014E4d000016A9sv*sd*bc*sc*i*
alias: pci:v000014E4d000016A5sv*sd*bc*sc*i*
alias: pci:v000014E4d0000168Asv*sd*bc*sc*i*
alias: pci:v000014E4d0000166Fsv*sd*bc*sc*i*
alias: pci:v000014E4d00001663sv*sd*bc*sc*i*
alias: pci:v000014E4d00001662sv*sd*bc*sc*i*
alias: pci:v000014E4d00001650sv*sd*bc*sc*i*
alias: pci:v000014E4d0000164Fsv*sd*bc*sc*i*
alias: pci:v000014E4d0000164Esv*sd*bc*sc*i*
depends: mdio,libcrc32c
intree: Y
vermagic: 3.19.0-22-generic SMP mod_unload modversions
parm: num_queues: Set number of queues (default is as a

number of CPUs) (int)
parm: disable_tpa: Disable the TPA (LRO) feature (int)
parm: int_mode: Force interrupt mode other than MSI-X (1

INT#x; 2 MSI) (int)
parm: dropless_fc: Pause on exhausted host ring (int)
parm: mrrs: Force Max Read Req Size (0..3) (for debug)

(int)
parm: debug: Default debug msglevel (int)

~ # ip -o link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue \

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000\

link/ether 98:be:94:58:20:34 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000\

link/ether 98:be:94:58:20:35 brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000\

link/ether 98:be:94:58:20:36 brd ff:ff:ff:ff:ff:ff
5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000\

link/ether 98:be:94:58:20:37 brd ff:ff:ff:ff:ff:ff

NOTE:
-------
Currently machine is available in shell prompt

Pls let me know if you any other details

Regards,
Indira