Moonshot nodes with Mellanox interfaces fail to deploy in maas 1.7

Does this happen with other architectures too? I think it's a dupe of bug 1378479 (whose title needs to change, it's mis-labelled).

This isn't just ARM - I hit this on X86 kvm also. My KVM's /etc/network/interfaces file only has:

"auto lo".

This is the problem.

We think another change fixed this problem as a side effect. We're publishing a beta 6 right now - if you could test it that would be good.

Nevermind - this is still broken in beta6.

Thanks for the update Jason, I didn't have a chance to test my other x86 nodes at the time. Please let me know if you would like me to smoke test 1.7 beta7 when avail.

Hi Sean - it's available now in the experimental PPA - please test! I've tested and it works for me.

Same for me, I upgraded to 1.7.0~beta6+bzr3232-0ubuntu1~trusty1 and the node I had problems with before came up correctly.

Confirmed for me as well, fixed with beta6. Thanks everyone!

i think I may have spoken to soon,

my arm64 nodes are giving me a hassle still, I'm just getting around to booting them up. my armhf & x64 nodes appear to be working. I've been on them all day.

https://pastebin.canonical.com/118930/

Using maas version: 1.7.0~beta6+bzr3260-0ubuntu1~trusty1 all MAAS server all-in-one metapackage

Sean,

Can you post the lshw output from your node? It should be available from the discovery data section at the bottom of the node page.

See pastebin:
https://pastebin.canonical.com/118933/

Sean, I only see one network interface in that lshw output.

        - node:
          id: network
          claimed: true
          class: network
          handle: PCI:0000:01:00.0
          - description:
            Ethernet interface
          - product:
            MT27500 Family [ConnectX-3]
          - vendor:
            Mellanox Technologies

Can you provide the XML lshw output? I wonder if something was lost in translation to yaml...

Can you also retest with the latest 1.7beta7 MAAS package?

Hey Jason,

I created a maas 1.7 beta 7 env and tested my mcdivitt

maas:
  Installed: 1.7.0~beta7+bzr3266-0ubuntu1~trusty1
  Candidate: 1.7.0~beta7+bzr3266-0ubuntu1~trusty1

Attached is the discovery data xml
https://pastebin.canonical.com/119179/

xml lshw output from maas as you requested
https://pastebin.canonical.com/119178/

I have captured all of the output from poweron -> failure of the mcdivitt console and attached it to this comment.

Sean - thanks for all the debug info - I understand what's going on now. Apparently the Mellanox driver is a bit weird - we have only one network node for it in lshw output, but it shows up as two interfaces.

From the mcdivitt-39 log above, we see there are two network interfaces:

ci-info: | eth1 | False | . | . | fc:15:b4:21:00:93 |
ci-info: | eth0 | True | 10.228.65.139 | 255.255.0.0 | fc:15:b4:21:00:92 |

And we see that eth0 is the one we pxe boot from:
ci-info: | eth0 | True | 10.228.65.139 | 255.255.0.0 | fc:15:b4:21:00:92 |

However, the lshw output includes eth1 and its MAC address:
       <logicalname>eth1</logicalname>
       <version>00</version>
       <serial>fc:15:b4:21:00:93</serial>

When MAAS generates /etc/network/interfaces files, it creates configurations for all interfaces it finds in /etc/network/interfaces, but makes only the interface that the node PXE boots from an auto/dhcp interface - the rest are manual.
Since eth0 doesn't show up in lshw output, it doesn't get an entry created.

The fix for this is to use the node's list of MAC addresses discovered at enlistment time rather than the lshw output. The discovery done at enlistment time uses 'ip addr' to list MAC addresses from interfaces, so it picks up the MAC address for both eth0 and eth1.

This is something we wanted to do already but were hoping to not have to do it for 1.7 at this point. This is the second critical bug we've hit in using lshw output to build the interfaces list though, so it's worth reconsidering.

Thinking about this some more, I'm not sure the approach discussed above is sufficient.

Since lshw output doesn't include one of the interfaces for this system, the only way to fix the issue is to stop using lshw and use something else - like 'ifconfig' or 'ip addr' output instead. Luckily, we have 'ip addr' output from enlistment that's used to populate the node's macaddress_set in the database.

However, the mac addresses we store in the database don't include interface names, while lshw does. We need the interface names because we have to create /etc/network/interfaces, which identifies each interface by name. If we assume interfaces are named eth0, eth1 etc and the OS names them em1, wlan2 etc then our /etc/network/interfaces config won't work.

Some things we could do:
- Store interface name during enlistment, along side MAC address. This won't work for nodes on existing systems that get upgraded as they would have to be deleted and reenlisted to get the necessary information.
- Generate 70-persistent-net-rules along with /etc/network/interfaces, and force the names to align this way by picking the names ourselves

Not considering 1.7, I think storing interface names during enlistment is definitely the right thing to do, in particular because lshw output can be unreliable (it used to not work at all on ARM IIRC, for instance). For existing nodes, I assume they either have the right information (i.e. lshw returned valid results) or they can be deleted and re-enlisted since they are half-broken anyway.

For 1.7, however, I'm not sure what to do, in particular because I'm not sure whether or why this regressed. Did we only start using lshw information in this release?

I've followed up offbug to check whether this may be a fixable driver bug.

Is there a reason we need to store the names of the interfaces? Can't we just bring up each interface using an index, eth0 - ethN. Or do some interfaces require a specific naming convention?

Just a quick note on the impact for Moonshot, without this fix, fastpath (curtin) MAAS deployment for McDivitt (ARM64 Moonshot cartridge) will be broken. However, we can still MAAS d-i deploy to McDivitt.

lshw appears to assume that two NICs that have the same bus/device/function are the same device. That isn't true for Mellanox on the m400 (McDivitt). By commenting out the "duplicate merging" code, I can cause lshw to emit correct data.

Do we know that MAAS would be OK if lshw were fixed? If so, should we reassign to Ubuntu lshw and get it off the MAAS critical path? I can provide a hacked lshw if we want to verify that MAAS would work correctly if lshw were fixed.

I also want to add that the anders cartridge , also has a mellanox interface, however the arch is x64 not arm64.

See the lshw output here:
https://pastebin.canonical.com/119279/

This x64 node currently does not have the any issue deploying with maas 1.7 beta 8.

On 10/23/2014 10:44 AM, dann frazier wrote:
> lshw appears to assume that two NICs that have the same
> bus/device/function are the same device. That isn't true for Mellanox on
> the m400 (McDivitt). By commenting out the "duplicate merging" code, I
> can cause lshw to emit correct data.
>
> Do we know that MAAS would be OK if lshw were fixed? If so, should we
> reassign to Ubuntu lshw and get it off the MAAS critical path? I can
> provide a hacked lshw if we want to verify that MAAS would work
> correctly if lshw were fixed.

MAAS would work if both interfaces were reported in the lshw output, but
it wouldn't hurt to test it to prove it.

Jason

On 10/23/2014 10:53 AM, Sean Feole wrote:
> I also want to add that the anders cartridge , also has a mellanox
> interface, however the arch is x64 not arm64.
>
> See the lshw output here:
> https://pastebin.canonical.com/119279/
>
> This x64 node currently does not have the any issue deploying with maas
> 1.7 beta 8.

This can be explained by the fact that for whatever reason, this anders
node's lshw output includes eth0, while the arm64 node's lshw output
includes eth1. Since the node PXE boots from eth0, it's the interface
we're trying to configure to come up with DHCP, and the one we need to
find in lshw output in order for that to work.

Jason

> Is there a reason we need to store the names of the interfaces? Can't we just bring up each interface using an index,
> eth0 - ethN. Or do some interfaces require a specific naming convention?

I don't think it's reasonable to think we can change system interface names without a lot of fallout.

m800 lshw (slayton armhf)
https://pastebin.canonical.com/119289/

m710 lshw (anders x64)
https://pastebin.canonical.com/119290/

m300 lshw (avaton x64)
https://pastebin.canonical.com/119291/

m400 lshw (mcdivitt arm64)
https://pastebin.canonical.com/119292/

I've filed an upstream bug with lshw with a patch. New bugs are moderated, but I'll add a URL when I have one.

I've written a branch for MAAS that should work around this issue, although it's a huge hack. I have unit tests developed using the lshw output Sean provided, but I don't have a real system to test it on.

lp:~jason-hobbs/maas/workaround-mellanox-lshw-bug

Sean - could you test this? The change is all in one file (not counting the unit tests). I think it'd be easiest to just manually apply this patch for that one file. You should be able to edit the file, restart apache, then successfully deploy the system.

Status changed to 'Confirmed' because the bug affects multiple users.

update for today:

I was able to successfully commission/deploy a mcdivitt arm64 node using the modifications that dann has proposed/installed onto my test maas 1.7 environment.

Attached is the complete log from Commissioning to Deployment.

Also available is the lshw output after these changes were made, we now see 2 ethernet devices.

https://pastebin.canonical.com/119448/

This bug was fixed in the package lshw - 02.17-1ubuntu1

---------------
lshw (02.17-1ubuntu1) vivid; urgency=medium

  * Resynchronise with Debian. Remaining changes:
     - Disable /dev/mem access for SMBIOS on aarch64 systems
     - Fix endianness issues with device-tree
     - Add DDR3 support for systems described by device-tree
  * Support multiple NICs sharing the same PCI function (LP: #1379591)
 -- dann frazier <email address hidden> Mon, 27 Oct 2014 13:22:38 -0600

MAAS Team discussed this today and agreed the fix should be in lshw, not in MAAS.

Hello Sean, or anyone else affected,

Accepted lshw into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/lshw/02.16-2ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Sean, or anyone else affected,

Accepted lshw into utopic-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/lshw/02.16-2ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Fix verified:

ubuntu@ms01a:~$ sudo dpkg -i lshw_02.16-2ubuntu1.1_arm64.deb
dpkg: warning: downgrading lshw from 02.16-2ubuntu1.2 to 02.16-2ubuntu1.1
(Reading database ... 80719 files and directories currently installed.)
Preparing to unpack lshw_02.16-2ubuntu1.1_arm64.deb ...
Unpacking lshw (02.16-2ubuntu1.1) over (02.16-2ubuntu1.2) ...
Setting up lshw (02.16-2ubuntu1.1) ...
Processing triggers for man-db (2.6.7.1-1) ...
ubuntu@ms01a:~$ sudo lshw -class net > pre-fix
ubuntu@ms01a:~$ sudo dpkg -i lshw_02.16-2ubuntu1.2_arm64.deb
(Reading database ... 80719 files and directories currently installed.)
Preparing to unpack lshw_02.16-2ubuntu1.2_arm64.deb ...
Unpacking lshw (02.16-2ubuntu1.2) over (02.16-2ubuntu1.1) ...
Setting up lshw (02.16-2ubuntu1.2) ...
Processing triggers for man-db (2.6.7.1-1) ...
ubuntu@ms01a:~$ sudo lshw -class net > post-fix
ubuntu@ms01a:~$ diff -u pre-fix post-fix
--- pre-fix 2014-11-04 11:23:20.564933999 -0500
+++ post-fix 2014-11-04 11:23:36.674933999 -0500
@@ -1,4 +1,4 @@
- *-network
+ *-network DISABLED
        description: Ethernet interface
        product: MT27500 Family [ConnectX-3]
        vendor: Mellanox Technologies
@@ -11,7 +11,7 @@
        width: 64 bits
        clock: 33MHz
        capabilities: pm vpd msix pciexpress bus_master cap_list ethernet physical tp
- configuration: autonegotiation=off broadcast=yes driver=mlx4_en driverversion=2.2-1 (Feb 2014) duplex=full firmware=2.30.3000 ip=10.229.41.200 latency=0 link=yes multicast=yes port=twisted pair speed=1Gbit/s
+ configuration: autonegotiation=off broadcast=yes driver=mlx4_en driverversion=2.2-1 (Feb 2014) duplex=full firmware=2.30.3000 latency=0 link=no multicast=yes port=twisted pair speed=1Gbit/s
        resources: irq:244 memory:a010800000-a0108fffff memory:a010000000-a0107fffff
   *-network:0
        description: Ethernet interface
@@ -32,14 +32,23 @@
   *-network:2
        description: Ethernet interface
        physical id: 3
+ bus info: pci@0000:01:00.0
+ logical name: eth0
+ serial: 2c:59:e5:36:9a:b2
+ size: 1Gbit/s
+ capabilities: ethernet physical tp
+ configuration: autonegotiation=off broadcast=yes driver=mlx4_en driverversion=2.2-1 (Feb 2014) duplex=full firmware=2.30.3000 ip=10.229.41.200 link=yes multicast=yes port=twisted pair speed=1Gbit/s
+ *-network:3
+ description: Ethernet interface
+ physical id: 4
        logical name: vnet1
        serial: fe:16:3e:fe:b1:a1
        size: 10Mbit/s
        capabilities: ethernet physical
        configuration: autonegotiation=off broadcast=yes driver=tun driverversion=1.6 duplex=full link=yes multicast=yes port=twisted pair speed=10Mbit/s
- *-network:3
+ *-network:4
        description: Ethernet interface
- physical id: 4
+ physical id: 5
        logical name: novatap
        serial: 46:df:52:39:2e:34
        size: 10Mbit/s

And for utopic:

ubuntu@ms01a:~$ sudo lshw -class net > pre-fix
ubuntu@ms01a:~$ echo "deb http://ports.ubuntu.com/ubuntu-ports utopic-proposed main" | sudo tee /etc/apt/sources.list
deb http://ports.ubuntu.com/ubuntu-ports utopic-proposed main
ubuntu@ms01a:~$ sudo apt-get update
Ign http://ports.ubuntu.com utopic-proposed InRelease
Get:1 http://ports.ubuntu.com utopic-proposed Release.gpg [933 B]
Get:2 http://ports.ubuntu.com utopic-proposed Release [215 kB]
Get:3 http://ports.ubuntu.com utopic-proposed/main arm64 Packages [10.3 kB]
Get:4 http://ports.ubuntu.com utopic-proposed/main Translation-en [8725 B]
Fetched 235 kB in 1s (208 kB/s)
Reading package lists... Done
ubuntu@ms01a:~$ sudo apt-get upgrade
sudo: unable to resolve host ms01a
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  lshw
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 195 kB of archives.
After this operation, 8192 B of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://ports.ubuntu.com/ubuntu-ports/ utopic-proposed/main lshw arm64 02.16-2ubuntu2.1 [195 kB]
Fetched 195 kB in 0s (332 kB/s)
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
 LANGUAGE = (unset),
 LC_ALL = (unset),
 LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
(Reading database ... 11075 files and directories currently installed.)
Preparing to unpack .../lshw_02.16-2ubuntu2.1_arm64.deb ...
Unpacking lshw (02.16-2ubuntu2.1) over (02.16-2ubuntu2) ...
ubuntu@ms01a:~$ sudo lshw -class net > post-fix
sudo: unable to resolve host ms01a
ubuntu@ms01a:~$ diff -u pre-fix post-fix
--- pre-fix 2014-11-05 22:12:31.434933999 +0000
+++ post-fix 2014-11-05 22:13:56.574933999 +0000
@@ -1,4 +1,4 @@
- *-network
+ *-network DISABLED
        description: Ethernet interface
        product: MT27500 Family [ConnectX-3]
        vendor: Mellanox Technologies
@@ -11,7 +11,7 @@
        width: 64 bits
        clock: 33MHz
        capabilities: pm vpd msix pciexpress bus_master cap_list ethernet physical tp
- configuration: autonegotiation=off broadcast=yes driver=mlx4_en driverversion=2.2-1 (Feb 2014) duplex=full firmware=2.30.3000 ip=10.229.41.200 latency=0 link=yes multicast=yes port=twisted pair speed=1Gbit/s
+ configuration: autonegotiation=off broadcast=yes driver=mlx4_en driverversion=2.2-1 (Feb 2014) duplex=full firmware=2.30.3000 latency=0 link=no multicast=yes port=twisted pair speed=1Gbit/s
        resources: irq:244 memory:a010800000-a0108fffff memory:a010000000-a0107fffff
   *-network:0
        description: Ethernet interface
@@ -32,14 +32,23 @@
   *-network:2
        description: Ethernet interface
        physical id: 3
+ bus info: pci@0000:01:00.0
+ logical name: et...

The verification of the Stable Release Update for lshw has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package lshw - 02.16-2ubuntu2.1

---------------
lshw (02.16-2ubuntu2.1) utopic-proposed; urgency=medium

  * Support multiple NICs sharing the same PCI function (LP: #1379591)
 -- dann frazier <email address hidden> Mon, 27 Oct 2014 20:35:20 -0600

This bug was fixed in the package lshw - 02.16-2ubuntu1.2

---------------
lshw (02.16-2ubuntu1.2) trusty-proposed; urgency=medium

  * Support multiple NICs sharing the same PCI function (LP: #1379591)
 -- dann frazier <email address hidden> Tue, 28 Oct 2014 09:08:51 -0600