maas fails to set a power type after enlistment on AMT based systems

Bug #1280486 reported by Jeff Lane 
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
MAAS
Triaged
High
Unassigned

Bug Description

EDIT: I initially filed this against the latest version of maas-test on trusty. But after bringing up my full maas server and trying the latest trusty version of maas, I have confirmed that this is actually a maas bug.

After enlistment, the node powers off, and is listed in the node list. However, no power type is set by the maas server, meaning every server I enlist has to be manually edited in order to be commissioned.

I have now seen this with both maas and maas-test on two different systems, one a NUC using AMT and one a SuperMicro using IPMI.

bladernr@maas-test:~$ sudo maas-test --bmc-ip 10.0.0.5 --bmc-username ADMIN --bmc-password insecure --maas-series trusty --series trusty eth0
2014-02-14 16:14:26,688 INFO Checking for KVM extensions.
2014-02-14 16:14:26,699 INFO Checking for virtualised hardware...
2014-02-14 16:14:26,751 INFO Scanning for unexpected DHCP servers on testing network...
2014-02-14 16:14:29,785 INFO Checking for running proxy instance...
2014-02-14 16:14:29,785 INFO Starting proxy...
2014-02-14 16:14:29,788 INFO Done starting proxy.
2014-02-14 16:14:29,807 INFO Downloading KVM image for series=trusty, arch=amd64...
2014-02-14 16:14:34,002 INFO Done downloading KVM image for series=trusty, arch=amd64.
2014-02-14 16:14:34,003 INFO Creating virtual machine fdf10b3c-95bc-11e3-bbd7-001f16355790, arch=amd64...
2014-02-14 16:14:36,508 INFO Done creating virtual machine fdf10b3c-95bc-11e3-bbd7-001f16355790, arch=amd64.
2014-02-14 16:14:36,509 INFO Waiting for the virtual machine to come up...
2014-02-14 16:14:57,033 INFO Virtual machine is running.
2014-02-14 16:14:57,034 INFO Waiting for cloud-init to finish its work...
2014-02-14 16:15:27,652 INFO Cloud-init run finished.
2014-02-14 16:15:27,652 INFO Configuring network interface on virtual machine fdf10b3c-95bc-11e3-bbd7-001f16355790...
2014-02-14 16:15:35,047 INFO Done configuring network interface on virtual machine fdf10b3c-95bc-11e3-bbd7-001f16355790
2014-02-14 16:15:37,749 INFO Virtual machine fdf10b3c-95bc-11e3-bbd7-001f16355790 is ready.
2014-02-14 16:15:53,851 INFO Installing MAAS (version 1.5+bzr1909-0ubuntu1)...
2014-02-14 16:17:44,616 INFO Done installing MAAS.
2014-02-14 16:17:54,738 INFO Importing MAAS PXE files series=trusty, architectures=amd64/generic i386/generic...
2014-02-14 16:21:36,461 INFO Done importing MAAS PXE Files.
2014-02-14 16:21:37,659 INFO MAAS server URL: http://192.168.122.148/MAAS/ username:admin, password:edlOZSRM
2014-02-14 16:21:37,659 INFO SSH login: sudo ssh -i /var/cache/maas-test/vm_ssh_id_rsa ubuntu@192.168.122.148
test_power_up_node (maastest.main.ConfiguredTestMAAS) ... ok
test_enlist_node (maastest.main.ConfiguredTestMAAS) ... ok
test_wait_node_down (maastest.main.ConfiguredTestMAAS) ... ok
test_commission_node (maastest.main.ConfiguredTestMAAS) ...

I ran maas-test as noted in the CLI above. It sent a power-on to the BMC of my server and the server PXE booted and succeeded in enlistment.

maas-test then went to the comissioning test and failed to power on the server.

Thanks to an update that provides the IP and login data for the MAAS vm that maas test creates, I was able to log ito the maas dashboard and take a look. It showed my server as a node ready to be comissioned. The maas dashboard ALSO, however was giving the warning about not having a power type set. I looked into the node data itself and sure enough, no power type was set.

I logged into the BMC manually and powered the server on and it boted and completed commissioning and passe the test.

So it seems that maas test sent IPMI commands to initiate the enlistment, but then failed to do likewise for the commissioning.

As a final check, just to make sure things weren't wonky on the server, I powered it on remotely via ipmitool from my maas-test machine:
ladernr@maas-test:~$ sudo ipmitool -H 10.0.0.5 -U ADMIN -P insecure power on
Chassis Power Control: Up/On

and off:
bladernr@maas-test:~$ sudo ipmitool -H 10.0.0.5 -U ADMIN -P insecure power off
Chassis Power Control: Down/Off

Jeff Lane  (bladernr)
summary: - maas-test failed to power system on to run commission test
+ maas fails to set a power type after enlistment
description: updated
affects: maas-test → maas
Revision history for this message
Julian Edwards (julian-edwards) wrote : Re: maas fails to set a power type after enlistment

This type of power discovery does not work on AMT (NUCs) yet. I think the super micro bug might have just been fixed, can you please try again with the package in dailybuilds and let me know if it works now.

https://launchpad.net/~maas-maintainers/+archive/dailybuilds

Changed in maas:
status: New → Incomplete
Revision history for this message
Jeff Lane  (bladernr) wrote :

Setting this to confirmed due to your comment about AMT.

However, I can also confirm that the current bits seem to have resolved the issue with IPMI based controllers.

Changed in maas:
status: Incomplete → Confirmed
Jeff Lane  (bladernr)
summary: - maas fails to set a power type after enlistment
+ maas fails to set a power type after enlistment on AMT based systems
Changed in maas:
status: Confirmed → Triaged
importance: Undecided → High
tags: added: server-hwe
tags: added: power
Raphaël Badin (rvb)
tags: added: amt
Revision history for this message
Joe Hicks (6oe) wrote :

I just tried to upgrade maas with the bits from ppa:maas-maintainers/daily-qa-ok and seem to have the same or a similar problem on a SuperMicro server. What should I check to confirm? I am a bit of a n00b here, so I apologize in advance.

Revision history for this message
Julian Edwards (julian-edwards) wrote : Re: [Bug 1280486] Re: maas fails to set a power type after enlistment on AMT based systems

On Monday 26 May 2014 04:39:24 you wrote:
> I just tried to upgrade maas with the bits from ppa:maas-maintainers
> /daily-qa-ok and seem to have the same or a similar problem on a
> SuperMicro server. What should I check to confirm? I am a bit of a n00b
> here, so I apologize in advance.

the daily-qa-ok PPA is a bit behind at the moment, can you try /stable
instead, I uploaded a new release this morning.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.