MAAS (amttool) cannot control AMT version > 8

Thanks for the bug report.

Do you think then that amttool should be fixed, or should maas use wsman instead? This obviously affects the bug tasks involved here.

I would say maas should use wsman instead - Intel has deprecated the SOAP API's used by amttool for quite a while according to the linked articles above.
My quick tests showed that the attached wsman script controlled the machines with v7, v8, and v9 ok.
v4 did not work (note however that the BIOS is extremely downlevel on that machine; I'm trying to update it today)
Perhaps the best way to deal with it is to add a "wsman" entry to the MAAS power control and default to that for AMT boxes - if one of these real old boxes shows up the user can switch to the AMT selection.

On 18 June 2014 14:24, Mark W Wenning <email address hidden> wrote:
> Perhaps the best way to deal with it is to add a "wsman" entry to the MAAS power control and default to that for AMT boxes - if one of these real old boxes shows up the user can switch to the AMT selection.

I'd be more inclined to just switch to wsman and leave it at that, or
have MAAS auto-discover which power control tool it should use and
then use that (rather than making the user choose). Two choices for
the same power type is a surefire road to the confusing-ui tag.

Fair enough, probably ought to just go to wsman then.

Update: v. 4 does work with wsman after updating BIOS to the latest level.

I'm with Graham, let's just make it work with minimal fuss. IIRC Dustin used to use wsman in the early days so perhaps he's still got some valid scripts.

On 18 June 2014 19:16, Mark W Wenning <email address hidden> wrote:
> Update: v. 4 does work with wsman after updating BIOS to the latest
> level.

Good to know. Thanks for testing it :)

Just FYI, as an added bonus, this is blocking the cert of one Dell system (the one Mark was testing that brought this out). That system fails cert for now, until wsman is in place and working within MAAS.

FYI and FWIW, Samantha ran amttool on her NUC, AMT version came back as v8.1.30 .
I have several machines (Dell laptops) that have versions v4, v7, and V8, and of course the T20 at v9.
Feel free to ping me if you need something tested.

Even though wsman looks like a better alternative than amttool, I'm a bit concerned to make MAAS rely on a bunch of packages (libwsman-client2 libwsman-curl-client-transport1 libwsman1 wsmancli) that are all in universe. Can we first get a member of the server team to evaluate if it's doable to have them in main first?

Note that amttool is also in universe to it wouldn't be that much of a regression to use something else also in universe… that being said, the good thing the amtterm had going for it is that is was a standalone package. We even toyed with the idea of rewritting it in Python an include it in MAAS.

See bug 1314629 for details.

On 24/06/14 02:04, Raphaël Badin wrote:
> Even though wsman looks like a better alternative than amttool, I'm a
> bit concerned to make MAAS rely on a bunch of packages (libwsman-client2
> libwsman-curl-client-transport1 libwsman1 wsmancli) that are all in
> universe. Can we first get a member of the server team to evaluate if
> it's doable to have them in main first?
>
> Note that amttool is also in universe to it wouldn't be that much of a
> regression to use something else also in universe… that being said, the
> good thing the amtterm had going for it is that is was a standalone
> package. We even toyed with the idea of rewritting it in Python an
> include it in MAAS.
>

Right, a Python re-write would be preferable here. We're only using a
*fraction* of the functionality right now, so the new code should be
very straightforward indeed.

On 24 June 2014 00:50, Julian Edwards <email address hidden> wrote:
>
> Right, a Python re-write would be preferable here. We're only using a
> *fraction* of the functionality right now, so the new code should be
> very straightforward indeed.

I spent some time yesterday poking around with wsman and I'd far
prefer that we bring the functionality that we need into MAAS and
maintain it properly ourselves. That's more work than a simple(ish)
shift to using wsman, but I think it's going to be far more robust in
the long term.

(Moving this back to "Triaged" since the work will need to be more
comprehensive than we have the bandwidth for right now).

Is there an ETA on this?
I am trying to certify Dell's T20 tower and this is a showstopper - this will be true for any future low-end Dell server as well.

FYI, guys, we do have some "helper" scripts that use wsman to poweron and off systems. MAAS could certainly copy that logic and build off of it.

See lp:orange-box in the usr/bin/wsman* scripts.

Thanks,
Dustin

I think an easy short-term fix could be simply to fall back onto the helper scripts from the AMT power template (perhaps detecting in there whether we are using AMT 9.0 or above). I don't think the dependency on universe should be a showstopper for that (we could add it as a Suggests and release-note the fact that the user needs to install those packages, with a UI hint as an added bonus if we have time for it).

If AMT becomes something which is used in more and more "real server hardware" we could look to making a native interface, but I don't really think that's the case at the moment.

Any known workarounds for this issue?

Any movement on this? kiko, I thought there was going to be a fix in December?

This is currently being investigated and worked on by the Server Hardware Enablement team.

Still waiting on access to hardware for debugging and verifying fix.

Note that we still want to port all the Bash templates to Python. If you're thinking about changing an existing Bash power driver, please see if porting it to Python is possible. It might sound like a lot of work but I believe it might not be as bad as it sounds; especially if the change you're planning to make requires doing things that are cumbersome to express and test in Bash.
Please see the virsh driver for an example of this (the core functionality is Python but it's still called from a Bash template, ./src/provisioningserver/drivers/hardware/virsh.py,. /src/provisioningserver/drivers/hardware/tests/test_virsh.py, ./etc/maas/templates/power/virsh.template).

Just to add my perspective and needs for this bug to be resolved. This issue currently gates certification on any system that uses AMT v9 . Bear in mind that without the MAAS requirements these systems would likely all pass hardware certification, so we're gating perfectly workable hardware because our provisioning system doesn't yet support ATM v9.

Newell's MP to MAAS that we worked on provides the necessary enablement to make AMT v9 work with maas for everything but in-band settings (creating a maas user and setting a maas password) which I'm not sure is possible at all in AMT anyway.

From a cert perspective, getting this merge reviewed and completed and packages built that include this enablement is pretty critical. FWIW, I don't need packages pushed into Trusty via Backports, just packages in either the Testing or Stable MAAS PPA will be sufficient for certification work.

One customer who ships servers based around AMT v9 has a test plan that has a deadline of March 16 to get two or three new AMT based servers certified for customer requirements.

I mentioned this this morning at the MAAS cross-team meeting, so I'm just adding notes here to preserve them in this bug.

Adding a task for Ubuntu to handle adding wsmancli to the Suggests line.

Verified on several VPro machines in the Dell lab, with different AMT firmware levels.

## PowerEdge T20 FW 9.0.21-build 1462 works

## Laptop: E6400 FW 4.2.60-build 1060 works
## Laptop: E6520 FW 7.1.4-build 1068 works
## Laptop: E6430 FW 8.0.1-build 1399 works
## Laptop: M4700 FW 8.1.40-build 1416 works

This bug was fixed in the package maas - 1.7.2+bzr3355-0ubuntu1

---------------
maas (1.7.2+bzr3355-0ubuntu1) vivid; urgency=medium

  * New upstream release, 1.7.2 bzr3355:
    - Support AMT Version > 8 (LP: #1331214)
    - Fix call to amttool when restarting a node to not fail
      disk erasing. (LP: #1397567)
    - Do not generate the 'option routers' stanza if router IP
      is None. (LP: #1415538)
    - Do not deallocate StaticIPAddress before node has powered
      off. (LP: #1403909)
    - Remove all OOPS reporting. (LP: #1405998)
    - Update node host maps when a sticky ip address is claimed
      over the API. (LP: #1423931)
  * debian/control:
    - Depends on ubuntu-cloudimage-keyring for region (LP: #1424287)
    - Depends on pxelinux instead of syslinux-dev (LP: #1433697)
  * Drop dependencies on python-oops* and add dependency to python-bson.
 -- Andres Rodriguez <email address hidden> Fri, 30 Jan 2015 11:58:07 +0000

Hello Mark, or anyone else affected,

Accepted maas into utopic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/maas/1.7.5+bzr3369-0ubuntu1~14.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

This issue has been verified to work both on upgrade and fresh install, and has been QA'd. Marking verification-done.

Tested with: Intel NUC Model:NUC5i5MYHE with VPro extension running AMT Version: 10.0.45

To get this working properly with MAAS Version 1.8.3+bzr4053-0ubuntu1 (trusty1) , I had to also install the wsmancli package, which MAAS appears to now take advantage of to properly control the device. This may impact some users who miss installation of that package -

sudo apt-get install wsmancli

Update

Hardware Tested : Intel NUC Model:NUC5i5MYHE with VPro extension running AMT Version: 10.0.45
MAAS Version : 1.8.3+bzr4053-0ubuntu1 (trusty1)

Unit properly can be commissioned in MAAS and is in ready state, however "power on" function does not work.

Akash,

You should file this as a new bug. Additionally, it would be interesting to see if you could get this working in MAAS 1.10 or MAAS trunk as that has the latest AMT power driver code.

Related is bug #1513198.

Newell,

I worked with mpontillo today, and figured out that I had completed a task in the 'wrong order of events' when setting up a node in MAAS. Apparently you cannot power on a node, after commisioning it and in "Ready" state. You have to deploy first.

Once adding my ssh key, I was successfully able to deploy on :
Hardware Tested : Intel NUC Model:NUC5i5MYHE with VPro extension running AMT Version: 10.0.45
MAAS version : MAAS Version 1.9.0+bzr4533-0ubuntu1 (trusty1)

Akash,

Ah, didn't catch that you had only commissioned the node. Thanks Mike!

On Mon, Jan 4, 2016 at 4:39 PM, Akash Chandrashekar <
<email address hidden>> wrote:

> Newell,
>
> I worked with mpontillo today, and figured out that I had completed a
> task in the 'wrong order of events' when setting up a node in MAAS.
> Apparently you cannot power on a node, after commisioning it and in
> "Ready" state. You have to deploy first.
>
> Once adding my ssh key, I was successfully able to deploy on :
> Hardware Tested : Intel NUC Model:NUC5i5MYHE with VPro extension running
> AMT Version: 10.0.45
> MAAS version : MAAS Version 1.9.0+bzr4533-0ubuntu1 (trusty1)
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1331214
>
> Title:
> MAAS (amttool) cannot control AMT version > 8
>
> Status in MAAS:
> Fix Committed
> Status in MAAS 1.7 series:
> Fix Committed
> Status in maas package in Ubuntu:
> Fix Released
>
> Bug description:
> While testing Dell Poweredge T20, I noticed that MAAS will not power
> on/off the node even if you type in the AMT values. (see bug #1280486
> )
>
> amttool shows:
> test@test-Latitude-14-Rugged-Extreme-7404:~$ amttool 192.168.0.117 info
> soap_init: http://192.168.0.117:16992/NetworkAdministrationService
> http://schemas.intel.com/platform/client/NetworkAdministration/2004/01###
> AMT info on machine '192.168.0.117' ###
> AMT version: 9.0.1
> 404 Not Found at /usr/bin/amttool line 244.
> test@test-Latitude-14-Rugged-Extreme-7404:~$
>
> A little googling (for example, https://software.intel.com/en-
> us/blogs/2012/12/01/intel-amt-wsman-interface-is-replacing-the-
> soapeoi-interface ) shows that intel has probably removed amttool
> capability in version 9 of AMT, in favor of WSMAN capability.
>
> I also have several other AMT boxes here, with v 7, 8, and 4, and have
> verified that amttool (and MAAS) works OK on those boxes.
>
> To further check this, I ran some (attached) wsman scripts and
> verified that they did, indeed, power the box on and off.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/maas/+bug/1331214/+subscriptions
>