cannot deploy ESXi to non-sda drive

added messages file captured during the VM deployment.

Reformatted config file - easier to read than the log file version.

Here is the proposed change.

https://pastebin.ubuntu.com/p/cvNd4Kbcky/

here is the physical server's full installation log for visibility
https://pastebin.ubuntu.com/p/Vktdf3NR2t/

Why do the disks on physical or virtual not have a serial number?

Looking at the physical logs, they certainly do have serial numbers.

2021-04-24T01:12:48-05:00 dell2 cloud-init[1606]: get_path_to_storage_volume for volume sda
2021-04-24T01:12:48-05:00 dell2 cloud-init[1606]: Processing serial 6842b2b01b006d0024858f55183677de via udev to 6842b2b01b006d0024858f55183677de
2021-04-24T01:12:48-05:00 dell2 cloud-init[1606]: lookup_disks found: ['wwn-0x6842b2b01b006d0024858f55183677de', 'scsi-36842b2b01b006d0024858f55183677de', 'wwn-0x6842b2b01b006d0024858f55183677de-part1', 'scsi-36842b2b01b006d0024858f55183677de-part1']
2021-04-24T01:12:48-05:00 dell2 cloud-init[1606]: lookup_disks realpath(wwn-0x6842b2b01b006d0024858f55183677de)=/dev/sda

Is it not included in your custom storage config? No serial numbers in the config.

> Here is the proposed change.
> https://pastebin.ubuntu.com/p/cvNd4Kbcky/

The risk here, especially with physical systems, is that the path (/dev/sda) is not reliable. That is, from boot to boot, the path to a specific disk may differ (sda may point to a different physical device).

MAAS certainly knows (physical system) or can construct (virtual) disks with serial numbers to ensure consistent and reliable deployments.

I would suggest this is not a bug, but rather configuration error (missing serial number in disk config).

I think I understand what's happening. I'll finish up a proposed fix this morning and test it and report back.

The virtual case listed in the original report doesn't have serials. I have been told by Lee Trager that we can assume we get the serial value in the virtual case. I can't ask MAAS to supply a value it doesn't have.

The physical case, in the newer log, does have the serial, and still has the problem. Diff pending that should address that item.

*that we CAN'T assume we get the serial value in the virtual case

Ryan, probably a bit of misunderstanding.
The absence of serial only happens in case of the VM deployment.
With the physical server the serial is always in place.
I originally thought that I can reproduce the problem with the VM (still waiting for the requestor to share the full log), but later I found a way to deploy onto a physical server and I shared a log from it in my last comment.

Here is my updated and untested proposal. I still plan to test this this morning.
https://pastebin.ubuntu.com/p/DfQqmqDZQJ/

> Curtin doesn't respect "custom" config of the storage and follows "simple" flow and deploys the OS on "sda", more precisely to the first block device in the array (see last two lines of the log below)

MAAS and curtin only support writing custom images as opaque blobs with DD. When the image is dd-* type, curtin deploys via meta_simple as it's going to just write out the image to the specified target device.

As with all curtin configurations you should specify the disk serial number of the target disk on which you expect to install your image.

> but later I found a way to deploy onto a physical server and I shared a
> log from it in my last comment.

Thanks; this is helpful.

'storage': {'config': [
    {'id': 'sda', 'model': 'PERC H700', 'name': 'sda',
      'serial': '6842b2b01b006d0024858f55183677de',
      'type': 'disk', 'wipe': 'superblock'},
    {'grub_device': True, 'id': 'sdb', 'model': 'MP0402H',
     'name': 'sdb', 'ptable': 'gpt', 'serial': '581270040222',
     'type': 'disk', 'wipe': 'superblock'},
...
]

Two disks identified, and required them to be wiped. Then the
dd-image will take the meta_simple path and sdb is selected as the
target (due to grub_device: True).

Then here's the bug:

'custom' mode but multiple devices given. using first found
2021-04-24T01:12:50-05:00 dell2 cloud-init[1606]: mode is 'custom'. multiple devices given. using '/dev/sda' (first available)

In the code, we example args.devices which was set to both disks
due to including both of them in the storage config for wiping.

The logic which looks through the list of devices when more than
one is supplied did not account for diskPath being set.

The cleanest fix here is to only select from the devices list if
devpath did not get set.

diff --git a/curtin/commands/block_meta.py b/curtin/commands/block_meta.py
index cf6bc025..e2f37201 100644
--- a/curtin/commands/block_meta.py
+++ b/curtin/commands/block_meta.py
@@ -2007,17 +2007,20 @@ def meta_simple(args):
     elif len(devices) == 0 and devpath:
         devices = [devpath]

- if len(devices) > 1:
- if args.devices is not None:
- LOG.warn("'%s' mode but multiple devices given. "
- "using first found", args.mode)
- available = [f for f in devices
- if block.is_valid_device(f)]
- target = sorted(available)[0]
- LOG.warn("mode is '%s'. multiple devices given. using '%s' "
- "(first available)", args.mode, target)
+ if devpath is not None:
+ target = devpath
     else:
- target = devices[0]
+ if len(devices) > 1:
+ if args.devices is not None:
+ LOG.warn("'%s' mode but multiple devices given. "
+ "using first found", args.mode)
+ available = [f for f in devices
+ if block.is_valid_device(f)]
+ target = sorted(available)[0]
+ LOG.warn("mode is '%s'. multiple devices given. using '%s' "
+ "(first available)", args.mode, target)
+ else:
+ target = devices[0]

Oh launchpad...

https://paste.ubuntu.com/p/kGTD4zws5p/

> Here is my updated and untested proposal. I still plan to test this this morning.
> https://pastebin.ubuntu.com/p/DfQqmqDZQJ/
>
> * Some disks may not have serial indicated, and that's OK. Use the
> supplied path instead if present.

I would avoid relaxing this to prevent using the wrong disk. MAAS knows the serial
numbers of the disks.

> * grub_device means we mean to image THAT device, so we shouldn't keep
> the entire list of devices as plausible image targets.

grub_device can be set on more than one disk. In which case we still need to handle the case where we've been given to valid targets and selecting one.

+ if i.get("grub_device"):
+ serial = i.get("serial")
+ if serial:
+ devices = [block.lookup_disk(serial)]
+ LOG.info("choosing device %s based on grub_device flag "
+ "and serial", devices[0])
+ break

This is the general idea; as soon as we meet the selection criteria
we can exit the loop and we have THE specified disk. Note that
block.lookup_disk(serial) may not find the disk in which case it throws
a ValueError; I suggest we continue searching of lookup fails.

Above is the attached log from our use case that Dan requested

Michael, thanks for the log. I think it looks similar to the https://bugs.launchpad.net/curtin/+bug/1925722/comments/4 log so I think we're on the right track. This was helpful as it provides some level of confirmation that we're chasing the right issue.

This bug is fixed with commit def7d5b1 to curtin on branch master.
To view that commit see the following URL:
https://git.launchpad.net/curtin/commit/?id=def7d5b1

This bug is believed to be fixed in curtin in version 21.3. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

Thank you.

Do you know which version of MAAS that version of curtin is included with, how to find out, or how to update curtin within a MAAS installation? I've had a quick web search but struggling to find out.