Bug #1771382 “ds-identify: fails to recognize NoCloud datasource... : Bugs : cloud-init

Revision history for this message

Martin Steigerwald (ms-proact) wrote on 2018-05-16:

#1

Download full text (5.4 KiB)

I now cloned a VM from the Cloud Init image that I "fixed" by running ds-identify --force, but I am running into the same thing again:

Cloud Init is disabled again, since ds-identify fails to run blkid command:

slestemplate:~ # cat /run/cloud-init/ds-identify.log
[… Hostname should have been changed to sles1 by Cloud Init …]
[up 11.01s] ds-identify
policy loaded: mode=search report=false found=all maybe=all notfound=disabled
no datasource_list found, using default: MAAS ConfigDrive NoCloud AltCloud Azure Bigstep CloudSigma CloudStack DigitalOcean AliYun Ec2 GCE OpenNebula OpenStack OVF SmartOS Scaleway Hetzner IBMCloud
ERROR: failed running [127]: blkid -c /dev/null -o export
DMI_PRODUCT_NAME=Standard PC (i440FX + PIIX, 1996)
DMI_SYS_VENDOR=QEMU
DMI_PRODUCT_SERIAL=
DMI_PRODUCT_UUID=0CE53BB3-A48E-44F7-9EC7-3C339E9C80D3
PID_1_PRODUCT_NAME=unavailable
DMI_CHASSIS_ASSET_TAG=
FS_LABELS=unavailable:error
ISO9660_DEVS=unavailable:error
KERNEL_CMDLINE=BOOT_IMAGE=/boot/vmlinuz-4.4.73-5-default root=/dev/mapper/sys0-rootfs splash=silent quiet showopts console=tty0 console=ttyS0,115200
VIRT=kvm
UNAME_KERNEL_NAME=Linux
UNAME_KERNEL_RELEASE=4.4.73-5-default
UNAME_KERNEL_VERSION=#1 SMP Tue Jul 4 15:33:39 UTC 2017 (b7ce4e4)
UNAME_MACHINE=x86_64
UNAME_NODENAME=slestemplate
UNAME_OPERATING_SYSTEM=GNU/Linux
DSNAME=
DSLIST=MAAS ConfigDrive NoCloud AltCloud Azure Bigstep CloudSigma CloudStack DigitalOcean AliYun Ec2 GCE OpenNebula OpenStack OVF SmartOS Scaleway Hetzner IBMCloud
MODE=search
ON_FOUND=all
ON_MAYBE=all
ON_NOTFOUND=disabled
pid=1138 ppid=1133
is_container=false
ec2 platform is 'Unknown'.
No ds found [mode=search, notfound=disabled]. Disabled cloud-init [1]
[up 11.05s] returning 1

Which leads to:
slestemplate:~ # cat /run/cloud-init/cloud-init-generator.log
/usr/lib/systemd/system-generators/cloud-init-generator normal=/run/systemd/generator early=/run/systemd/generator.early late=/run/systemd/generator.late
kernel command line (/proc/cmdline): BOOT_IMAGE=/boot/vmlinuz-4.4.73-5-default root=/dev/mapper/sys0-rootfs splash=silent quiet showopts console=tty0 console=ttyS0,115200
kernel_cmdline found unset
etc_file found unset
default found enabled
checking for datasource
ds-identify rc=1
ds-identify _RET=notfound
cloud-init is enabled but no datasource found, disabling
already disabled: no change needed [no /run/systemd/generator.early/multi-user.target.wants/cloud-init.target

However just using it again with the force option fixes the issue:

slestemplate:~ # /usr/lib/cloud-init/ds-identify --force
slestemplate:~ # cat /run/cloud-init/ds-identify.log
[up 11.01s] ds-identify
policy loaded: mode=search report=false found=all maybe=all notfound=disabled
no datasource_list found, using default: MAAS ConfigDrive NoCloud AltCloud Azure Bigstep CloudSigma CloudStack DigitalOcean AliYun Ec2 GCE OpenNebula OpenStack OVF SmartOS Scaleway Hetzner IBMCloud
ERROR: failed running [127]: blkid -c /dev/null -o export
DMI_PRODUCT_NAME=Standard PC (i440FX + PIIX, 1996)
DMI_SYS_VENDOR=QEMU
DMI_PRODUCT_SERIAL=
DMI_PRODUCT_UUID=0CE53BB3-A48E-44F7-9EC7-3C339E9C80D3
PID_1_PRODUCT_NAME=unavailable
DMI_CHASSIS_ASSET_TAG=
FS_LABELS=unavailable:error
ISO9660_DEVS=unavaila...

I now cloned a VM from the Cloud Init image that I "fixed" by running ds-identify --force, but I am running into the same thing again:

Cloud Init is disabled again, since ds-identify fails to run blkid command:

slestemplate:~ # cat /run/cloud-init/ds-identify.log
[… Hostname should have been changed to sles1 by Cloud Init …]
[up 11.01s] ds-identify 
policy loaded: mode=search report=false found=all maybe=all notfound=disabled
no datasource_list found, using default: MAAS ConfigDrive NoCloud AltCloud Azure Bigstep CloudSigma CloudStack DigitalOcean AliYun Ec2 GCE OpenNebula OpenStack OVF SmartOS Scaleway Hetzner IBMCloud
ERROR: failed running [127]: blkid -c /dev/null -o export
DMI_PRODUCT_NAME=Standard PC (i440FX + PIIX, 1996)
DMI_SYS_VENDOR=QEMU
DMI_PRODUCT_SERIAL=
DMI_PRODUCT_UUID=0CE53BB3-A48E-44F7-9EC7-3C339E9C80D3
PID_1_PRODUCT_NAME=unavailable
DMI_CHASSIS_ASSET_TAG=
FS_LABELS=unavailable:error
ISO9660_DEVS=unavailable:error
KERNEL_CMDLINE=BOOT_IMAGE=/boot/vmlinuz-4.4.73-5-default root=/dev/mapper/sys0-rootfs splash=silent quiet showopts console=tty0 console=ttyS0,115200
VIRT=kvm
UNAME_KERNEL_NAME=Linux
UNAME_KERNEL_RELEASE=4.4.73-5-default
UNAME_KERNEL_VERSION=#1 SMP Tue Jul 4 15:33:39 UTC 2017 (b7ce4e4)
UNAME_MACHINE=x86_64
UNAME_NODENAME=slestemplate
UNAME_OPERATING_SYSTEM=GNU/Linux
DSNAME=
DSLIST=MAAS ConfigDrive NoCloud AltCloud Azure Bigstep CloudSigma CloudStack DigitalOcean AliYun Ec2 GCE OpenNebula OpenStack OVF SmartOS Scaleway Hetzner IBMCloud
MODE=search
ON_FOUND=all
ON_MAYBE=all
ON_NOTFOUND=disabled
pid=1138 ppid=1133
is_container=false
ec2 platform is 'Unknown'.
No ds found [mode=search, notfound=disabled]. Disabled cloud-init [1]
[up 11.05s] returning 1

Which leads to:
slestemplate:~ # cat /run/cloud-init/cloud-init-generator.log 
/usr/lib/systemd/system-generators/cloud-init-generator normal=/run/systemd/generator early=/run/systemd/generator.early late=/run/systemd/generator.late
kernel command line (/proc/cmdline): BOOT_IMAGE=/boot/vmlinuz-4.4.73-5-default root=/dev/mapper/sys0-rootfs splash=silent quiet showopts console=tty0 console=ttyS0,115200
kernel_cmdline found unset
etc_file found unset
default found enabled
checking for datasource
ds-identify rc=1
ds-identify _RET=notfound
cloud-init is enabled but no datasource found, disabling
already disabled: no change needed [no /run/systemd/generator.early/multi-user.target.wants/cloud-init.target

However just using it again with the force option fixes the issue:

slestemplate:~ # /usr/lib/cloud-init/ds-identify --force
slestemplate:~ # cat /run/cloud-init/ds-identify.log 
[up 11.01s] ds-identify 
policy loaded: mode=search report=false found=all maybe=all notfound=disabled
no datasource_list found, using default: MAAS ConfigDrive NoCloud AltCloud Azure Bigstep CloudSigma CloudStack DigitalOcean AliYun Ec2 GCE OpenNebula OpenStack OVF SmartOS Scaleway Hetzner IBMCloud
ERROR: failed running [127]: blkid -c /dev/null -o export
DMI_PRODUCT_NAME=Standard PC (i440FX + PIIX, 1996)
DMI_SYS_VENDOR=QEMU
DMI_PRODUCT_SERIAL=
DMI_PRODUCT_UUID=0CE53BB3-A48E-44F7-9EC7-3C339E9C80D3
PID_1_PRODUCT_NAME=unavailable
DMI_CHASSIS_ASSET_TAG=
FS_LABELS=unavailable:error
ISO9660_DEVS=unavailable:error
KERNEL_CMDLINE=BOOT_IMAGE=/boot/vmlinuz-4.4.73-5-default root=/dev/mapper/sys0-rootfs splash=silent quiet showopts console=tty0 console=ttyS0,115200
VIRT=kvm
UNAME_KERNEL_NAME=Linux
UNAME_KERNEL_RELEASE=4.4.73-5-default
UNAME_KERNEL_VERSION=#1 SMP Tue Jul 4 15:33:39 UTC 2017 (b7ce4e4)
UNAME_MACHINE=x86_64
UNAME_NODENAME=slestemplate
UNAME_OPERATING_SYSTEM=GNU/Linux
DSNAME=
DSLIST=MAAS ConfigDrive NoCloud AltCloud Azure Bigstep CloudSigma CloudStack DigitalOcean AliYun Ec2 GCE OpenNebula OpenStack OVF SmartOS Scaleway Hetzner IBMCloud
MODE=search
ON_FOUND=all
ON_MAYBE=all
ON_NOTFOUND=disabled
pid=1138 ppid=1133
is_container=false
ec2 platform is 'Unknown'.
No ds found [mode=search, notfound=disabled]. Disabled cloud-init [1]
[up 11.05s] returning 1
[up 544.63s] ds-identify --force
policy loaded: mode=search report=false found=all maybe=all notfound=disabled
no datasource_list found, using default: MAAS ConfigDrive NoCloud AltCloud Azure Bigstep CloudSigma CloudStack DigitalOcean AliYun Ec2 GCE OpenNebula OpenStack OVF SmartOS Scaleway Hetzner IBMCloud
DMI_PRODUCT_NAME=Standard PC (i440FX + PIIX, 1996)
DMI_SYS_VENDOR=QEMU
DMI_PRODUCT_SERIAL=
DMI_PRODUCT_UUID=0CE53BB3-A48E-44F7-9EC7-3C339E9C80D3
PID_1_PRODUCT_NAME=unavailable
DMI_CHASSIS_ASSET_TAG=
FS_LABELS=cidata
ISO9660_DEVS=/dev/sr0=cidata
KERNEL_CMDLINE=BOOT_IMAGE=/boot/vmlinuz-4.4.73-5-default root=/dev/mapper/sys0-rootfs splash=silent quiet showopts console=tty0 console=ttyS0,115200
VIRT=kvm
UNAME_KERNEL_NAME=Linux
UNAME_KERNEL_RELEASE=4.4.73-5-default
UNAME_KERNEL_VERSION=#1 SMP Tue Jul 4 15:33:39 UTC 2017 (b7ce4e4)
UNAME_MACHINE=x86_64
UNAME_NODENAME=slestemplate
UNAME_OPERATING_SYSTEM=GNU/Linux
DSNAME=
DSLIST=MAAS ConfigDrive NoCloud AltCloud Azure Bigstep CloudSigma CloudStack DigitalOcean AliYun Ec2 GCE OpenNebula OpenStack OVF SmartOS Scaleway Hetzner IBMCloud
MODE=search
ON_FOUND=all
ON_MAYBE=all
ON_NOTFOUND=disabled
pid=2762 ppid=2314
is_container=false
check for 'NoCloud' returned found
ec2 platform is 'Unknown'.
Found single datasource: NoCloud
[up 544.70s] returning 0

I can also reproduce the behavior on the template VM just by rebooting it.

It appears that the initial check for NoCloud data source during boot fails. So the caching may not be the issue here. Retitling.

summary:

- ds-identify caches errors
+ ds-identify: fails to recognize NoCloud datasource on boot, ERROR:
+ failed running [127]: blkid -c /dev/null -o export

Revision history for this message

Martin Steigerwald (ms-proact) wrote on 2018-05-16: Re: ds-identify: fails to recognize NoCloud datasource on boot, ERROR: failed running [127]: blkid -c /dev/null -o export

#2

The command reliable works here on the fully booted VM:

slestemplate:~ # blkid -c /dev/null -o export >/dev/null ; echo $?
0

DEVNAME=/dev/sda1
UUID=[…]
TYPE=LVM2_member
PARTUUID=[…]

DEVNAME=/dev/sr0
UUID=2018-05-16-12-18-22-00
LABEL=cidata
TYPE=iso9660

DEVNAME=/dev/mapper/0QEMU_QEMU_HARDDISK_drive-scsi0
PTUUID=[…]
PTTYPE=dos

DEVNAME=/dev/mapper/0QEMU_QEMU_HARDDISK_drive-scsi0-part1
UUID=Tlko7f-sFXz-tgEn-3Pks-m652-XxVK-tl5dIB
TYPE=LVM2_member
PARTUUID=[…]

DEVNAME=/dev/mapper/sys0-rootfs
UUID=[…]
UUID_SUB=[…]
TYPE=btrfs

I think I am just going to hardcode the datasource now. Either via configuration file if possible or by hacking the shell script.

Revision history for this message

Martin Steigerwald (ms-proact) wrote on 2018-05-16:

#3

Download full text (4.9 KiB)

I ran blkid command 10 times in a row and always got return code 0. So I don´t get why:

    out=$(blkid -c /dev/null -o export) || {
        ret=$?
        error "failed running [$ret]: blkid -c /dev/null -o export"
        DI_FS_LABELS="$UNAVAILABLE:error"
        DI_ISO9660_DEVS="$UNAVAILABLE:error"
        return $ret
    }

is not working on boot. It looks like correct shell code. Only idea I have:

It is run to early for the ISO device to become available. Okay, testing for this with this change:

slestemplate:~ # git diff /usr/lib/cloud-init/ds-identify.orig /usr/lib/cloud-init/ds-identify
diff --git a/usr/lib/cloud-init/ds-identify.orig b/usr/lib/cloud-init/ds-identify
index 9a2db5c..2083734 100755
--- a/usr/lib/cloud-init/ds-identify.orig
+++ b/usr/lib/cloud-init/ds-identify
@@ -199,14 +199,24 @@ read_fs_info() {
         return
     fi
     local oifs="$IFS" line="" delim=","
- local ret=0 out="" labels="" dev="" label="" ftype="" isodevs="" uuids=""
- out=$(blkid -c /dev/null -o export) || {
+ local ret=1 out="" labels="" dev="" label="" ftype="" isodevs="" uuids=""
+ local attempt=1
+ while [ $ret -ne 0 -a $attempt -le 10 ]; do
+ out=$( blkid -c /dev/null -o export 2>&1 )
+ ret=$?
+ if [ $ret -ne 0 ]; then
+ error "failed running [$ret]: blkid -c /dev/null -o export, attempt: $attempt, output: $out"
+ sleep 2
+ fi
+ let attempt++;
+ done
+ if [ $ret -ne 0 ]; then
         ret=$?
         error "failed running [$ret]: blkid -c /dev/null -o export"
         DI_FS_LABELS="$UNAVAILABLE:error"
         DI_ISO9660_DEVS="$UNAVAILABLE:error"
         return $ret
- }
+ fi
     # 'set --' will collapse multiple consecutive entries in IFS for
     # whitespace characters (\n, tab, " ") so we cannot rely on getting
     # empty lines in "$@" below.

Which gets me:

slestemplate:~ # cat /run/cloud-init/ds-identify.log | head -7
[up 8.80s] ds-identify
policy loaded: mode=search report=false found=all maybe=all notfound=disabled
no datasource_list found, using default: MAAS ConfigDrive NoCloud AltCloud Azure Bigstep CloudSigma CloudStack DigitalOcean AliYun Ec2 GCE OpenNebula OpenStack OVF SmartOS Scaleway Hetzner IBMCloud
ERROR: failed running [127]: blkid -c /dev/null -o export, attempt: 1, output: /usr/lib/cloud-init/ds-identify: line 205: blkid: command not found
ERROR: failed running [127]: blkid -c /dev/null -o export, attempt: 2, output: /usr/lib/cloud-init/ds-identify: line 205: blkid: command not found
ERROR: failed running [127]: blkid -c /dev/null -o export, attempt: 3, output: /usr/lib/cloud-init/ds-identify: line 205: blkid: command not found
ERROR: failed running [127]: blkid -c /dev/null -o export, attempt: 4, output: /usr/lib/cloud-init/ds-identify: line 205: blkid: command not found

Which may just mean that during startup via Systemd

slestemplate:~ # type blkid
blkid is /sbin/blkid

is not in path.

And well now I learned from bash manpagethat is exactly what the bash error code tells me (Manpage: bash(1)):

       If a command is not found, the child process created
       to execute it returns a status of 127. If a command
       is foun...

I ran blkid command 10 times in a row and always got return code 0. So I don´t get why:

out=$(blkid -c /dev/null -o export) || {
        ret=$?
        error "failed running [$ret]: blkid -c /dev/null -o export"
        DI_FS_LABELS="$UNAVAILABLE:error"
        DI_ISO9660_DEVS="$UNAVAILABLE:error"
        return $ret
    }

is not working on boot. It looks like correct shell code. Only idea I have:

It is run to early for the ISO device to become available. Okay, testing for this with this change:

slestemplate:~ # git diff /usr/lib/cloud-init/ds-identify.orig /usr/lib/cloud-init/ds-identify
diff --git a/usr/lib/cloud-init/ds-identify.orig b/usr/lib/cloud-init/ds-identify
index 9a2db5c..2083734 100755
--- a/usr/lib/cloud-init/ds-identify.orig
+++ b/usr/lib/cloud-init/ds-identify
@@ -199,14 +199,24 @@ read_fs_info() {
         return
     fi
     local oifs="$IFS" line="" delim=","
-    local ret=0 out="" labels="" dev="" label="" ftype="" isodevs="" uuids=""
-    out=$(blkid -c /dev/null -o export) || {
+    local ret=1 out="" labels="" dev="" label="" ftype="" isodevs="" uuids=""
+    local attempt=1
+    while [ $ret -ne 0 -a $attempt -le 10 ]; do
+       out=$( blkid -c /dev/null -o export 2>&1 )
+       ret=$?
+       if [ $ret -ne 0 ]; then
+               error "failed running [$ret]: blkid -c /dev/null -o export, attempt: $attempt, output: $out"
+               sleep 2
+       fi
+       let attempt++;
+    done 
+    if [ $ret -ne 0 ]; then
         ret=$?
         error "failed running [$ret]: blkid -c /dev/null -o export"
         DI_FS_LABELS="$UNAVAILABLE:error"
         DI_ISO9660_DEVS="$UNAVAILABLE:error"
         return $ret
-    }
+    fi
     # 'set --' will collapse multiple consecutive entries in IFS for
     # whitespace characters (\n, tab, " ") so we cannot rely on getting
     # empty lines in "$@" below.

Which gets me:

slestemplate:~ # cat /run/cloud-init/ds-identify.log  | head -7
[up 8.80s] ds-identify 
policy loaded: mode=search report=false found=all maybe=all notfound=disabled
no datasource_list found, using default: MAAS ConfigDrive NoCloud AltCloud Azure Bigstep CloudSigma CloudStack DigitalOcean AliYun Ec2 GCE OpenNebula OpenStack OVF SmartOS Scaleway Hetzner IBMCloud
ERROR: failed running [127]: blkid -c /dev/null -o export, attempt: 1, output: /usr/lib/cloud-init/ds-identify: line 205: blkid: command not found
ERROR: failed running [127]: blkid -c /dev/null -o export, attempt: 2, output: /usr/lib/cloud-init/ds-identify: line 205: blkid: command not found
ERROR: failed running [127]: blkid -c /dev/null -o export, attempt: 3, output: /usr/lib/cloud-init/ds-identify: line 205: blkid: command not found
ERROR: failed running [127]: blkid -c /dev/null -o export, attempt: 4, output: /usr/lib/cloud-init/ds-identify: line 205: blkid: command not found

Which may just mean that during startup via Systemd

slestemplate:~ # type blkid
blkid is /sbin/blkid

is not in path.

And well now I learned from bash manpagethat is exactly what the bash error code tells me (Manpage: bash(1)):

If a command is not found, the child process created
       to execute it returns a status of 127.  If a command
       is found but is not executable, the return status is
       126.

But it does not seem that the systemd generator is being run on reboot, cause I added:

slestemplate:~ # diff -u cloud-init-generator.orig  /usr/lib/systemd/system-generators/cloud-init-generator
--- cloud-init-generator.orig   2018-05-16 13:08:41.302467498 +0200
+++ /usr/lib/systemd/system-generators/cloud-init-generator     2018-05-16 13:22:26.661939261 +0200
@@ -1,6 +1,8 @@
 #!/bin/sh
 set -f
 
+echo "PATH: $PATH" > /root/path
+
 LOG=""
 DEBUG_LEVEL=1
 LOG_D="/run/cloud-init"

at the beginning of it, yet got no output in /tmp, after reboot, while when running it manually I get the output. So it appears on reboot something else is calling it and this does not have /sbin in path.

I have no clue what else might be calling it:

slestemplate:/etc # grep -ir "ds-identify" .

slestemplate:/usr/lib/systemd # grep -ir "ds-identify"

only reports that system-generators/cloud-init-generator.

Also nothing in

slestemplate:/var # LANG=en grep -ir "ds-identify" .
Binary file ./lib/rpm/Packages matches
Binary file ./lib/rpm/Basenames matches
Binary file ./lib/mlocate/mlocate.db matches

So I am done with it for now and will just hardcode the path in ds-identify to /sbin/blkid.

And voila, this finally works. After a few dozens of attempts and reboots I finally at least have found the root cause and a work-around. I think to be really portable it ds-identify needs to try harder to find blkid, cause hard coding it to UsrMerge /usr/sbin/blkid is going to break on Debian and Ubuntu als long as UsrMerge is not done there. Or one might use /sbin/blkid at this is hard-linked on SLES 12 and RHEL 7 to /usr/sbin – and I bet these hardlinks better to be kept around for decades.

Gosh, this works. This finally works. Retitling again and adding patch.

summary:

- ds-identify: fails to recognize NoCloud datasource on boot, ERROR:
- failed running [127]: blkid -c /dev/null -o export
+ ds-identify: fails to recognize NoCloud datasource on boot cause it does
+ not have /sbin in $PATH and thus does not find blkid

Revision history for this message

Martin Steigerwald (ms-proact) wrote on 2018-05-16:

#4

I had 18.2-1.1.x86_64 but

http://download.opensuse.org/repositories/Cloud:/Tools/SLE_12_SP3/x86_64/cloud-init-18.2-2.1.x86_64.rpm

is also affected.

Revision history for this message

In bugzilla.opensuse.org/ #1093501, Martin Steigerwald (martin-steigerwald) wrote on 2018-05-16:

#6

Please see upstream bug report for all the details on this:

ds-identify: fails to recognize NoCloud datasource on boot cause it does not have /sbin in $PATH and thus does not find blkid
https://bugs.launchpad.net/cloud-init/+bug/1771382

Minimal patch to fix the issue:

slestemplate:~ # diff -u ds-identify.orig /usr/lib/cloud-init/ds-identify
--- ds-identify.orig 2018-05-16 13:34:06.376646777 +0200
+++ /usr/lib/cloud-init/ds-identify 2018-05-16 13:47:59.215541889 +0200
@@ -200,7 +200,7 @@
     fi
     local oifs="$IFS" line="" delim=","
     local ret=0 out="" labels="" dev="" label="" ftype="" isodevs="" uuids=""
- out=$(blkid -c /dev/null -o export) || {
+ out=$(/sbin/blkid -c /dev/null -o export) || {
         ret=$?
         error "failed running [$ret]: blkid -c /dev/null -o export"
         DI_FS_LABELS="$UNAVAILABLE:error"

Of course with UsrMerge you could also use /usr/sbin/blkid.

As stated in upstream bug report I have not the slightest idea what it calling ds-identify during boot. I thought it would be the systemd cloud-init generator, but I added debug output to it and it apparently is not called. For all the gory details see the upstream bug report.

Proper fix might be to make sure blkid is in $PATH.

Revision history for this message

In bugzilla.opensuse.org/ #1093501, Martin Steigerwald (martin-steigerwald) wrote on 2018-05-16:

#7

Created attachment 770432
minimal patch to add path to ds-identify

Revision history for this message

In bugzilla.opensuse.org/ #1093501, Martin Steigerwald (martin-steigerwald) wrote on 2018-05-16:

#8

Created attachment 770433
patch with additional debug output + reattempting

I first thought about blkid might be called to early in the boot time, but that is not true. 127 is bash´s return code for command not found. Still attaching it as a reference.

Revision history for this message

Martin Steigerwald (ms-proact) wrote on 2018-05-16:

#5

I have no idea how to attach patches as files here, but I attached them in downstream bug report:

https://bugzilla.opensuse.org/show_bug.cgi?id=1093501

Bug Watch Updater (bug-watch-updater) on 2018-05-16

Changed in cloud-init (openSUSE):
importance:	Unknown → Medium
status:	Unknown → Confirmed

Revision history for this message

In bugzilla.opensuse.org/ #1093501, Robert Schweikert (rjschwei) wrote on 2018-05-16:

#10

We will not take this patch.

At present it is not understood why PATH is not always part of the environment when the generator runs. This is being investigated.

For example the generator that spawns ds-identify also runs in the SUSE published images in AWS and there is no problem with finding blkid.

Revision history for this message

In bugzilla.opensuse.org/ #1093501, Martin Steigerwald (martin-steigerwald) wrote on 2018-05-16:

#11

(In reply to Robert Schweikert from comment #3)
> We will not take this patch.

Fair enough, I will use it nonetheless, cause with it I have it working *now*.

> At present it is not understood why PATH is not always part of the
> environment when the generator runs. This is being investigated.

Interesting. That SLES 12 image has some service pack migrations behind it already. Its an minimal image I installed and adapted myself for training purposes. It has only minimal adaptions in configuration:

These are the files I checked in to git repo – my changes are limited to these files (I did not adapt os-release and so on of course):

etc/SuSE-release
etc/cloud
etc/cloud/cloud.cfg
etc/cloud/cloud.cfg.d
etc/cloud/cloud.cfg.d/05_logging.cfg
etc/default
etc/default/grub
etc/dracut.conf.d
etc/dracut.conf.d/90-hostonly.conf
etc/fstab
etc/group
etc/hostname
etc/hosts
etc/os-release
etc/passwd
etc/resolv.conf
etc/screenrc
etc/snapper
etc/snapper/configs
etc/snapper/configs/root
etc/sysconfig
etc/sysconfig/network
etc/sysconfig/network/ifcfg-eth0
etc/sysconfig/network/routes
etc/zypp
etc/zypp/repos.d
etc/zypp/repos.d/Cloud_Tools.repo
etc/zypp/repos.d/SLES12-SP3_12.3-0.repo

Revision history for this message

Martin Steigerwald (ms-proact) wrote on 2018-05-16:

#9

Downstream is investigating why PATH is not always in the environment of cloud-init systemd generator. I never clearly noted: The OpenSUSE Build Service cloud-init 18.2 are still experimental.

Revision history for this message

In bugzilla.opensuse.org/ #1093501, Robert Schweikert (rjschwei) wrote on 2018-05-16:

#12

After receiving additional input and based on what was already known I have decided to drop the generator from our package.

The generator, which in turn runs ds-identify, where the problem is created by using blkid, speeds up the boot process in cases where cloud-init shold not be running in the first place. The chain of events is as follows:

generator runs ds-identify
if ds-identify cannot find a data source the cloud-init services are disabled

With cloud-init disabled the boot is sped up as no Python code gets executed.

A reasonable assumption is that the person installing and enabling cloud-init knows they run in environment where cloud-init is needed. Thus looking for the data source twice, once in ds-identify and then again in the cloud-init Python code is not really an advantage.

Dropping the generator avoids the problem with blkid and it avoids looking for the data source twice, once in shell code and once in Python code.

Change is on the way to Factory

Revision history for this message

In bugzilla.opensuse.org/ #1093501, Swamp-a (swamp-a) wrote on 2018-05-16:

#13

This is an autogenerated message for OBS integration:
This bug (1093501) was mentioned in
https://build.opensuse.org/request/show/609843 Factory / cloud-init

Bug Watch Updater (bug-watch-updater) on 2018-05-17

Changed in cloud-init (openSUSE):
status:	Confirmed → Fix Released

Revision history for this message

Scott Moser (smoser) wrote on 2018-05-17:

#14

For what its worth, the caching of ds-identify result is due to cloud-init-generator being called multiple times in a boot and thus ds-identify being called multiple times. We wanted to avoid 'blkid' calls to re-search disks during high IO process as boot.

we do have intent to make ds-identify more stand-alone useful. In doing that it would make sense to have the systemmd-generator use a "--respect-previous-run" or something and only then cache it.

Revision history for this message

Scott Moser (smoser) wrote on 2018-05-17:

#15

I've put a merge proposal up.
https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/345786

that will ensure that PATH is set to include common locations.

I really am interesetd in cloud-init doing the right thing in all cases, and thus I would like to have ds-identify enabled in suse and am willing to carry the change there so that we can assume a sane PATH. Even though I think a sane PATH should be set by the system rather than any program that expected to execute other programs.

Scott Moser (smoser) on 2018-05-18

Changed in cloud-init:
status:	New → Confirmed
importance:	Undecided → Low

Revision history for this message

In bugzilla.opensuse.org/ #1093501, Martin Steigerwald (ms-proact) wrote on 2018-05-22:

#18

Just removing the generator leads to cloud-init services not being run at startup on my SLES 12 SP 3 VM with

slestemplate:~ # find /etc/systemd | grep cloud
/etc/systemd/system/cloud-init.target.wants
/etc/systemd/system/cloud-init.target.wants/cloud-final.service
/etc/systemd/system/cloud-init.target.wants/cloud-config.service
/etc/systemd/system/cloud-init.target.wants/cloud-init-local.service
/etc/systemd/system/cloud-init.target.wants/cloud-init.service

All services are enabled according to systemctl status SERVICE

slestemplate:~ # rpm -qa | grep cloud-init
cloud-init-config-suse-18.2-3.1.x86_64
cloud-init-18.2-3.1.x86_64

slestemplate:~ # systemctl status cloud-init.target
● cloud-init.target - Cloud-init target
   Loaded: loaded (/usr/lib/systemd/system/cloud-init.target; static; vendor preset: disabled)
   Active: inactive (dead)
slestemplate:~ # systemctl status cloud-config.target
● cloud-config.target - Cloud-config availability
   Loaded: loaded (/usr/lib/systemd/system/cloud-config.target; static; vendor preset: disabled)
   Active: inactive (dead)

The output I get is:

slestemplate:~ # systemctl |grep cloud

When I do:

slestemplate:~ # systemctl start cloud-init.target
slestemplate:~ # systemctl | grep cloud
cloud-config.service loaded active exited Apply the settings specified in cloud-config
cloud-final.service loaded active exited Execute cloud user/final scripts
cloud-init-local.service loaded active exited Initial cloud-init job (pre-networking)
cloud-init.service loaded active exited Initial cloud-init job (metadata service crawler)
cloud-config.target loaded active active Cloud-config availability
cloud-init.target loaded active active Cloud-init target

So it is still not working out of the box. I enabled all services and the targets with systemctl enable. It is not obvious for me to enable cloud init in case the Systemd generator does not do it. I thought I did, but apparently I did not.

Just removing the generator leads to cloud-init services not being run at startup on my SLES 12 SP 3 VM with

slestemplate:~ # find /etc/systemd | grep cloud
/etc/systemd/system/cloud-init.target.wants
/etc/systemd/system/cloud-init.target.wants/cloud-final.service
/etc/systemd/system/cloud-init.target.wants/cloud-config.service
/etc/systemd/system/cloud-init.target.wants/cloud-init-local.service
/etc/systemd/system/cloud-init.target.wants/cloud-init.service

All services are enabled according to systemctl status SERVICE

slestemplate:~ # rpm -qa | grep cloud-init
cloud-init-config-suse-18.2-3.1.x86_64
cloud-init-18.2-3.1.x86_64

slestemplate:~ # systemctl status cloud-init.target
● cloud-init.target - Cloud-init target
   Loaded: loaded (/usr/lib/systemd/system/cloud-init.target; static; vendor preset: disabled)
   Active: inactive (dead)
slestemplate:~ # systemctl status cloud-config.target
● cloud-config.target - Cloud-config availability
   Loaded: loaded (/usr/lib/systemd/system/cloud-config.target; static; vendor preset: disabled)
   Active: inactive (dead)

The output I get is:

slestemplate:~ # systemctl  |grep cloud

When I do:

slestemplate:~ # systemctl start cloud-init.target
slestemplate:~ # systemctl | grep cloud
cloud-config.service                                                                        loaded active exited    Apply the settings specified in cloud-config
cloud-final.service                                                                         loaded active exited    Execute cloud user/final scripts
cloud-init-local.service                                                                    loaded active exited    Initial cloud-init job (pre-networking)
cloud-init.service                                                                          loaded active exited    Initial cloud-init job (metadata service crawler)
cloud-config.target                                                                         loaded active active    Cloud-config availability
cloud-init.target                                                                           loaded active active    Cloud-init target

So it is still not working out of the box. I enabled all services and the targets with systemctl enable. It is not obvious for me to enable cloud init in case the Systemd generator does not do it. I thought I did, but apparently I did not.

Revision history for this message

In bugzilla.opensuse.org/ #1093501, Martin Steigerwald (ms-proact) wrote on 2018-05-22:

#19

Also please note that according

https://bugs.launchpad.net/cloud-init/+bug/1771382/comments/15

upstream developer Scott Moser would like to see cloud-init doing the sane thing in all cases and added a merge proposal for setting the PATH to common locations. He would like to see ds-identify enabled in SUSE.

Revision history for this message

Martin Steigerwald (ms-proact) wrote on 2018-05-22:

#16

Scott, I mentioned your comment about having a sane PATH everywhere and ds-identify enabled in SUSE at the SUSE bugtracker. Their change to just remove the generator did not yield the expected result on my SLES 12 SP 3 VM. Cloud Init is simply not started at all then. Reported there.

Revision history for this message

In bugzilla.opensuse.org/ #1093501, Martin Steigerwald (ms-proact) wrote on 2018-05-22:

#20

(In reply to Martin Steigerwald from comment #8)
> Just removing the generator leads to cloud-init services not being run at
> startup on my SLES 12 SP 3 VM with
>
> slestemplate:~ # find /etc/systemd | grep cloud
> /etc/systemd/system/cloud-init.target.wants
> /etc/systemd/system/cloud-init.target.wants/cloud-final.service
> /etc/systemd/system/cloud-init.target.wants/cloud-config.service
> /etc/systemd/system/cloud-init.target.wants/cloud-init-local.service
> /etc/systemd/system/cloud-init.target.wants/cloud-init.service
[…]
> slestemplate:~ # systemctl status cloud-init.target
> ● cloud-init.target - Cloud-init target
> Loaded: loaded (/usr/lib/systemd/system/cloud-init.target; static; vendor
> preset: disabled)
> Active: inactive (dead)
[…]
> So it is still not working out of the box. I enabled all services and the
> targets with systemctl enable. It is not obvious for me to enable cloud init
> in case the Systemd generator does not do it. I thought I did, but
> apparently I did not.

With the work-around

slestemplate:/etc/systemd/system # mv cloud-init.target.wants/* multi-user.target.wants/

cloud-init is started on boot.

Seems cloud-init.target is never triggered. I am not really experienced with targets in Systemd.

Revision history for this message

In bugzilla.opensuse.org/ #1093501, Robert Schweikert (rjschwei) wrote on 2018-05-22:

#21

- Please set the datasource in your cloud.cfg file
- Disable the target
- Enable the services

cloud.cfg should contain something along these lines:

datasource_list: [ NoCloud,......, None ]

systemctl disable cloud-init.target
systemctl enable cloud-init-local
systemctl enable cloud-init
systemctl enable cloud-config
systemctl enable cloud-final

or if you are building images with kiwi add the following to config.sh

suseInsertService cloud-init-local
suseInsertService cloud-init
suseInsertService cloud-config
suseInsertService cloud-final

For the next version 18.3 I expect a better solution as upstream has a pending patch for the PATH issue

Revision history for this message

Scott Moser (smoser) wrote on 2018-05-22:

#17

An upstream commit landed for this bug.

To view that commit see the following URL:
https://git.launchpad.net/cloud-init/commit/?id=b4ae0e1f

Changed in cloud-init:
status:	Confirmed → Fix Committed

Bug Watch Updater (bug-watch-updater) on 2018-05-22

Changed in cloud-init (openSUSE):
status:	Fix Released → Confirmed

Revision history for this message

In bugzilla.opensuse.org/ #1093501, Robert Schweikert (rjschwei) wrote on 2018-05-22:

#22

OK, this is clearly putting the burden on the user and not really what we want to do. I've pulled the upstream patch to address the PATH issue. New cloud-init on it's way to Factory and available in Cloud:Tools

Revision history for this message

In bugzilla.opensuse.org/ #1093501, Swamp-a (swamp-a) wrote on 2018-05-22:

#23

This is an autogenerated message for OBS integration:
This bug (1093501) was mentioned in
https://build.opensuse.org/request/show/611409 Factory / cloud-init

Bug Watch Updater (bug-watch-updater) on 2018-05-23

Changed in cloud-init (openSUSE):
status:	Confirmed → Fix Released

Revision history for this message

In bugzilla.opensuse.org/ #1093501, Swamp-a (swamp-a) wrote on 2018-05-23:

#25

openSUSE-RU-2018:1407-1: An update that has two recommended fixes can now be installed.

Category: recommended (moderate)
Bug References: 1089824,1093501
CVE References:
Sources used:
openSUSE Leap 15.0 (src): cloud-init-18.2-lp150.2.3.1

Revision history for this message

In bugzilla.opensuse.org/ #1093501, Martin Steigerwald (ms-proact) wrote on 2018-05-24:

#26

(In reply to Robert Schweikert from comment #12)
> OK, this is clearly putting the burden on the user and not really what we
> want to do. I've pulled the upstream patch to address the PATH issue. New
> cloud-init on it's way to Factory and available in Cloud:Tools

Working out of the box with:

slestemplate:~ # rpm -qa | grep cloud
cloud-init-18.2-4.1.x86_64
cloud-init-config-suse-18.2-4.1.x86_64

Thank you, Robert.

Revision history for this message

Martin Steigerwald (ms-proact) wrote on 2018-05-24:

#24

Fix confirmed to work with:

slestemplate:~ # rpm -qa | grep cloud
cloud-init-18.2-4.1.x86_64
cloud-init-config-suse-18.2-4.1.x86_64

Thank you, Scott.

Revision history for this message

In bugzilla.opensuse.org/ #1093501, Swamp-a (swamp-a) wrote on 2018-06-05:

#27

This is an autogenerated message for OBS integration:
This bug (1093501) was mentioned in
https://build.opensuse.org/request/show/614273 15.0 / cloud-init

Revision history for this message

In bugzilla.opensuse.org/ #1093501, Swamp-a (swamp-a) wrote on 2018-06-07:

#28

SUSE-RU-2018:1575-1: An update that has 10 recommended fixes can now be installed.

Category: recommended (moderate)
Bug References: 1069635,1072811,1080595,1084509,1084749,1085787,1089824,1092637,1093501,997614
CVE References:
Sources used:
SUSE Linux Enterprise Module for Public Cloud 12 (src): cloud-init-18.2-37.14.1
SUSE CaaS Platform ALL (src): cloud-init-18.2-37.14.1
OpenStack Cloud Magnum Orchestration 7 (src): cloud-init-18.2-37.14.1

Revision history for this message

In bugzilla.opensuse.org/ #1093501, Swamp-a (swamp-a) wrote on 2018-06-08:

#29

openSUSE-RU-2018:1609-1: An update that has 10 recommended fixes can now be installed.

Category: recommended (moderate)
Bug References: 1069635,1072811,1080595,1084509,1084749,1085787,1089824,1092637,1093501,997614
CVE References:
Sources used:
openSUSE Leap 42.3 (src): cloud-init-18.2-37.1

Revision history for this message

In bugzilla.opensuse.org/ #1093501, Swamp-a (swamp-a) wrote on 2018-06-08:

#30

openSUSE-RU-2018:1613-1: An update that has two recommended fixes can now be installed.

Category: recommended (moderate)
Bug References: 1089824,1093501
CVE References:
Sources used:
openSUSE Leap 15.0 (src): cloud-init-18.2-lp150.2.6.1

Revision history for this message

Scott Moser (smoser) wrote on 2018-06-20: Fixed in cloud-init version 18.3.

#31

This bug is believed to be fixed in cloud-init in version 18.3. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

Changed in cloud-init:
status:	Fix Committed → Fix Released

Revision history for this message

Dimitri John Ledkov (xnox) wrote on 2018-09-11:

#32

Systemd by default executes things, with execv, not execve. Hence the default environment is not available. However, cloud-init generator is executed by /bin/sh which does that a built-in default path

$ lxc launch images:opensuse/15.0 test-sh-built-in-path

$ lxc exec test-sh-built-in-path -- env -u PATH /bin/sh -c 'echo $PATH'
/usr/local/bin:/usr/bin:/bin:.

On ubuntu, it is instead:

$ env -u PATH /bin/dash -c 'echo $PATH'
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

$ env -u PATH /bin/bash -c 'echo $PATH'
/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:.

Maybe you want to report a bug against Suse's default /bin/sh about this....

Also /bin/dash and /bin/bash differences are akward....

Revision history for this message

In bugzilla.opensuse.org/ #1093501, Dimitri John Ledkov (xnox) wrote on 2018-09-11:

#33

Systemd by default executes things, with execv, not execve. Hence the default environment is not available. However, cloud-init generator is executed by /bin/sh which does has a built-in default path

$ lxc launch images:opensuse/15.0 test-sh-built-in-path

$ lxc exec test-sh-built-in-path -- env -u PATH /bin/sh -c 'echo $PATH'
/usr/local/bin:/usr/bin:/bin:.

No idea if it is intentional, or not, that "sbin" is excluded there.

Scott Moser (smoser) on 2018-09-11

description:

updated

Revision history for this message

James Falcon (falcojr) wrote on 2023-05-11:

#34

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3182

Affects		Status	Importance	Assigned to	Milestone
	cloud-init	Fix Released	Low	Unassigned
	cloud-init (openSUSE)	Fix Released	Medium	auto-bugzilla.opensuse.org #1093501

cloud-init

ds-identify: fails to recognize NoCloud datasource on boot cause it does not have /sbin in $PATH and thus does not find blkid

Bug Description

Related branches

Other bug subscribers

Remote bug watches