iSCSI installation on root too complicated

Can you check whether this works in 9.10? This sounds a bit like something that might have been caused by bug 457767, and I'd like to find out what's still broken.

I tried this with 9.10 server amd64 final ISO, but the problems stay:

=========

PROBLEM #1: no suitable initrd.img file is provided that allows for iSCSI-on-root booting, though it is possible to create one after installing 9.10 to a VM for example and create a custom initrd.img file, like i did.

mkdir ~/tmp
cd ~/tmp
gunzip < /boot/initrd.img-2.6.31-14-server | cpio -i --make-directories

touch etc/iscsi.initramfs
cp -p /lib/modules/2.6.31-14-server/kernel/crypto/crc32c.ko lib/modules/2.6.31-14-server/kernel/crypto/
cp -p /lib/modules/2.6.31-14-server/kernel/lib/libcrc32c.ko lib/modules/2.6.31-14-server/kernel/lib/
cp -p /sbin/iscsid sbin/
cp -p /sbin/iscsistart sbin/

find ./ | cpio -H newc -o > ~/initrd-iscsi.cpio
gzip ~/initrd-iscsi.cpio
mv ~/initrd-iscsi.cpio.gz ~/initrd-iscsi.img
// now move the initrd-iscsi.img in your home directory to the TFTP server, and make sure it is used by pxelinux.cfg config file

note 1) this procedure has to be performed each time the initrd.img file is updated by the Ubuntu update system; otherwise it would not receive the updates the update system thinks its making. The local /boot/initrd.img file would not be used; only the initrd.img file provided by the TFTP server is used.

note 2) the etc/iscsi.initramfs file needs to exist, even though being an empty file. Not creating this file would cause the pxelinux.cfg iscsi login data to not be used. If you prefer getting the iSCSI authentication data from pxelinux.cfg file like i did, simply create an empty file by using the touch command in the above example. Putting the authentication data in the /etc/iscsi.initramfs file instead might be more secure, however. But only barrely as this is unencrypted and located on the TFTP server as well; which does not provide for any authentication by itself.

=========

PROBLEM #2: after having a suitable initrd.img file, iSCSI booting works. But upon rebooting or shutting down, the rc.d shutdown script configuration causes a bug that shuts down networking before the (iscsi-mounted) root filesytem is unmounted, causing an unclean shutdown with possible dataloss/corruption. It also requires a manual reset to continue. To fix this, i used the following commands:

cd /etc/rc0.d/
rm S35networking
mv S41open-iscsi S80open-iscsi
cd /etc/rc6.d/
rm S35networking
mv S41open-iscsi S80open-iscsi

Note 1) the file S41open-iscsi may be called differently; i do not remember 100% correctly; but anyway the open-iscsi file in the /etc/rc0.d/ and /etc/rc6.d/ directories have to called S80open-iscsi to allow clean shutdown
Note 2) removing the S35networking may have side effects i'm unaware of; it does seem to work without issues for me. I don't really understand why the network interface has to be "down" before shutting down anyway. The only thing important is that the root filesystem is unmounted cleanly, and the network interface has to be up long enough to make that happen. Removing the network shutdown script seemed the most simple solution to me.

=========

After coping with these two problems, diskless Ubuntu systems work. To get a desktop, i used the following commands:
sudo apt-get install ubuntu-desktop
reboot

Hope this helps ...

Could you tell me whether /etc/iscsi/iscsi.initramfs exists, and if so what it contains? Furthermore, could you attach /var/log/installer/syslog?

By the way, the reason I'm asking these questions is that the installer is *already* supposed to set things up such that the initrd is generated with iSCSI support if the root filesystem is on iSCSI. (We don't do it by default because we have very aggressive boot time targets and every additional chunk in the initramfs slows things down; in any case, there's no point putting iSCSI support in the initramfs unless the configuration is also in place, and you're seeing this bug precisely because the configuration is not in place for some reason.) You're phrasing your comments in terms of a feature request; what I'm saying is that the feature is already present, but for some reason it is not working for you, hence my requests for specific files.

I agree that the networking setup is wrong. The least invasive fix for this will be to extend the check for network filesystems that's already in that script to detect root-on-iSCSI as well.

Could you file a separate bug report with your TFTP suggestions, please? We'd have to think about them in some more depth, and they're not specific to iSCSI.

Sadly, iSCSI support currently won't fit on the desktop CD, and is unlikely to be higher priority than other things the desktop team want to do ...

This bug was fixed in the package netbase - 4.35ubuntu3

---------------
netbase (4.35ubuntu3) lucid; urgency=low

  * debian/netbase.init:
    - Don't deconfigure network interfaces if /etc/iscsi/iscsi.initramfs
      exists, since that probably indicates root on iSCSI (LP: #447377).
 -- Colin Watson <email address hidden> Thu, 03 Dec 2009 16:05:30 +0000

sub.mesa: Any progress on getting me the files with more information that I requested earlier? I'd like to make sure that this is cleaned up for Lucid, but I will need that information to do so. Thanks in advance.

@Colin: I don't know if sub.mesa already has solved this, but I encountered just the same frustration and problems als he did :-)

I think the most frustrating part is, that there is no documentation or any note in the installer regarding iSCSI on root installations, other than that it CAN and DOES install the system on an iSCSI-based root filesystem, but no instructions or actions on where to find an initramfs image that can be used to make mounting the iSCSI volume doing the boot process possible :). In other words: The Ubuntu 9,10 release notes state explicitly iSCSI support has been improved and installation of root on iSCSI is possible, but it is practically not useable without a lot of hassle.

In an earlier response to this bug, you mention a file called "/etc/iscsi/iscsi.initramfs". When I get home and have the time (but ASAP) I will see if this file exists on the newly-installed system .Is this file useable as a ready to use initrd or do you still have to do something to create one? But even if it exists, there still is no easy way to get this distributed to a boot medium.

Scenario:
* Instantiate a Virtual Machine (eg. in VirtualBox) with no local storage attached
* Boot the Ubuntu server 9.10 ISO
* During installation, login to an iSCSI target and let the system install to it.
* After installation, the system wants to reboot

This is where the problem start: The system reboots and you have no way of reaching any initramfs, other than mounting the root filesystem on the iSCSI target from another system and (worst case) chroot to it if creating an initrd is still needed.

In my opinion (and I think many people will agree), the installation scenario should continue:

* After installation, an extra step is performed, in which the system will create the right initrd needed to boot this particular system (with the modules of the NIC that was in use during iSCSI login for installation, with the iSCSI modules and login procedure, anything to be able to mount the root FS).
* An option is given to SCP the newly created initrd and kernel to another system, or even put it on a USB key.

Any thoughts on this? :-). I am happy to test any howto/procedure you can give me, or a pointer to a place sub.mesa or I might not have searched for documentation.

@Colin Watson:

OK, just looked up the "etc/iscsi/iscsi.initramfs" file and it only shows 4 lines containing the details of my iSCSI target (target name, target IP, etc.).

Also, the initramfs configuration does not show anything at all that might indicate instructions to the mkinitramfs command to include the right iSCSI setup procedure in the initrd-stage.... :(

I also included the installer syslog :)

Colin, my apologies for my late response. I found some time to play with iSCSI again and used a fresh setup with final Ubuntu 9.10 server, without much issues.

The problems i had appear to originate from the date Ubuntu 9.10 was still in RC-phase; even after upgrading them as the final came out. I just setup a completely new Ubuntu 9.10 server final iscsi install and the initrd.img and vmlinuz written to the iscsi filesystem are indeed usable and work for me. The problem as eymey explained, is getting that initrd.img and vmlinuz (2 files) to the TFTP server used for booting. I'll file a separate bug on that, as you requested.

I simply used an initrd i already had to boot the system, then copied the /vmlinuz and /initrd.img files (which are symlinks and kept up-to-date to the currently installed kernel/initrd.img) to the TFTP server which pxelinux reads before the kernel is even started. An interum workaround would be to put the initial two files of a blank installation somewhere on the ftp server. I don't know about the 'dynamic' creation of that /etc/iscsi/iscsi.initramfs file though. I booted with an initrd from a different system, which would need to have different data in that file, and it still worked. Possibly because its dynamically generated when booting as it gets the data from pxelinux.cfg file which gives all iscsi information, snippet:

  APPEND ip=bootp root=/dev/sda1 iscsi_initiator=X iscsi_target_name=X iscsi_target_ip=X iscsi_target_group=1 iscsi_target_port=3260 iscsi_username=X iscsi_password=X initrd=/tftp/initrd.img
(removed actual data by X for privacy)

I believe this data is used to generate the /etc/iscsi/iscsi.initramfs file dynamicaly/automatically when the script activates upon boot.

I then upgraded using sudo apt-get update followed by sudo apt-get dist-upgrade, and could copy the 2 files to my remote server again, and booting worked fine. I created a small script i could run everytime after installing ubuntu updates that involve kernel/initrd changes. I simply run ~/updateiscsi and its copied to my remote server. So, i do not have to generate an initrd each time after upgrading; thus the auto-generated initrd.img is fine when using ubuntu 9.10 final.

So while it works, there are lots of rough edges:
- How to initially boot after installing; you need the initrd.img and vmlinuz files on the iscsi system but you can't get to that easily.
- After updating you need to manually synchronise your vmlinuz and initrd.img files to the TFTP server used for booting
- /etc/rc0.d and /etc/rc6.d hacks are still required for clean shutdown on iscsi-on-root systems
- /etc/network/interfaces must be manually created and set to static ip configuration; it doesn't appear to pickup the gateway information from the DHCP server initally
- If any local disks are present which you may or may not want to use as additional storage - not as root/system filesystem for Ubuntu - you need to change the pxelinux.cfg file to set root=/dev/sdb1 (for 1 local disk present) or to root=/dev/sdc1 (2 local disks present). Using UUIDs here probably won't work. Ideally, we would want a script that automatically checks which of the /dev/ entries is an i...

mercedes, please don't close bugs (a) without comment (b) without discussion with the responsible developer (c) when they're not actually really all-the-way fixed ...