deploy-ironic ramdisks do not boot: Failed to execute /init (error -28)

The trusty ramdisk fails to boot on a VM /w 512MB of RAM. Increasing that to 1GB fixes the issue, tho IMO the ramdisk release shouldn't impose such memory requirements.

After a little experimenting, this might point to a kernel or qemu issue of some kind:

* the resulting trusty initramfs boots fine using a kernel saucy 3.12 kernel (using either a kernel resulting from a DIB run or a stock kernel from /boot).

*a saucy initramfs built with the same elements fails /w the same -28 error using a trusty kernel (from /w DIB run or stock).

* if I bump the VM's memory from 512MB to 620MB (arbitrary?) the kernel successfully execs init

I've uploaded a compressed initramfs that triggers this to http://no-carrier.net/~adam/lp1311987/ I'm able to reproduce by simply booting it with qemu using the standard Ubuntu trusty vmlinuz-3.13.0-24-generic kernel.

Worth noting that the corresponding stock initramfs initrd.img-3.13.0-24-generic has no problems booting in a VM with 512MB

Setting to medium for now because 512 M instances seem like kind of an edge case, but if this ends up being a blocker for moving to trusty then it should be escalated. Although it's not clear that this is a bug in dib, so we may want to open a high priority bug for whatever the broken component actually is in that case.

This turns out to be a kernel issue, either upstream or in Ubuntu. Bisected to the following, which was released in 3.12.

commit 16203a7a9422315bc929461503e3a046459ea5ff
Author: Rob Landley <email address hidden>
Date: Wed Sep 11 14:26:12 2013 -0700

    initmpfs: make rootfs use tmpfs when CONFIG_TMPFS enabled

    Conditionally call the appropriate fs_init function and fill_super
    functions. Add a use once guard to shmem_init() to simply succeed on a
    second call.

    (Note that IS_ENABLED() is a compile time constant so dead code
    elimination removes unused function calls when CONFIG_TMPFS is disabled.)

I agree this is an edge case for tripleO but not so much for general Ironic usage, as admins should be free to declare nodes with as little memory as they want, and DIB produced deploy-ironic ramdisks should be expected to work.

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1311987

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

So there's a later commit which adds a command-line option to force use of ramfs:

commit 6e19eded3684dc184181093af3bff2ff440f5b53
Author: Rob Landley <email address hidden>
Date: Wed Sep 11 14:26:13 2013 -0700

    initmpfs: use initramfs if rootfstype= or root= specified

    Command line option rootfstype=ramfs to obtain old initramfs behavior, and
    use ramfs instead of tmpfs for stub when root= defined (for cosmetic
    reasons).

Can you try adding rootfstype=ramfs to the kernel parameters and see if this works? Would you be able to do this for cases where you really need ramfs?

Thanks for the tip Seth.. That appears to work, thanks! I reached out to Rob and he pointed out that, when using tmpfs, the fs will fill up at 50% memory usage, so the ENOSPC is to be expected here. Now that the cause is known, I think we can get away with bumping the default and minimum required memory to 1GB for our ramdisks and we should be okay across the different kernel versions / rootfstypes.

Reviewed: https://review.openstack.org/90821
Committed: https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=74eafa4b49dcf2f76b7627739d26e457fa1775db
Submitter: Jenkins
Branch: master

commit 74eafa4b49dcf2f76b7627739d26e457fa1775db
Author: Adam Gandelman <email address hidden>
Date: Mon Apr 28 14:06:42 2014 -0700

    Bump minimum default RAM for Ironic nodes to 1GB

    Set the default RAM for Ironic BM VMs to 1GB to ensure DIB produced deployment
    ramdisks function correctly across older and newer kernels, regardless of
    rootfstype.

    Change-Id: Ifc15a4430aa63c3599e7fd76f5116dc7b728ead0
    Closes-bug: #1311987

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/100698

Reviewed: https://review.openstack.org/100698
Committed: https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=cd1cf1b41753a0d05a814f2a7817522b7f7e7b91
Submitter: Jenkins
Branch: stable/icehouse

commit cd1cf1b41753a0d05a814f2a7817522b7f7e7b91
Author: Adam Gandelman <email address hidden>
Date: Mon Apr 28 14:06:42 2014 -0700

    Bump minimum default RAM for Ironic nodes to 1GB

    Set the default RAM for Ironic BM VMs to 1GB to ensure DIB produced deployment
    ramdisks function correctly across older and newer kernels, regardless of
    rootfstype.

    Change-Id: Ifc15a4430aa63c3599e7fd76f5116dc7b728ead0
    Closes-bug: #1311987
    (cherry picked from commit 74eafa4b49dcf2f76b7627739d26e457fa1775db)

help me to problem init not taited 3.13.0-39-generic #66-ubuntu.... boot cd I can not work with the actual. when going to reinstall will always appear (init). I also can not get into Ubuntu ...

subair, Can you open a new bug with a full description of the problem you are having, including the steps you took to get there so we can reproduce? This one is pretty specific to booting the deploy ramdisk on machines with less than 1GB of RAM.

THANKS!