ipxe fails on Centos 7 (inc: command not found)

Bug #1507738 reported by Dan Prince
22
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Ironic
Fix Released
High
Dmitry Tantsur
tripleo
Invalid
Undecided
Unassigned

Bug Description

Tried to use the latest Ironic packages with an instack undercloud today (on Centos 7). Undercloud instances then failed to boot. I attached to the console and saw the following boot messages:

http://172.19.0.3/8088/boot.ipxe... ok
inc: command not found
Could not boot image: Exec format error (http://ipxe.org/2e022001)

----

This looks to be related to the recent boot.ipxe change here:

http://git.openstack.org/cgit/openstack/ironic/commit/?id=1c28eb79a5d1b25fb685e2f48449c51e905766b5

Reverting that change seems to resolve the issue on Centos 7.

My guess is that Centos 7 has a version of /usr/share/ipxe/undionly.kpxe that is now too old to work w/ Ironic. Users would need to copy one over from Fedora or build a new one from ipxe source to use the latest Ironic code again.

Dan Prince (dan-prince)
Changed in tripleo:
importance: Undecided → Critical
status: New → Triaged
Changed in ironic:
status: New → Triaged
importance: Undecided → High
Changed in ironic:
assignee: nobody → Lucas Alvares Gomes (lucasagomes)
Revision history for this message
Dmitry Tantsur (divius) wrote :

My take is that we either make it configurable or politely ask CentOS to update :) reverting the original patch will break some other users..

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ironic (master)

Fix proposed to branch: master
Review: https://review.openstack.org/244530

Changed in ironic:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/250745

Changed in ironic:
assignee: Lucas Alvares Gomes (lucasagomes) → Dmitry Tantsur (divius)
Dmitry Tantsur (divius)
tags: added: liberty-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ironic (master)

Reviewed: https://review.openstack.org/250745
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=f968e1a22d3ed928278a04f0da9124678474ff0a
Submitter: Jenkins
Branch: master

commit f968e1a22d3ed928278a04f0da9124678474ff0a
Author: Dmitry Tantsur <email address hidden>
Date: Fri Nov 27 11:59:32 2015 +0100

    Fall back to old boot.ipxe behaviour if inc command is not found

    We introduced looping over all NIC's as on older iPXE ROM's ${mac} variable
    does not always point to the PXE-booting NIC. However, CentOS/RHEL ship
    even older iPXE ROM that does not even have inc command at all.

    This patch restores the old behaviour of relying on ${mac} variable when
    inc command cannot be executed. It's still not perfect and might break
    sometimes, but it won't be completely broken. Nothing is changes for users
    for which inc command is already working.

    Change-Id: I30b8c0e4ab076684ea9baa042e5a9a4f44da9a3a
    Closes-Bug: #1507738

Changed in ironic:
status: In Progress → Fix Committed
Changed in ironic:
status: Fix Committed → Fix Released
Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/ironic 4.3.0

This issue was fixed in the openstack/ironic 4.3.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ironic (stable/liberty)

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/254759

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on ironic (master)

Change abandoned by Lucas Alvares Gomes (<email address hidden>) on branch: master
Review: https://review.openstack.org/244530
Reason: We've fixed it another way

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ironic (stable/liberty)

Reviewed: https://review.openstack.org/254759
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=35c7f81fa80c44e22d660d0d57f2419430408f43
Submitter: Jenkins
Branch: stable/liberty

commit 35c7f81fa80c44e22d660d0d57f2419430408f43
Author: Lucas Alvares Gomes <email address hidden>
Date: Fri Oct 9 11:26:13 2015 +0100

    iPXE: Fix assumption that ${mac} is the MAC of the NIC it's booting

    The ${mac} variable not necessarily points to the MAC address of the NIC
    we are booting from. This patch updates the boot.ipxe script to make it
    more robust by interating over all NICs found in the machine and boot
    from the first one capable of booting (the one that contains a
    configuration file in the server). If no NICs are capable of booting the
    script will fail just like before telling the user that the network boot
    failed because we couldn't find any configuration for the present NICs.

    In case the inc command is not found, the boot.ipxe will fall back to
    the old behavior.

    This backport is the combination of 2 commits (squashed):

    (cherry picked from commit 1c28eb79a5d1b25fb685e2f48449c51e905766b5)
    (cherry picked from commit f968e1a22d3ed928278a04f0da9124678474ff0a)

    We need to squash both patches because they must merge at the same time
    to avoid breakages.

    Closes-Bug: #1507738
    Closes-Bug: #1504482
    Change-Id: I5b8f0740fa42f611e84b66dc7f6578cffe02760a

tags: added: in-stable-liberty
Revision history for this message
Steven Hardy (shardy) wrote :

I don't think there's anything to fix on the TripleO side here, consuming the latest ironic gets us the required fix, so marking the tripleo bug invalid.

Changed in tripleo:
status: Triaged → Invalid
importance: Critical → Undecided
Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/ironic 4.2.3

This issue was fixed in the openstack/ironic 4.2.3 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.