If CONFIG_NO_HZ_FULL is selected in RT 3.10 kernel, udhcpc process hangs while attempting to obtain IP address and boot sequence fails to complete.

Bug #1215507 reported by Gary S. Robertson
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linaro-networking
Fix Released
High
Gary S. Robertson

Bug Description

In kernel version 3.10.6 with PREEMPT_RT_FULL patch set 3.10.6.rt3 applied, if CONFIG_NO_HZ_FULL=Y the udhcpc client process hangs while attempting to obtain an IP address, and the boot startup sequence fails to complete. The same kernel without PREEMPT_RT_FULL patches applied & selected is able to boot successfully with CONFIG_NO_HZ_FULL=Y, and the PREEMPT_RT_FULL-enabled kernel boots successfully if CONFIG_NO_HZ_FULL=N.

Revision history for this message
Gary S. Robertson (gary-robertson) wrote :

Here is the log output from the hangup:
[ 237.449129] INFO: task udhcpc:1554 blocked for more than 120 seconds.
[ 237.454040] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 237.461730] udhcpc D 8044e6fd 0 1554 1546 0x00000000

Changed in linaro-networking:
importance: Undecided → High
status: New → Confirmed
Revision history for this message
viresh kumar (viresh.kumar) wrote :

Its working for me and I don't see any issues on my local board. Please see what's missing in my .config (attached.)

Revision history for this message
Mike Holmes (mike-holmes) wrote :

I did a diff - attached, it looks like there are a few differences, I also wondered if you were using Ubuntu or the OE filesystem.

Revision history for this message
Gary S. Robertson (gary-robertson) wrote : Re: [Bug 1215507] Re: If CONFIG_NO_HZ_FULL is selected in RT 3.10 kernel, udhcpc process hangs while attempting to obtain IP address and boot sequence fails to complete.

Kim pointed out Viresh was using Ubuntu - which apparently doesn't use
udhcpc - so it should avoid any udhcpc problems as I understand it.

On Mon, Aug 26, 2013 at 12:31 PM, Mike Holmes <email address hidden>wrote:

> I did a diff - attached, it looks like there are a few differences, I
> also wondered if you were using Ubuntu or the OE filesystem.
>
>
> ** Attachment added: "diff of lng RT.config with Vireshes .config"
>
> https://bugs.launchpad.net/linaro-networking/+bug/1215507/+attachment/3788518/+files/diff.txt
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1215507
>
> Title:
> If CONFIG_NO_HZ_FULL is selected in RT 3.10 kernel, udhcpc process
> hangs while attempting to obtain IP address and boot sequence fails to
> complete.
>
> Status in Linaro networking Group:
> Confirmed
>
> Bug description:
> In kernel version 3.10.6 with PREEMPT_RT_FULL patch set 3.10.6.rt3
> applied, if CONFIG_NO_HZ_FULL=Y the udhcpc client process hangs while
> attempting to obtain an IP address, and the boot startup sequence
> fails to complete. The same kernel without PREEMPT_RT_FULL patches
> applied & selected is able to boot successfully with
> CONFIG_NO_HZ_FULL=Y, and the PREEMPT_RT_FULL-enabled kernel boots
> successfully if CONFIG_NO_HZ_FULL=N.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/linaro-networking/+bug/1215507/+subscriptions
>

Revision history for this message
Anders Roxell (aroxell) wrote :

yo can have CONFIG_NO_HZ_FULL=y when you build 3.10.9-rt5 or 3.10.10-rt7, the link points to my repo that I have been testing with.
https://git.linaro.org/gitweb?p=people/anders.roxell/linux-linaro-lng.git;a=shortlog;h=refs/heads/linux-lng-preempt-rt-3.10.10-rt7

Revision history for this message
Anders Roxell (aroxell) wrote :
Revision history for this message
Gary S. Robertson (gary-robertson) wrote :

Just nit-picking but did you check the .config to verify that
PREEMPT_RT_FULL made it all the way through the configuration process and
was selected in the final .config? I ask because we had problems with this
initially and had to add a patch to arch/arm/Kconfig to enable it (commit
dbe80efe81a3d9fb541c88284553c046a4e9e958)... and I haven't checked
personally because I'm still hamstrung by metered bandwidth during prime
time hours, so I can't clone your repo to look at the commit history till
after midnight local time. If PREEMPT_RT_FULL wasn't available in
arch/arm/Kconfig then it might give a misleading impression that upstream
changes had fixed the udhcpc issue when NO_HZ_FULL was selected.

Revision history for this message
Mike Holmes (mike-holmes) wrote :

I got this
mike@fedora1:/mnt/SSD/ci_build/workspace$ cat kernel_config_arndale | grep
PREEMPT
CONFIG_TREE_PREEMPT_RCU=y
CONFIG_PREEMPT_RCU=y
CONFIG_PREEMPT_NOTIFIERS=y
CONFIG_PREEMPT=y
CONFIG_PREEMPT_RT_BASE=y
CONFIG_HAVE_PREEMPT_LAZY=y
CONFIG_PREEMPT_LAZY=y
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT__LL is not set
# CONFIG_PREEMPT_RTB is not set
CONFIG_PREEMPT_RT_FULL=y
CONFIG_PREEMPT_COUNT=y
# CONFIG_DEBUG_PREEMPT is not set
# CONFIG_PREEMPT_TRACER is not set

On 30 August 2013 12:36, Gary S. Robertson <email address hidden>wrote:

> Just nit-picking but did you check the .config to verify that
> PREEMPT_RT_FULL made it all the way through the configuration process and
> was selected in the final .config? I ask because we had problems with this
> initially and had to add a patch to arch/arm/Kconfig to enable it (commit
> dbe80efe81a3d9fb541c88284553c046a4e9e958)... and I haven't checked
> personally because I'm still hamstrung by metered bandwidth during prime
> time hours, so I can't clone your repo to look at the commit history till
> after midnight local time. If PREEMPT_RT_FULL wasn't available in
> arch/arm/Kconfig then it might give a misleading impression that upstream
> changes had fixed the udhcpc issue when NO_HZ_FULL was selected.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1215507
>
> Title:
> If CONFIG_NO_HZ_FULL is selected in RT 3.10 kernel, udhcpc process
> hangs while attempting to obtain IP address and boot sequence fails to
> complete.
>
> Status in Linaro networking Group:
> Confirmed
>
> Bug description:
> In kernel version 3.10.6 with PREEMPT_RT_FULL patch set 3.10.6.rt3
> applied, if CONFIG_NO_HZ_FULL=Y the udhcpc client process hangs while
> attempting to obtain an IP address, and the boot startup sequence
> fails to complete. The same kernel without PREEMPT_RT_FULL patches
> applied & selected is able to boot successfully with
> CONFIG_NO_HZ_FULL=Y, and the PREEMPT_RT_FULL-enabled kernel boots
> successfully if CONFIG_NO_HZ_FULL=N.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/linaro-networking/+bug/1215507/+subscriptions
>

Revision history for this message
Gary S. Robertson (gary-robertson) wrote :
Download full text (3.4 KiB)

Excellent! Great news!

On Fri, Aug 30, 2013 at 12:33 PM, Mike Holmes <email address hidden>wrote:

> I got this
> mike@fedora1:/mnt/SSD/ci_build/workspace$ cat kernel_config_arndale | grep
> PREEMPT
> CONFIG_TREE_PREEMPT_RCU=y
> CONFIG_PREEMPT_RCU=y
> CONFIG_PREEMPT_NOTIFIERS=y
> CONFIG_PREEMPT=y
> CONFIG_PREEMPT_RT_BASE=y
> CONFIG_HAVE_PREEMPT_LAZY=y
> CONFIG_PREEMPT_LAZY=y
> # CONFIG_PREEMPT_NONE is not set
> # CONFIG_PREEMPT_VOLUNTARY is not set
> # CONFIG_PREEMPT__LL is not set
> # CONFIG_PREEMPT_RTB is not set
> CONFIG_PREEMPT_RT_FULL=y
> CONFIG_PREEMPT_COUNT=y
> # CONFIG_DEBUG_PREEMPT is not set
> # CONFIG_PREEMPT_TRACER is not set
>
>
> On 30 August 2013 12:36, Gary S. Robertson
> <email address hidden>wrote:
>
> > Just nit-picking but did you check the .config to verify that
> > PREEMPT_RT_FULL made it all the way through the configuration process and
> > was selected in the final .config? I ask because we had problems with
> this
> > initially and had to add a patch to arch/arm/Kconfig to enable it (commit
> > dbe80efe81a3d9fb541c88284553c046a4e9e958)... and I haven't checked
> > personally because I'm still hamstrung by metered bandwidth during prime
> > time hours, so I can't clone your repo to look at the commit history till
> > after midnight local time. If PREEMPT_RT_FULL wasn't available in
> > arch/arm/Kconfig then it might give a misleading impression that upstream
> > changes had fixed the udhcpc issue when NO_HZ_FULL was selected.
> >
> > --
> > You received this bug notification because you are subscribed to the bug
> > report.
> > https://bugs.launchpad.net/bugs/1215507
> >
> > Title:
> > If CONFIG_NO_HZ_FULL is selected in RT 3.10 kernel, udhcpc process
> > hangs while attempting to obtain IP address and boot sequence fails to
> > complete.
> >
> > Status in Linaro networking Group:
> > Confirmed
> >
> > Bug description:
> > In kernel version 3.10.6 with PREEMPT_RT_FULL patch set 3.10.6.rt3
> > applied, if CONFIG_NO_HZ_FULL=Y the udhcpc client process hangs while
> > attempting to obtain an IP address, and the boot startup sequence
> > fails to complete. The same kernel without PREEMPT_RT_FULL patches
> > applied & selected is able to boot successfully with
> > CONFIG_NO_HZ_FULL=Y, and the PREEMPT_RT_FULL-enabled kernel boots
> > successfully if CONFIG_NO_HZ_FULL=N.
> >
> > To manage notifications about this bug go to:
> > https://bugs.launchpad.net/linaro-networking/+bug/1215507/+subscriptions
> >
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1215507
>
> Title:
> If CONFIG_NO_HZ_FULL is selected in RT 3.10 kernel, udhcpc process
> hangs while attempting to obtain IP address and boot sequence fails to
> complete.
>
> Status in Linaro networking Group:
> Confirmed
>
> Bug description:
> In kernel version 3.10.6 with PREEMPT_RT_FULL patch set 3.10.6.rt3
> applied, if CONFIG_NO_HZ_FULL=Y the udhcpc client process hangs while
> attempting to obtain an IP address, and the boot startup sequence
> fails to complete. The same kernel without PREEMPT_RT_FULL patches
> applied & selected is able to b...

Read more...

Revision history for this message
Gary S. Robertson (gary-robertson) wrote :

This behavior was corrected by upstream patches as of kernel version 3.10.9-rt5, and is presently in place in the LNG git repository with current linux-lng-preempt-rt branch version 3.10.10-rt7. Release of this kernel is expected for the 13.09 release cycle.

Changed in linaro-networking:
status: Confirmed → Fix Released
status: Fix Released → Fix Committed
Revision history for this message
Maxim Uvarov (maxim-uvarov) wrote :

Confirm that rt kernel compiles and boots.

Revision history for this message
Anders Roxell (aroxell) wrote :

if this is fixed we need to re-enable no_hz_full

Revision history for this message
Gary S. Robertson (gary-robertson) wrote :

Yes we just need to add the config fragment back in to the RT kernel build
script. The config fragment itself has remained available the whole time
for use in local builds. I see you have already posted an email to the
mailing list regarding the patch. Since the underlying problem has already
been verified as fixed, your patch merely re-enables the feature for our
daily CI builds and tests. I think we need to get this pushed today to
fall within the deadline for changes made during the current release cycle
(even though strictly speaking this is not a modification to the kernel
itself - just to the CI build scripts). Since you and Fathi have been more
active in maintaining the build scripts than I have I think it should be
okay for you to do the push... but if you prefer that I do it please send
me a link to your repo so I can push the commit directly rather than having
to fool with git-am, etc..

On Thu, Oct 17, 2013 at 8:14 AM, Anders Roxell <email address hidden>wrote:

> if this is fixed we need to re-enable no_hz_full
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> Matching subscriptions: LNG all
> https://bugs.launchpad.net/bugs/1215507
>
> Title:
> If CONFIG_NO_HZ_FULL is selected in RT 3.10 kernel, udhcpc process
> hangs while attempting to obtain IP address and boot sequence fails to
> complete.
>
> Status in Linaro networking Group:
> Fix Committed
>
> Bug description:
> In kernel version 3.10.6 with PREEMPT_RT_FULL patch set 3.10.6.rt3
> applied, if CONFIG_NO_HZ_FULL=Y the udhcpc client process hangs while
> attempting to obtain an IP address, and the boot startup sequence
> fails to complete. The same kernel without PREEMPT_RT_FULL patches
> applied & selected is able to boot successfully with
> CONFIG_NO_HZ_FULL=Y, and the PREEMPT_RT_FULL-enabled kernel boots
> successfully if CONFIG_NO_HZ_FULL=N.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/linaro-networking/+bug/1215507/+subscriptions
>

Changed in linaro-networking:
assignee: nobody → Gary S. Robertson (gary-robertson)
Changed in linaro-networking:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.