[public] bad: scheduling from the idle thread!

Note: logs scroll for about 50s before the systems actually reboots, after the "Restarting system" message and before the system is cycled.

Can anybody upload a full log from power-on. We need to understand from which point this unexpected schedule is happening. Thanks!

Well, I get it already. Thanks Ike and the remote access. :)

However, I didn't see this "scheduling" error in the log on the board (xx.xx.37.252). Good or bad luck... :(

Here's fresh log with Linux version 3.2.0-1602-armadaxp (buildd@nasl) (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu4) ) #5-Ubuntu SMP Thu Apr 5 13:27:03 UTC 2012 (Ubuntu 3.2.0-1602.5-armadaxp 3.2.14):

Thanks. I saw that. And it's really strange that this error
disappeared after the board reboot.

Could you paste a full log which shows (from boot-up) when the first error
message is printed? It's helpful for us to understand which device driver
is likely to cause this unexpected scheduling (within its interrupt
handler).

On Sat, Apr 7, 2012 at 2:25 AM, MMorana <email address hidden> wrote:

> Here's fresh log with Linux version 3.2.0-1602-armadaxp (buildd@nasl)
> (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu4) ) #5-Ubuntu SMP Thu
> Apr 5 13:27:03 UTC 2012 (Ubuntu 3.2.0-1602.5-armadaxp 3.2.14):
>
> ** Attachment added: "armadaxp-lp959604.log"
>
> https://bugs.launchpad.net/eilt/+bug/959604/+attachment/3017702/+files/armadaxp-lp959604.log
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/959604
>
> Title:
> [public] bad: scheduling from the idle thread!
>
> Status in The Eilt project:
> Confirmed
> Status in “linux-armadaxp” package in Ubuntu:
> Confirmed
>
> Bug description:
> Build Version/Date: Linux armadaxp3 3.2.0-1600-armadaxp #3-Ubuntu SMP Fri
> Mar 16 19:14:51 UTC 2012 armv7l armv7l armv7l GNU/Linux
> Release: 12.04
> Environment used for testing: armadaxp3
>
> Summary: bad: scheduling from the idle thread!
>
> Steps to Reproduce: reboot and watch the console
>
> Expected result: no console message flood
>
> Actual result: console floods with bad scheduling message
>
> [ 347.493449] bad: scheduling from the idle thread!
> [ 347.494177] bad: scheduling from the idle thread!
> [ 347.494332] bad:/0x24)
> <4>[ 347.512264] [<c06d0d6[ 347.512855] bad: scheduling from the idle
> thread!
> (un[ 347.520263] [<c0015a5c>] (unwind_backtrace+0x0/0xec) from [<c06[
> 347.520735] bad: scheduling from the idle thread!
> [ 347.521098] bad: scheduling from the idle thread!
> [ 347.521119] bad: scheduling from the idl] (dequeue[ 347.539644] bad:
> scheduling from the idle thread!
> [ 347.540004] bad: scheduling from the idle thread!
> [ 347.540025] bad: scheduling from the idle thread!
> [ 347.540586] bad: scheduling from the idle thrc)
> <4[ 347.558693] bad: scheduling from the idle thread!
> [ 347.558715] bad: scheduling from the idle thread!
> [ 347.559477] bad: scheduling from the idle thread!
> [ 347.559585] bad: scheduling from the idle threa28)[ 347.577554] bad:
> scheduling from the idle thread!
> [ 347.578037] bad: scheduling from the idle thread!
> [ 347.578439] bad: scheduling from the idle thread!
> [ 347.578460] bad: scheduling from the idle thread8>[ 347.596572] bad:
> scheduling from the idle thread!
> [ 347.597253] bad: scheduling from the idle thread!
> [ 347.597268] bad: scheduling from the idle thread!
> [ 347.597584] bad: scheduling from the idle thread!
> [ 34706cd7f4>] (0x6cd7f4)
> <3>[ 347.616061] bad: sch[ 347.616079] bad: scheduling from the idle
> thread!
> [ 347.616101] bad: scheduling from the idle thread!
> [ 347.616939] bad: scheduling from the idle thread!
> [ 347+0x3c/0x4c) from [<c004fc24>] (dequeue_task+0xc[ 347.634937] bad:
> scheduling from the idle thread!
> [ 347...

@Li Li, the problem appears during reboot (shutdown) and that log contained the startup and the shutdown logs - here's another one in startup-shutdown order for you:

^ Linux armadaxp3 3.2.0-1602-armadaxp #5-Ubuntu SMP Thu Apr 5 13:27:03 UTC 2012 armv7l armv7l armv7l GNU/Linux

Thanks for the new log. It looks like a certain driver failed to clean up itself and reschedule happened within its interrupt handler.

hi guys,

would you also try helping me understand any special kernel configuration ? have you guys compiled with anything special ?
from this thread seem that guys are not doing anything with the system after prompt, is this the case ? any special setup ?

Seif

@Seif,

There's nothing special that's causing the problem, it just happens on shutdown/reboot.

The kernel lives here: https://launchpad.net/ubuntu/+source/linux-armadaxp/ and http://ports.ubuntu.com/ubuntu-ports/pool/main/l/linux-armadaxp/.

Have you tried to recreate it?

here u go guys.
this should fix it ....btw this issues was not in the original drop.

Seif

Thank you, Seif. :)

@MMorana, has Seif's patch fixed this issue?

Hi MMorana, please find the patched kernel at https://chinstrap.canonical.com/~lli5/959604/linux-image-3.2.0-1602-armadaxp_3.2.0-1602.5_armhf.deb. Thanks!

@Li, your patched kernel does not exhibit this problem:

...
[ 40.606710] md0: detected capacity change from 0 to 19994247168
udevd[222]: inotify_add_watch(6, /dev/dm-0, 10) failed: Invalid argument

done.
[ 44.238967] pool #0: pkt_size=1536, buf_size=1632 - 2048 of 2048 buffers added

Ubuntu 12.04 LTS n4 ttyS0

n4 login: ubuntu
Password:
Last login: Sun May 6 00:38:01 BST 2012 on ttyS0
Welcome to Ubuntu 12.04 LTS (GNU/Linux 3.2.0-1602-armadaxp armv7l)

 * Documentation: https://help.ubuntu.com/
unubuntu@n4:~$ uname -a
Linux n4 3.2.0-1602-armadaxp #5 SMP Tue Apr 24 02:17:06 UTC 2012 armv7l armv7l armv7l GNU/Linux
ubuntu@n4:~$ md5sum /boot/uI
md5sum: /boot/uI: No such file or directory
ubuntu@n4:~$ md5sum /boot/uImage
bc919e5071962e60ae787628a66e41f2 /boot/uImage
ubuntu@n4:~$ md5sum linux-image-3.2.0-1602-armadaxp_3.2.0-1602.5_armhf.deb
c848b5c20dbe0b8afb3dc7d2f4be9b6c linux-image-3.2.0-1602-armadaxp_3.2.0-1602.5_armhf.deb
ubuntu@n4:~$ sudo reboot

Broadcast message from ubuntu@n4
        (/dev/ttyS0) at 0:39 ...

The system is going down for reboot NOW!
ubuntu@n4:~$ * Stopping MD monitoring service mdadm --monitor [ OK ]
 * Asking all remaining processes to terminate... [ OK ]
 * All processes ended within 1 seconds.... [ OK ]
 * Deconfiguring network interfaces... [ OK ]
 * Deactivating swap... [ OK ]
 * Will now restart
[ 85.323522] Restarting system.

BootROM 1.15
Booting from SPI flash
DDR3 Training Sequence - Ver 2.2.0
DDR3 Training Sequence - Number of DIMMs detected: 2
DDR3 Training Sequence - Ended Successfully
...

@MMorana, I didn't see "bad: scheduling from the idle thread!" in the new log you posted. Are you pasting comments about another bug of "MS card could make kernel panic"? :)

@Li, I tested your kernel with the patch and there were no messages in the log so the patch fixed the problem it appears.

Aha, I saw you pasted a new log and misunderstood that. Thank you and Seif a lot! :)

update the patch with author's signed-off

Hi Seif,

Another possible reason is CONFIG_HOTPLUG_CPU is enabled and I checked defconfig its also enabled.

config HOTPLUG_CPU
        bool "Support for hot-pluggable CPUs (EXPERIMENTAL)"
        depends on SMP && HOTPLUG && EXPERIMENTAL
        help
          Say Y here to experiment with turning CPUs off and on. CPUs
          can be controlled through /sys/devices/system/cpu.

So it looks to turn off cpu hotplug is also a solution if we do not support for cpu hotplug with Armada XP.

Could you confirm this?

Thanks a lot.

Enabling this hotplug config provides the possibility to save more power (as well as introducing advanced error
reporting and correction capabilities theoretically). Meanwhile, according to kernel document, SMP support for suspend/resume is a work in progress. I would suggest we enable this by default to find and fix any potential bugs.

sorry for the delay on this one.
we have done the same work on an earlier kernel, I'll work on porting, and testing this patch your latest stable source.
Patch should be ready in the next couple of days.

Seif

@Seif, cool~

Hi, any update?

@Ike, no updates from me. So I am assuming this patch will go in, I believe there are others asking for it. What's the ETA for Precise?

Me, either.

So the attached patch is not good for SRU and we should wait for a better fix?

@Seif, could you please provide an update as to what you would like to see happen? Thank you.

I commited this patch in the precise git repo, and is in the 3.2.0-1604.7 package that will go into precise-proposed as soon as the previous upload gets promoted to updates.
I verified the deb built in the Canonical kernel team PPA and indeed it no longer prints these messages on the screen and reboots faster (about 20 s)

I have verified that this fix is in 3.2.0-1605 and that the "bad: scheduling from the idle thread!" no longer shows up during a reboot.

ubuntu@n2:~$ uname -a
Linux n2 3.2.0-1605-armadaxp #8-Ubuntu SMP Sat Jul 7 22:36:46 UTC 2012 armv7l armv7l armv7l GNU/Linux

This bug was fixed in the package linux-armadaxp - 3.2.0-1605.8

---------------
linux-armadaxp (3.2.0-1605.8) precise-proposed; urgency=low

  * Release Tracking Bug
   -LP: #1020888

  [ Jani Monoses ]

  * Fix warnings.
  * Adapt to new sched_clock API

  [ Ubuntu: 3.2.0-27.43 ]

  * No change upload to fix .ddeb generation in the PPA.
  * Release Tracking Bug
    - LP: #1020016

  [ Ubuntu: 3.2.0-27.42 ]

  * Release Tracking Bug
    - LP: #1020016
  * PACKAGING: add .gnu_debuglink sections to .ko files
    - LP: #669641
  * [Config] Updateconfigs
    - LP: #1008345
  * SAUCE: (upstreamed) [media] ene_ir: Fix driver initialisation
    - LP: #1014800
  * SAUCE: ocfs2: Fix NULL pointer dereferrence in
    __ocfs2_change_file_space
    - LP: #1006012
  * SAUCE: net: calxedaxgmac: enable rx cut-thru mode
    - LP: #1008345
  * SAUCE: EDAC: Add support for the highbank platform memory
    - LP: #1008345
  * SAUCE: EDAC: add support for highbank platform L2 cache ecc
    - LP: #1008345
  * (pre-stable): bcma: add ext PA workaround for BCM4331 and BCM43431
    - LP: #925577
  * SAUCE: ALSA: hda - Fix power-map regression for HP dv6 & co
    - LP: #1013183
  * [Config] Enable CONFIG_CGROUPS for highbank
    - LP: #1014692
  * Revert "net: maintain namespace isolation between vlan and real device"
    - LP: #1013723
  * x86/amd: Re-enable CPU topology extensions in case BIOS has disabled it
    - LP: #1009087
  * hwmon: (k10temp) Add support for AMD Trinity CPUs
    - LP: #1009086
  * hwmon: (fam15h_power) Increase output resolution
    - LP: #1009086
  * Input: wacom - use BTN_TOOL_FINGER to indicate touch device type
    - LP: #1009435
  * Input: wacom - use switch statement for wacom_tpc_irq()
    - LP: #1009435
  * Input: wacom - isolate input registration
    - LP: #1009435
  * Input: wacom - wireless monitor framework
    - LP: #1009435
  * Input: wacom - create inputs when wireless connect
    - LP: #1009435
  * Input: wacom - wireless battery status
    - LP: #1009435
  * Input: wacom - check for allocation failure in probe()
    - LP: #1009435
  * Input: wacom - add basic Intuos5 support
    - LP: #1009435
  * Input: wacom - add Intuos5 Touch Ring/ExpressKey support
    - LP: #1009435
  * Input: wacom - add Intuos5 Touch Ring LED support
    - LP: #1009435
  * Input: wacom - add Intuos5 multitouch sensor support
    - LP: #1009435
  * iommu/amd: Add workaround for event log erratum
    - LP: #1013723
  * MIPS: BCM63XX: Add missing include for bcm63xx_gpio.h
    - LP: #1013723
  * cifs: Include backup intent search flags during searches {try #2)
    - LP: #1013723
  * sunrpc: fix loss of task->tk_status after rpc_delay call in
    xprt_alloc_slot
    - LP: #1013723
  * exofs: Fix CRASH on very early IO errors.
    - LP: #1013723
  * cifs: fix oops while traversing open file list (try #4)
    - LP: #1013723
  * Fix dm-multipath starvation when scsi host is busy
    - LP: #1013723
  * ixp4xx: fix compilation by adding gpiolib support
    - LP: #1013723
  * drm/i915: properly handle interlaced bit for sdvo dtd conversion
    - LP: #1013723
  * drm/i915: enable vdd when switching off the eDP panel
    - LP: #1013723
  * drm/i915: Ignore LVDS o...