Powertop reports huge number of wakeups and drains battery

The machine is a Dell Latitude D630. (Centrino Duo T7300, Santa Rosa chipset)

Since the C0 residency is 50% plus whatever process wake-ups I can see, I would hazard a guess that the kernel has woken up one of the cores in my dual core CPU and not let it sleep again.

Hi,

I encounter the same issue on a d630. After a while the wakeups count jumps from less than 100 to ~6K per seconds.
I've switched to console removing all unused processes and kernel modules but the wake-ups count is still around 6K per seconds and C0 residency is 50%.

It looks like a kind of idle-wakeups "bug" into the kernel.

Try putting the pcmcia and yenta_socket modules in the blacklist
[ In the file /etc/modprobe.d/blacklist append
blacklist pcmcia
blacklist yenta_socket
] and reboot.
Don't try unloading them, you need to not load them at all.
This will obviously disable pcmcia, but if you don't use it then you will save power.
If you want to use it once in a while just load the modules by hand
[ sudo modprobe pcmcia yenta_socket].
I will do some search to see what is the exact module to blame (these ones depend on others)

Thanks, it solves this issue.

In my case this is not a definitive solution because I use a pcmcia device (external 3G card).

Is there anything I can do to help ?

I wrote a script to see which are the modules loaded.
The module to blame should be one of these....

pcmcia 41388 0
pcmcia_core 40980 3 pcmcia,yenta_socket,rsrc_nonstatic
rsrc_nonstatic 14080 1 yenta_socket
yenta_socket 27532 1

So, what I need you to do is to make a boot with the usual blacklists and then load the modules one by one using
sudo modprobe module_name
where module_name each time one of these:
yenta_socket rsrc_nonstatic pcmcia pcmcia_core
Please use this order (or at least make sure pcmcia_core is the last one and rsrc_nonstatic after yenta_socket).

So the process is:
* Boot with the blacklists
* You load the first module
* Wait some time (e.g. 10 minutes) to see if it increases the wakeups
* If not, you load the second one, you wait, etc
* When you find the module to blame just let us know here to forward the bug to the appropriate place.

I hope the instructions are clear. Good luck :)

Thanks for the detailed instructions.

The culprit appears to be yenta_socket!

Obviously, yenta_socket is causing this behavior.

After a few tests it appears that loading yenta_socket makes the number of wake ups jump to 6K/s, but unloading it then reloading it reset the number of wakeups to a standard value.

Does it make difference to have the modules loaded in boot time or after?
Does it make difference if you unload them later?
Does it happen every time?

If you are sure that we have a certain study case, we can forward the bug to the upstream.

> Does it make difference to have the modules loaded in boot time or after?

Not for me.

> Does it make difference if you unload them later?

If I unload yenta_socket the wakeups stays high.

> Does it happen every time?

Yes.

Thanks, HTH!
J.

After further investigations, here are the results of my tests.

- The phenomenon is reproducible and happens every time.
- It makes no differences if the modules are loaded at boot time or after.
- loading yenta_socket makes the phenomenon appear after a random period of time.
- unloading yenta_socket doesn't do anything and wakeups rate stays high.
- reloading yenta_socket while rate is high makes it goes low.
- Then, removing yenta_socket while wakeups rate is low make it stay low.

The measures and comments are in attachment.

I've deactivated one core but the behavior is exactly the same.

The power consumption resulting of this high level of wakeups is around 2W on my laptop.

Hope this helps,

Please post: powertop --dump

if you still have this issue.

Here it is.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jean-Baptiste, please provide your:
lspci -vv

Jean-Baptiste Lallement schrieb:
> Here it is.
>
> ** Attachment added: "powertop--dump.out"
> http://launchpadlibrarian.net/10467820/powertop--dump.out
>

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHQdmkVVCoNUmKuAcRAlHuAJ9QXm9U+tGFRqhJnWg9uBkG/DAi7wCgpmQc
fxmIOyUCXEOOS4mC0580rpY=
=1V5a
-----END PGP SIGNATURE-----

You've got it.

Ok. This is know issue on <email address hidden> mailinglist but at the moment no solution available.

I am assigning this bug to the 'ubuntu-kernel-team' per their bug policy. For future reference you can learn more about their bug policy at https://wiki.ubuntu.com/KernelTeamBugPolicies .

Thank you for taking the time to report this bug and helping to make Ubuntu better. This bug does not meet the criteria for a stable release update and is being marked as Won't Fix for the kernel version the bug was submitted about. You can learn more about the stable release update process at https://wiki.ubuntu.com/StableReleaseUpdates .
However, the issue that you reported is one that should be possible to test with the live environment of the Desktop CD of the development release - Hardy Heron. It would help us greatly if you could test with it so we can work on getting it fixed in the actively developed kernel. You can find out more about the development release at http://www.ubuntu.com/testing/ .
If you do decide to test with the development release of Ubuntu please comment on this bug report and include at least the minimal information requested at http://wiki.ubuntu.com/KernelTeamBugPolicies . Thanks again and we appreciate your help.

The same problem is in Hardy (which I upgraded to last night as it
happens), I will open a new bug.

Output for "powertop --dump" on Hardy attached.

Hardy dmesg

Hardy -lspci vvnn

This is most curious. Can you send me the output put of the following commands:

vmstat 1 > vmstat.log

(run this for 60 seconds or so)

and also:

top -b > top.log

(again, run for 60 seconds or so)

hopefully from this I get a better idea of what's sucking up the power.

I can confidently say it's not yenta_socket again as I still have it blacklisted.

So, here's the vmstat:

and the top

Hi again,

can you try the following:

sudo bash -c "(echo 0 >/proc/timer_stats; echo 1 > /proc/timer_stats ; sleep 10; cat < /proc/timer_stats)" > timer_stats.log

This will do a 10 second sample of timer activity. From this way may see which process(es) are a bit heavy handed on waking up the processor from idle.

Attached.

It seems to be an order of magnitude out compared to what powertop is saying for total wakeups.

I think it's the kernel doing it, not a process, like the yenta_socket problem for Gutsy, and powertop does not list kernel events.

Hi Julian,

It seems interesting that the timer information in timer_stats.log seems to show a big difference between the timer events occurring in the kernel and what powertop is reporting. Somehow I believe the kernels own metrics in /proc/timer_stats more than powertop at this point. Could powertop be recording some events incorrectly?

I'm pretty sure powertop is correct (doesn't it use the CPU itself to get the stats?). The reason being that the temperature sensors rise by about 10-15 degrees and my battery life is shot to bits :)

Can you reproduce it at all?

This line from powertop also piqued my interest:
  57.0% (403.1) <kernel IPI> : Rescheduling interrupts

What's going on there?

Hi Julian,

"Rescheduling interrupts" is where the scheduler reschedules a process to another core to try and load balance a system across cores. I large amount of these could indicate processes are ping-ponging back and forth between cores, which is not helpful.

I suggest enabling multicore power savings; it may be worth trying. To do so:

echo 1 > /sys/devices/system/cpu/sched_mc_power_savings

Then re-run powertop and see if this helps and let me know the results - thanks.

On a Hardy installtion, when the system is idle, please run the following:

 for i in 1 2 3 4 5 6 7 8 9 10 ; do powertop -d -t 10 | grep Wakeups-from-idle ; done

..and attach the results. I'd like to see how powertop reports wakeups over a period of time on an idle hardy installation. Thanks

Unfortunately the sched_mc_power_savings change makes no difference :(

I noticed that immediately after booting, the wakeups look normal (around 100/sec). It's only after I Ieave Skype running with a call on my (USB) headset that it makes these wake-ups go skyrocketing (even after I shut down Skype), so something in that combination of usage is tickling some bug,

Wakeups-from-idle per second : 14480.5 interval: 10.0s
Wakeups-from-idle per second : 5699.9 interval: 10.0s
Wakeups-from-idle per second : 18543.5 interval: 10.0s
Wakeups-from-idle per second : 20464.1 interval: 10.0s
Wakeups-from-idle per second : 5950.8 interval: 10.0s
Wakeups-from-idle per second : 9812.5 interval: 10.0s
Wakeups-from-idle per second : 7553.7 interval: 10.0s
Wakeups-from-idle per second : 11080.6 interval: 10.0s
Wakeups-from-idle per second : 14264.8 interval: 10.0s
Wakeups-from-idle per second : 5957.5 interval: 10.0s

Hi,

I can confirm what Julian is saying.

I've monitored w-ups and battery consumption. After a period of time the wakeups rate jumps from normal (~ 300/s) to a very high level (~ 9000/s). In the meantime, battery consumption increased by 35%.

During all this time, I was reading news in liferea.

You'll find in attachment the results of this test.

Jean-Baptiste: Can you attach a copy of the script or source to the code that generated the wakeup_test2.log? Thanks

Bizarrely, this problem has now gone away for me. I've kept an eye on powertop for the 2 days since it went and not once have I seen it heading over 1000 wakeups per second unless I am doing something intensive. There must have been an update that fixed it, but who knows what given the huge number of daily updates at this point in the cycle.

My only concern now is that this:
  52.6% (116.4) <kernel IPI> : Rescheduling interrupts

seems to be causing most of the wakeups now. This did not happen on Gutsy. For example, I am now around 200-250 wakeups per second on a quiesced machine, whereas on Gutsy it was around 100. This equates to a good watt of power.

hi guys,

just by chance i read this bug report a couple of minutes ago. i installed powertop and let it run. result >20000 wakeups/s. blacklisted yenta_socket and now i have ~150 wakeups/s.

i tested on latest hardy i386. i will attach the usual stuff. anything else you would like me to test?

!!!! ups. sorry. the above files are from the wrong computer sorry.!!!

the right ones are below....

Unfortunately, this problem has started happening for me again :(

When I get some time, I'll try and unload some kernel modules and see if I can pinpoint which is causing the problem.

exactt: the yenta_socket blacklisting fix only works on Gutsy.

@Julian Edwards: As i wrote: the "fix" works on hardy. at least for me...

After many many tests (latest with 8.04 / kernel 2.6.24-12-generic) I'm still unable to find an event making the system go into this state. Sometimes it occurs after a few minutes, and sometimes not for hours. Obviously this is very strange.

Blacklisting yenta_socket doesn't fix the problem. But one thing is for sure, when it occurs, modprobing yenta_socket reset the wakeups rate to a standard value.

This bug is a showstopper for laptop users as this event drains 400mA (as a comparison backlight at full power pumps 500mA on DELL D630)

I'll continue this discussion on the power ML and let you know if there is any progress.

BTW, in attachment, a small script to monitor wakeups and battery life.

Regards,

Confirming bug for Hardy beta on Lenovo T61.
Workaround (blacklisting yenta_socket and pcmcia) works for me. Wakeups: ~6000 before, ~25 after.

Blacklisting yenta_socket and pcmcia does not fix this problem, it merely averts it temporarily until some event happens that causes the bad state again.

Is there any news on this? There's a kernel freeze in 10 days and I would hate for this to be still a bug when Hardy is released (more the the point, my battery will hate me).

Hi,

Just a long shot, but can you reboot with the kernel boot option acpi=noirq

if it improves things then it may be that some ACPI DSTD weirdness is causing this problem.

Colin

just added acpi=noirq to the kernel command line and tested. sadly the wakeups are still way too high ( > 11000).

cheers

I added acpi=noirq and removed "quiet" to watch the console during boot and so far, three hours later, it's not gone into crazy mode. This has happened before though where it didn't go wrong for a while, so I will keep an eye on it.

root@potpal:~# dmesg|grep command
[ 0.000000] Kernel command line: root=UUID=49bb501f-9dcb-4ce4-908f-2888f2313e99 ro splash acpi=noirq

exactt, can you do this grep just to make sure the kernel picked up your boot option?

i already did the dmesg|grep command and acpi=noirq was in it...

Hi there,

Some things to try:

1. See what interrupts are most active:

(cat /proc/interrupts; sleep 10; cat /proc/interrupts) > interrupts.log

and then using the attached awk script, do the following:

awk -f dualcore.awk < interrupts.log > irq_delta.log

This will give some idea of the most active interrupts on your system.

2. Obtain a log of busy processes using top:

top -b -d 10 -n 6 > top.log

3. See overall system activity (I'm interested in the context switches/second) using vmstat :

vmstat 1 -n 60 > vmstat.log

4. and attach all these logs to the bug report.

Thanks, Colin

Note: I'd also like you refer you to https://bugs.launchpad.net/ubuntu/+source/linux/+bug/177895 which discusses some of these points at length.

here you go...

Colin, that was an interesting read on bug 177895.

I think that in those comments there are two separate issues:
1. People complaining about "<kernel IPI> : Rescheduling interrupts"
2. Some people noticing the larger interrupt count caused by the issue on this bug report (possibly yenta_socket).

Anyway, I've got the three logs to attach here. I made sure that when I was capturing the logs that powertop had seen the system go into "crazy" mode with 10-20k interrupts/sec.
I can't see *anything* unusual in the logs :( It's looking very much like a kernel bug *somewhere*, particularly since reloading yenta_socket will fix it, albeit temporarily.

Cheers.

Julian,

Yep, from the logs you supplied the system looks quite normal. I would expect that the irq_delta.log would have captured some kind of abnormal activity. My initial thought is that I should now put some debug into PowerTop to see where it thinks the system is busy. Meanwhile I will also look at what the yenta_socket.

Did the system go "crazy" with the acpi=noirq boot option?

Thanks for the feedback. Colin

Colin

acpi=noirq removed the problems but exactt above says he still had the problem! Weird - the only difference in what we did was that I removed "quiet" - I can't believe that would make any difference but I've seen weirder things happen :)

I'll do some more experiments shortly and let you know if I can definitely say acpi=noirq on its own affects anything or not.

Cheers.

just rechecked with "acpi=noirq" and without "quiet". it does not fix the problem for me.

exactt,

The acpi=noirq works for some people as their BIOS ACPI DSTD is sometimes not quite correct and which can confuse drivers that don't get the correct IRQ setup, so it's a useful rune to try to factor out dodgy BIOS settings. However, now we know it does not work for you we need to look at a different way of cornering this issue. I hope to get back to you early next week on this.

Colin

Have a MacBook Pro 17 (not Santa Rosa) running Heron and started playing with powertop and am seeing some of the same issues. My "kernel IPI" interrupts are taking a good 50%+ of my wakeups although I am only seeing 300-900 rather than in the >1000. What should be normal at this point? Powertop seems to think you can get a system down to <20 wakeups and I pretty much however around the 300-900 mark.

acpi=noirq is working well for me (Dell D630 as in the second bug comment)

Does this have any knock-on effects?

Hi Julian,

If acpi=noirq seems to solve the problem then it's most probable that your ACPI DSTD is causing the problem. One could look and see if an updated ACPI DSTD is available and either load this at boot time, or see if there is a BIOS upgrade for your machine. A good place to look for updated DSTD's is http://acpi.sourceforge.net/dsdt/view.php?manufacturer=Dell&name=Latitude+D630

There is a discussion of overriding one's DSDT (generic discussion) at : http://www.lesswatts.org/projects/acpi/overridingDSDT.php

I'd refer you to https://help.ubuntu.com/community/ACPIBattery for notes on how to compile and load a new DSDT for Ubuntu.

Mind you, if acpi=noirq works for you then you don't have to got to the lengths of overriding the DSTD, just edit you grub loader menu /boot/grub/menu.lst (using your favourite editor but sudo'd) to make it a permanent change. Note that it's not necessarily guaranteed that the updated DSTD will solve this particular issue, it just may be worth looking at just in case it solves other issues too.

Colin

wtgee

> Have a MacBook Pro 17 (not Santa Rosa) running Heron and started playing with powertop and am seeing some of the same issues. My "kernel IPI"
> interrupts are taking a good 50%+ of my wakeups although I am only seeing 300-900 rather than in the >1000. What should be normal at this point?
> Powertop seems to think you can get a system down to <20 wakeups and I pretty much however around the 300-900 mark.

Good question on what is normal. The thing to see is how many interrupts and/or wakeups are occurring on your machine and see if that looks reasonable. For example, running Opera will push the number of wakeups to a very high number and this will push up the number of rescheduling interrupts. Looking at /proc/interrupts will show you the overall interrupt activity. Looking at top will show you the busiest apps. Looking at the cs field of vmstat gives some idea of context switches a second which could show you how busy your system is. Each person has different hardward and a different mix of apps they run, all of which add to the mix of how many wakeups and interrupts a second they see on their system. The scheduler will try to schedule the load across cores and hence you will see a certain level of rescheduling interrupts related to a function of wakeups and interrupts on your system.

A rule of thumb is if you boot your machine into single user mode and look at the activity from powertop and it looks very busy then look to see if the problem is due an interrupt misconfiguration (you will see that if you look at /proc/interrupts over a period of time). If that is so, then it could be (maybe) a BIOS misconfiguration.

Otherwise, system activity is generally due to some over aggressive wakeups in some applications, and perhaps one needs to stop these one by one to see which one may be the culprit.

Hope this helps as a starting point.

Colin

Colin

Thanks very much for all the info. I had a look into my BIOS version, saw that it was A01 and A08 is available. I excitedly upgraded but it's not fixed this problem :(

I know absolutely nothing about DSDT so I am going to leave the acpi=noirq on my menu.lst for now, I need my laptop working for Launchpad development :)

I still consider this a kernel bug because:
a) It happened under gutsy but was controlled by blacklisting yenta_socket. This workaround no longer works for hardy and the problem is in fact a lot worse.
b) The "broken" DSDT doesn't cause problems when Vista is booted on the same laptop. Yeah I know - as much as I tried to convince Dell to give me a bare laptop they wouldn't :)

Cheers

Hi Julian,

This is a moot point about it being a kernel problem. It appears that the if the kernels does exactly as the ACPI configurations then we see the problem, however, telling the kernel to ignore the ACPI IRQ config we don't see the problem. This does not mean it's a kernel issue per sa, it means that the ACPI config is probably not quite right. The fact that Vista does it right may mean that Vista may ignore the ACPI IRQ routing much like when we tell the Linux kernel to do so by the acpi=noirq setting.

If you provide the output from:

lspc -v
cat /proc/interrupts

from your machine booted in acpi=noirq configuration and also in the default configuration then I maybe able to see what is wrong with the BIOS ACPI IRQ routing and perhaps suggest a better workaround. However, these are workarounds for your BIOS ACPI settings, there is no kernel fix as this is not a kernel bug in my opinion.

Thank you.

Hi exactt,

Apologies for not getting back to you sooner - I wanted to work through Julian's issues first. Can you generate an irq_delta.log without the acpi=noirq when powertop reports thousands of "Rescheduling Interrupts" a second - thanks.

Colin

hi again,

i made 2 deltas. one with the option and the other one without.

thx for taking the time to fix this issue!

Hi exactt,

There is minimal difference between the two results, apart from the IRQ 0 being a little more busy on the acpi_noirq configuration. I really cannot see many rescheduling interrupts either, you are getting about 20 to 45 per second which is very low. I suggest that when powertop reports excessive "Rescheduling Interrupts" to do a capture of system activity using the dualcore.awk script and redo the vmstat and top commands described earlier to capture a busy system.

Thank you, Colin

Hi Colin,

Disabling IO-APIC (kernel boot option noapic ) works for me and avoid going into crazy mode.

acpi=noirq is working well for me too ( DELL D630 A06) but it generates some warning messages during boot:

======= /var/log/messages =======
Apr 14 16:31:31 minquiers2 kernel: [ 14.829946] sysfs: duplicate filename 'bridge' can not be created
Apr 14 16:31:31 minquiers2 kernel: [ 14.829949] WARNING: at /build/buildd/linux-2.6.24/fs/sysfs/dir.c:424 sysfs_add_one()
Apr 14 16:31:31 minquiers2 kernel: [ 14.829951] Pid: 1, comm: swapper Not tainted 2.6.24-15-generic #1
Apr 14 16:31:31 minquiers2 kernel: [ 14.829954] [sysfs_add_one+0x9f/0xe0] sysfs_add_one+0x9f/0xe0
Apr 14 16:31:31 minquiers2 kernel: [ 14.829960] [usbcore:sysfs_create_link+0x8a/0x6f0] sysfs_create_link+0x8a/0x110
Apr 14 16:31:31 minquiers2 kernel: [ 14.829964] [shpchp:pci_bus_add_devices+0x9e/0x4f0] pci_bus_add_devices+0x9e/0x130
Apr 14 16:31:31 minquiers2 kernel: [ 14.829968] [pcibios_scan_root+0x1b/0x80] pcibios_scan_root+0x1b/0x80
Apr 14 16:31:31 minquiers2 kernel: [ 14.829972] [pci_legacy_init+0x51/0x100] pci_legacy_init+0x51/0x100
Apr 14 16:31:31 minquiers2 kernel: [ 14.829976] [kernel_init+0x131/0x320] kernel_init+0x131/0x320
Apr 14 16:31:31 minquiers2 kernel: [ 14.829979] [ret_from_fork+0x6/0x20] ret_from_fork+0x6/0x20
Apr 14 16:31:31 minquiers2 kernel: [ 14.829983] [kernel_init+0x0/0x320] kernel_init+0x0/0x320
Apr 14 16:31:31 minquiers2 kernel: [ 14.829985] [kernel_init+0x0/0x320] kernel_init+0x0/0x320
Apr 14 16:31:31 minquiers2 kernel: [ 14.829988] [kernel_thread_helper+0x7/0x10] kernel_thread_helper+0x7/0x10
[...]
Apr 14 16:31:31 minquiers2 kernel: [ 14.959425] PCI: No IRQ known for interrupt pin A of device 0000:00:01.0. Please try using pci=biosirq.
Apr 14 16:31:31 minquiers2 kernel: [ 14.959455] PCI: No IRQ known for interrupt pin A of device 0000:00:1c.0. Please try using pci=biosirq.
Apr 14 16:31:31 minquiers2 kernel: [ 14.959486] PCI: No IRQ known for interrupt pin B of device 0000:00:1c.1. Please try using pci=biosirq.
Apr 14 16:31:31 minquiers2 kernel: [ 14.959518] PCI: No IRQ known for interrupt pin B of device 0000:00:1c.5. Please try using pci=biosirq.
[...]
Apr 14 16:31:31 minquiers2 kernel: [ 15.553799] pcie_portdrv_probe->Dev[8086:2a01] has invalid IRQ. Check vendor BIOS
======= =======

I haven't check if it's harmless or not.

Hence, disabling ACPI or IO-APIC for IRQ routing makes the problem disappear. Regarding this, can we still consider this as a ACPI issue ( broken DSDT) or a IRQ routing bug or kernel bug or else ?

lspci -v and cat /proc/interrupts with and without acpi=noirq follows

sweet :-D

the "noapic" option "fixed" the problem for me. got ~300 wake-ups/s now.

thx

Jean-Baptiste,

Thank you for your contribution here - you've found appropriate workaround for a IRQ routing bug (in the BIOS?). I suspect this is an ACPI issue and not a kernel issue, mainly because the kernel is trying to follow the settings as defined in the BIOS and hence it can only do what it is told. We could put a lot of effort here to probably find there is no kernel bug, but a ACPI misconfiguration. If one is feeling adventurous enough, the final step is to find out the buggy settings and override them with a new DSTD (links have been supplied above in one of my postings on the steps required).

I will mark this this as "Won't Fix" tomorrow as I am confident this is not a kernel issue. If you think this is unsatisfactory, please let me know and we can discuss what to do next.

Regards,

Colin.

Now added a Wiki page to describe troubleshooting of this problem.

https://help.ubuntu.com/community/ReschedulingInterrupts

Hi Colin,

I'd like to hack DSDT if I had time and skill.

Regarding ACPI issue and quoting ACPI project ( http://www.lesswatts.org/projects/acpi/overridingDSDT.php ) :
"In the early days of Linux/ACPI, DSDT modifications were common to work around both BIOS bugs and Linux bugs. However, the stated goal of the Linux/ACPI project today is that Linux should run on un-modified firmware. Thus, the DSDT database at the old http://acpi.sourceforge.net web site is now largely a historical artifact. "

As "Linux should run on un-modified firmware", this may be considered as a kernel or ACPI bug.

However, I will also try the latest vanilla kernel as issues like "Rescheduling interrupts" and huge number of interrupts due to cpuidle bug have been adressed in 2.6.25-rc and see if it changes something to this behaviour.

Thank you for your documentation effort as it may help many people with wake-ups issues. But the title of your Wiki page is a little bit confusing and not totally related to what we are facing here. As said Julian in his comment above, the problem is not only due to a high level of rescheduling interrupts. It's a huge number of wakeups that we can't account for ( > 10K Wups/s but "Rescheduling interrupts" is around 100/s) you're page should distinguish both problems.

Regards

Hi there Colin

Jean-Baptiste is right, and I feel quite strongly that "Won't Fix" is an inappropriate result to this bug. To quote a cliche, expecting my granny to hack menu.lst or download a BIOS update or fix the DSDT is a non-starter. The kernel should work properly on this particular type of broken hardware (and in fact before Gutsy it did not exhibit this problem), especially since other OSes appear capable of doing so.

Also as Jean-Baptiste says, the *extremely* high number of interrupts is nothing to do with the "rescheduling interrupts" wake-up, this is clearly a problem with yenta_socket, which has exacerbated the problem from the Gutsy 5k wakeups/sec to the Hardy 20k wakeups/sec.

Best regards
J

I just discovered that adding acpi=noirq has a rather unfortunate consequence.

When resuming from suspend, my USB ports no longer work.

acpi=noirq can't really be considered a viable work around.

Cheers
J

Hi Julian,

Point taken about the "Won't Fix" - I will change it back to "In progress": was premature of me, so apologies for that. The difficulty we have that if there is something wrong in the DSDT and the kernel uses this to configure the hardware, then we are at the mercy of the DSDT to be correct, else we get these sort of issues. I will investigate why this does not occur with gutsy.

As for the yenta_socket, I will dig into this, but again I suspect that if it is being misconfigured from the BIOS APIC settings, then we are going to see problems unless we fix the BIOS or tell the kernel to ignore the BIOS.

I agree with the statement that ' "Linux should run on un-modified firmware", this may be considered as a kernel or ACPI bug' - if it's an ACPI bug we are at the mercy of the manufacturer getting things right. If it's a kernel bug in yenta_socket or the APIC handling then I will look into an appropriate quirk to fix it.

Colin.

Julian,

The yenta_socket kernel module has a module parameter that is worth trying:

pwr_irqs_off=1

so can you reload the module with this to see if this helps. The module parameter has the oblique note stating:

    "Force IRQs off during power-on of slot. Use only when seeing IRQ storms!"

which may help, but I am unsure if this is the once-on power on of the device, or if the device is powered on/off in many times over an extended period, hence you see the sporadic IRQ storms.

Colin

Sorry for the delay in replying - I've tried pwr_irqs_off=1 and it doesn't help I'm afraid :(

I've tried it too and it doesn't help either.

There are many threads on power@bughost talking about this issue.

This one from a Ubuntu user is interesting http://www.bughost.org/pipermail/power/2008-March/001327.html. He met the same issue on a Lenovo T8300.

He has made some experiments with unloading modules to the bare minimum with no results : http://www.bughost.org/pipermail/power/2008-March/001339.html

Except if DELL and Lenovo BIOSes are the same this is definitely not a BIOS issue.

BTW, I've tried the freshly released 2.6.25 and results are quite poor:
- Kernel IPI is around 15/s which is good
- Total Wakeups/s is unchanged compared to 2.6.24 :(

Hi Jean-Baptiste,

The power@bughost issue was in fact related to bug 177895, which turned out to be a few issues, one of which was Opera killing the laptop with excessive amounts of wakeups.. it's quite a drawn out saga. As for the WikiPage, it was originally written on the back of addressing bug 177895, so it's not entirely applicable for all cases, I just thought it may be pulling a reference into this bug report just incase it helped. I will review the WikiPage in respect to this bug and try and make it more usable.

I suggest booting with the noapic kernel option and doing:

(cat /proc/interrupts; sleep 10; cat /proc/interrupts) > interrupts.log

and then using the dualcore.awk awk script (attached much earlier in the bug report), and do the following:

awk -f dualcore.awk < interrupts.log > irq_delta.log

And repeat this with the normal kernel boot (without the noapic option). Attach the results and I will see if there is anything we can do to check the IRQ routing.

Colin

Julian, have you tried the noapic kernel boot option for this problem?

I've made 2 irq delta with and without noapic. The log without noapic option has been made with a wakeups rate of 22K/s

Thanks for your help

Removing linux-source-2.6.24 task since beginning with Hardy, kernel bugs should be reported against the "linux" package which this already is. Thanks.

Hi Colin.

Was absent for a bit but have to come back here with some clearing.
You said: "The power@bughost issue was in fact related to bug 177895, which turned out to be a few issues, one of which was Opera killing the laptop with excessive amounts of wakeups.. it's quite a drawn out saga."

It is not related. Not in a way you're writing here. Yep - my overloaded opera with milions of mails, usenet and rss news counted in tousands causes up to 8k wakeups per second (with 4000~8000 context switches / s). It is quite isolated issue and can be omitted. There is also another problem - massive amout of interrupts from nowhere. Maybe it's a bug in hardware/misonfigured or broken BIOS/ broken ACPI table / interrupt strom and many more. Problem was gone for about a month, but today it reappeared.. Without any couse i've got 40 thousand of wakeups. So.. this bug still exists in latest hardy kernel.

* In logs opera is running in background (using it to write this comment) so 8k of cs is quite normal and not related to main issue

2.6.24-17-generic

Colin, finally I tried booting with noapic. The problem appears to go away (as with acpi=noirq) and it doesn't freeze the machine with excessive USB activity (like acpi=noirq).

I'll try suspending and resuming next and see how that goes.

I did another interrupt diff, see attached.

IRQ diff with no boot params and 20k wakeups/sec

I have this problem ( 73.1% (491.0) wakeups makes <kernel IPI> : Rescheduling interrupts ). I am running on macbook 2,1 .. hardy 2.6.24-16-generic ..

any fixes of this problem ? Does i need to try vanilla kernel ?

Actually i know compiling kernel (huge cpu usage) .. So <kernel IPI> : Rescheduling interrupts uses just about 200 wakeups ..
But then system idle kernel IPI produces 500 wakeups !! So tha'ts the point .. scheduler is not suitable for idle process .. and it's better for big cpu usage .. that's that i think

Hi Colin - noapic seems to not break suspend/resume so I'm quite happy again :-)

I have this problem with a Lenovo Thinkpad T60.
I am using Hardy and the latest ati driver (fglrx version 8.5)

Passing of options to the kernel at boot as suggested above and on the wiki did not work for me.
A couple of things did have an effect however:

1) If I use Openbox as my window manager/desktop rather than gnome
the number of Rescheduling interrupts is heavily reduced at least until a graphics intensive program is launched. The strong reduction is also apparent if I log into a bare terminal instead of gnome. This suggests that X or fglrx is misbehaving

2) If I upgrade my kernel to the kernel-ppa (launchpad) 2.6.25 the number of Rescheduling interrupts drops hugely (to 20 per second for normal use). Unfortunately with this kernel my wireless card does not work (problem with iwl3945) nor does sound.... I tried compiling my own version of this kernel (2.6.25.2) with identical results.

So given that the 2.6.25 kernel appears to solve this problem I strongly suspect a kernel scheduling problem in the standard hardy kernel.

Hi,

i got the same behavior with a Compal HEL80. At the beginning, i had something around 30k wakeups per second. I tweaked the system a little bit, disabled yenta, hci_usb and all pcmcia related stuff and got only a very poor improvement.
Then i tried the acpi=noirq option, with no impact. Because i read that other people had problems putting their laptop into standby with acpi=noirq, i tried it.
And, surprise: after returning from standby, powertop displayed me something between 15 and 40 wakeups per second ... nice ;)
I restarted ubuntu without the acpi=noirq option and had the same behavior again ... before standby: > 30k wups/s, after standby: something around 30.
Sometimes acpi causes ~100 wakeups, but even while typing and using an usb-mouse and an usb-keyboard, the wakeup-counter stays around 75~125 wups/s

Maybe this could help you too fix the "problem" ... and the important question: shows your ubuntu the same behavior or am i only "lucky".

with kind regards
Markus

PS: I'm using the "stock" kernel: 2.6.24-17-generic #1 SMP Thu May 1 14:31:33 UTC 2008 i686 GNU/Linux

I tried to use acpi=noirq on macbook (second revision). wakeups much less, but keyboard gets very slow(typing) .. I removed acpi=noirq and keyboard works fast again ..

I looked to 2.6.25-rc2 ( i compiled myself ). It's has same problem.

I am talking about my macbook (rev.2 )

I tried to use nosmp and maxcpus=0 parameters. Same as acpi=noirq. Keyboard and touchpad was very slow(not usable at all) ! I checked /proc/cpu_info and saw just one core. So it's OK. That's i wanted. I started powertop and saw tha't wakeups 100 (MAX!). Ok but laptop using about 16-17-18 watts of energy. it's the same like regular ubuntu system with smp .. Sow now i don't know that to think .. using just one core+relative small amount of wakeups and usage of energy is the same(bit less).

I tested this on 2.6.24-17-generic.

Just my two cents. I accidentally installed 32-bit version (out of habit) on my new Latitude D630, and noticed the same huge mispresentation of wakeups. But now with 64-bit Ubuntu the problem is not there, so it's probably something that depends on the architecture-speficic code in Linux.

I'm now having ca. 50 wakeups per second on idle if Bluetooth is disabled (sudo hciconfig hci0 down). Bluetooth unfortunately creates 100 wakeups per second by itself.

Hello again Colin

My happiness with "noapic" has waned, because I am now getting hard lockups requiring a power cycle. It seems to happen when I am using Skype and waggle my USB mouse around, so I guess it's causing a lot of interrupts and overwhelming something.

Are we any closer to a *real* fix for this issue for us people stupid enough to have installed the 32 bit kernel? :)

Cheers.

On further small information:
It seems that i'm having the big amount of wakeups only when i'm running on AC.
After i plugged the battery in and the AC off, all went well!

I so for some days two new patches. Thirst one may help you ( pcmcia:
irq probe can be done without risking an IRQ storm):

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=42c59208219a2d43f0dde94bebc68c20b95b13ce

author
Linus Torvalds
<email address hidden>

Mon, 14 Jul 2008
20:24:39 +0000 (13:24
-0700)
committer
Linus Torvalds
<email address hidden>

Mon, 14 Jul 2008
20:24:39 +0000 (13:24
-0700)
commit
42c59208219a2d43f0dde94bebc68c20b95b13ce
tree
ff20941f83a92ffb4224c95ddee9b7eb225ed958
tree | snapshot
parent
dddec01eb8e2b56267b37a6f9f0997a64b4e0b2a
commit | diff
parent
727c6742c29e46177951fdc8f6758085e03bb981
commit | diff
Merge git://git./linux/kernel/git/brodo/pcmcia-2.6

* git://git.kernel.org/pub/scm/linux/kernel/git/brodo/pcmcia-2.6: (23 commits)
  pcmcia: Fix ide-cs sparse warning
  pcmcia: ide-cs debugging bugfix
  pcmcia: allow for longer CIS firmware files
  pcmcia: cm40x0 cdev lock_kernel() pushdown
  pcmcia: (re)move {pcmcia,pccard}_get_status
  pcmcia: kill IN_CARD_SERVICES
  pcmcia: Remove unused header file code
  pcmcia: remove unused bulkmem.h
  pcmcia: simplify pccard_validate_cis
  pcmcia: carve out ioctl adjust function to pcmcia_ioctl
  pcmcia: irq probe can be done without risking an IRQ storm
  pcmcia: Fix ti12xx_2nd_slot_empty always failing
  pcmcia: check for pointer instead of pointer address
  pcmcia: switch cm4000_cs.c to unlocked_ioctl
  pcmcia: simplify rsrc_nonstatic attributes
  pcmcia: add support CompactFlash PCMCIA support for Blackfin.
  pcmcia: remove version.h
  pcmcia: cs: kill thread_wait
  pcmcia: i82365.c: check request_irq return value
  pcmcia: fix Alchemy warnings

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=1adb0850a1254333d81e64121c80af100c6d6e06

author
Thomas Gleixner
<email address hidden>

Mon, 28 Apr 2008
15:01:56 +0000 (17:01
+0200)
committer
Thomas Gleixner
<email address hidden>

Fri, 2 May 2008 11:40:34
+0000 (13:40 +0200)
commit
1adb0850a1254333d81e64121c80af100c6d6e06
tree
61835b06e78eb6f556c038ceabc706440f339d3a
tree | snapshot
parent
886c35fbcf6fb2eee15687efc2d64d99b6ad9a4a
commit | diff
genirq: reenable a nobody cared disabled irq when a new driver arrives

Uwe Kleine-Koenig has some strange hardware where one of the shared
interrupts can be asserted during boot before the appropriate driver
loads. Requesting the shared irq line from another driver result in a
spurious interrupt storm which finally disables the interrupt line.

I have seen similar behaviour on resume before (the hardware does not
work anymore so I can not verify).

Change the spurious disable logic to increment the disable depth and
mark the interrupt with an extra flag which allows us to reenable the
interrupt when a new driver arrives which requests the same irq
line. In the worst case this will disable the irq again via the
spurious trap, but there is a decent chance that the new driver is the
one which can handle the already asserted interrupt and makes the box
usable again.

Eric Biederman said further: This case also happens on a regular basis
in kdump kernels where we deliberately don't shutdown the hardware
before starting the new...

commit 635416ef393e8cec5a89fc6c1de710ee9596a51e
Author: Alan Cox <email address hidden>
Date: Mon Jun 16 14:35:15 2008 +0200

    pcmcia: irq probe can be done without risking an IRQ storm

    Nowdays you can ask for an IRQ to be allocated but not enabled, when
PCMCIA
    was written this was not true and this feature is thus not used

    [<email address hidden>: add comment and ifdef to avoid
compilation
     breakage at least on alpha]
    Signed-off-by: Alan Cox <email address hidden>
    Signed-off-by: Andrew Morton <email address hidden>
    Signed-off-by: Dominik Brodowski <email address hidden>

diff --git a/drivers/pcmcia/pcmcia_resource.c
b/drivers/pcmcia/pcmcia_resource.c
index c8f77b8..78af594 100644
--- a/drivers/pcmcia/pcmcia_resource.c
+++ b/drivers/pcmcia/pcmcia_resource.c
@@ -812,6 +812,15 @@ int pcmcia_request_irq(struct pcmcia_device *p_dev,
irq_req
                type = IRQF_SHARED;

 #ifdef CONFIG_PCMCIA_PROBE
+
+#ifdef IRQ_NOAUTOEN
+ /* if the underlying IRQ infrastructure allows for it, only
allocate
+ * the IRQ, but do not enable it
+ */
+ if (!(req->Attributes & IRQ_HANDLE_PRESENT))
+ type |= IRQ_NOAUTOEN;
+#endif /* IRQ_NOAUTOEN */
+
        if (s->irq.AssignedIRQ != 0) {
                /* If the interrupt is already assigned, it must be the
same */
                irq = s->irq.AssignedIRQ;

The Ubuntu Kernel Team is planning to move to the 2.6.27 kernel for the upcoming Intrepid Ibex 8.10 release. As a result, the kernel team would appreciate it if you could please test this newer 2.6.27 Ubuntu kernel. There are one of two ways you should be able to test:

1) If you are comfortable installing packages on your own, the linux-image-2.6.27-* package is currently available for you to install and test.

--or--

2) The upcoming Alpha5 for Intrepid Ibex 8.10 will contain this newer 2.6.27 Ubuntu kernel. Alpha5 is set to be released Thursday Sept 4. Please watch http://www.ubuntu.com/testing for Alpha5 to be announced. You should then be able to test via a LiveCD.

Please let us know immediately if this newer 2.6.27 kernel resolves the bug reported here or if the issue remains. More importantly, please open a new bug report for each new bug/regression introduced by the 2.6.27 kernel and tag the bug report with 'linux-2.6.27'. Also, please specifically note if the issue does or does not appear in the 2.6.26 kernel. Thanks again, we really appreicate your help and feedback.

Samsung NP-R20 notebook (Intel Core Duo T2350, ATI RS600ME+SB600 chipset, Radeon Xpress 1250 IGP, 2GB RAM), BIOS/MICOM 15SH

Full specs: http://www.samsungpc.com/gb/support/r20/specs/r20spec.pdf

Intrepid with kernel 2.6.27-7-generic
Linux r20 2.6.27-7-generic #1 SMP Tue Nov 4 19:33:20 UTC 2008 i686 GNU/Linux

PowerTOP reports ~50000(!) wakeups-from-idle per second, while the six top causes for wakeups shown in the default view only add up to approx. 100. Avg residency for an idling system is ~60% "C0 (cpu running)", ~40% C2.

Adding the noapic kernel option results in only the aforementioned ~100 wakeups being listed, and ~99% C2. Cooling fan then alternates between low and medium speed, while it never slows down without noapic.

Per a decision made by the Ubuntu Kernel Team, bugs will longer be assigned to the ubuntu-kernel-team in Launchpad as part of the bug triage process. The ubuntu-kernel-team is being unassigned from this bug report. Refer to https://wiki.ubuntu.com/KernelTeamBugPolicies for more information. Thanks.

Unfortunately it seems this bug is still an issue. Can you confirm this issue exists with the most recent Jaunty Jackalope 9.04 release - http://www.ubuntu.com/news/ubuntu-9.04-desktop . Please let us know your results. Thanks.

Since I switched to using amd64 builds on my Core2, instead of i386, the problem went away. So sorry, I can't tell you if it's still a problem for i386.

I'm still seeing what I described above with a clean install of 9.04 (2.6.28-11-generic).

@luminous blue variable:

it is curious that powertop is recording a high level of wakeups per second, but not showing which is the offending culprit that is doing this high level of wakeups. I will consult the powertop folks about this.

Yes, this is still an issue. I am running Karmic amd64 on a Thinkpad X200, and had ca. 100-200 "Rescheduling Interrupts" wakeups per second. After booting with "acpi=noirq", these were reduced to ca. 10-20/s, saving me another 0.5W of power consumption.

What negative side effects should I expect from "acpi=noirq"?

Hmm, I noticed this side effect: On both my X200 and another X61s, the USB devices sometimes do not work after initialization by the kernel when acpi=noirq is set. I tried rebooting, power-cycling etc., but the only remedy was to remove acpi=noirq.

This bug was filed against a series that is no longer supported and so is being marked as Won't Fix. If this issue still exists in a supported series, please file a new bug.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.