Let me apologise in advance for not being able to narrow this one down further, but it's something in the kernel... so my user space tools aren't helping me track this down.
I have a thinkpad T43 (1.87GHz P-M). The system ran great when I was running breezy. When I upgraded to dapper (final release), the system was slugish, even after a fresh bootup. The cpu monitor applet showed cpu spikes every few seconds, lasting about one second (or one slice on the cpu monitor). How often these spikes happen, gets progressively worse and worse the more you use the system. The frequency of these spikes eventually gets so bad the cpu won't spin down to 800MHz, and will stay at 1.06GHz, then it'll get worse, and evenutally just stay stuck at 1.87GHz. Even after you quit all of your apps and sit at a blank desktop.
Running top shows no cpu being used (except for top, which runs about .3% cpu or so). vmstat however, shows the *real* picture. Here is my system after a fresh bootup, but sitting in X windows:
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
0 0 0 375608 187040 230140 0 0 0 14 1126 217 5 15 80 0
2 0 0 375608 187048 230132 0 0 0 3 1124 212 5 23 73 0
0 0 0 375608 187056 230124 0 0 0 4 1112 200 23 12 65 0
0 0 0 375600 187064 230116 0 0 0 3 1093 230 12 16 72 0
0 0 0 375600 187072 230108 0 0 0 3 1090 237 8 27 65 0
0 0 0 375608 187072 230108 0 0 0 13 1089 221 21 14 64 0
Note all the user and system time. After using my computer (mostly webbrowsing, maybe a bunch of ssh sessions) for about an hour it looks like this:
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
0 0 0 217664 200236 256792 0 0 0 90 1217 911 65 16 19 0
0 0 0 217788 200244 256784 0 0 0 15 1390 979 56 24 20 0
0 0 0 219300 200252 256776 0 0 0 4 1388 1700 61 20 18 0
1 0 0 219300 200260 256768 0 0 0 14 1297 1223 61 21 18 0
0 0 0 219300 200264 256764 0 0 0 6 1126 633 58 18 23 0
1 0 0 220664 200264 256764 0 0 0 25 1354 1658 65 20 15 0
Now it's mostly NOT idle.
I still have my 2.6.12 kernel (from breezy) installed. If I boot back into that kernel, everything is competely fine. This problem is totally not present. I have rebooted a dozen times, and it's not just some flakey thing, it happens everytime I boot the 2.6.15-23 kernel, and NEVER when I boot the 2.6.12-9 kernel. (I am using a 686 kernel for both).
To make matters worse, I upgraded to the 2.6.12-25 kernel last night. This kernel is basically unusable on my T43. The kernel still has the "spikey" cpu problem... But now when I use an aterm (after a freshbootup noless) the system is so sluggish it can't keep up with my typing. The graphics when it opens a new window (the rectangle coming out from the icon when you click it) is so slow it takes a full second to render. Some apps can keep up with my typing, for example firefox. It seems 'okay'. However, very wierd.. If I drag the firefox window around the screen (even if there are no windows below it) it takes nearly my ENTIRE CPU (according to the CPU applet). However top shows Xorg only taking 7%. Top shows 50% system time being taken. If I boot back into the -23 kernel, this particular problem goes away. So again, another problem that seems kernel related.
So if you can think of anythign I can do to try to debug this... I will happily perform tests. I can't figure out where to go from here. I tried killing all of the user space programs last night (including X)... still seeing the cpu spikes.
At this point, none of the dapper kernels are really usable for me (although the -23 kernel is atleast somewhat useful if I'm not on battery). I have to use the breezy kernel now, although some of my devices don't work now that the rest of the system has been dapperized.
Can confirm 2.6.15-25-686 being sluggish on my Sony VAIO VGN-FS295VP. Can't really remember how the breezy kernel was, used dapper throughout development. 2.6.15-23-686 is much faster for me.