Linux 3.2 freezes system on FUJITSU ESPRIMO P7935

this bug may be related to or duplicate of bugs #891830 and #911210

The problem still persists with mainline kernels downloaded from http://kernel.ubuntu.com/~kernel-ppa/mainline/daily/2012-01-09-precise/

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 914161

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Thank you for taking the time to file a bug report on this issue.

However, given the number of bugs that the Kernel Team receives during any development cycle it is impossible for us to review them all. Therefore, we occasionally resort to using automated bots to request further testing. This is such a request.

We have noted that there is a newer version of the development kernel than the one you last tested when this issue was found. Please test again with the newer kernel and indicate in the bug if this issue still exists or not.

You can update to the latest development kernel by simply running the following commands in a terminal window:

    sudo apt-get update
    sudo apt-get upgrade

If the bug still exists, change the bug status from Incomplete to Incomplete. If the bug no longer exists, change the bug status from Incomplete to Fix Released.

If you want this bot to quit automatically requesting kernel tests, add a tag named: bot-stop-nagging.

 Thank you for your help, we really do appreciate it.

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

I did reinstall the same machine with debian/wheezy and was unable to reproduce this with the kernel from experimental:

Linux faui49i 3.2.0-rc7-amd64 #1 SMP Wed Dec 28 14:29:59 UTC 2011 x86_64 GNU/Linux

Installing the kernel from http://kernel.ubuntu.com/~kernel-ppa/mainline/daily/2012-01-09-precise/ does trigger this bug again.

Maybe the bug was introduced after 3.2.0-rc7?

I'm reinstalling ubuntu oneiric again and will test with kernels from
http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.2-rc6-precise/

Thanks for the update. It would be great to know if running 3.2.0-rc6 makes the bug go away. If it does not, it would be great if you can test some prior rc kernels, to identify when the regression was introduced.

Thank you for taking the time to file a bug report on this issue.

However, given the number of bugs that the Kernel Team receives during any development cycle it is impossible for us to review them all. Therefore, we occasionally resort to using automated bots to request further testing. This is such a request.

We have noted that there is a newer version of the development kernel than the one you last tested when this issue was found. Please test again with the newer kernel and indicate in the bug if this issue still exists or not.

You can update to the latest development kernel by simply running the following commands in a terminal window:

    sudo apt-get update
    sudo apt-get upgrade

If the bug still exists, change the bug status from Incomplete to Confirmed. If the bug no longer exists, change the bug status from Incomplete to Fix Released.

If you want this bot to quit automatically requesting kernel tests, add a tag named: bot-stop-nagging.

 Thank you for your help, we really do appreciate it.

Oneiric with the kernel from http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.2-rc6-precise/ does trigger this issue.

Oneric with kernel from http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.1.8-precise/ does not seem to trigger this issue.

Oneric with kernel from http://kernel.ubuntu.com/~kernel-ppa/mainline/daily/2012-01-12-precise/ does trigger this issue.

Oneiric with kernel from http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.2-rc1-oneiric/ does not seem to trigger this issue.

Oneiric with kernel from http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.2-rc2-oneiric/ does not seem to trigger this issue.

Oneiric with kernel from http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.2-rc4-oneiric/ does not seem to trigger this issue.

note, I didn't test rc3 because that kernel failed to build an no packages are available.

Oneiric with the kernel from http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.2-rc5-precise/ does trigger this issue

Interestingly this issue *is* reproducible with http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.2-rc4-precise/

Does this indicate that the issue is toolchain related?

Thank you for taking the time to file a bug report on this issue.

However, given the number of bugs that the Kernel Team receives during any development cycle it is impossible for us to review them all. Therefore, we occasionally resort to using automated bots to request further testing. This is such a request.

We have noted that there is a newer version of the development kernel than the one you last tested when this issue was found. Please test again with the newer kernel and indicate in the bug if this issue still exists or not.

You can update to the latest development kernel by simply running the following commands in a terminal window:

    sudo apt-get update
    sudo apt-get upgrade

If the bug still exists, change the bug status from Incomplete to Confirmed. If the bug no longer exists, change the bug status from Incomplete to Fix Released.

If you want this bot to quit automatically requesting kernel tests, add a tag named: bot-stop-nagging.

 Thank you for your help, we really do appreciate it.

Sorry, the symptoms still happens with 3.2.0-9.16.

As indicated above, I suspect a toolchain issue.

Reinhard, thanks for all the testing.

Can you confirm the following:
v3.2-rc2-oneiric GOOD
v3.2-rc4-oneiric GOOD
v3.2-rc4-precise BAD
v3.2-rc5-precise BAD

I'll see what changes there are between v3.2-rc4-oneiric and v3.2-rc4-precise. I'll also perform a bisect if needed and generate some test kernels, if you have time to test them?

Hi Joseph,

I have tested the following two kernels just today:

v3.2-rc4-oneiric GOOD
v3.2-rc4-precise BAD

I'm happy to test additional test kernels on my test machine. Thank you so much for investigating this issue, since we have a pool with a number of machines of this type.

Reinhard,

This would indicate the issue is caused in the kernel config files, since that is the only difference between these two kernels. I'll do a diff of the config files to see what all the differences are.

One additional test would be to test 3.2.0-3.7 vs 3.2.0-2.6, since 3.2.0-2.7 was when the kernel was re-based to v3.2-rc4. The kernels are available at:

3.2.0-2.6:
https://launchpad.net/ubuntu/+source/linux/3.2.0-2.6/+build/2976027

3.2.0-3.7:
https://launchpad.net/ubuntu/+source/linux/3.2.0-3.7/+build/2981087

On Do, Jan 26, 2012 at 18:34:59 (CET), Joseph Salisbury wrote:

> This would indicate the issue is caused in the kernel config files,
> since that is the only difference between these two kernels. I'll do a
> diff of the config files to see what all the differences are.

Right, that also seems plausible to me.

> One additional test would be to test 3.2.0-3.7 vs 3.2.0-2.6, since
> 3.2.0-2.7 was when the kernel was re-based to v3.2-rc4. The kernels are
> available at:
>
> 3.2.0-2.6:
> https://launchpad.net/ubuntu/+source/linux/3.2.0-2.6/+build/2976027
>
> 3.2.0-3.7:
> https://launchpad.net/ubuntu/+source/linux/3.2.0-3.7/+build/2981087

I can confirm that both these kernels exhibit the behavior descriped in
the bug description of this bug.

IOW: both are BAD.

--
Gruesse/greetings,
Reinhard Tartler, KeyID 945348A4

To make things more interesting, I've just tested https://launchpad.net/ubuntu/precise/amd64/linux-image-3.2.0-1-generic/3.2.0-1.3 on this machine and that kernel also exhibits the described symptoms above.

That is interesting. Kernel 3.2.0-1.3 was re-based to upstream v3.2-rc2. I recall you did not hit this bug when testing mainline v3.2-rc4, in comment #36.

There are all of the kernels for Precise listed on the page:
https://launchpad.net/ubuntu/precise/+source/linux

Would it be possible for you to test some of these kernels? We already know 3.2.0-1.3 is bad, so maybe test 3.2.0-1.1:
https://launchpad.net/ubuntu/+source/linux/3.2.0-1.1

The goal is to find the first kernel version that exhibited this bug, so we can bisect and identify the exact commit.

Oneiric with the kernel from https://launchpad.net/ubuntu/+source/linux/3.2.0-1.1 does trigger this issue

Oneiric with the kernel from
https://launchpad.net/ubuntu/precise/amd64/linux-image-3.1.0-2-generic/3.1.0-2.3
does not seem to trigger this issue.

Is there anything else I could collect data-wise?

Hi Reinhard,

Sorry for the delay. I will build some additional test kernels to continue the bisect. I'll post a link to the test kernel shortly.

It looks like the following config file changes were made between v3.1.0-2.3 and v3.2.0-1.1:

  * [Config] enforcer -- ensure CONFIG_FAT_FS is built-in on arm
  * [Config] Enable PCI_IOV on powerpc
  * [Config] Temporarily disable CONFIG_PASEMI_MAC on powerpc
  * [Config] updateconfigs after select ARM_AMBA
  * [Config] Temporarily disable CONFIG_KVM_BOOK3S_32 on powerpc
  * [Config] Enable CONFIG_EXT2_FS=m
  * [Config] Build in CONFIG_SATA_AHCI=y
  * [Config] Enable EVENT_POWER_TRACING_DEPRECATED=y for powertop
  * [Config] Built-in xen-netfront and xen-blkfront
  * [Config] CONFIG_USB_XHCI_HCD=y
  * [Config] CONFIG_R6040=m
  * [Config] Consolidated amd64 server flavour into generic
  * [Config] updateconfigs after rebase to 3.2-rc1
  * [Config] Disabled dm-raid4-5
  * [Config] Disabled ndiswrapper
  * [Config] Disable vt6656
  * [Config] exclude ppp-modules for virtual flavour
  * [Config] CONFIG_MEMSTICK_R592=m

I'll investigate further and will post a test kernel. I may be able to build a v3.2.0-1.1 test kernel using the v3.1.0-2.3 config file. That will confirm that one of the config options about causes this bug.

I created a test kernel, which is available at:
http://people.canonical.com/~jsalisbury/lp914161

Can you test that kernel and report back if the bug still exists or not?

I've been running the kernel from http://people.canonical.com/~jsalisbury/lp914161/ kernel for over 3h now, and it does not exhibit the bug:

>> uname -a
Linux faui49i 3.2.0-1-generic #1~lp914161v1 SMP Thu Feb 16 21:16:57 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
>> uptime
  15:26:09 up 3:19, 1 user, load average: 0.00, 0.01, 0.05

As an extra saftey check, I've tried kernel v3.2.0-1.1 again, and it exposed the bug after a couple of minutes of uptime.

That means that a config file change between 3.1.0-2.3 and 3.2.0-1.1 caused this bug.

When the bug happens, do you get any output on the screen? It would be very helpful if we could capture a trace or a screen shot of the panic.

Unfortunately, there isn't any output of the screen. No matter if in X or on the console, the screen just freezes in the sense that it is not updated anymore and the content at the time of the freeze becomes static. There is no visual corruption in X11 either.

I'm not even sure if the kernel actually panics, or if it "just" hangs in some ISR or something.

Thanks for the update, Reinhard.

I will bisect through the kernel config options to identify which specific one is causing this bug. I will have another test kernel posted shortly.

I created another test kernel, which is available at:
http://people.canonical.com/~jsalisbury/lp914161

Can you test that kernel and report back if the bug still exists or not?

the first of the two testkernel seems to be GOOD:

>> uname -a
Linux faui49i 3.2.0-1-generic #1~lp914161v1 SMP Thu Feb 16 21:16:57 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
tartler-@faui49i:~
>> uptime
 12:44:06 up 2:52, 1 user, load average: 0.00, 0.01, 0.05

I'm going to test the 'v2' one now.

Also the 'v2' kernel seems to be GOOD:

root@faui49i:~# uname -a
Linux faui49i 3.2.0-1-generic #1~lp914161v2 SMP Fri Feb 17 20:45:25 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
root@faui49i:~# uptime
 13:35:44 up 49 min, 1 user, load average: 0.00, 0.01, 0.05

hm. If I had to guess, I would pick CONFIG_USB_XHCI_HCD. Is there a command-line option to disable this driver at boot time?

There is no way to disable that specific option. However, I can build a test kernel with just that option disabled. I'll post a link to it shortly.

I created another test kernel, which is available at:
http://people.canonical.com/~jsalisbury/lp914161

This test kernel has the CONFIG_USB_XHCI_HCD option set to m, which is how it is set in v3.1.0-2.3.

Can you test that kernel and report back if the bug still exists or not?

Hmrpf. wrong pick, this kernel again triggers the symptom after a couple of minutes:

unaroot@faui49i:~# uname -a
Linux faui49i 3.2.0-1-generic #1~lp914161v3 SMP Tue Feb 21 16:26:36 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

Thanks for the update, Reinhard. It was definitely worth a test. I'll continue to bisect the kernel config options and have another test kernel built shortly.

I created another test kernel, which is available at:
http://people.canonical.com/~jsalisbury/lp914161

Can you test that kernel and report back if the bug still exists or not?

Still BAD:

root@faui49i:~# uname -a
Linux faui49i 3.2.0-1-generic #1~lp914161v4 SMP Tue Feb 21 22:42:16 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

This kernel again hangs after a couple of minutes of uptime.

Thanks for the update.

That's a bit of good news that the v4 kernel is bad, since the v2 kernel was good. I enabled the following config parameters in the v4 kernel:

CONFIG_IRQ_REMAP=y
 CONFIG_INTEGRITY=y
CONFIG_INTEL_IOMMU=y
CONFIG_INTEL_IOMMU_DEFAULT_ON=y
CONFIG_INTEL_IOMMU_FLOPPY_WA=y

I'll build a test kernel with the IOMMU options disabled like they are in 3.1.0-2.3.

I created another test kernel, which is available at:
http://people.canonical.com/~jsalisbury/lp914161

Can you test that kernel and report back if the bug still exists or not?

This kernel appears to be GOOD:

root@faui49i:~# uname -a
Linux faui49i 3.2.0-1-generic #1~lp914161v5 SMP Thu Feb 23 16:45:58 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
root@faui49i:~ $ uptime
 09:49:46 up 43 min, 1 user, load average: 0.00, 0.01, 0.05

I'll continue running this kernel for rest of today, just to be sure, but other than that, I'd say 45min already classifies as good.

Thanks, Reinhard.

I'm building one additional kernel. This v6 kernel will have the following re-enable:
CONFIG_IRQ_REMAP=y
CONFIG_INTEGRITY=y

I will leave the following options disabled still to confirm they are the cause of this bug:
CONFIG_INTEL_IOMMU
CONFIG_INTEL_IOMMU_DEFAULT_ON
CONFIG_INTEL_IOMMU_FLOPPY_WA

Actually a really quick test would be to boot back with the latest precise kernel, but use the following boot option:
intel_iommu=off

Unfortunately, I have to leave now for the weekend and will be able to restart testing on monday.

Nevertheless, I've booted the latest precise kernel without intel_iommu=off to verify that the bug still persists:

root@faui49i:~# uname -a
Linux faui49i 3.2.0-17-generic #26-Ubuntu SMP Fri Feb 17 21:35:49 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
root@faui49i:~# uptime
 19:27:55 up 24 min, 1 user, load average: 0.00, 0.01, 0.06

So it *seems* the bug does no longer happen, but I really need to keep the machine running for a longer amount of time to be sure. Nevertheless, the conclusio of bug #907377 was to disable INTEL_IOMMU by default AFAIUI the changelog:

  * Revert "SAUCE: dmar: disable if ricoh multifunction detected"
  * [Config] Disable CONFIG_INTEL_IOMMU_DEFAULT_ON
    - LP: #907377, #911236
  * [Config] Enable CONFIG_IRQ_REMAP

Therefore, I believe that the option intel_iommu=off is now used by default. Will this option be kept for precise release?

I've been running the latest precise kern now for over 6h without problems.

However, passing 'intel_iommu=on' to the kernel command line does make the problem re-appear. So this indeed seems to be a bug in the intel iommu driver.

@Reinhard

We intend to keep iommu disabled by default. So it appears this bug is resolved since you have to purposely enable intel_iommu to make the bug come back.

I will mark the bug as "Fix Release" since it is resolved in the latest Precise kernel.

On Mo, Feb 27, 2012 at 19:02:19 (CET), Joseph Salisbury wrote:

> We intend to keep iommu disabled by default. So it appears this bug is
> resolved since you have to purposely enable intel_iommu to make the bug
> come back.

I'm glad to hear that!

> I will mark the bug as "Fix Release" since it is resolved in the latest
> Precise kernel.

Nevertheless, this bug was big fun to work on. It has been a pleasure to
work with you here :-)

--
Gruesse/greetings,
Reinhard Tartler, KeyID 945348A4

I enjoyed working with you as well, Reinhard!