2.6.12-9-686 and SMP kernel and Ubuntu Live CD Panic During Boot with DPT/Adaptec I2O Controller - Cause: Loads both dpt_i2o and the i2o drivers.

To reproduce - boot a machine with an Adaptec 2100S (and presumably any other DPT I2O SCSI card) with current Kubuntu Live CD (Ubuntu Live CD should do
the same) and it will panic in the same way.

I didn't get enough time last night to rebuild the initrd, will try again this evening.

Striping out the I2O subsystem from the cpio archive created by initramfs
doesn't work - it then whinges about not being able to mount the root device.

Creating an initrd with mkinitrd works though as it doesn't try to include I2O
in the first place. It now boots to the point where the init scripts start (far
further than before).

Unfortunately the machine *then* crashes after mounting filesystems with the
same panic as before, presumably because something else (like hotplug) tries to
load those devices. Unfortunately I couldn't see far back enough to confirm
that it was hotplug doing this and now the machine hangs at hotplug. :-(

I'm going to go grab a coffee to see if hotplug eventually completes and if not
boot back into the 2.6.10 kernel from Hoary and see if it passes hotplug there
or whether something has broken terminally now.

Chris

OK - hotplug didn't exit under 2.6.12-9-686-smp, but booting into the
2.6.10-5-686-smp kernel from Hoary was fine.

I then completely removed the I2O subsystem by moving
/lib/modules/2.6.12-9-686-smp/kernel/drivers/message/i2o out of the modules tree
and reran depmod for that version.

Rebooting into 2.6.12-9-686-smp then finally worked!

Any ideas on how to avoid this happening in future ?

Changed severity to "critical" as the bugzilla docs say that's what crashes should be marked as.

Apologies for not noticing that sooner!

Please see https://wiki.ubuntu.com/HelpingWithBugs

(In reply to comment #5)

> Please see https://wiki.ubuntu.com/HelpingWithBugs

Thanks Matt - sorry about that.

Given that I've figured out (hopefully) how to blacklist the I2O subsystem in udev, does
anyone have any hints on how to either stop mkinitramfs from loading it or on how to
force the kernel packages to use mkinitrd rather than mkinitramfs ?

(In reply to comment #6)

> Given that I've figured out (hopefully) how to blacklist the I2O subsystem in udev,

Argh - of course that should be hotplug, not udev..

(In reply to comment #1)
> To reproduce - boot a machine with an Adaptec 2100S
> (and presumably any other DPT I2O SCSI card) with
> current Kubuntu Live CD (Ubuntu Live CD should do
> the same) and it will panic in the same way.

Flight CD 1 (Dapper Drake) Live CD panics as well on this machine.

Same problem, erroneously tries to load both dpt_i2o and the I2O subsystem.

Chris

This bug has been fixed in the latest kernel in our Dapper release. There are no
plans to fix this in breezy.

(In reply to comment #9)

> This bug has been fixed in the latest kernel in our Dapper release. There are no
> plans to fix this in breezy.

Hi Ben,

Thanks so much for this good news, when the next test live CD arrives I'll try that out and let
you know how it goes here.

Can I ask how this was fixed please ?

thanks again!
Chris

(In reply to comment #10)
> Can I ask how this was fixed please ?

Upstream fixed the i2o drivers so that they didn't ignore the return value of
request_irq(), thus allowing them to both be loaded at once (even if one doesn't
end up working).

(In reply to comment #9)

> This bug has been fixed in the latest kernel in our Dapper release. There are no
> plans to fix this in breezy.

Just tried Dapper Flight CD 2.

I'm afraid that I still get a crash during boot on my system, although now
with what looks like an Oops rather than a panic, this time it's saying
"scheduling while atomic".

The system appears to boot normally and problems only start when it reaches the
"Detecting and activating hardware..." script.

The Adaptec DPT I2O driver appears to load with no problems, finds the RAID-5,
then the I2O subsystem tries to load and fails because it's already claimed, and
then the system crashes.

Here's the text of the lead up and the failure from the Dapper Flight CD 2 boot
with the "quiet" option removed. Thankfully I could still scroll back unlike
previous panics so I could get the messages prior to the crash. I've had to
transcribe manually so I've ommitted the leading timestamps and any typos
are likely to be mine.

TID 008: Vendor: ADAPTEC Device: AIC-7899 Rev: 00000001
TID 519: Vendor: ADAPTECT Device: RAID-5 Rev: 370F

scsi0 : Vendor: Adaptec Model:2100S FW: 370F
  Vendor: ADAPTEC Model: RAID-5 Rev: 370F
  Type: Direct-Access ANSI SCSI revision: 02
SCSI device sda: 143372288 512-byte hdwr sectors (73407MB)
SCSI device sda: drive cache: write back
SCSI device sda: 143372288 512-byte hdwr sectors (73407MB)
SCSI device sda: drive cache: write back
 sda: sda1
sd 0:0:0:0: Attached scsi disk sda
I2O subsystem v1.288
i2o: max drivers = 8
i2o: checking for PCI I2O controllers...
ACPI: PCI Interrupt 0000:02:01.1[A] -> GSI 21 (level, low) -> IRQ 193
iop0: controller found (0000:02:01.1)
PCI: Unable to reserve mem region #1:2000000@dc000000 for device 0000:02:01.1
iop0: device already claimed
iop0: DMA / IO allocation for I2O controller failed
ACPI: PCI interrupt for device 0000:02:01.1 disabled
dpti0: Trying to Abort: cmd=42
scheduling while atomic: scsi_eh_0/0xffffffff/3654
 [<c02e7722>] schedule+0x5c2/0x690
 [<c011bab3>] vprintk+0x1d3/0x340
 [<f8a3164e>] adpt_i2o_post_wait+0x1ae/0x250 [dpt_i2o]
 [<c0117b20>] default_wake_function+0x0/0x20
 [<f8a309c2>] adpt_abort+0x92/0x120 [dpt_i2o]
 [<f8a1134e>] scsi_eh_abort_cmds+-x4e/0x100 [scsi_mod]
 [<f8a1217b>] scsi_unjam_host+0xab/0x200 [scsi_mod]
 [<f8a122d0>] scsi_error_handler+0x0/0x120 [scsi_mod]
 [<f8a12399>] scsi_error_handler+0xc9/0x120 [scsi_mod]
 [<c012fdf3>] kthread+0x93/0xa0
 [<c012fd60>] kthread+0x0/0xa0
 [<c0101385>] kernel_thread_helper+0x5/0x10

It then says the following:

                                                          [fail]
 * Loading modules... [ ok ]
 * Setting the system clock [ ok ]
 * Cleaning up ifupdown... [ ok ]
 * Setting the system clock [ ok ]
device-mapper: 4.4.0-ioctl (2005-01-12) initialised: <email address hidden>
 * Setting up LVM Volume Groups... [ ok ]
 * Starting Enterprise Volume Management System...

and hangs with the cursor at the right hand side of the screen,
waiting for the EVMS script to...

Can you see if you can get me more of the dmesg, around where it loads the
dpt_i2o driver?

(In reply to comment #13)

> Can you see if you can get me more of the dmesg, around where it loads the
> dpt_i2o driver?

I'll give it a go, I started the first one at the point I first noticed it
loading the DPT drivers, before that there's lots about loading drivers for
USB, parallel port, audio, etc, etc. Anything you want me to look for in
particular (e.g. IRQ messages or ACPI messages) ?

I ask because it's quite a few pages of error-prone typing which was why I
didn't attempt it previously.

cheers,
Chris

(In reply to comment #13)

> Can you see if you can get me more of the dmesg, around where it loads the
> dpt_i2o driver?

My bad - I missed this section completely the first time around!

[DVB frontend loads just prior to this]
ACPI: PCI Interrupt 0000:00:1f.5[B] -> GSI 17 (level, low) -> IRQ 217
Loading Adaptec I2O RAID: Version 2.4 Build 5go
Detecting Adaptec I2O RAID controllers...
ACPI: PCI Interrupt: 0000:02:01.1[A] -> GSI 21 (level, low) -> IRQ 209
Adaptec I2O RAID controller 0 at f8f00000 size=100000 irq=209
intel8x0_measure_ac97_clock: measured 152630 usecs
intel8x0" clocking to 48000
Intel 810 + AC97 Audio, version 1.01, 03:24:11 Dec 13 2005
dpti: If you have a lot of devices this could take a few minutes.
dpti0: Reading the hardware resource table.

and then it carries on with the TID 008 etc.. that I reported previously.

So to my novice eye it looks like ACPI is disabling the interrupt
for the Adaptec card just before the abort and the Oops.

If you need more detail of the modules loading before that point
then let me know.

thanks!
Chris

Just found that by doing ALT-SYSRQ-I to kill
current tasks I can force it to continue the boot,
including loading X and starting KDE, but obviously
some things aren't happy because DHCP doesn't happen.

However, it does mean that I can now mount a floppy and
grab useful bits out of /proc as well as as much dmesg
output that is left at that point, which I'll attach in
a few minutes.

Chris

Created an attachment (id=5364)
Output of lspci -vv

Output of lspci -vv from DD Flight CD 2 when finally booted.

Created an attachment (id=5365)
Output of dmesg

Output of dmesg from DD Flight CD 2 after finally getting it to boot.

Created an attachment (id=5366)
/proc/interrupts

Contents of /proc/interrupts after DD FCD2 boot

Created an attachment (id=5367)
/proc/iomem

Created an attachment (id=5368)
/proc/ioports

Created an attachment (id=5369)
/proc/dma

Created an attachment (id=5370)
/proc/cpuinfo

Thanks. After being able to look at the dmesg in more detail, your observation
is correct. When i2o gets loaded, and notices the device is in use, it attempts
to back out. Upon doing so, it calls pci_disable_device(). I used some examples
elsewhere, which showed that it was incorrect in doing this. So I changed it to
only make this call for every failure except the case where the device is in use
(this case).

Will be fixed in 2.6.15-9.11

(In reply to comment #24)
> Thanks. After being able to look at the dmesg in more detail, your observation
> is correct. When i2o gets loaded, and notices the device is in use, it attempts
> to back out. Upon doing so, it calls pci_disable_device(). I used some examples
> elsewhere, which showed that it was incorrect in doing this. So I changed it to
> only make this call for every failure except the case where the device is in use
> (this case).

That's great news, thanks so much for that. I presume the I2O subsystem changes
haven't led to it doing the same ?

> Will be fixed in 2.6.15-9.11

How easy would it be for me to modify the ISO image I've downloaded to use
the new kernel for burning and testing ? I did a quick bit of Googling
with no success.

(In reply to comment #25)

> That's great news, thanks so much for that. I presume the I2O subsystem changes
> haven't led to it doing the same ?

s/I2O subsystem/DPT driver/

ENOCAFFEINE, sorry.

2.6.15-9 is uploaded, so this bug is resolved.

(In reply to comment #27)

> 2.6.15-9 is uploaded, so this bug is resolved.

Wonderful - is there any way I can modify the current
Live CD to test this to avoid having to wait for the
next Flight CD release ?

(In reply to comment #28)
> (In reply to comment #27)
>
> > 2.6.15-9 is uploaded, so this bug is resolved.
>
> Wonderful - is there any way I can modify the current
> Live CD to test this to avoid having to wait for the
> next Flight CD release ?

Easiest way is to install breezy and dist-upgrade to dapper.

(In reply to comment #29)
>
> Easiest way is to install breezy and dist-upgrade to dapper.

I'm already running Breezy, but I'd rather not upgrade to Dapper
as I'm using the box for useful things and don't want to risk
breaking it at present - would much rather just use a Live CD.

I'll refer the question to my local Linux group in case anyone's
got any bright ideas there.. :-)

thanks!
Chris

I can confirm that this is now fine - finally upgraded to Dapper. Thanks!