random crashes caused by Xen kernel bug (radix-tree.c)

Bug #172632 reported by quam
18
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Won't Fix
Medium
Ubuntu Xen Team
linux-source-2.6.22 (Ubuntu)
Won't Fix
Undecided
Unassigned

Bug Description

The bug occurs at random intervals, uptime was about 20 days when the bug occured last.
The host is used as webserver running apache2, php5, mysql, no special configuration.
Filesystem is ext3 on a lvm partition using kernel software raid.
Please find below the output from the syslog.

------------[ cut here ]------------
kernel BUG at /build/buildd/linux-source-2.6.22-2.6.22/debian/build/custom-source-xen/lib/radix-tree.c:447!
invalid opcode: 0000 [#1]
SMP
Modules linked in: af_packet ipt_REJECT xt_tcpudp nf_conntrack_ipv4 xt_state iptable_filter ip_tables x_tables nf_conntrack_ftp nf_conntrack nfnetlink ipv6 evdev ext3 jbd mbcache capability commoncap
CPU: 0
EIP: 0061:[radix_tree_tag_set+152/160] Not tainted VLI
EFLAGS: 00010007 (2.6.22-14-xen #1)
EIP is at radix_tree_tag_set+0x98/0xa0
eax: c648e140 ebx: 00000002 ecx: 00000001 edx: c14cba00
esi: c648e13c edi: 00000000 ebp: c648e14c esp: c1393d80
ds: 007b es: 007b fs: 00d8 gs: 0000 ss: 0069
Process pdflush (pid: 61, ti=c1392000 task=c712f9f0 task.ti=c1392000)
Stack: 00000001 c14cba00 c648e140 c1489d80 c648e13c 00000000 c648e14c c0158823
       00000000 c1489d80 00000000 00000000 c648e098 c019d786 c011a628 00000000
       c7c38c14 c112b2d0 c906e250 c1489d80 c768b480 c14df4a0 00000000 c4d826c8
Call Trace:
 [test_set_page_writeback+163/256] test_set_page_writeback+0xa3/0x100
 [__block_write_full_page+438/864] __block_write_full_page+0x1b6/0x360
 [__wake_up+56/80] __wake_up+0x38/0x50
 [<c906e250>] ext3_get_block+0x0/0x100 [ext3]
 [block_write_full_page+247/256] block_write_full_page+0xf7/0x100
 [<c906e250>] ext3_get_block+0x0/0x100 [ext3]
 [<c906fae1>] ext3_ordered_writepage+0xf1/0x1a0 [ext3]
 [<c906c920>] bget_one+0x0/0x10 [ext3]
 [__writepage+8/48] __writepage+0x8/0x30
 [write_cache_pages+532/784] write_cache_pages+0x214/0x310
 [__writepage+0/48] __writepage+0x0/0x30
 [do_IRQ+64/112] do_IRQ+0x40/0x70
 [generic_writepages+32/48] generic_writepages+0x20/0x30
 [do_writepages+73/80] do_writepages+0x49/0x50
 [__writeback_single_inode+147/960] __writeback_single_inode+0x93/0x3c0
 [schedule_timeout+83/208] schedule_timeout+0x53/0xd0
 [schedule+854/2304] schedule+0x356/0x900
 [sync_sb_inodes+382/576] sync_sb_inodes+0x17e/0x240
 [writeback_inodes+153/208] writeback_inodes+0x99/0xd0
 [wb_kupdate+133/240] wb_kupdate+0x85/0xf0
 [pdflush+0/608] pdflush+0x0/0x260
 [pdflush+328/608] pdflush+0x148/0x260
 [wb_kupdate+0/240] wb_kupdate+0x0/0xf0
 [kthread+66/112] kthread+0x42/0x70
 [kthread+0/112] kthread+0x0/0x70
 [kernel_thread_helper+7/16] kernel_thread_helper+0x7/0x10
 =======================
Code: 8b 0c 24 8b 42 04 ba 01 00 00 00 83 c1 14 d3 e2 85 d0 75 09 8b 4c 24 08 09 d0 89 41 04 83 c4 0c 89 f8 5b 5e 5f 5d c3 0f 0b eb fe <0f> 0b eb fe 8d 74 26 00 55 89 c5 57 56 53 83 ec 48 89 54 24 08
EIP: [radix_tree_tag_set+152/160] radix_tree_tag_set+0x98/0xa0 SS:ESP 0069:c1393d80

Revision history for this message
Leann Ogasawara (leannogasawara) wrote :

The Hardy Heron kernel was recently uploaded for testing. We'd really appreciate it if you could try testing with this newer kernel and verify if this issue still exists. Unfortunately, the Hardy Heron Alpha1 LiveCD was released with the older 2.6.22 kernel. You'll have to manually install the newer Hardy Heron kernel in order to test. This should not be the case for Alpha2. However, here are the instructions to install (if you choose to do so):

1) edit the file /etc/apt/sources.list and add the following line:

deb http://archive.ubuntu.com/ubuntu hardy main restricted

2) sudo apt-get update
3) sudo apt-get install linux-image-2.6.24-1-generic
4) reboot and select the new kernel from the grub menu

After you've tested, please feel free to revert back - ie boot into the old kernel, sudo apt-get remove linux-image-2.6.24-1-generic, and remove the line from /etc/apt/sources.list . Please update this report with your results. Thanks in advance!

Changed in linux:
importance: Undecided → Medium
status: New → Incomplete
Revision history for this message
Nico (pub-1g6) wrote :

Similar oops with 2.6.22-14, in domU with XEN :

exact version of kernel :
kernel: [ 0.000000] Linux version 2.6.22-14-xen (buildd@palmer) (gcc version 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)) #1 SMP Mon Oct 15 00:35:38 GMT 2007 (Unofficial)

I do not see XEN kernel made for 2.6.24 ? , as requested in last comment

[245619.262174] BUG: unable to handle kernel paging request at virtual address 00100100
[245619.262191] printing eip:
[245619.262194] c019d6b1
[245619.262203] 0102e000 -> *pde = 00000000:1577f001
[245619.262207] 04338000 -> *pme = 00000000:00000000
[245619.262214] Oops: 0000 [#1]
[245619.262216] SMP
[245619.262223] Modules linked in: reiserfs dm_mirror dm_snapshot dm_mod fuse apparmor commoncap
[245619.262242] CPU: 0
[245619.262243] EIP: 0061:[<c019d6b1>] Not tainted VLI
[245619.262244] EFLAGS: 00010297 (2.6.22-14-xen #1)
[245619.262258] EIP is at __block_write_full_page+0xe1/0x360
[245619.262262] eax: c16a1180 ebx: 00100100 ecx: 00000000 edx: 0000000c
[245619.262266] esi: 00000ee1 edi: 00000000 ebp: c17963f0 esp: c46bfde4
[245619.262272] ds: 007b es: 007b fs: 00d8 gs: 0000 ss: 0069
[245619.262277] Process pdflush (pid: 26176, ti=c46be000 task=cae979f0 task.ti=c46be000)
[245619.262282] Stack: 00000110 00000110 c46bfe88 c745c6f0 c01a1d30 c16a1180 00000000 0013ffff
[245619.262298] 00000000 c162c2f4 00001000 00000001 c17963f0 c16a1180 00000000 c019da27
[245619.262313] c46bff70 00000000 c46bff70 c01a1d30 c16a1180 c1796494 0000000b c46bff70
[245619.262330] Call Trace:
[245619.262335] [<c01a1d30>] blkdev_get_block+0x0/0x70
[245619.262346] [<c019da27>] block_write_full_page+0xf7/0x100
[245619.262351] [<c01a1d30>] blkdev_get_block+0x0/0x70
[245619.262357] [<c0157a38>] __writepage+0x8/0x30
[245619.262364] [<c0157ef4>] write_cache_pages+0x214/0x310
[245619.262370] [<c0157a30>] __writepage+0x0/0x30
[245619.262377] [<c0158010>] generic_writepages+0x20/0x30
[245619.262384] [<c015804b>] do_writepages+0x2b/0x50
[245619.262389] [<c01980c3>] __writeback_single_inode+0x93/0x3c0
[245619.262401] [<c02ff1f6>] schedule+0x356/0x900
[245619.262410] [<c019877e>] sync_sb_inodes+0x17e/0x240
[245619.262416] [<c0198c49>] writeback_inodes+0x99/0xd0
[245619.262423] [<c0158715>] wb_kupdate+0x85/0xf0
[245619.262429] [<c0158ab0>] pdflush+0x0/0x260
[245619.262434] [<c0158bf8>] pdflush+0x148/0x260
[245619.262441] [<c0158690>] wb_kupdate+0x0/0xf0
[245619.262447] [<c0136312>] kthread+0x42/0x70
[245619.262454] [<c01362d0>] kthread+0x0/0x70
[245619.262461] [<c0105927>] kernel_thread_helper+0x7/0x10
[245619.262469] =======================
[245619.262471] Code: 24 24 eb 21 77 06 3b 74 24 1c 76 1f 90 0f ba 33 01 90 0f ba 2b 00 8b 5b 04 39 5c 24 24 74 68 83 c6 01 83 d7 0 0 3b 7c 24 20 73 d9 <8b> 03 a8 20 75 e5 8b 03 a8 02 74 df 8b 54 24 28 3b 53 14 0f 85
[245619.262563] EIP: [<c019d6b1>] __block_write_full_page+0xe1/0x360 SS:ESP 0069:c46bfde4

Revision history for this message
Brian Murray (brian-murray) wrote :

I am assigning this bug to the 'ubuntu-kernel-team' per their bug policy. For future reference you can learn more about their bug policy at https://wiki.ubuntu.com/KernelTeamBugPolicies .

Changed in linux-source-2.6.22:
assignee: nobody → ubuntu-kernel-team
Revision history for this message
Christian Bayle (christian-bayle) wrote :

I had exactly the same bug on a 2.6.24-17-xen kernel, except error is at line 460
It seems that renaming /lib/tls in /lib/tls.disabled solved the problem, but after I had the
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/229033 bug
seem to be solved by "blacklist ipv6" in /etc/modprobe.d/blacklist .
I could reproduce the bug easily just by stressing the dom0 with 2 concurrents debuild, one on kernel package, one on xen-tools

Changed in linux:
assignee: nobody → ubuntu-xen
status: Incomplete → Triaged
Revision history for this message
Leann Ogasawara (leannogasawara) wrote :

The Ubuntu Kernel Team is planning to move to the 2.6.27 kernel for the upcoming Intrepid Ibex 8.10 release. As a result, the kernel team would appreciate it if you could please test this newer 2.6.27 Ubuntu kernel. There are one of two ways you should be able to test:

1) If you are comfortable installing packages on your own, the linux-image-2.6.27-* package is currently available for you to install and test.

--or--

2) The upcoming Alpha5 for Intrepid Ibex 8.10 will contain this newer 2.6.27 Ubuntu kernel. Alpha5 is set to be released Thursday Sept 4. Please watch http://www.ubuntu.com/testing for Alpha5 to be announced. You should then be able to test via a LiveCD.

Please let us know immediately if this newer 2.6.27 kernel resolves the bug reported here or if the issue remains. More importantly, please open a new bug report for each new bug/regression introduced by the 2.6.27 kernel and tag the bug report with 'linux-2.6.27'. Also, please specifically note if the issue does or does not appear in the 2.6.26 kernel. Thanks again, we really appreicate your help and feedback.

Revision history for this message
Launchpad Janitor (janitor) wrote : Kernel team bugs

Per a decision made by the Ubuntu Kernel Team, bugs will longer be assigned to the ubuntu-kernel-team in Launchpad as part of the bug triage process. The ubuntu-kernel-team is being unassigned from this bug report. Refer to https://wiki.ubuntu.com/KernelTeamBugPolicies for more information. Thanks.

Revision history for this message
Amit Kucheria (amitk) wrote :

This bug was reported a while ago but there hasn't been any recent comments or updates. Is this still an issue with the latest pre-release of Jaunty 9.04? Refer to http://www.ubuntu.com/testing/jaunty/beta . Please let us know.

Also, this seems to be specific to Xen.

summary: - random crashes caused by kernel bug (radix-tree.c)
+ random crashes caused by Xen kernel bug (radix-tree.c)
Changed in linux (Ubuntu):
status: Triaged → Incomplete
Revision history for this message
Sergio Zanchetta (primes2h) wrote :

The 18 month support period for Gutsy Gibbon 7.10 has reached its end of life -
http://www.ubuntu.com/news/ubuntu-7.10-eol . As a result, we are closing the
linux-source-2.6.22 kernel task. It would be helpful if you could test the
new Jaunty Jackalope 9.04 release and confirm if this issue remains -
http://www.ubuntu.com/getubuntu/releasenotes/904overview. If the issue still exists with the Jaunty
release, please update this report by changing the Status of the "linux (Ubuntu)"
task from "Incomplete" to "New". Also please be sure to run the command below
which will automatically gather and attach updated debug information to this
report. Thanks in advance.

apport-collect -p linux-image-2.6.28-11-generic 172632

Changed in linux-source-2.6.22 (Ubuntu):
status: Confirmed → Won't Fix
Revision history for this message
Brad Figg (brad-figg) wrote : Unsupported series, setting status to "Won't Fix".

This bug was filed against a series that is no longer supported and so is being marked as Won't Fix. If this issue still exists in a supported series, please file a new bug.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: Incomplete → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.