chromeos 3.8 kernel fails to boot when compiled with gcc-4.8

I did builds on Chromebook running armhf/saucy:

hrw@krolik:~$ gcc --version
gcc (Ubuntu/Linaro 4.8.0-6ubuntu1) 4.8.0
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

How far into the boot process does the bad build fail? Do you get any output at all on the serial console (with earlyprintk)?

It does not display anything on screen. But combination of Power+Refresh (F3) keys == hard reset and then mount "pstore" somewhere and it should have console log of failed kernel.

Attached is log from failed build (done with older gcc-4.8).

Status changed to 'Confirmed' because the bug affects multiple users.

Used 4.8.0 from OpenEmbedded - failed same way:

[ 0.153366] Switching to clocksource mct-frc
[ 0.168712] AppArmor: AppArmor Filesystem Enabled
[ 0.173472] Unable to handle kernel NULL pointer dereference at virtual address 00000001
[ 0.173491] pgd = 80004000
[ 0.173500] [00000001] *pgd=00000000
[ 0.173515] Internal error: Oops: 5 [#1] SMP ARM
[ 0.173525] Modules linked in:
[ 0.173539] CPU: 0 Not tainted (3.8.0 #1)
[ 0.173554] PC is at kmem_cache_alloc_trace+0x78/0x168
[ 0.173566] LR is at kmem_cache_alloc_trace+0x40/0x168
[ 0.173577] pc : [<800f43d4>] lr : [<800f439c>] psr: 20000113
[ 0.173577] sp : ef0bfe00 ip : ef0bfe00 fp : ef0bfe34
[ 0.173591] r10: 000013b7 r9 : 00000000 r8 : 00000080
[ 0.173601] r7 : 802a614c r6 : 000000d0 r5 : 00000001 r4 : ef001200
[ 0.173611] r3 : 00000000 r2 : 80795640 r1 : 0112b000 r0 : 00000000
[ 0.173622] Flags: nzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment kernel
[ 0.173635] Control: 10c5387d Table: 4000406a DAC: 00000015
[ 0.173645] Process swapper/0 (pid: 1, stack limit = 0xef0be240)
[ 0.173655] Stack: (0xef0bfe00 to 0xef0c0000)
[ 0.173667] fe00: 8052df44 80261980 00000110 00002665 ef2df9c0 00000003 ef33a664 807e29d2
[ 0.173681] fe20: 00000000 807e29d4 ef0bfe54 ef0bfe38 802a614c 800f4368 ef008400 ef2df9c0
[ 0.173695] fe40: 00000003 00000001 ef0bfe94 ef0bfe58 802a6ec8 802a60a8 00000001 00000003
[ 0.173708] fe60: 00000001 ef008654 80030328 00000000 00000014 80891614 80766b3c 80808ac0
[ 0.173722] fe80: 00000000 ef0be020 ef0bfeb4 ef0bfe98 80766330 802a6df0 00000001 00000000
[ 0.173735] fea0: 808914f0 8055b1ec ef0bfed4 ef0bfeb8 80766800 807662f8 80662b8b ef0bfec8
[ 0.173749] fec0: 80891048 00000000 ef0bfef4 ef0bfed8 80765e08 807666a8 80664023 ef0bfefc
[ 0.173762] fee0: 0000000e 80892200 ef0bff1c ef0bfef8 80766bf4 80765d0c 8066309b ef0bff08
[ 0.173776] ff00: 80762db0 00000005 80773f48 8078efd0 ef0bff5c ef0bff20 80008758 80766b48
[ 0.173789] ff20: 00000005 00000005 000000ae 806cd10c 8004ea48 00000005 80773f48 8078efd0
[ 0.173803] ff40: 80808ac0 80808ac0 000000ae 00000000 ef0bff94 ef0bff60 8074ba78 800086c4
[ 0.173816] ff60: 00000005 00000005 8074b2c0 8052df58 80808ac0 80522608 00000000 00000000
[ 0.173830] ff80: 00000000 00000000 ef0bffac ef0bff98 80522624 8074b974 00000000 00000000
[ 0.173843] ffa0: 00000000 ef0bffb0 8000e118 80522614 00000000 00000000 00000000 00000000
[ 0.173856] ffc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 0.173869] ffe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
[ 0.173895] [<800f43d4>] (kmem_cache_alloc_trace+0x78/0x168) from [<802a614c>] (con_insert_unipair+0xb0/0xfc)
[ 0.173916] [<802a614c>] (con_insert_unipair+0xb0/0xfc) from [<802a6ec8>] (con_set_default_unimap+0xe4/0x174)
[ 0.173936] [<802a6ec8>] (con_set_default_unimap+0xe4/0x174) from [<80766330>] (console_map_init+0x44/0x58)
[ 0.173953] [<80766330>] (console_map_init+0x44/0x58) from [<80766800>] (vty_init+0x164/0x1a8)
[ 0.173969] [<80766800>] (vty_init+0x164/0x1a8) from [<80765e08>] (tty_init+0x108/0x144)
[ 0.173984] [<807...

Not exactly the same, the fault address is different.

Could you post a disassembly of kmem_cache_alloc_trace?

00003404 <kmem_cache_alloc_trace>:
    3404: e1a0c00d mov ip, sp
    3408: e92ddff0 push {r4, r5, r6, r7, r8, r9, sl, fp, ip, lr, pc}
    340c: e24cb004 sub fp, ip, #4
    3410: e24dd00c sub sp, sp, #12
    3414: e52de004 push {lr} ; (str lr, [sp, #-4]!)
    3418: ebfffffe bl 0 <__gnu_mcount_nc>
    341c: e59f3140 ldr r3, [pc, #320] ; 3564 <kmem_cache_alloc_trace+0x160>
    3420: e1a0700e mov r7, lr
    3424: e1a04000 mov r4, r0
    3428: e1a06001 mov r6, r1
    342c: e1a08002 mov r8, r2
    3430: e5933000 ldr r3, [r3]
    3434: e2033010 and r3, r3, #16
    3438: e1130001 tst r3, r1
    343c: 0a000000 beq 3444 <kmem_cache_alloc_trace+0x40>
    3440: ebfffffe bl 0 <_cond_resched>
    3444: ee1d1f90 mrc 15, 0, r1, cr13, cr0, {4}
    3448: e5942000 ldr r2, [r4]
    344c: e0813002 add r3, r1, r2
    3450: e593a004 ldr sl, [r3, #4]
    3454: e7915002 ldr r5, [r1, r2]
    3458: e3550000 cmp r5, #0
    345c: 1a000005 bne 3478 <kmem_cache_alloc_trace+0x74>
    3460: e1a00004 mov r0, r4
    3464: e1a01006 mov r1, r6
    3468: e1a02007 mov r2, r7
    346c: ebfffffe bl 838 <calculate_sizes+0x328>
    3470: e1a05000 mov r5, r0
    3474: ea00001a b 34e4 <kmem_cache_alloc_trace+0xe0>
    3478: e5943014 ldr r3, [r4, #20]
    347c: e7952003 ldr r2, [r5, r3]
    3480: e10f9000 mrs r9, CPSR
    3484: f10c0080 cpsid i
    3488: e5943000 ldr r3, [r4]
    348c: ee1d0f90 mrc 15, 0, r0, cr13, cr0, {4}
    3490: e7903003 ldr r3, [r0, r3]
    3494: e1530005 cmp r3, r5
    3498: 0a000001 beq 34a4 <kmem_cache_alloc_trace+0xa0>
    349c: e3a03000 mov r3, #0
    34a0: ea00000a b 34d0 <kmem_cache_alloc_trace+0xcc>
    34a4: e5943000 ldr r3, [r4]
    34a8: e283c004 add ip, r3, #4
    34ac: e790c00c ldr ip, [r0, ip]
    34b0: e15c000a cmp ip, sl
    34b4: 1afffff8 bne 349c <kmem_cache_alloc_trace+0x98>
    34b8: e7802003 str r2, [r0, r3]
    34bc: e28cc001 add ip, ip, #1
    34c0: e5943000 ldr r3, [r4]
    34c4: e2833004 add r3, r3, #4
    34c8: e780c003 str ip, [r0, r3]
    34cc: e3a03001 mov r3, #1
    34d0: e121f009 msr CPSR_c, r9
    34d4: e3530000 cmp r3, #0
    34d8: 0affffda beq 3448 <kmem_cache_alloc_trace+0x44>
    34dc: e5943014 ldr r3, [r4, #20]
    34e0: f7d2f003 pld [r2, r3]
    34e4: e3160902 tst r6, #32768 ; 0x8000
    34e8: 0a000006 beq 3508 <kmem_cache_alloc_trace+0x104>
    34ec: e3550000 cmp r5, #0
    34f0: 0a000004 beq 3508 <kmem_cache_alloc_trace+0x104>
    34f4: e5941010 ldr r1, [r4, #16]
    34f8: e3510000 cmp r1, #0
    34fc: 0a000001 beq 3508 <kmem_cache_alloc_trace+0x104>
    3500: e1a00005 mov r0, r5
    3504: ebfffffe bl 0 <__memzero>
    3508: e59f3058 ldr r3, [pc, #88] ; 3568 <kmem_cache_alloc_trace+0x164>
    350c: e594900c ldr r9, [r4, #12]
    3510: e5932004 ldr r2, [r3, #4]
    3514: e3520000 cmp r2, #0
    3518: 0a00000e beq 3558 <kmem_cache_alloc_trace+0x154>
    351c: e5934010 ldr r4, [r3, #16]
    3520: e3540000 cmp r4, #0
    3524: 0a00000b beq 3558 <kmem_cache_alloc_trace+0x154>
    3528: e2844008 add r4, r4, #8
    352c: e58d9000 str r9, [sp]
    3530: e1a03008 mov r3, r8
    3534: e58d6004 str r6, [sp, #4]
    3538: e1a01007 mov r1, r7
    353c: e514c008...

Which log does that match?

Could you try a non-smp build?

It is for #6 one.

Will try non-smp

no-smp boot failed

nosmp copy:

00002fa8 <kmem_cache_alloc_trace>:
    2fa8: e1a0c00d mov ip, sp
    2fac: e92ddbf0 push {r4, r5, r6, r7, r8, r9, fp, ip, lr, pc}
    2fb0: e24cb004 sub fp, ip, #4
    2fb4: e24dd008 sub sp, sp, #8
    2fb8: e52de004 push {lr} ; (str lr, [sp, #-4]!)
    2fbc: ebfffffe bl 0 <__gnu_mcount_nc>
    2fc0: e59f3128 ldr r3, [pc, #296] ; 30f0 <kmem_cache_alloc_trace+0x148>
    2fc4: e1a0700e mov r7, lr
    2fc8: e1a05000 mov r5, r0
    2fcc: e1a06001 mov r6, r1
    2fd0: e1a08002 mov r8, r2
    2fd4: e5933000 ldr r3, [r3]
    2fd8: e2033010 and r3, r3, #16
    2fdc: e1130001 tst r3, r1
    2fe0: 0a000000 beq 2fe8 <kmem_cache_alloc_trace+0x40>
    2fe4: ebfffffe bl 0 <_cond_resched>
    2fe8: e5953000 ldr r3, [r5]
    2fec: e593c004 ldr ip, [r3, #4]
    2ff0: e5934000 ldr r4, [r3]
    2ff4: e3540000 cmp r4, #0
    2ff8: 1a000005 bne 3014 <kmem_cache_alloc_trace+0x6c>
    2ffc: e1a00005 mov r0, r5
    3000: e1a01006 mov r1, r6
    3004: e1a02007 mov r2, r7
    3008: ebfffffe bl 2f4 <ksize+0xc0>
    300c: e1a04000 mov r4, r0
    3010: ea000016 b 3070 <kmem_cache_alloc_trace+0xc8>
    3014: e5953014 ldr r3, [r5, #20]
    3018: e7942003 ldr r2, [r4, r3]
    301c: e10f0000 mrs r0, CPSR
    3020: f10c0080 cpsid i
    3024: e5953000 ldr r3, [r5]
    3028: e5931000 ldr r1, [r3]
    302c: e1510004 cmp r1, r4
    3030: 1a000008 bne 3058 <kmem_cache_alloc_trace+0xb0>
    3034: e5931004 ldr r1, [r3, #4]
    3038: e151000c cmp r1, ip
    303c: 1a000005 bne 3058 <kmem_cache_alloc_trace+0xb0>
    3040: e5832000 str r2, [r3]
    3044: e2811001 add r1, r1, #1
    3048: e5953000 ldr r3, [r5]
    304c: e5831004 str r1, [r3, #4]
    3050: e3a03001 mov r3, #1
    3054: ea000000 b 305c <kmem_cache_alloc_trace+0xb4>
    3058: e3a03000 mov r3, #0
    305c: e121f000 msr CPSR_c, r0
    3060: e3530000 cmp r3, #0
    3064: 0affffdf beq 2fe8 <kmem_cache_alloc_trace+0x40>
    3068: e5953014 ldr r3, [r5, #20]
    306c: f7d2f003 pld [r2, r3]
    3070: e3160902 tst r6, #32768 ; 0x8000
    3074: 0a000006 beq 3094 <kmem_cache_alloc_trace+0xec>
    3078: e3540000 cmp r4, #0
    307c: 0a000004 beq 3094 <kmem_cache_alloc_trace+0xec>
    3080: e5951010 ldr r1, [r5, #16]
    3084: e3510000 cmp r1, #0
    3088: 0a000001 beq 3094 <kmem_cache_alloc_trace+0xec>
    308c: e1a00004 mov r0, r4
    3090: ebfffffe bl 0 <__memzero>
    3094: e59f3058 ldr r3, [pc, #88] ; 30f4 <kmem_cache_alloc_trace+0x14c>
    3098: e595900c ldr r9, [r5, #12]
    309c: e5932004 ldr r2, [r3, #4]
    30a0: e3520000 cmp r2, #0
    30a4: 0a00000e beq 30e4 <kmem_cache_alloc_trace+0x13c>
    30a8: e5935010 ldr r5, [r3, #16]
    30ac: e3550000 cmp r5, #0
    30b0: 0a00000b beq 30e4 <kmem_cache_alloc_trace+0x13c>
    30b4: e2855008 add r5, r5, #8
    30b8: e58d9000 str r9, [sp]
    30bc: e1a03008 mov r3, r8
    30c0: e58d6004 str r6, [sp, #4]
    30c4: e1a01007 mov r1, r7
    30c8: e515c008 ldr ip, [r5, #-8]
    30cc: e1a02004 mov r2, r4
    30d0: e5150004 ldr r0, [r5, #-4]
    30d4: e12fff3c blx ip
    30d8: e4953008 ldr r3, [r5], #8
    30dc: e3530000 cmp r3, #0
    30e0: 1afffff4 bne 30b8 <kmem_cache_alloc_trace+0x110>
    30e4: e2...

I am pretty sure this is a GCC 4.8 issue I have almost identical log when compiling an msm-8960 kernel with GCC-4.8
[ 0.350923,0] Unable to handle kernel NULL pointer dereference at virtual address 00000001
[ 0.351014,0] pgd = c0004000
[ 0.351136,0] [00000001] *pgd=00000000
[ 0.351258,0] Internal error: Oops: 5 [#1] PREEMPT SMP
[ 0.351350,0] Modules linked in:
[ 0.351533,0] CPU: 0 Not tainted (3.0.31-ge95971c #1)
[ 0.351594,0] PC is at kmem_cache_alloc_trace+0xa4/0x1a0
[ 0.351747,0] LR is at con_insert_unipair+0xa0/0xec
[ 0.351808,0] pc : [] lr : [] psr: 60000093
[ 0.351808,0] sp : c8c45ef8 ip : c0c29d38 fp : 00000000
[ 0.352021,0] r10: 00002cf8 r9 : 00000080 r8 : c0410a18
[ 0.352083,0] r7 : c8c44038 r6 : 000000d0 r5 : c8c00200 r4 : 00000001
[ 0.352205,0] r3 : c8c44000 r2 : 20000013 r1 : 028ac000 r0 : c0053540
[ 0.352327,0] Flags: nZCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment kernel
[ 0.352418,0] Control: 10c5787d Table: 8020406a DAC: 00000015

Kernel compiles and boots fine with GCC 4.7.
https://github.com/razrqcom-dev-team/android_kernel_motorola_msm8960-common/tree/cm-10.1-gcc47

I know this is off topic but hopefully it helps get to the bottom of the root issue with GCC 4.8 and arm cross-compile

Looks like a SLUB freelist is being corrupted at some point prior to the crash. Perhaps turning on CONFIG_SLUB_DEBUG (or some other memory debugging option) will catch the fault sooner.

I will take a look at it after 22nd May as my Chromebook goes back to shop to get speakers fixed.

It seems the same problem also applies to the 3.4.0-5 kernel in the Ubuntu repository. The kernel image in the 13.04 release (https://launchpad.net/ubuntu/+source/linux-chromebook/3.4.0-5/+build/4314955) works for me, but the one in the 13.10 and 14.04 releases (https://launchpad.net/ubuntu/+source/linux-chromebook/3.4.0-5.1/+build/4575433) does not. According to the build log, the newer image was built with gcc 4.8.

won't fix in gcc-4.8. Please recheck with GCC 4.9