Strange out of memory on pandaboard

Interestingly without swap but with mem=768M I got past the failure point.

 with mem=768M and no swap make bootstrap failed a bit later on "virtual memory exhausted: Cannot allocate memory", nothing in /var/log/messages

Trying with mem=768M and swap activated.

Even if swap is totally unused it got past the previous failure point:

Swap: 524284k total, 0k used, 524284k free, 265836k cached

More tomorrow...

As Nicolas Pitre pointed in the past, this could be a side effect of using CONFIG_VMSPLIT_2G and not CONFIG_VMSPLIT_3G.

Can you point me the instructions to reproduce this bug? Will change my kernel configuration to see if I can reproduce it here with different setups.

With swap reactivated GCC bootstrap succeeded but I got regular backtraces "swapper: page allocation failure. order:3, mode:0x4020", see attached /var/log/messages.

I installed on a 4 GB SD card following instructions here:

http://pandaboard.org/content/resources/getting-started
http://omappedia.org/wiki/Ubuntu_Pre-built_Binaries_Guide
http://cdimage.ubuntu.com/ubuntu-netbook/ports/releases/10.10/release/ubuntu-netbook-10.10-preinstalled-netbook-armel+omap4.img.gz

Then I did apt-get update / apt-get upgrade, reboot, then I edited /boot/boot.script to remove GUI and add memory:

# cat /proc/version_signature
Ubuntu 2.6.35-903.19-omap4 2.6.35.3
# cat /boot/boot.script
        fatload mmc 0:1 0x80000000 uImage
        fatload mmc 0:1 0x81600000 uInitrd
        setenv bootargs quiet text ro elevator=noop vram=32M mem=768M root=UUID=b5d2dfb1-270c-4966-abe6-dfe7a2a17efd fixrtc smsc95xx.macaddr=32:57:F8:93:E1:CD
        bootm 0x80000000 0x81600000
# cat /etc/rc.local

sysctl -w vm.overcommit_memory=2
echo 0 > /proc/sys/kernel/randomize_va_space
mount -t nfs pc2:/mnt/s2000c/nfs/44/home /home
exit 0

Two openvpn running, /home/ over NFS as above I installed nfs-kernel-server.

With the original mem= mem= setting with swap activated I got backtraces by just doing:

cd /home/user
svn co svn://gcc.gnu.org/svn/gcc/trunk

(this will use 1.3 Gbyte of space)

With no swap I don't get any backtraces but user space command start getting NULL out of malloc.

If it's easier for you just send me your ssh public key I will install it on my pandaboard on root.

Even when using CONFIG_VMSPLIT_3G and 768M I was able to reproduce the lack of memory exception. Now need to check why the usb driver is requiring so much memory.

second freeze this night. no blinking LED, nothing on the serial console, last top image and tail -f /var/log/messages attached.

My board froze multiple times during testing of GCC bootstrap

It's ubuntu 10.10 with natty kernel now (the issue were present with 10.10 kernel too)

root@gcc44:~# cat /boot/boot.script
        fatload mmc 0:1 0x80000000 uImage
        fatload mmc 0:1 0x81600000 uInitrd
        setenv bootargs text ro elevator=noop vram=32M mem=768M root=UUID=b5d2dfb1-270c-4966-abe6-dfe7a2a17efd fixrtc smsc95xx.macaddr=32:57:F8:93:E1:CD
        bootm 0x80000000 0x81600000
root@gcc44:~# uname -a
Linux gcc44 2.6.35-1101-omap4 #3-Ubuntu SMP PREEMPT Tue Dec 14 16:05:04 UTC 2010 armv7l GNU/Linux
root@gcc44:~# md5sum /usr/lib/x-loader/omap4430panda/MLO
ddb53314e61be6bcd1cfbe043f584f0e /usr/lib/x-loader/omap4430panda/MLO
root@gcc44:~# cat /proc/cmdline
text ro elevator=noop vram=32M mem=768M root=UUID=b5d2dfb1-270c-4966-abe6-dfe7a2a17efd fixrtc smsc95xx.macaddr=32:57:F8:93:E1:CD
root@gcc44:~# cat /proc/version
Linux version 2.6.35-1101-omap4 (buildd@crabapple) (gcc version 4.5.2 (Ubuntu/Linaro 4.5.1-12ubuntu1) ) #3-Ubuntu SMP PREEMPT Tue Dec 14 16:05:04 UTC 2010
root@gcc44:~# cat /proc/version
version version_signature
root@gcc44:~# cat /proc/version_signature
Ubuntu 2.6.35-1101.3-omap4 2.6.35.3

As root:

echo 0 > /proc/sys/kernel/randomize_va_space
mount -t nfs pc2:/mnt/s2000c/nfs/44/home /home

Install GCC and GCC build depencies.

As user, copy the attached b44.sh to /home/user then

cd /home/user
svn co svn://gcc.gnu.org/svn/gcc/trunk
nohup ./b44.sh >& b44.log < /dev/null &

The board should freeze after a while.

Hey Laurent,

Finally setup my panda to do the same thing, so far i haven't been seeing this issue.. (I'm not using the mem=768M boot arg)
(MMC contains uImage/uInitrd rootfs on external usb sata drive)

My own kernel: (http://rcn-ee.net/deb/squeeze/v2.6.37-x0/)

voodoo@panda-a1-1gb:~$ cat /proc/meminfo | grep MemTotal
MemTotal: 766560 kB

cat /boot/uboot/boot.cmd
echo "Debug: NetInstall"
setenv bootcmd 'mmc init; fatload mmc 0:1 0x80300000 uImage; fatload mmc 0:1 0x81600000 uInitrd; bootm 0x80300000 0x81600000'
setenv bootargs console=ttyO2,115200n8 root=/dev/sda1 ro buddy=${buddy} mpurate=${mpurate}
boot

voodoo@panda-a1-1gb:~$ uname -a
Linux panda-a1-1gb 2.6.37-x0 #1 SMP PREEMPT Thu Jan 6 02:29:06 UTC 2011 armv7l GNU/Linux

Build logs from the last trunk/c test:
http://rcn-ee.homeip.net:81/dl/gcc/SVN-168617-trunk-c-panda-a1-1gb-GOOD/

It's currently halfway thru the test suite after successfully building trunk/c&c++

http://rcn-ee.homeip.net:81/dl/gcc/SVN-168633-trunk-c-c++-panda-a1-1gb/ (link will change after testsuite completion)

Regards,

Hi Robert,

Interesting. From the logs it looks like you're not using "-j 2", am I right? I managed to get the freeze without -j and C only bootstrap but it took a few iterations. Also are you using NFS for the build?

I've relaunched a build without NFS.

Ricardo, any news on this?

Sincerely,

Hi Laurent,

Yeah, i didn't have any '-j' parameters in those builds (also had 'nice' enabled)...

I just updated my script to use "-j2" and disabled "nice" so we shall see what it does..

No, I'm not using NFS, (i really should), I'm using a locally mounted usb sata drive (160G/7200rpm), also contains the rootfs..

Regards,

There are some differences in Robert's setup...he's using a .37 kernel(from kernel.org?) and a A1 board, while Laurent has an EA1. I don't think the board revision has something to do, but the kernel may.

Hi Laurent,

It survived the "-j2" pass,

http://rcn-ee.homeip.net:81/dl/gcc/SVN-168690-trunk-c-c++-panda-a1-1gb/

It's running the testsuite now, and i have it setup to try "-j 3" on the next run..

Based on my log date stamps:

"-j 2" = 7 hours 27 minutes
"no -j setting and nice" = 13 hours 44 minutes

I was monitoring some of the build with top, it never really it the swap file..

Raúl - Yeah, it's very vanilla mainline 2.6.37 + the omap4 usb/ehci patches that where merged into 2.6.38 ( so no fancy DSS2 video, musb/otg, etc.. )

http://bazaar.launchpad.net/~beagleboard-kernel/%2Bjunk/2.6-stable/files/head:/patches/panda/

Regards,

Without NFS, using /home on a 16GB USB key, I got a freeze again after a bit less than 3 days of svn update/-j 3 bootstrap/check c,ada in loop.

Robert, if you have kernel binary + instructions on how to install (on top of my current ubuntu) I'd gladly test the newer kernel.

Hi Laurent,

For a maverick userspace, on the panda board download and run this script:

wget http://rcn-ee.net/deb/maverick/v2.6.37-x1/install-me.sh

/bin/bash install-me.sh

The script will then download that version, install it and backup the older ubuntu uImage and uInitrd's on the mmc card:

uImage_old
uInitrd_old

Notes on the Known issues, with 2.6.37 mainline..

The dvi/dss2 stuff doesn't work yet.. So you can only use the serial port and ssh..
Console serial is: "/dev/ttyO2"
Onboard ethernet is: usb0

Annoying kernel error that i'm currently debugging: "smsc95xx 1-1.1:1.0: usb0: kevent 2 may have been dropped"
(it only seems to stall the usb0 network for just a moment but so far this week it's recovered every time)

An extra note with ubuntu userspace.. if you upgrade the userspace and ubuntu pushes a newer kernel, it will run a script to isntall a new uImage/uInitrd combo..

To block it:

cat > /etc/flash-kernel.conf <<EOF
#!/bin/sh
UBOOT_PART=/dev/mmcblk0p1

DIST=\$(lsb_release -cs)

case "\$DIST" in
    lucid)
            exit 0
        ;;
    maverick)
            FLASH_KERNEL_SKIP=yes
        ;;
    natty)
            FLASH_KERNEL_SKIP=yes
        ;;
esac

EOF

Regards,

I first did a simple test: leave the pandaboard idle, after 5 days no crash.

Robert, I followed your instructions but after reboot I have only the serial console and no network. It seems because my boot.script has smsc95xx.macaddr=32:57:F8:93:E1:CD and it looks like it blocks loading of the module with the newer kernel:

[ 474.915527] smsc95xx: Unknown parameter `macaddr'

I've found no way to tell modprobe to not pass macaddr.

After I edit /boot/boot.script to remove this option when I run flash-kernel I get "Unsupported platform."
So I looked at flash-kernel code and ran manually the line to update boot.scr and managed to reboot with network this time (but another MAC address ...)

Then it looks like I lost some memory (I removed the line mem= and

Mem: 489252k total, 88104k used, 401148k free, 5700k buffers

Here is my /proc/cmdline:

# cat /proc/cmdline
text ro root=UUID=b5d2dfb1-270c-4966-abe6-dfe7a2a17efd console=ttyO2,115200n8

I removed "elevator=noop vram=32M mem=768M" since you made no mention of those.

What is your /proc/cmdline?

Thanks!

Hi Laurent,

I think the "macaddr" parameter is a ubuntu sauce only fix that isn't in mainline.. On my panda, I take care of it thru "/etc/network/interfaces" with:

# The primary network interface
allow-hotplug usb0
iface usb0 inet dhcp
hwaddress ether 62:55:bf:8d:05:28

Odd.. I don't lose that much and get very close to the 768M...

Mem: 766560k total, 421156k used, 345404k free, 11312k buffers
Swap: 1534172k total, 10384k used, 1523788k free, 338336k cached

My /proc/cmdline:

console=ttyO2,115200n8 root=/dev/sda1 ro buddy= mpurate=

(Safe to ignore buddy= and mpurate= as they are from using the same script as an beagle/beaglexm..)

Just for reference, i'm using git version of x-loader from: (the Dec 23 commit)

http://gitorious.org/x-loader/x-loader

and mainline u-boot v2010.12..

Regards,

Robert for reference with the new kernel I get a board freeze in less than an hour.

I don't understand why when I boot with no option I get only 490M.

Robert and Laurent,

How about your U-Boot, does it recognize the whole 1G memory or just 512M memory. Without any mem= option in kernel cmdline, kernel will get such memory information from U-Boot, IMHO.

My U-Boot recognize the whole 1G memory, then even without mem=1G, Kernel will get the whole 1G physical memory on board.

Thanks,
-Bryan

Hi Laurent,

On the beagleboard list, we found a hack/workaround for the smsc95xx kevent spam in dmesg, that seems to lock up the ethernet interface:

Add to /etc/sysctl.conf:
vm.min_free_kbytes = 8192

So far, after 13 hours, my panda hasn't had a single kevent while building gcc and running the gcc testsuite, with a setting of "-j 3"..

Regards,

On Mon, Feb 14, 2011 at 9:36 AM, Robert Nelson <email address hidden> wrote:
> Hi Laurent,
>
> On the beagleboard list, we found a hack/workaround for the smsc95xx
> kevent spam in dmesg, that seems to lock up the ethernet interface:
>
> Add to /etc/sysctl.conf:
> vm.min_free_kbytes = 8192
>
> So far, after 13 hours, my panda hasn't had a single kevent while
> building gcc and running the gcc testsuite, with a setting of "-j 3"..

Is it still freezing even after adding this to sysctl.conf?

Hi Ricardo,

Nope, i'm not seeing any sudden freeze's or lockup's..

Panda's dmesg running gcc/testsuite before:

http://rcn-ee.homeip.net:81/dl/gcc/sys/dmesg-170061-2.6.38-rc4-d3-panda-a1-1gb.log
(lot's of kevent, stall, oom's, etc..)

After:
http://rcn-ee.homeip.net:81/dl/gcc/sys/dmesg-170116-2.6.38-rc4-d3-panda-a1-1gb.log

My xM's smsc95xx is showing the exact issue, but still hasn't completed the gcc/testsuite run. (figure another half day..)

(before)
http://rcn-ee.homeip.net:81/dl/gcc/sys/dmesg-170105-2.6.38-rc4-d3-beagle-xma-512mb.log

Regards,

Hi, just pinging :). Should I try a more recent system image?

@laurent: if by any chance you could try on a recent natty daily image, we now have 2.6.38 kernel in the daily's... at least we would know whether or not the problem is still there.

@Nicolas I'm downloading 20110321 omap4 images from http://cdimage.ubuntu.com/ubuntu-netbook/daily-preinstalled/current/

54040512b1835e83d0096779d6c70c01 *natty-preinstalled-netbook-armel+omap4.img.gz

I'll be testing that within a week. Let me know if another image would be better. Thanks!

add "vm.min_free_kbytes = 8192" to /etc/sysctl.conf and retry, see lp746137: the backtraces are the same and it's a nic driver/memory fragmentation issue.

2.6.38-1207 (mem=463M@0x80000000 mem=512M@0xA0000000 = 943MB for system) during build of kernel:

[ 2945.114135] cc1: page allocation failure. order:3, mode:0x4020
[ 2945.114196] [<c005a420>] (unwind_backtrace+0x0/0xe0) from [<c0107e98>] (__alloc_pages_nodemask+0x5a0/0x6b4)
[ 2945.114227] [<c0107e98>] (__alloc_pages_nodemask+0x5a0/0x6b4) from [<c0107fbc>] (__get_free_pages+0x10/0x24)
[ 2945.114257] [<c0107fbc>] (__get_free_pages+0x10/0x24) from [<c012f494>] (kmalloc_order_trace+0x20/0xc8)
[ 2945.114288] [<c012f494>] (kmalloc_order_trace+0x20/0xc8) from [<c04882fc>] (__alloc_skb+0x50/0xe0)
[ 2945.114318] [<c04882fc>] (__alloc_skb+0x50/0xe0) from [<c038b2ec>] (rx_submit+0x24/0x1cc)
[ 2945.114349] [<c038b2ec>] (rx_submit+0x24/0x1cc) from [<c0395524>] (usb_hcd_giveback_urb+0xa0/0xec)
[ 2945.114379] [<c0395524>] (usb_hcd_giveback_urb+0xa0/0xec) from [<c03a1650>] (ehci_urb_done+0xa8/0xb4)
[ 2945.114410] [<c03a1650>] (ehci_urb_done+0xa8/0xb4) from [<c03a5488>] (qh_completions+0xb4/0x3d8)
[ 2945.114440] [<c03a5488>] (qh_completions+0xb4/0x3d8) from [<c03a5d94>] (scan_async+0x90/0x144)
[ 2945.114440] [<c03a5d94>] (scan_async+0x90/0x144) from [<c03a6504>] (ehci_work+0x30/0x88)
[ 2945.114471] [<c03a6504>] (ehci_work+0x30/0x88) from [<c03a83f0>] (ehci_irq+0x2f4/0x35c)
[ 2945.114471] [<c03a83f0>] (ehci_irq+0x2f4/0x35c) from [<c0394e90>] (usb_hcd_irq+0x38/0x7c)
[ 2945.114501] [<c0394e90>] (usb_hcd_irq+0x38/0x7c) from [<c00d89b4>] (handle_IRQ_event+0x9c/0x1b4)
[ 2945.114532] [<c00d89b4>] (handle_IRQ_event+0x9c/0x1b4) from [<c00da5dc>] (handle_level_irq+0xdc/0x160)
[ 2945.114562] [<c00da5dc>] (handle_level_irq+0xdc/0x160) from [<c004e088>] (asm_do_IRQ+0x88/0xc8)
[ 2945.114593] [<c004e088>] (asm_do_IRQ+0x88/0xc8) from [<c05bb128>] (__irq_usr+0x48/0x120)
[ 2945.114593] Exception stack(0xee37ffb0 to 0xee37fff8)
[ 2945.114593] ffa0: 00000001 00000019 00000000 00000000
[ 2945.114624] ffc0: 00000001 007cee74 42e3db80 00788550 00000001 00000000 00788550 42e3db80
[ 2945.114624] ffe0: 00000074 bed13ff0 000fd831 000fbbee 800f0030 ffffffff
[ 2945.114654] Mem-info:
[ 2945.114654] Normal per-cpu:
[ 2945.114654] CPU 0: hi: 186, btch: 31 usd: 156
[ 2945.114654] CPU 1: hi: 186, btch: 31 usd: 38
[ 2945.114654] HighMem per-cpu:
[ 2945.114685] CPU 0: hi: 90, btch: 15 usd: 54
[ 2945.114685] CPU 1: hi: 90, btch: 15 usd: 3
[ 2945.114685] active_anon:152413 inactive_anon:156 isolated_anon:0
[ 2945.114715] active_file:26523 inactive_file:34377 isolated_file:0
[ 2945.114715] unevictable:0 dirty:37 writeback:0 unstable:0
[ 2945.114715] free:4833 slab_reclaimable:6423 slab_unreclaimable:1905
[ 2945.114715] mapped:3752 shmem:352 pagetables:1798 bounce:0
[ 2945.114746] Normal free:18596kB min:3416kB low:4268kB high:5124kB active_anon:414312kB inactive_anon:0kB active_file:92764kB inactive_file:119716kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:730112kB mlocked:0kB dirty:136kB writeback:0kB mapped:8456kB shmem:8kB slab_reclaimable:25692kB slab_unreclaimable:7620kB kernel_stack:2688kB pagetables:7192kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 2945.114776] lowmem_r...

Laurent, with both bug 746137 and bug 633227 now fixed, can you try to reproduce this issue again? Tomorrow's image (or beta-2) should be OK for the test.

Hi Marcin Juszkiewicz,

If out of memory always happened when you build something over a NFS, it is very possibly caused by usbnet,
which seems to be shown from #28.

The issue has been discussed in maillist recently:

       http://marc.info/?t=130406641000006&r=1&w=2
       http://marc.info/?t=130471082300010&r=1&w=2

but still have not a good solution to fix this, I will try to figure out a patch to address the issue.

thanks,
Ming Lei

Ming Lei: I have hard drive connected on USB. But I use ethernet during build cause I control board remotely.

Hi Marcin Juszkiewiczm,

You mean that you are building the kernel on the filesystem over usb storage, and at the same time
you are controlling from remote, then oom happened?

If so, could you disconnect ethernet connection(usbnet) to build kernel on usb storage and
see if the oom can happen?

thanks,

I think the attachment patch can fix the issue, so anyone would like to verify it on your board?

Does this happen with Linaro kernels as well?

Used to happen also with the Linaro kernel, but would be good to test with latest TI LT one (linux-linaro-lt-omap from https://launchpad.net/~linaro-maintainers/+archive/overlay).

Ming Lei, did you try publishing this patch at lkml or similar to have more feedback?

I see very similar errors when doing anything which causes a lot of I/O to an external USB hard drive. I see this bug is marked fixed, but there's no indication of which updated package fixes it, so I can't confirm.

Matt, are you using at least 11.10 ? Natty may still have the bug.

Jani, yes, the machine in question is running 11.04. Can you confirm which package version is expected to fix the bug? Or is it unknown whether the bug was fixed?

Similar bugs were fixed so hopefully yours is not a different one. I only followed these bugs from afar, never had the issue, just noticed that several were fixed before Oneiric released. So I would try the ti-omap4 kernel from Oneiric (3.0 based IIRC)

@mdz: please check that

flag@flag-desktop:~$ sysctl vm.min_free_kbytes
vm.min_free_kbytes = 32768

since N we had this fix in casper, and a file was created in /etc/sysctl.d accordingly.
If you upgraded from a previous release/daily/beta/etcetc, it could be that you didn't get that fix/file, so create it manually.
From Q this is fixed at the kernel level, so no need to do anything there.

Beware that since P we switched to armhf so you won't be able to upgrade to the latest and greatest, so if i would be in you i would consider a reinstallation with a recent P/Q armhf image.

s/casper/jasper-initramfs/

;)

Yes, I do have that setting, and I see these errors regardless of how it is
set.

I've ordered a new SD card so I can try upgrading to a later version. What
version of Ubuntu would you recommend for this board as the most stable?

On Wed, Sep 26, 2012 at 12:58:42PM -0000, Paolo Pisati wrote:
> @mdz: please check that
>
> flag@flag-desktop:~$ sysctl vm.min_free_kbytes
> vm.min_free_kbytes = 32768
>
> since N we had this fix in casper, and a file was created in /etc/sysctl.d accordingly.
> If you upgraded from a previous release/daily/beta/etcetc, it could be that you didn't get that fix/file, so create it manually.
> >From Q this is fixed at the kernel level, so no need to do anything there.
>
> Beware that since P we switched to armhf so you won't be able to upgrade
> to the latest and greatest, so if i would be in you i would consider a
> reinstallation with a recent P/Q armhf image.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/690370
>
> Title:
> Strange out of memory on pandaboard
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/linux-ti-omap4/+bug/690370/+subscriptions

--
 - mdz

I no longer have this problem with ubuntu-12.04-preinstalled-server-armhf+omap4.img.gz

[Expired for linux-ti-omap4 (Ubuntu) because there has been no activity for 60 days.]