Amd64 ubuntu build hogs memory due to libs built with excessive alignment requirement

See http://lists.ubuntu.com/archives/ubuntu-devel/2005-October/012640.html

I am not sure why this was marked as invalid. The discussion on the developer
list kind of stalled, but as far as I can see, the conclusion was that it isn't
as bad as it seems, most of that memory actually isn't being wasted, it's just
reported as such.

This issue still needs resolved because at the very least, it gives the
appearence that a lot of memory is being wasted, which can cause the casual user
to look at ubuntu and decide not to use it because it is "wasting tons of memory".

The tool chains on build daemons or wherever the problem is needs to be fixed so
as not to set the rediculous 1 MB alignment requirement.

Does the bug still exist in the latest version of (k)ubuntu, Dapper Drake 6.06.1 ? Please try Dapper and give us an update if you can. Thanks.

Also, can you reproduce it on Edgy?

Andrew Ash wrote:
> Also, can you reproduce it on Edgy?

Yes. I just looked through the mem maps of a running xmms process that
is listed as taking 114 MB of Virtual Memory. I found several offending
libs:

/usr/lib/libdrm.so.2.0.0 has a 20 bit alignment requirement in its
program header according to objdump -x. This causes a 1 MB mapping to
be made even though the entire file size is only 31 KB.

/usr/lib/libvorbis.so.0.3.1 is another offender causing a 1 MB mapping
with a total file size of only 172 KB.

/usr/lib/libXfixes.so.3.1.0 is using a meg mapping for a 21 kb lib
/usr/lib/libXi.so.6.0.0 uses a meg for only 33k
/usr/lib/libz.so.1.2.3 wastes an extra meg for an 89 KB lib

You get the idea, there are a lot of them like this. If you see a
"2**20" in the objdump of the file, it has this problem.

I just came up with the following test for effected files and results for /lib. It came up with many duplicates due to the multiple symbolic names for each library though.

root@ubuntu:/lib# for file in `ls *.so.*` ; do if objdump -x $file | grep -q -e '2\*\*20' ; then echo $file ; fi ; done
ld-linux-x86-64.so.2
libacl.so.1
libacl.so.1.1.0
libanl.so.1
libatm.so.1
libatm.so.1.0.0
libattr.so.1
libattr.so.1.1.0
libblkid.so.1
libblkid.so.1.0
libbrlapi.so.0.4
libbrlapi.so.0.4.1
libBrokenLocale.so.1
libbz2.so.1
libbz2.so.1.0
libbz2.so.1.0.3
libcap.so.1
libcap.so.1.10
libcfont.so.0
libcfont.so.0.0.0
libcidn.so.1
libcom_err.so.2
libcom_err.so.2.1
libconsole.so.0
libconsole.so.0.0.0
libcrypt.so.1
libc.so.6
libctutils.so.0
libctutils.so.0.0.0
libdal-0.3.so.0
libdal-0.3.so.0.0.0
libdb1.so.2
libdb.so.2
libdevmapper.so.1.01
libdevmapper.so.1.02
libdiscover.so.1
libdiscover.so.1.0.0
libdl.so.2
libe2p.so.2
libe2p.so.2.3
libevms-2.5.so.0
libevms-2.5.so.0.5
libext2fs.so.2
libext2fs.so.2.4
libgcc_s.so.1
libhandle.so.1
libhandle.so.1.0.3
libhistory.so.4
libhistory.so.4.3
libhistory.so.5
libhistory.so.5.1
libiw.so.28
libm.so.6
libncurses.so.5
libncurses.so.5.5
libncursesw.so.5
libncursesw.so.5.5
libnsl.so.1
libnss_compat.so.2
libnss_dns.so.2
libnss_files.so.2
libnss_hesiod.so.2
libnss_nisplus.so.2
libnss_nis.so.2
libpamc.so.0
libpamc.so.0.79
libpam_misc.so.0
libpam_misc.so.0.79
libpam.so.0
libpam.so.0.79
libparted-1.6.so.12
libparted-1.6.so.12.0.9
libparted-1.6.so.13
libparted-1.6.so.13.11.1
libparted-1.7.so.1
libparted-1.7.so.1.0.0
libpopt.so.0
libpopt.so.0.0.0
libpthread.so.0
libreadline.so.4
libreadline.so.4.3
libreadline.so.5
libreadline.so.5.1
libreiserfs-0.3.so.0
libreiserfs-0.3.so.0.0.0
libresolv.so.2
librt.so.1
libselinux.so.1
libsepol.so.1
libslang.so.1
libslang.so.1.4.9
libslang.so.2
libslang.so.2.0.6
libss.so.2
libss.so.2.0
libsysfs.so.1
libsysfs.so.1.0.3
libsysfs.so.2
libsysfs.so.2.0.0
libthread_db.so.1
libusb-0.1.so.4
libusb-0.1.so.4.4.4
libusplash.so.0
libutil.so.1
libuuid.so.1
libuuid.so.1.2
libvolume_id.so.0
libvolume_id.so.0.65.0
libwrap.so.0
libwrap.so.0.7.6

I'm sure that info will be handy for the devs. I've subscribed the AMD64 team to this bug, since I don't really have the power or knowledge to fix problems. If anything new comes up, keep us posted.

Thanks

I don't think this should still be marked as Needs Info, so I am marking as Confirmed.

Has any work been done on fixing this in feisty?

In current gutsy, /lib is ok (no such problem) but /usr/lib still has libs with this problem.
Some beeing Firewire related (libavc/rom1394), the closed source nvidia driver too.

All impacted packages on my system :
libavc1394-0
libfaac0
libgdbm3
libgnomecups1.0-1
liblzo1
libmng1
libsidplay1
libsoundtouch1c2
libsp1c2
libsphinx2g0
nvidia-glx-new

I suppose those are packages that haven't seen an update in a while, so they were not rebuilt. Or no one reported the problem to their owner.

Should these be reported as bugs to the individual packages?

I appear to have 31 incorrect ones in /usr/lib on my system.

In current Hardy alpha 6 the problem in /lib seems to be fixed completely. However still some libraries in /usr/lib have it:

libavc1394.so.0
libavc1394.so.0.3.0
libBPM.so.1
libBPM.so.1.0.0
libgdbm_compat.so.3
libgdbm_compat.so.3.0.0
libgdbm.so.3
libgdbm.so.3.0.0
libmng.so.1
libmng.so.1.1.0.9
librom1394.so.0
librom1394.so.0.3.0
libsidplay.so.1
libsidplay.so.1.0.3
libSoundTouch.so.1
libSoundTouch.so.1.0.0
libtar.so.0
libtar.so.0.0.0

I stumbled across this bug looking for something else, but I looked at system monitor on my AMD64 Gusty machine and it seems to report garbage numbers. Screenshot of sys-monitor attached. I'm guessing its somewhat related?

Ryan,

You're looking for bug no. 148325, "Certain processes show incorrect memory usage when using xgl," a known xserver-xgl issue.

Best,
Jonathan T.

I just ran the test in /usr/lib with Ubutun 8.04 (kernel 2.6.24-19-generic amd64) and get a couple more libs than Jim Quode. The extra ones seem mostly to be the nvidia drivers. I've removed the symlinks from the output:

libavc1394.so.0.3.0
libBPM.so.1.0.0
libcuda.so.173.14.05
libfame-0.9.so.1.0.0
libgdbm_compat.so.3.0.0
libgdbm.so.3.0.0
libGLcore.so.173.14.05
libGL.so.173.14.05
liblzo.so.1.0.0
libmng.so.1.1.0.9
libnvidia-cfg.so.173.14.05
libnvidia-tls.so.173.14.05
librom1394.so.0.3.0
libsidplay.so.1.0.3
libSoundTouch.so.1.0.0
libtar.so.0.0.0
libXvMCNVIDIA.so.173.14.05

for file in `find /usr/lib -type f -name '*.so.*'`; do if objdump -x $file | grep -q -e '2\*\*20' ; then echo $file ; fi ; done

Run the above to avoid seeing symlinks in the output. On my 8.10 desktop with fglrx installed, this outputs:
/usr/lib/libfglrx_pp.so.1.0
/usr/lib/libifp.so.4.1.4
/usr/lib/libavc1394.so.0.3.0
/usr/lib/librom1394.so.0.3.0
/usr/lib/libGL.so.1.2
/usr/lib/libfglrx_gamma.so.1.0
/usr/lib/libXvBAW.so.1.0
/usr/lib/libglitz-glx.so.1.0.0
/usr/lib/libgdbm_compat.so.3.0.0
/usr/lib/libglitz.so.1.0.0
/usr/lib/libfglrx_dm.so.1.0
/usr/lib/libfglrx_tvout.so.1.0
/usr/lib/libmng.so.1.1.0.9
/usr/lib/libgdbm.so.3.0.0

I'm on Jaunty too.
Here is my output :
draco@12:13:03 [0] <0>:~/tmp$ for file in `find /usr/lib -type f -name '*.so.*'`; do if objdump -x $file | grep -q -e '2\*\*20' ; then echo $file ; fi ; done | sort
/usr/lib/libavc1394.so.0.3.0
/usr/lib/libcuda.so.180.44
/usr/lib/libGLcore.so.180.44
/usr/lib/libGL.so.180.44
/usr/lib/libmac.so.2.0.0
/usr/lib/libmng.so.1.1.0.9
/usr/lib/libnvidia-tls.so.180.44
/usr/lib/librom1394.so.0.3.0
/usr/lib/libvdpau_nvidia.so.180.44
/usr/lib/libvdpau.so.180.44
/usr/lib/libvdpau_trace.so.180.44
/usr/lib/libXvMCNVIDIA.so.180.44
/usr/lib/nvidia/libnvidia-cfg.so.180.44
/usr/lib/tls/libnvidia-tls.so.180.44
/usr/lib/xorg/modules/extensions/libglx.so.180.44

Most of those libs are from the NVidia drivers (cuda, nvpau ...).

The above one-liners do not pick up on files that end in ".so". Moreover, there are two such files that are not in fact shared objects....they're ASCII text. Also, since some of the vendor-specific objects are discovered to reintroduce the problem, it would be good to look in their directories.

The following version examines all shared objects for the alignment of interest in the directories of interest, and emits the name of the package that owns it (or whatever dpkg-query's error string is for unowned files). I've evolved and tested this over a few hours, and it seems to pick up everything.

warning: This version is not for the faint of CPU. It will run at least 6 minutes on a modern machine before emitting anything at all (due to the queuing of sort). It has not been optimized in any way, but it does spread across CPUs and I did like the self-regulating nature of its CPU usage.

for file in `find {/usr,}/lib{32,} -type f`; do if file $file | grep -q -e 'shared object'; then if objdump -x $file | grep -q -e '2\*\*20'; then dpkg-query -S `echo $file`; fi; fi; done | grep -o -e '.*:' | sort | uniq

Output to follow....

For a recently updated Jaunty:

for file in `find {/usr,}/lib{32,} -type f`; do if file $file | grep -q -e 'shared object'; then if objdump -x $file | grep -q -e '2\*\*20'; then dpkg-query -S `echo $file`; fi; fi; done | grep -o -e '.*:' | sort | uniq

diversion by xorg-driver-fglrx from:
diversion by xorg-driver-fglrx to:
gkrellm-volume:
gkrellmwho2:
libavc1394-0:
libglitz1:
libglitz-glx1:
libmng1:
libnet0:
libphysfs-1.0-0:
sun-java6-bin:
xorg-driver-fglrx:
xorg-driver-fglrx, libgl1-mesa-glx:
xserver-xorg-core, xorg-driver-fglrx:

Some had quite a few files.

For a recently updated Hardy, the output was:

libgdbm3:
libtext-iconv-perl:
...and a number of non-deb .sos from VMware server.

For a recently updated Karmic, the output was:

libglitz1:
libglitz-glx1:
sun-java6-bin:

This is my output for Lucid Lynx Alpha 2+:

libglitz1:
libglitz-glx1:
nvidia-173:
nvidia-current:

Thank you for posting.

Is this still an issue in Maverick?

I think it is still an issue. If I try to use a 64-bit livecd in a VM with 512 MB of RAM, the VM can't run very long before the OOM killer comes into play.

I just tried with today's daily build. Running the live session, plus Firefox, plus the installer (but not doing anything in any of them) was enough to make it start killing processes.

Or, high memory usage is currently a problem. Not sure if it's because of this, or if this can cause much extra memory usage.

I'll investigate.

It is now even worse in Maverick. The alignment has increased from 2**20 to 2**21.

Usage of virtual address space is not the same as usage of physical pages. A process using up 100+ MB of virtual memory isn't actually a problem. For comparison, on my machine, a bit of mental arithmetic shows that the top few processes shown by System Monitor have over 8 GB of memory mapped, whereas the real usage is around 1.4GB.

If you are trying to measure the physical memory usage of processes, you should use the "Resident Memory" column instead, as far as I can tell.

The aligment of 2**21 is already used in some Lucid packages currently.

Here the command I use :
for file in `find {/usr,}/lib{32,} -xdev -type f -iname '*.so.*'`; do if file $file | grep -q -e 'shared object'; then if objdump -x "$file" | egrep -q '2\*\*[0-9]{2}'; then apt-file search "$file" ; fi; fi; done | sort

Simon Ruggier is correct. Virtual addressing has nothing to do with physical address space used.

This "empty" space doesn't even take up swap space, let alone virtual memory. It does take up some page tables but that's it.

Having the extra virtual address space there is actually a good thing. It reduces vulnerability to buffer overflows, and similar issues.

However, I must agree that 1MB is a bit big, especially for libraries that are only a few kilobytes. Generally 4KB-64KB of space is enough to prevent issues. However, considering the Linux Userland virtual address space on AMD64 is currently 128TB, I don't think there is any shortage of virtual address space in the near future.

There may be other reasons for providing this extra space that I don't know about. Some of these libraries might allocate additional memory for certain things, and this would provide plenty of space for them to do so while keeping the same block of memory.

The space can not be "allocated" and made use of at runtime; it is permanently off limits. It also does nothing for buffer overflows since they are done on the stack.

Yes, but the space is also unallocated.
Allocation of physical memory is done far smaller memory blocks, hence the "zero space" is just empty allocation space. It's not using physical RAM. This is because memory mapping is done through sparse allocation on 64bit mapping, where the virtual memory layer is then mapped to a physical memory layer, in small pages, with most of those pages simply referencing nothing.

It's much like non existing phone numbers don't need lines and buildings to go to them. Unused sparse memory allocation is pretty much the same. The program is just like a business getting all the phone numbers in a 100 number range and then only using 25 of them. So long as there are enough phone numbers to go around it doesn't make there be any more lines being used.

Sure, it's a problem. There is no benefit, and it wastes resources to map in this manner. However, it doesn't hemorrhage memory like you would think.

More importantly, if any 32bit libraries are built in this manner, 32bit program virtual memory space is a very precious thing when running memory demanding 32bit legacy programs, such as games under wine that require a single 4GB flat memory, and wasting the shared virtual memory space of a 32bit program dramatically reduces the flat memory space accessible by such programs. This is probably a far bigger concern than 64bit virtual address space, which there is honestly so much of (64TB of allocation space in the current standard mapping, more on some large servers.)

It would be far more beneficial to map to 4K or 64K depending on the size of the library. Only insanely big libraries should be mapped to the 1MB barrier.

I can list the real consiquences of 1MB alignment of small libraries, and they are pretty bad:
1. Cache Prefetch performance issues.
2. Bulk Swap Transfers: If the shared libraries for a program aren't in contiguous space, they can't be swapped in bulk as one chunk.
3. Increased VM overhead.

You are right, there is no reason for the 1MB alignment and it wastes some resources. However, VM handling, particularly the D_Paged function which enables sparse memory mapping and dynamic paging, should prevent it from using nearly as much physical memory as you think.

Personally, I think that the memory alignment should be to 4KB, 64KB or 2MB depending on object size on systems with 4KB pages, and 64KB or 2MB on systems with 64KB pages. It should be using a waste factor calculation that weighs waste factor over increased allocation costs.

The reason for the option 2MB size is for large page allocation. Specifically, if a program is using hundreds of megabytes of memory, it is better to allocate them in 2MB aligned blocks so that the kernel can reallocate them dynamically into huge pages. This allows reduced VM overhead in both page indexing and processor overhead, reduces memory map fragmentation on both the physical and virtual layers, provides for faster cache transfers on modern CPUs, and faster transfers in and out of swap.

I agree that there is absolutely no reason to use a 1MB alignment. It is too big to be beneficial for small allocations, and too small to be beneficial for larg...

Even on large libraries, a 2 M alignment does not help. If a library maps a section larger than 2 M then it will still be able to use large pages for that mapping, even if its alignment requirement for the section is only 1 byte. The alignment requirement just reserves extra address space that goes unused ( and won't be converted to a large page since part of that 2 M space maps to nothing ).

Basically there is absolutely no reason to ever force an alignment larger than 4k.