Comment 28 for bug 655831

A weekly fatal unable to fork error message from /etc/cron.weekly/apt-xapian-index on a server with 512MB memory with a web server and database server running leads to this bug. That cron job calls /usr/sbin/update-apt-xapian-index, which is a Python script that imports axi and axi.indexer. Those Python packages are the heart of the problem.

On an idle system with 512M memory and 768M swap, while /etc/cron.weekly/apt-xapian-index was running, I used free to see memory and swap used. It rose to then peaked at this:

       total used free
Mem: 503528 497736 5792
Swap: 786428 231196 555232

Total used memory is 728M.

After ending, it immediately dropped back to this:

       total used free
Mem: 503528 259160 244368
Swap: 786428 224484 561944

Total used memory is 483M.

I conclude that apt-xapian-index consumes the difference, which is 245M.

Running "apt-cache stats" I see at the end "Total space accounted for: 26.0 M".

Therefore, it takes 245M to sort and index 26M of information. This seems conclusive that the algorithms, containers, and/or functions chosen are very inefficient. It should definitely not require 10 times the memory space of what is being indexed.

The solution is to change the sorting algorithm in the python axi and axi.indexer modules. The first priority is to switch to an algorithm that consumes a whole lot less memory (i.e., each step of the algorithm keeps less objects in memory), and it will stop crashing and stop thrashing (memory swapping to disk).

The second priority is of lesser importance (because renice can solve a lot of the effect), which is to switch to an algorithm that takes a lot less time to run (i.e., takes fewer steps to complete), and it will stop consuming so much CPU for so long.