out of memory Fetching revisions:Inserting stream:Estimate on AIX

Hi Adi,

Thanks for the report.

Please attach the traceback from ~/.bzr.log and if possible the run with -Dmem_dump.

istr seeing something like this being due to a unix api quirk rather than actually being out of memory.

Hi,

Here is a new run on another branch:

$ bzr branch -Dmem_dump sftp://repo.tld/home/project/bzr/server/devel
Connected (version 2.0, client OpenSSH_5.1p1)
SSH <email address hidden> password:
Authentication (password) successful!
Secsh channel 1 opened.
[chan 1] Opened sftp connection (server version 3)
bzr: out of memory | Fetching revisions:Inserting stream:Estimate 1092/3759
Memory dumped to /tmp/bzr_memdumpJjQyLe.json

Here is the bzr_log:
http://dl.dropbox.com/u/174543/bzr_log

And here is the memory dump:
http://dl.dropbox.com/u/174543/bzr_memdumpJjQyLe.json.tar.bz2

Cheers,
Adi

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 9/23/2011 9:23 AM, Martin Pool wrote:
> Hi Adi,
>
> Thanks for the report.
>
> Please attach the traceback from ~/.bzr.log and if possible the run
> with -Dmem_dump.
>
> istr seeing something like this being due to a unix api quirk
> rather than actually being out of memory.
>
> ** Changed in: bzr Importance: Undecided => Medium
>
> ** Changed in: bzr Status: New => Incomplete
>
> ** Tags added: aix memory
>

If it is AIX, my guess is that we are missing a place where
"malloc(0)" returns NULL rather than a pointer like a lot of GNU based
systems. I know we had to fix up some patience-diff code for that case.

Given the traceback, I'm suspecting the create_delta_index_from_delta
code.

Specifically:
max_num_entries = (src->size - 1) / RABIN_WINDOW;

/* allocate an array to hold whatever entries we find */
entries = malloc(sizeof(*entry) * max_num_entries);
if (!entries) /* malloc failure */
    return DELTA_OUT_OF_MEMORY;

If src->size is less than 17, then that ends up as (16-1) / 16 = 0.
The algorithm still works today, because all the rest of the code just
skips, and doesn't try to assign anything into the entries list.
However, we could just do:

if (max_num_entries == 0) {
  *fresh = old_index;
  return DELTA_OK;
}

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk58XZMACgkQJdeBCYSNAANCkQCffLbMJ97M6cv+xA6bOnSAY6oQ
PggAn3x0i5uvD32HWZ0aj4+ceiiwoT3J
=Q3f4
-----END PGP SIGNATURE-----

John's analysis and proposed solution looks dead on, I'll put up a branch with that change.

Adi: If possible could you test the branch linked with the fix?

If you run `bzr selftest -s bt.test_groupcompress after compiling the pyrex extensions, you should get test failures currently, and they should go away with the new branch.

Hi,

Sorry for the delay.
I have tested the branch and everything looks OK.

Many thanks for this quick fix.

Cheers,
Adi