Out of memory error in _ensure_content on auto repacking

The commit is triggering an automatic repack, you can separate the two operations by doing an explicit 'bzr pack'.
Can you try that and report whether it's still failing ?

Thanks Vincent. I eventually grew so frustrated I deleted the repository and started afresh. I will see if I can restore it and try what you say, just to report the result. I'll report back in a few days.

Vincent, I have recovered enough files to recreate the issue.
The command:
  bzr pack
Produces the output:
  bzr: out of memory

Ok, so it's indeed a repack problem. I thought John commented on the issue and even had a patch to propose but his comment is not there and I can't find it back in my mail either...

@Greg: Do you have a reproducing recipe ?

@vila you're probably thinking about bug #605480

I believe I can reproduce the issue at will.
I don't have any unique steps to recreate the issue on a stable repository. I am also unable to share the repository as it is part of a proprietary commercial code base.

I believe I'm seeing this same bug by a different route. I'm getting an out of memory error on the exact same line when it tries to decompress a text larger than 1 GB. Seems to be directly related to this python bug: http://bugs.python.org/issue8571

My repo at work is the one that exhibits the bug. I'm planning to figure out which python versions contain the patch this weekend and test on Monday.

I was wrong about it being related to that python bug. My python is patched for that bug. I have managed to track it down further that might aid someone else to reproduce it.

A little while ago, I accidentally added my "tags" file. It is over 1 GB in size, but is highly compressible, down to around 54 MB. Now whenever there is a need to decompress that file, such as switching to a branch that contains that file, or doing a bzr check --repo, it fails with out of memory. Doing an auto repack would also hit it, which would explain why Greg is seeing it.

@jam: ping, this is targeted at 2.3.0 freezing tomorrow, should we retarget ?

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2/2/2011 4:56 AM, Vincent Ladeuil wrote:
> @jam: ping, this is targeted at 2.3.0 freezing tomorrow, should we
> retarget ?
>

Just remove the target.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk1J9GsACgkQJdeBCYSNAANvIACdEzXbaYtQRGbtQPYL36wCZsA1
VcIAoNXSKYjmit+rTUGPUd1LIGRLYV33
=uL5r
-----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 3/2/2011 1:52 AM, Martin Pool wrote:
> ** Summary changed:
>
> - Out of memory error on auto repacking
> + Out of memory error in _ensure_content on auto repacking
>

_ensure_content is the extraction side of things. (Decompress the zlib
data into raw groupcompress instructions that can be used to generate
the fulltexts.)

Just adding some context.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk1uWXQACgkQJdeBCYSNAAPcgQCgkGmv6o2pDpPc0RoeWni44QfA
GQ8AnR+MW/t/F8YhajrnxbCr+9HaGxVZ
=IIz/
-----END PGP SIGNATURE-----

Does anyone have a workaround on this? Is there a process to cleanse my tree without starting from scratch?

As a workaround, you can do the following:
bzr pack nosmart+<url to your repository>

Thanks treaves! Unfortunately I get the response:
bzr: out of memory

I watch the process in top in a separate console and see that bzr crashes when I reach about 500M of resident memory usage.

Jay: sorry, that worked for me. Are you running the command from a local machine? Maybe that is the difference. It is my server that has this defect, and the command I pasted actually performs the pack on the client.

Its all remote... I finally got past the issue by using "push" to send the tree into the server instead of "pull" or "update". I have 3 remote servers, 2 servers that produce a bzr: out of memory and one server that will not (each has 2Gb), but the server that works is our dev server, running a much lighter load. I guess I could try to get more memory, but this really shouldn't be happening. If it gets the point where we have to rebuild the tree from scratch I might jump ship over to svn if no one has fixed this yet. Kinda frustrating when your trying to deploy new software.... Thanks treaves...

Problem resurfaced. Prevented deployment of code release. I've switched myself and my team of 5 developers to svn.

I'm having this problem, too.

Bzr is running on a Windows 7 (32 bit) with service pack 1.

There is one commit (pack) with ~ 2.5 GB which is probably causing the memory error. Both 'bzr pack' and 'bzr pack nosmart' fail with out of memory:

C:\nnn\mmm>bzr pack
bzr: out of memory
**** entering debugger
> e:\nnn\mmm\bzrlib\groupcompress.pyo(293)_create_z_content_from_chunks()

A normal commit (bzr commit) fails at: bzrlib\groupcompress.pyo(151)_ensure_content()

Is there any workaround I can try?

I just associated a branch which might help. The idea is that if it gets a text which is large, rather than creating a delta index which overlays every byte, it 'skips' some. This should reduce the best-possible compression it can compute, but should lower peak memory as well. I tested it with MAX_NUM_ENTRIES = 1000 (which would mean that texts >16kB would be under-sampled). And the resulting bzr repository was 80,488kB rather than 79,892kB.

I don't have a great feeling for where the best cutoff is, etc.

If you want to test it, I suggest you copy your repository to a temp location, build this bzr, and then run:
 test/bzr pack temp/repo -Dmemory

That should give some information about whether this is worthwhile or not.

Note, the change is not well tested, but I did do some sanity tests. As such, I wouldn't run it on your live repository, but it certainly would help give information about whether it is worth pursuing this further.

The other workaround is to just switch to a 64-bit machine and run "bzr pack" there. It should only need to be run on a single machine one time, and from then the clients should be ok with the one-large-pack for a while.

John, thanks for your reply. I've tried it with a 64-bit machine (Win 7 SP1), but it failed with "out of memory" too. This machine has only 2 GB RAM, though.

Is there any way to prevent bzr from repackaging? Removing the pack-file causes it to crash immediately.

Encountering this same problem again right now I believe. Can't check in changes.
Note this error occurs attempting to check in even a single small file, so something must be corrupted. I am not trying to check in large files and the workstation is 64bit Windows with 24GB ram.

Bazaar Explorer commit status window looks like:

Run command: bzr commit -m "blah blah"
Committing to: V:/Proj/Bob/
modified src/BobLocal/file1.h
aborting commit write group: MemoryError()
bzr: out of memory
Use -Dmem_dump to dump memory to a file.

Running command 'bzr check -v' results in out of memory error.

I'm afraid I will have to abandon Bazaar if this issue can't be fixed real soon. I can't accept this level of instability and my coworkers are angry. Another friendly developer is about to abandon for the same reason. This is very unfortunate as I love Bazaar and admire design goals. I realize free software means waiting for volunteer activity. I'd pay for improvement if it were possible.

My work around for this issue once again was to backup local tree, delete the local repository, pull another version, then copy back the local tree to preserve code and versioning.

Now, with a new repository and after 10 commits, bzr tries to repack and fails again with an out of memory error. Starting anew I would lose all history.

Is it possible to increase the number of commits after a repack happens?

For those following this bug, <https://code.launchpad.net/~jameinel/bzr/2.4-max-entries-gc-602614/+merge/61072> adds a fix in to bzr 2.4 that should reduce memory consumption when repacking. If you would like to test 2.4b3 or the nightly ppa and let us know if it fixes it for you or not, that would be good.