Huge memory usage for bzr branch/checkout

~$ free -m
             total used free shared buffers cached
Mem: 1010 109 901 0 5 32
-/+ buffers/cache: 71 939
Swap: 1906 397 1509
rune@rao:~/bzr.repo$ du -sh
9,9G .
rune@rao:~/bzr.repo$ bzr info
Shared repository (format: pack-0.92)
Location:
  shared repository: .

So I have a shared repository taking up 10GB and less than 10GB memory.
This gives me problems (I run out of memory), when I run bzr check (and other commands).

rune@rao:~/bzr.repo$ cd data
rune@rao:~/bzr.repo/data$ bzr check
bzr: ERROR: bzrlib.errors.KnitCorrupt: Knit <bzrlib.knit._PackAccess object at 0x86de70c> corrupt: While reading {<email address hidden>} got MemoryError()

Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 834, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 790, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 492, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 2408, in run
    check(branch_obj, verbose)
  File "/usr/lib/python2.5/site-packages/bzrlib/check.py", line 254, in check
    repo_result = branch.repository.check([branch.last_revision()])
  File "/usr/lib/python2.5/site-packages/bzrlib/decorators.py", line 127, in read_locked
    return unbound(self, *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 1847, in check
    return self._check(revision_ids)
  File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 1851, in _check
    result.check()
  File "/usr/lib/python2.5/site-packages/bzrlib/check.py", line 79, in check
    self.check_one_rev(rev_id)
  File "/usr/lib/python2.5/site-packages/bzrlib/check.py", line 190, in check_one_rev
    self._check_revision_tree(rev_id)
  File "/usr/lib/python2.5/site-packages/bzrlib/check.py", line 234, in _check_revision_tree
    ie.check(self, rev_id, inv, tree)
  File "/usr/lib/python2.5/site-packages/bzrlib/inventory.py", line 375, in check
    self._check(checker, rev_id, tree)
  File "/usr/lib/python2.5/site-packages/bzrlib/inventory.py", line 598, in _check
    if self.text_sha1 != w.get_sha1(self.revision):
  File "/usr/lib/python2.5/site-packages/bzrlib/knit.py", line 760, in get_sha1
    return self.get_sha1s([version_id])[0]
  File "/usr/lib/python2.5/site-packages/bzrlib/knit.py", line 764, in get_sha1s
    record_map = self._get_record_map(version_ids)
  File "/usr/lib/python2.5/site-packages/bzrlib/knit.py", line 1121, in _get_record_map
    self._data.read_records_iter(records):
  File "/usr/lib/python2.5/site-packages/bzrlib/knit.py", line 2639, in read_records_iter
    content, digest = self._parse_record(version_id, data)
  File "/usr/lib/python2.5/site-packages/bzrlib/knit.py", line 2545, in _parse_record
    % (version_id, e.__class__.__name__, str(e)))
KnitCorrupt: Knit <bzrlib.knit._PackAccess object at 0x86de70c> corrupt: While reading {<email address hidden>...

rune@rao:~/data$ bzr info
Standalone tree (format: pack-0.92)
Location:
  branch root: .

Related branches:
  parent branch: /home/rune/bzr.repo/data

This branch already has a lot of the data, but trying to pull more revisions fail, because of memory problems.
This is on the same machine as above, so I can't do a bzr check to see if such a failed pull breaked anything.

rune@rao:~/data$ bzr pull
Using saved location: /home/rune/bzr.repo/data/
bzr: ERROR: exceptions.MemoryError:

Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 834, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 790, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 492, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 652, in run
    possible_transports=possible_transports)
  File "/usr/lib/python2.5/site-packages/bzrlib/decorators.py", line 165, in write_locked
    return unbound(self, *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/workingtree.py", line 1570, in pull
    possible_transports=possible_transports)
  File "/usr/lib/python2.5/site-packages/bzrlib/decorators.py", line 165, in write_locked
    return unbound(self, *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/branch.py", line 1714, in pull
    run_hooks=run_hooks)
  File "/usr/lib/python2.5/site-packages/bzrlib/decorators.py", line 165, in write_locked
    return unbound(self, *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/branch.py", line 1513, in pull
    self.update_revisions(source, stop_revision, overwrite=overwrite)
  File "/usr/lib/python2.5/site-packages/bzrlib/decorators.py", line 165, in write_locked
    return unbound(self, *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/branch.py", line 1465, in update_revisions
    self.fetch(other, stop_revision)
  File "/usr/lib/python2.5/site-packages/bzrlib/decorators.py", line 165, in write_locked
    return unbound(self, *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/branch.py", line 266, in fetch
    pb=nested_pb)
  File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 949, in fetch
    return inter.fetch(revision_id=revision_id, pb=pb, find_ghosts=find_ghosts)
  File "/usr/lib/python2.5/site-packages/bzrlib/decorators.py", line 165, in write_locked
    return unbound(self, *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 2759, in fetch
    revision_ids).pack()
  File "/usr/lib/python2.5/site-packages/bzrlib/repofmt/pack_repo.py", line 589, in pack
    return self._create_pack_from_packs()
  File "/usr/lib/python2.5/site-packages/bzrlib/repofmt/pack_repo.py", line 722, in _create_pack_from_packs
    self._copy_text_texts()
  File "/usr/lib/python2.5/site-packages/bzrlib/repofmt/pack_repo.py", line 686, in _copy_text_texts
    self.new_pack.text_index, readv_group_iter, total_items))
  File "/usr/lib/python2.5/site-packages/bzrlib/repofmt/pack_repo.py", line 807, in _copy_nodes_gr...

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Olberd wrote:
> rune@rao:~/data$ bzr info
> Standalone tree (format: pack-0.92)
> Location:
> branch root: .
>
> Related branches:
> parent branch: /home/rune/bzr.repo/data
>
> This branch already has a lot of the data, but trying to pull more revisions fail, because of memory problems.
> This is on the same machine as above, so I can't do a bzr check to see if such a failed pull breaked anything.
>
> rune@rao:~/data$ bzr pull
> Using saved location: /home/rune/bzr.repo/data/
...

> bzr 1.3.1rc1 on python 2.5.1.final.0 (linux2)
> arguments: ['/usr/bin/bzr', 'pull']
> encoding: 'UTF-8', fsenc: 'UTF-8', lang: 'en_DK.UTF-8'
> plugins:
> launchpad /usr/lib/python2.5/site-packages/bzrlib/plugins/launchpad [unknown]
> *** Bazaar has encountered an internal error.
> Please report a bug at https://bugs.launchpad.net/bzr/+filebug
> including this traceback, and a description of what you
> were doing when the error occurred.

^- I highly recommend upgrading from bzr 1.3.1.

In bzr 2.0 there is a new repository format where we have focused more
directly on memory consumption (as well as performance).

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkr0SXcACgkQJdeBCYSNAAN0+gCgmisFG+ZHuCIg5UHuaz9XpWjm
VK0AnjN2wJ1w6cws0QXBgivm64VfFrGp
=a9p5
-----END PGP SIGNATURE-----

But upgrading to the new format is also problematic. See bug #490770.
So should this bug be dependent on bug #490770, because the suggested workaround is dependent on that bug?
A fix of this bug is not dependent on bug #490770, only the workaround (workaround == upgrade).

So I finally got my repository upgraded to the new format (see how in bug #490770), but I still have problems branching.

test.bzr.2$ bzr info
Shared repository with trees (format: 2a)
Location:
  shared repository: .
test.bzr.2$ cd ../test.bzr.3
test.bzr.3$ bzr info
Shared repository (format: 2a)
Location:
  shared repository: .
test.bzr.3$ bzr branch ~/test.bzr.2/data
Fetching revisions:Get stream source
Segmentation fault (core dumped)source

Unfortunately, the core dump is too large to be analyzed according to apport.

On 12 April 2010 21:39, Olberd <email address hidden> wrote:
> So I finally got my repository upgraded to the new format (see how in
> bug #490770), but I still have problems branching.
>
> test.bzr.2$ bzr info
> Shared repository with trees (format: 2a)
> Location:
>  shared repository: .
> test.bzr.2$ cd ../test.bzr.3
> test.bzr.3$ bzr info
> Shared repository (format: 2a)
> Location:
>  shared repository: .
> test.bzr.3$ bzr branch ~/test.bzr.2/data
> Fetching revisions:Get stream source
> Segmentation fault (core dumped)source
>
> Unfortunately, the core dump is too large to be analyzed according to
> apport.

Please run this command under gdb, ie

gdb /usr/bin/python
run /usr/bin/bzr branch ~/test.bzr.2/data

(adjust paths to taste)

if that aborts then type

bt

and paste the results.

--
Martin <http://launchpad.net/~mbp/>

Do you prefer large outputs, such as that below, in a comment or an attachment?
I will keep the gdb session running (since it takes a while to get to the error), so feel free to ask me to investigate further.

(gdb) run /usr/bin/bzr branch ~/test.bzr.2/data test2
Starting program: /usr/bin/python /usr/bin/bzr branch ~/test.bzr.2/data test2
[Thread debugging using libthread_db enabled]
- Fetching revisions:Get stream source
Program received signal SIGSEGV, Segmentation fault.
0x00007ffff2306de8 in create_delta_index ()
   from /usr/lib/python2.6/dist-packages/bzrlib/_groupcompress_pyx.so
(gdb) bt
#0 0x00007ffff2306de8 in create_delta_index ()
   from /usr/lib/python2.6/dist-packages/bzrlib/_groupcompress_pyx.so
#1 0x00007ffff230546c in ?? ()
   from /usr/lib/python2.6/dist-packages/bzrlib/_groupcompress_pyx.so
#2 0x00000000004a290d in PyEval_EvalFrameEx ()
#3 0x00000000004a40e0 in PyEval_EvalCodeEx ()
#4 0x00000000004a245f in PyEval_EvalFrameEx ()
#5 0x00000000004a40e0 in PyEval_EvalCodeEx ()
#6 0x00000000004a245f in PyEval_EvalFrameEx ()
#7 0x00000000005299ce in ?? ()
#8 0x000000000049e0ee in PyEval_EvalFrameEx ()
#9 0x00000000004a2e47 in PyEval_EvalFrameEx ()
#10 0x00000000004a40e0 in PyEval_EvalCodeEx ()
#11 0x00000000004a245f in PyEval_EvalFrameEx ()
#12 0x00000000004a2e47 in PyEval_EvalFrameEx ()
#13 0x00000000004a2e47 in PyEval_EvalFrameEx ()
#14 0x00000000004a2e47 in PyEval_EvalFrameEx ()
#15 0x00000000004a40e0 in PyEval_EvalCodeEx ()
#16 0x000000000052bf2d in ?? ()
#17 0x000000000041d6e7 in PyObject_Call ()
#18 0x00000000004254ff in ?? ()
#19 0x000000000041d6e7 in PyObject_Call ()
#20 0x000000000047068e in ?? ()
---Type <return> to continue, or q <return> to quit---
#21 0x000000000046b92f in ?? ()
#22 0x000000000041d6e7 in PyObject_Call ()
#23 0x00000000004a199c in PyEval_EvalFrameEx ()
#24 0x00000000004a40e0 in PyEval_EvalCodeEx ()
#25 0x000000000052bf2d in ?? ()
#26 0x000000000041d6e7 in PyObject_Call ()
#27 0x00000000004a0f31 in PyEval_EvalFrameEx ()
#28 0x00000000004a40e0 in PyEval_EvalCodeEx ()
#29 0x00000000004a245f in PyEval_EvalFrameEx ()
#30 0x00000000004a40e0 in PyEval_EvalCodeEx ()
#31 0x00000000004a245f in PyEval_EvalFrameEx ()
#32 0x00000000004a40e0 in PyEval_EvalCodeEx ()
#33 0x00000000004a245f in PyEval_EvalFrameEx ()
#34 0x00000000004a40e0 in PyEval_EvalCodeEx ()
#35 0x000000000052bf2d in ?? ()
#36 0x000000000041d6e7 in PyObject_Call ()
#37 0x00000000004a0f31 in PyEval_EvalFrameEx ()
#38 0x00000000004a40e0 in PyEval_EvalCodeEx ()
#39 0x000000000052bf2d in ?? ()
#40 0x000000000041d6e7 in PyObject_Call ()
#41 0x00000000004a0f31 in PyEval_EvalFrameEx ()
#42 0x00000000004a40e0 in PyEval_EvalCodeEx ()
#43 0x000000000052bf2d in ?? ()
---Type <return> to continue, or q <return> to quit---
#44 0x000000000041d6e7 in PyObject_Call ()
#45 0x00000000004a0f31 in PyEval_EvalFrameEx ()
#46 0x00000000004a40e0 in PyEval_EvalCodeEx ()
#47 0x000000000052bf2d in ?? ()
#48 0x000000000041d6e7 in PyObject_Call ()
#49 0x00000000004a0f31 in PyEval_EvalFrameEx ()
#50 0x00000000004a40e0 in PyEval_EvalCodeEx ()
#51 0x000000000052be30 in ?? ()
#52 0x000000000041d6e7 in PyObject_Call ()
#53 0x000000...

On 13 April 2010 20:15, Olberd <email address hidden> wrote:
> Do you prefer large outputs, such as that below, in a comment or an attachment?

Up to about a hundred lines is fine in a comment, beyond that I would
put it in an attachment.

> I will keep the gdb session running (since it takes a while to get to the error), so feel free to ask me to investigate further.
>
> (gdb) run /usr/bin/bzr branch ~/test.bzr.2/data test2
> Starting program: /usr/bin/python /usr/bin/bzr branch ~/test.bzr.2/data test2
> [Thread debugging using libthread_db enabled]
> - Fetching revisions:Get stream source
> Program received signal SIGSEGV, Segmentation fault.
> 0x00007ffff2306de8 in create_delta_index ()
>   from /usr/lib/python2.6/dist-packages/bzrlib/_groupcompress_pyx.so
> (gdb) bt
> #0  0x00007ffff2306de8 in create_delta_index ()
>   from /usr/lib/python2.6/dist-packages/bzrlib/_groupcompress_pyx.so

OK, I think that makes it fairly clear. You could also run the
command 'disassemble' here.

Thanks
--
Martin <http://launchpad.net/~mbp/>

Presumably something in create_delta_index is either badly handling
out-of-memory, or allocating but then not coping well with a large
array. I inspected this and can't see anything wrong at the moment so
it would help to know what line specifically is failing, either by
seeing the disassembly or by rebuilding the extensions with debug
symbols.

'memsize' is an unsigned long but would be better as a size_t: ulong
is not guaranteed to be big enough.

The requested disassemble.

btw I don't think I posted this info for the latest crash.
Bazaar (bzr) 2.1.1
  Python interpreter: /usr/bin/python 2.6.4
  Python standard library: /usr/lib/python2.6
  Platform: Linux-2.6.31-20-generic-x86_64-with-Ubuntu-9.10-karmic
  bzrlib: /usr/lib/python2.6/dist-packages/bzrlib
  Bazaar configuration: /home/rune/.bazaar
  Bazaar log file: /home/rune/.bzr.log

From
 *** 2.1.1-1~bazaar2~karmic 0
        500 http://ppa.launchpad.net karmic/main Packages
deb http://ppa.launchpad.net/bzr/ppa/ubuntu karmic main

Noticed the log file info in the last comment. So here is the relevant log entry.

If you need the info with debugging symbols, which package is it that I need symbols for?

On 14 April 2010 19:12, Olberd <email address hidden> wrote:
> If you need the info with debugging symbols, which package is it that I
> need symbols for?

Are you installing from Ubuntu, or bzr's ppa, or something else?

You need the symbols for bzr.

If you built from source, running

make CFLAGS='-g'

should probably do it.

--
Martin <http://launchpad.net/~mbp/>

Actually I have included the version information in comment #10.
But the package firefox-lp-improvements has probably removed much of the text from comment #10 on your computer, it did on mine anyway.
I've installed from bzr's ppa.

I recompiled using debuild and pkg-create-dbgsym on the source of the package I have installed.
Then I installed the resulting bzr-dbgsym package.
But that doesn't change the output of the gdb backtrace, because gdb only tries to load the symbols for python, not for the bzr stuff.

I tried loading the relevant symbols manually, but that didn't help:
(gdb) symbol-file /usr/lib/debug/usr/lib/python2.6/dist-packages/bzrlib/_groupcompress_pyx.so
Load new symbol table from "/usr/lib/debug/usr/lib/python2.6/dist-packages/bzrlib/_groupcompress_pyx.so"? (y or n) y
Reading symbols from /usr/lib/debug/usr/lib/python2.6/dist-packages/bzrlib/_groupcompress_pyx.so...done.
(gdb) bt
#0 0x00007ffff2306de8 in create_delta_index ()
   from /usr/lib/python2.6/dist-packages/bzrlib/_groupcompress_pyx.so
#1 0x00007ffff230546c in ?? ()
   from /usr/lib/python2.6/dist-packages/bzrlib/_groupcompress_pyx.so
#2 0x00000000004a290d in ?? ()
#3 0x00007fff3e82e7d7 in ?? ()
#4 0x0000000000159700 in ?? ()
#5 0x00000000ffffff01 in ?? ()
#6 0x00007ffff702c82e in *__GI___libc_malloc (bytes=8904864) at malloc.c:3638
#7 0x0000000005046098 in ?? ()
#8 0x00000000019478c8 in ?? ()
#9 0x00000000016935d0 in ?? ()
#10 0x00000000016a1a64 in ?? ()
#11 0x0000000000000000 in ?? ()

Finally got around to recompiling without optimizations.

Starting program: /usr/bin/python /usr/bin/bzr branch ~/test.bzr.2/data test3data
[Thread debugging using libthread_db enabled]
- Fetching revisions:Get stream source
Program received signal SIGSEGV, Segmentation fault.
0x00007ffff22f2af9 in pack_delta_index (hash=0x7ffcef822010, hsize=33554432,
    num_entries=115046781, old_index=0x7ffddaf1e010) at bzrlib/diff-delta.c:344
344 *packed_entry++ = entry->entry;
(gdb) l
339 *packed_entry++ = *old_entry;
340 }
341 }
342 }
343 for (entry = hash[i]; entry; entry = entry->next) {
344 *packed_entry++ = entry->entry;
345 }
346 /* TODO: At this point packed_entry - packed_hash[i] is the number of
347 * records that we have inserted into this hash bucket.
348 * We should *really* consider doing some limiting along the
(gdb) p *packed_entry
$2 = {ptr = 0x7ffd65a55fd0 "\004\232Z\207\377\177", src = 0x7ffd10721d70,
  val = 416214768}
(gdb) p i
$3 = 9909109
(gdb) p hsize
$4 = 33554432
(gdb) p entry
$5 = (struct unpacked_index_entry *) 0x4df1237bff2c999c
(gdb) p *entry
Cannot access memory at address 0x4df1237bff2c999c
(gdb) p *hash[i]
$7 = {entry = {
    ptr = 0x8e3089a7b28697c2 <Address 0x8e3089a7b28697c2 out of bounds>,
    src = 0x7854880e606623e0, val = 834984091}, next = 0x4df1237bff2c999c}
(gdb) p *(packed_entry-1)
$9 = {ptr = 0x8e3089a7b28697c2 <Address 0x8e3089a7b28697c2 out of bounds>,
  src = 0x7854880e606623e0, val = 834984091}

Thanks very much for that gdb trace.

Wow:

0x00007ffff22f2af9 in pack_delta_index (hash=0x7ffcef822010, hsize=33554432, num_entries=115046781, old_index=0x7ffddaf1e010) at bzrlib/diff-delta.c:344

Those are very big numbers for hsize and num_entries. On a 64-bit system I think that would need about 6 GB of memory.

As Martin pointed out earlier, the memsize variable is "unsigned int", but should be size_t. Does "unsigned int" end up as only holding 32 bits on a 64-bit system? If so, it would probably try allocating a mere 2GB or so, possibly succeed, and then (probably) segfault when trying to populate that memory. Fixing memsize to be a size_t would perhaps help cause a more sensible failure, but it would still fail.

I wonder why the hsize and num_entries are so huge in the first place?

This comment is in the spirit of retesting the original bug report from 2008.

$ uname -a
Linux binghamton 2.6.34.7-56.fc13.i686 #1 SMP Wed Sep 15 03:33:58 UTC 2010 i686 i686 i386 GNU/Linux
$ bzr --version
Bazaar (bzr) 2.1.2
  Python interpreter: /usr/bin/python 2.6.4
  Python standard library: /usr/lib/python2.6
  Platform: Linux-2.6.34.7-56.fc13.i686-i686-with-fedora-13-Goddard
  bzrlib: /usr/lib/python2.6/site-packages/bzrlib
  Bazaar configuration: /home/ben/.bazaar
  Bazaar log file: /home/ben/.bzr.log

I tried repeating the command:
   bzr branch lp:mysql-server/6.0
...but 6.0 isn't there anymore.

So I tried running:
  $ bzr branch lp:mysql-server/5.5

Running top alongside shows the resident memory usage of bzr to rise to 353MB before it is finished.

I'm not running x86_64 like the original bug report.

bing пишет:
> This comment is in the spirit of retesting the original bug report from
> 2008.
>
> $ uname -a
> Linux binghamton 2.6.34.7-56.fc13.i686 #1 SMP Wed Sep 15 03:33:58 UTC 2010 i686 i686 i386 GNU/Linux
> $ bzr --version
> Bazaar (bzr) 2.1.2

You should test at least bzr 2.2.1 today.

This bug is also affecting bzr 2.1.1-1 in Ubuntu 10.04.2 LTS on x86_64.

  status confirmed

This affects me too while working lp:gcc. Simply branching or doing checkouts is often needed on PCs with little RAM to just fetch a specific revision of the software and compile/test. So there should be at least one way (be it --lightweight or --stacked I don't care) that would actually work and need no more memory than wget. :-)

I am unable to checkout lp:mysql-server on a machine with 16 GB of ram!

[ 971.087983] Out of memory: Kill process 1694 (bzr) score 956 or sacrifice child
[ 971.087986] Killed process 1694 (bzr) total-vm:15870528kB, anon-rss:15588084kB, file-rss:16kB
[bpowers@fina src]$ bzr --version
Bazaar (bzr) 2.5.0
  Python interpreter: /usr/bin/python 2.7.3
  Python standard library: /usr/lib64/python2.7
  Platform: Linux-3.5.0-rc3+-x86_64-with-fedora-17-Beefy_Miracle

Note the rss usage: 15588084kB

2.5.0 indeed still has memory leak issues. Gave my summary here - https://bugs.launchpad.net/bzr-svn/+bug/191731/comments/25

Including a workaround now (add large swap space)

Eduardo, on what basis are you marking this as fixed?