many get_parent_map calls during walk to common revisions

Bug #435048 reported by Martin Pool
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Bazaar
Confirmed
Medium
Unassigned

Bug Description

While pulling from lp:bzr-svn into my copy, with the remote in 2a and my branch in a pack format, I see it make very slow progress with many small get_parent_map calls. I eventually interrupted it in

175.807 Traceback (most recent call last):
  File "/home/mbp/bzr/trunk/bzrlib/commands.py", line 842, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/home/mbp/bzr/trunk/bzrlib/commands.py", line 1037, in run_bzr
    ret = run(*run_argv)
  File "/home/mbp/bzr/trunk/bzrlib/commands.py", line 654, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/home/mbp/bzr/trunk/bzrlib/builtins.py", line 1017, in run
    possible_transports=possible_transports, local=local)
  File "/home/mbp/bzr/trunk/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/home/mbp/bzr/trunk/bzrlib/workingtree.py", line 1611, in pull
    local=local)
  File "/home/mbp/bzr/trunk/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/home/mbp/bzr/trunk/bzrlib/branch.py", line 948, in pull
    possible_transports=possible_transports, *args, **kwargs)
  File "/home/mbp/bzr/trunk/bzrlib/branch.py", line 3194, in pull
    _override_hook_target=_override_hook_target)
  File "/home/mbp/bzr/trunk/bzrlib/branch.py", line 3071, in pull
    overwrite=overwrite, graph=graph)
  File "/home/mbp/bzr/trunk/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/home/mbp/bzr/trunk/bzrlib/branch.py", line 896, in update_revisions
    overwrite, graph)
  File "/home/mbp/bzr/trunk/bzrlib/branch.py", line 3014, in update_revisions
    self.target.fetch(self.source, stop_revision)
  File "/home/mbp/bzr/trunk/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/home/mbp/bzr/trunk/bzrlib/branch.py", line 579, in fetch
    pb=pb)
  File "/home/mbp/bzr/trunk/bzrlib/repository.py", line 1695, in fetch
    find_ghosts=find_ghosts, fetch_spec=fetch_spec)
  File "/home/mbp/bzr/trunk/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/home/mbp/bzr/trunk/bzrlib/repository.py", line 3413, in fetch
    pb=pb, find_ghosts=find_ghosts)
  File "/home/mbp/bzr/trunk/bzrlib/fetch.py", line 81, in __init__
    self.__fetch()
  File "/home/mbp/bzr/trunk/bzrlib/fetch.py", line 103, in __fetch
    search = self._revids_to_fetch()
  File "/home/mbp/bzr/trunk/bzrlib/fetch.py", line 172, in _revids_to_fetch
    find_ghosts=self.find_ghosts)
  File "/home/mbp/bzr/trunk/bzrlib/decorators.py", line 138, in read_locked
    result = unbound(self, *args, **kwargs)
  File "/home/mbp/bzr/trunk/bzrlib/repository.py", line 1522, in search_missing_revision_ids
    revision_id, find_ghosts)
  File "/home/mbp/bzr/trunk/bzrlib/decorators.py", line 138, in read_locked
    result = unbound(self, *args, **kwargs)
  File "/home/mbp/bzr/trunk/bzrlib/repository.py", line 3479, in search_missing_revision_ids
    return self._walk_to_common_revisions([revision_id])
  File "/home/mbp/bzr/trunk/bzrlib/repository.py", line 3435, in _walk_to_common_revisions
    next_revs_part, ghosts_part = searcher.next_with_ghosts()
  File "/home/mbp/bzr/trunk/bzrlib/graph.py", line 1303, in next_with_ghosts
    self._advance()
  File "/home/mbp/bzr/trunk/bzrlib/graph.py", line 1313, in _advance
    found, ghosts, next, parents = self._do_query(self._next_query)
  File "/home/mbp/bzr/trunk/bzrlib/graph.py", line 1336, in _do_query
    parent_map = self._parents_provider.get_parent_map(revisions)
  File "/home/mbp/bzr/trunk/bzrlib/graph.py", line 99, in get_parent_map
    new_found = parents_provider.get_parent_map(remaining)
  File "/home/mbp/bzr/trunk/bzrlib/graph.py", line 167, in get_parent_map
    parent_map = self._get_parent_map(needed_revisions)
  File "/home/mbp/bzr/trunk/bzrlib/remote.py", line 1390, in _get_parent_map_rpc
    verb, args, body)
  File "/home/mbp/bzr/trunk/bzrlib/remote.py", line 72, in _call_with_body_bytes_expecting_body
    method, args, body_bytes)
  File "/home/mbp/bzr/trunk/bzrlib/smart/client.py", line 167, in call_with_body_bytes_expecting_body
    method, args, body=body, expect_response_body=True)
  File "/home/mbp/bzr/trunk/bzrlib/smart/client.py", line 78, in _call_and_read_response
    expect_body=expect_response_body),
  File "/home/mbp/bzr/trunk/bzrlib/smart/message.py", line 299, in read_response_tuple
    self._wait_for_response_args()
  File "/home/mbp/bzr/trunk/bzrlib/smart/message.py", line 264, in _wait_for_response_args
    self._read_more()
  File "/home/mbp/bzr/trunk/bzrlib/smart/message.py", line 277, in _read_more
    bytes = self._medium_request.read_bytes(next_read_size)
  File "/home/mbp/bzr/trunk/bzrlib/smart/medium.py", line 456, in read_bytes
    return self._read_bytes(count)
  File "/home/mbp/bzr/trunk/bzrlib/smart/medium.py", line 468, in _read_bytes
    return self._medium.read_bytes(count)
  File "/home/mbp/bzr/trunk/bzrlib/smart/medium.py", line 149, in read_bytes
    return self._read_bytes(bytes_to_read)
  File "/home/mbp/bzr/trunk/bzrlib/smart/medium.py", line 805, in _read_bytes
    bytes = osutils.until_no_eintr(self._read_from.read, bytes_to_read)
  File "/home/mbp/bzr/trunk/bzrlib/osutils.py", line 1855, in until_no_eintr
    return f(*a, **kw)
KeyboardInterrupt

Tags: hpss
Revision history for this message
Martin Pool (mbp) wrote :
Revision history for this message
John A Meinel (jameinel) wrote : Re: [Bug 435048] [NEW] many get_parent_map calls during walk to common revisions

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Martin Pool wrote:
> Public bug reported:
>
> While pulling from lp:bzr-svn into my copy, with the remote in 2a and my
> branch in a pack format, I see it make very slow progress with many
> small get_parent_map calls. I eventually interrupted it in
>

^- Isn't that backwards? Last I checked bzr-svn was in 1.9-rich-root
format...

...
> File "/home/mbp/bzr/trunk/bzrlib/fetch.py", line 103, in __fetch
> search = self._revids_to_fetch()
> File "/home/mbp/bzr/trunk/bzrlib/fetch.py", line 172, in _revids_to_fetch
> find_ghosts=self.find_ghosts)
> File "/home/mbp/bzr/trunk/bzrlib/decorators.py", line 138, in read_locked
> result = unbound(self, *args, **kwargs)
> File "/home/mbp/bzr/trunk/bzrlib/repository.py", line 1522, in search_missing_revision_ids
> revision_id, find_ghosts)

^- This means it was using the streaming fetch code path.

> File "/home/mbp/bzr/trunk/bzrlib/decorators.py", line 138, in read_locked
> result = unbound(self, *args, **kwargs)
> File "/home/mbp/bzr/trunk/bzrlib/repository.py", line 3479, in search_missing_revision_ids
> return self._walk_to_common_revisions([revision_id])
> File "/home/mbp/bzr/trunk/bzrlib/repository.py", line 3435, in _walk_to_common_revisions
> next_revs_part, ghosts_part = searcher.next_with_ghosts()

^- And I think the overhead was just in the _walk_to_common part.

This is supposed to grab the revisions, and a bit of extra history. That
said, I think Andrew has spent a lot more time optimizing 'push'
performance rather than 'pull' performance. And it is possible that some
more tweaking needs to be done here.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkq6I2oACgkQJdeBCYSNAAOaigCeOEJRlNuO2iglX8nWTCsAV6Jy
aScAoM/mqBKdp6mQNi9N566MVxh9Z+EP
=hDzG
-----END PGP SIGNATURE-----

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.