Bazaar

Bug #405251
Comment #5

Comment 5 for bug 405251

Revision history for this message

John A Meinel (jameinel) wrote on 2009-07-27: Re: [Bug 405251] [NEW] Huge data transfers/bad performance OVERALL

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Frits Jalvingh wrote:
> Public bug reported:
>
> (I mailed the mailing list about this problem earlier; see Re: Huge data
> transfers/bad performance on bzr pull??).
>
> I have a large repository where pulling small changes can easily take
> almost forever. This morning I did a pull and it took almost half an
> hour. The actual new commits in that pull changes about 40 small files.
> The total data pulled was > 100MB. An earlier pull that was stopped had
> already transfered another 150MB, to in total 250MB was pulled. All the
> while the bzr "python" process used 100% CPU on an Core i7 920
> system!!?!?!
>
> As was requested by the people on the mailing list, I started the 2nd
> pull with the -Dhpss parameter. I will add the log from that to this
> report.
>
> The pull was done with a bzr 1.17 client and an 1.16 server (both on
> Linux).
>
> In addition: since upgrading to 1.16 everywhere I see very bad
> performance all over the place (on Windows clients); commits taking 2 to
> 15(!) minutes; status taking > 2 minutes etc. No idea whether this is
> related; I'll post a separate bug for that.
>
> ** Affects: bzr
> Importance: Undecided
> Status: New
>

So there was a bug in transfers that would cause us to transfer too much
data sometimes, which was this fix in 1.17:

* pack <=> pack fetching is now done via a ``PackStreamSource`` rather
  than the ``Packer`` code. The user visible change is that we now
  properly fetch the minimum number of texts for non-smart fetching.
  (John Arbash Meinel)

However that was only for dumb transfers, not transfers using bzr+ssh.
And if your client was 1.17, that wouldn't be relevant anyway.

I also see some really large content being transferred:
1100.261 638842 byte part read

^- This sort of thing would generally indicate a single content that is
640kB, and I see a lot of them. Did you perhaps have some large objects
added, modified a few times, and then removed? I see some files being
versioned like:
[23682] 2009-07-27 12:26:47.673 INFO: +N CSharpProjects/lib/log4net.dll
[23682] 2009-07-27 12:26:47.673 INFO: +N
CSharpProjects/lib/nunit-console-runner.dll

^- And versioning DLLs often creates a lot of churn in large files.

I would also be curious to find out how large the repository is overall.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkpt0+MACgkQJdeBCYSNAAMuUACfYbdIfaSSgCcZwePABmmLRQBg
GHkAn3GAXKPfFOaIWWs5Ul6k+rnCMn93
=aPhz
-----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Frits Jalvingh wrote:
> Public bug reported:
> 
> (I mailed the mailing list about this problem earlier; see Re: Huge data
> transfers/bad performance on bzr pull??).
> 
> I have a large repository where pulling small changes can easily take
> almost forever. This morning I did a pull and it took almost half an
> hour. The actual new commits in that pull changes about 40 small files.
> The total data pulled was > 100MB. An earlier pull that was stopped had
> already transfered another 150MB, to in total 250MB was pulled. All the
> while the bzr "python" process used 100% CPU on an Core i7 920
> system!!?!?!
> 
> As was requested by the people on the mailing list, I started the 2nd
> pull with the -Dhpss parameter. I will add the log from that to this
> report.
> 
> The pull was done with a bzr 1.17 client and an 1.16 server (both on
> Linux).
> 
> In addition: since upgrading to 1.16 everywhere I see very bad
> performance all over the place (on Windows clients); commits taking 2 to
> 15(!) minutes; status taking > 2 minutes etc. No idea whether this is
> related; I'll post a separate bug for that.
> 
> ** Affects: bzr
>      Importance: Undecided
>          Status: New
>

So there was a bug in transfers that would cause us to transfer too much
data sometimes, which was this fix in 1.17:

However that was only for dumb transfers, not transfers using bzr+ssh.
And if your client was 1.17, that wouldn't be relevant anyway.

I also see some really large content being transferred:
1100.261                638842 byte part read

^- This sort of thing would generally indicate a single content that is
640kB, and I see a lot of them. Did you perhaps have some large objects
added, modified a few times, and then removed? I see some files being
versioned like:
[23682] 2009-07-27 12:26:47.673 INFO: +N  CSharpProjects/lib/log4net.dll
[23682] 2009-07-27 12:26:47.673 INFO: +N
CSharpProjects/lib/nunit-console-runner.dll

^- And versioning DLLs often creates a lot of churn in large files.

I would also be curious to find out how large the repository is overall.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkpt0+MACgkQJdeBCYSNAAMuUACfYbdIfaSSgCcZwePABmmLRQBg
GHkAn3GAXKPfFOaIWWs5Ul6k+rnCMn93
=aPhz
-----END PGP SIGNATURE-----