ErrorFromSmartServer - AbsentContentFactory (unfixable by users) error when pulling a branch from the mirrored area

This is probably due to a get_stream RPC with an "ancestry-of" argument failing on a stacked branch?

What ought to happen is fetching all the ancestry that the stacked branch has, then asking the fallback location for the ancestry of all the revisions that were referenced (but not present) in the first repo.

On Fri, 2009-04-03 at 01:00 +0000, Andrew Bennetts wrote:
> This is probably due to a get_stream RPC with an "ancestry-of" argument
> failing on a stacked branch?
>
> What ought to happen is fetching all the ancestry that the stacked
> branch has, then asking the fallback location for the ancestry of all
> the revisions that were referenced (but not present) in the first repo.

I think its a broken repository; perhaps due to the known stacking
issues that early bzr stacking support had?

-Rob

Whoa, so are you saying every branch in that project is essentially hosed?

On Fri, 2009-04-03 at 03:37 +0000, Jean-Francois Roy wrote:
> Whoa, so are you saying every branch in that project is essentially
> hosed?

No, not at all. I'm speculating about the reason you get this error. We
obviously need to check in some detail. And even if I'm correct it would
indicate a single branch is in trouble, not all.

One immediate test that can be done is to try
'bzr pull http://bazaar.launchpad.net/~rivenx/rivenx/domes' which will
not use the smart server code path; if there is a bug in our smart
server support this would work.

Could you please try that?

-Rob

I think there's something funky going on with bzr.dev:

jfroy:Projects bahamut$ pwd
/Volumes/Crossroads/bahamut/Documents/Projects
jfroy:Projects bahamut$ bzr pull http://bazaar.launchpad.net/~rivenx/rivenx/domes
bzr: ERROR: Not a branch: "/Volumes/Crossroads/bahamut/Documents/Projects/".

I in fact can't seem to be able to pull using any protocol (bzr+ssh, lp, http).

OK nevermind the above.

'bzr branch http://bazaar.launchpad.net/~rivenx/rivenx/domes'

completed successfully.

I can reproduce this locally with a copy of your branches from Launchpad (copied via SFTP).

It's not a regression in bzr.dev, it's a bug that's new in 1.13. It seems to stem from the fact that with the new smart server verbs some of the "which records to fetch" logic runs on the server rather than the client, and in the context they run on the server the server does not have any knowledge of the fallback (stacked-on) repository. This is turn reveals a bug in working on a stacked repository that doesn't have its fallbacks set.

Specifically, fileids_altered_by_revision_ids is returning records that aren't altered in a particular revision, and this happens because that revision's parent is a ghost in this repo (but would be present in the fallback). The fix is probably to make the server return placeholder "you need this record, but I don't have it" records for those records. That basically amounts to serialising AbsentContentFactory on the wire, I guess.

1.13 clients won't understand that reply, but 1.13 clients are already failing in this case because of the server bug, so there's no new backwards compatibility issue.

After extending the server to handle this case, I'll make backports of the fix for 1.13 and 1.14rc1. We'll want that fix backported to Launchpad's deployment and probably to Jaunty as well. We may want to release a 1.13.2.

I'm bumping this up to Critical; even though 1.13 had this bug too, I don't think we should release 1.14 without a fix for this.

My whole team is using 1.13, I'm using 1.14 nightly builds, and am getting this traceback on every single branch that I try to merge this week - I've had to drop back to using bzr 1.12 to be able to keep working.

On Sun, 2009-04-05 at 16:09 +0000, Elliot Murphy wrote:
> My whole team is using 1.13, I'm using 1.14 nightly builds, and am
> getting this traceback on every single branch that I try to merge this
> week - I've had to drop back to using bzr 1.12 to be able to keep
> working.

Yes, we're preparing a fix+hotfix for 1.13 servers.

-Rob

Excellent, thanks!

Ok, root cause analysis:
 - stacking currently does not require enough data be available to generate inventory deltas for the first revision in the stacking branch
 - the new server fetch code depends [reasonably so] on being able to generate inventory deltas for the revisions present in the stacking branch

Fixes:
Immediately:
1) we will modify bzr so that enough data will be stored in new stacked branches. This will cause a 1.13.1 client pushing to a fixed server to push enough data.
2) We also plan to change the client code to anticipate that this is needed so that a fixed client pushing to a 1.13 server will upload enough data. This isn't strictly needed and if too complex we'll not do it.

Long term:
We'll change some deeper aspects of streaming push and pull to allow the server to report errors like this that occur in a cleaner way, and because they will only occur when something is actually wrong, we won't need massive complexity in the client to handle it.

My lp:~spiv/bzr/stacking-inventory branch has the client-side fix; pushing a stacked branch with this client should result in a branch that even a 1.13.1 client and server will work with. It's a pretty small, self-contained fix.

We still need to make a fix for the server to trigger 1.13.1 clients to send the missing data (part 1 of Robert's plan above).

So all this talks about fixes to the client and server in order to push new branches.

What about a fix that will allow me to pull a branch that was created by a 1.13.1 client and server? Is that part of this?

Tim:

In general, a smart server can't generate a valid stream for an affected branch, because it doesn't have enough data to distinguish between "complete delta for a revision" and "silently missing crucial parts".

For Launchpad, we can fix affected branches by filling in the missing data (i.e. transferring the required parent inventories) from the stacking location, which is also on Launchpad. We should definitely do this after getting fixed clients out there (and cherry-picking a fix to make LP's server ask unfixed clients to send the necessary keys). If we fix up the data in hosted branches after applying the fix to the server, then all branches will be fixed, and all new branches will be fine as well.

Well, it will still be possible to make broken branches with unfixed clients via SFTP, but those can always be retrieved via SFTP (because in that case the client is able to find the missing data itself, by looking at the fallback repository, whereas the server is unable to use fallbacks).

I suppose pre-1.13 clients will also continue to push up incomplete stacked branches too, because they don't use the RPC that will cause the "please send more keys" response from the server. Hopefully there won't be many of those for long, because 1.13 is much faster at network operations.

On Thu, 2009-04-09 at 04:18 +0000, Tim Penhey wrote:
> So all this talks about fixes to the client and server in order to push
> new branches.
>
> What about a fix that will allow me to pull a branch that was created by
> a 1.13.1 client and server? Is that part of this?

In addition to Andrew's reply - its not generally possible to fix this.

All clients from 1.6 through to 1.13.2 and 1.14 (assuming this fix gets
in) will generate broken stacked branches on sftp;, and 1.6 through to
1.13 will generate broken stacked branches on bzr+ssh.

Fortunately, stacking is slow and painful before 1.13, and therefor folk
are likely to upgrade :).

-Rob

Rob pointed me to this bug when I told him about this:

stewart@willster:~/drizzle$ bzr branch lp:~eday/drizzle/eday-dev
bzr: ERROR: bzrlib.errors.ErrorFromSmartServer: Error received from smart server: ('NoSuchRevision',)

Traceback (most recent call last):
  File "/home/stewart/lib/python/bzrlib/commands.py", line 727, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/home/stewart/lib/python/bzrlib/commands.py", line 922, in run_bzr
    ret = run(*run_argv)
  File "/home/stewart/lib/python/bzrlib/commands.py", line 559, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/home/stewart/lib/python/bzrlib/builtins.py", line 1122, in run
    source_branch=br_from)
  File "/home/stewart/lib/python/bzrlib/bzrdir.py", line 1142, in sprout
    result_repo.fetch(source_repository, revision_id=revision_id)
  File "/home/stewart/lib/python/bzrlib/repository.py", line 1493, in fetch
    find_ghosts=find_ghosts, fetch_spec=fetch_spec)
  File "/home/stewart/lib/python/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/home/stewart/lib/python/bzrlib/repository.py", line 3057, in fetch
    pb=pb, find_ghosts=find_ghosts)
  File "/home/stewart/lib/python/bzrlib/fetch.py", line 84, in __init__
    self.__fetch()
  File "/home/stewart/lib/python/bzrlib/fetch.py", line 110, in __fetch
    self._fetch_everything_for_search(search)
  File "/home/stewart/lib/python/bzrlib/fetch.py", line 138, in _fetch_everything_for_search
    stream, from_format, [])
  File "/home/stewart/lib/python/bzrlib/repository.py", line 3926, in insert_stream
    return self._locked_insert_stream(stream, src_format)
  File "/home/stewart/lib/python/bzrlib/repository.py", line 3954, in _locked_insert_stream
    for substream_type, substream in stream:
  File "/home/stewart/lib/python/bzrlib/remote.py", line 1734, in missing_parents_chain
    stream = self._get_stream(source, search)
  File "/home/stewart/lib/python/bzrlib/remote.py", line 1706, in _get_stream
    (path, self.to_format.network_name()), search_bytes)
  File "/home/stewart/lib/python/bzrlib/remote.py", line 71, in _call_with_body_bytes_expecting_body
    self._translate_error(err, **err_context)
  File "/home/stewart/lib/python/bzrlib/remote.py", line 688, in _translate_error
    self.bzrdir._translate_error(err, repository=self, **context)
  File "/home/stewart/lib/python/bzrlib/remote.py", line 129, in _translate_error
    _translate_error(err, bzrdir=self, **context)
  File "/home/stewart/lib/python/bzrlib/remote.py", line 2497, in _translate_error
    raise NoSuchRevision(find('branch'), err.error_args[0])
  File "/home/stewart/lib/python/bzrlib/remote.py", line 2481, in find
    raise err
ErrorFromSmartServer: Error received from smart server: ('NoSuchRevision',)

bzr 1.15dev on python 2.5.2 (linux2)
arguments: ['/home/stewart/bin/bzr', 'branch', 'lp:~eday/drizzle/eday-dev']
encoding: 'UTF-8', fsenc: 'UTF-8', lang: 'en_AU.UTF-8'
plugins:
  automv /home/stewart/.bazaar/plugins/automv [unknown]
  bisect /home/stewart/.bazaar/plugins/bisect [1.1.0.pre.0]
  bzrtools /home/stewart/.bazaar/plugins...

It may be worth checking for this on branch unlock, and if detected
fixing it programatically. At least for a few months.

Any updates on this?

Hi Stewart,

I've seen that one too. I haven't dug too deeply, but I think that's actually a different (although related) bug. I've filed bug 360791 for it. I've put a preliminary analysis and possible workarounds for that bug in the description of bug 360791.

(Let's use bug 360791 to for any further discussion of "Error received from smart server: ('NoSuchRevision',)" with 1.13+ servers, rather than cluttering this one.)

The client-side fix for this has now landed in bzr.dev.

So the infestations of this bug are:

With both client and server > 1.14rc1 there's no problem.
If the client is >=1.13 && <=1.14rc1 you'll be susceptible to this problem, and may hit it eventually.

This was introduced in 1.13 and discovered when Launchpad upgraded to 1.13. The cause of the problem goes back to 1.6 (where stacking was added) but it can only be seen when there's a streaming pull from a stacked repository, which was added in 1.13.

We were trying to have a way to have a new server cause the client to do the right thing, but at present it only works sometimes. In other cases it will cause an assertion in the client, which is still better than having the client think it's pushed and get it wrong. However, it may be possible to fix this.

That seems to mean this must be fixed in jaunty, probably with a minimal 1.13.2.

Merged into 1.14rc2. Thanks.

After upgrading Bazaar to 0.14rc2, I'm still seeing this:

$ bzr merge lp:~thisfred/ubunet/contacts-api
bzr: ERROR: bzrlib.errors.ErrorFromSmartServer: Error received from smart server: ('error', "'AbsentContentFactory' object has no attribute 'get_bytes_as'")

Traceback (most recent call last):
  File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 727, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 922, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 559, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.6/dist-packages/bzrlib/builtins.py", line 3527, in run
    location, revision, remember, possible_transports, pb)
  File "/usr/lib/python2.6/dist-packages/bzrlib/builtins.py", line 3634, in _get_merger_from_branch
    other_revision_id, base_revision_id, other_branch, base_branch)
  File "/usr/lib/python2.6/dist-packages/bzrlib/merge.py", line 204, in from_revision_ids
    merger.set_other_revision(other, other_branch)
  File "/usr/lib/python2.6/dist-packages/bzrlib/merge.py", line 345, in set_other_revision
    self._maybe_fetch(other_branch, self.this_branch, self.other_rev_id)
  File "/usr/lib/python2.6/dist-packages/bzrlib/merge.py", line 362, in _maybe_fetch
    target.fetch(source, revision_id)
  File "/usr/lib/python2.6/dist-packages/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/lib/python2.6/dist-packages/bzrlib/branch.py", line 522, in fetch
    pb=pb)
  File "/usr/lib/python2.6/dist-packages/bzrlib/repository.py", line 1493, in fetch
    find_ghosts=find_ghosts, fetch_spec=fetch_spec)
  File "/usr/lib/python2.6/dist-packages/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/lib/python2.6/dist-packages/bzrlib/repository.py", line 3057, in fetch
    pb=pb, find_ghosts=find_ghosts)
  File "/usr/lib/python2.6/dist-packages/bzrlib/fetch.py", line 84, in __init__
    self.__fetch()
  File "/usr/lib/python2.6/dist-packages/bzrlib/fetch.py", line 110, in __fetch
    self._fetch_everything_for_search(search)
  File "/usr/lib/python2.6/dist-packages/bzrlib/fetch.py", line 138, in _fetch_every...

Reopened per ~thisfred (who obviously meant to say 1.14rc2)

Nicola: do you know if that branch was pushed with 1.14rc2 (and never any older version)? If so this is expected behaviour (unfortunately), not another case of this bug.

Even with 1.14rc2 on both the client and the server you will still get this error from stacked branches pushed with older bzr versions, because those versions can push stacked branches that are incomplete. i.e. the "AbsentContentFactory" error is the symptom, not the bug. The bug occurs silently during "push", but is only noticed later when doing a pull/merge/etc.

(We should improve the server to give a better error to the client in this scenario, because presumably old clients will keep creating stacked branches for a while to come, but we should track that in a separate bug.)

Yeah, I meant to say 1.14rc2, sorry. By the way, I'm teknico (~teknico.net on LP, ugh), I was trying to merge a branch by thisfred. :-)

> Nicola: do you know if that branch was pushed with 1.14rc2
> (and never any older version)? If so this is expected behaviour
> (unfortunately), not another case of this bug.

AFAIK, that branch was pushed at least yesterday, before today's upgrade to 1.14rc2, so no. I'll ask thisfred confirmation later, when I get hold of him.

As a workaround, I downgraded to 1.13.1, and get no errors anymore.

We do use stacked branches. What's the fix for the real bug? Do we need to change something in the local, stacked branches? Does anything in bzr code need to change?

I've updated the bug description to hopefully make the fix for affected branches more explicit: currently the only way to fix an affected branch is to delete it and push it again with a fixed client (such as 1.14rc2). Some other workarounds are listed in the bug description too.

Thanks for the update, it sounds like this isn't a new instance of this bug, so I'll close this bug again.

This should repair a branch.

import bzrlib.branch
from bzrlib.revision import NULL_REVISION
b = bzrlib.branch.Branch.open(argv[1])

def main():
    b.lock_read()
    if not b.repository._fallback_repositories:
        return
    raw_r = b.repository.bzrdir.open_repository()
    raw_r.lock_write()
    try:
      revs = raw_r.all_revision_ids()
      rev_parents = raw_r.get_graph().get_parent_map(revs)
      needed = set()
      map(needed.update, rev_parents.itervalues())
      needed.discard(NULL_REVISION)
      needed = set((rev,) for rev in needed)
      needed = needed - raw_r.inventories.keys()
      print "Missing inventories: %r" % needed
      if not needed:
        return
      assert raw_r._format.network_name() ==
b.repository._format.network_name()
      stream = b.repository.inventories.get_record_stream(needed,
'topological', True)
      raw_r.start_write_group()
      try:
        raw_r.inventories.insert_recort_stream(stream)
      except:
        raw_r.abort_write_group()
        raise
      else:
        raw_r.commit_write_group()
    finally:
      raw_r.unlock()

main()

Merged into 1.13.2, I cannot change the bug task Status, so posting here. See http://permalink.gmane.org/gmane.comp.version-control.bazaar-ng.general/57030 for my question regarding changing Status.

Today we upgraded bzr to 1.14~rc2-1~bazaar1~jaunty1.

Then we run "bzr push --overwrite" on both branches (the one being merged, and the one receiving the merge), but the problem was still there.

We then removed both branches from LP and pushed them again, but still no luck.

Ubuntu One is definitely still seeing problems with this, as Nicola indicated above.

I don't know what the proper thing to do with the status is on this, but I'm going to chime in with a "this isn't entirely fixed" opinion at this point.

On Thu, 2009-04-23 at 08:58 +0000, Nicola Larosa wrote:
>
> Then we run "bzr push --overwrite" on both branches (the one being
> merged, and the one receiving the merge), but the problem was still
> there.

This wouldn't fix the problem.

> We then removed both branches from LP and pushed them again, but still
> no luck.

This should have. Have you tried my fix script?

-Rob

when I try that script on an affected branch, I get

Traceback (most recent call last):
   File "/home/rmcbride/foo.py", line 3, in <module>
     b = bzrlib.branch.Branch.open(argv[1])
NameError: name 'argv' is not defined

I'm guessing there's a parameter I should be passing

Robert Collins wrote:
> On Thu, 2009-04-23 at 08:58 +0000, Nicola Larosa wrote:
>> Then we run "bzr push --overwrite" on both branches (the one being
>> merged, and the one receiving the merge), but the problem was still
>> there.
>
> This wouldn't fix the problem.
>
>> We then removed both branches from LP and pushed them again, but still
>> no luck.
>
> This should have. Have you tried my fix script?
>
> -Rob
>

On Thu, 2009-04-23 at 22:20 +0000, Rick McBride wrote:
> when I try that script on an affected branch, I get
>
> Traceback (most recent call last):
> File "/home/rmcbride/foo.py", line 3, in <module>
> b = bzrlib.branch.Branch.open(argv[1])
> NameError: name 'argv' is not defined
>
>
> I'm guessing there's a parameter I should be passing

New version that should fix that.

And yes, run as
python foo.py lp:BRANCHONLP

Robert Collins wrote:
> On Thu, 2009-04-23 at 22:20 +0000, Rick McBride wrote:
>> when I try that script on an affected branch, I get
>>
>> Traceback (most recent call last):
>> File "/home/rmcbride/foo.py", line 3, in <module>
>> b = bzrlib.branch.Branch.open(argv[1])
>> NameError: name 'argv' is not defined
>>
>>
>> I'm guessing there's a parameter I should be passing
>
> New version that should fix that.
>
> And yes, run as
> python foo.py lp:BRANCHONLP
>
>
> ** Attachment added: "foo.py"
> http://launchpadlibrarian.net/25904342/foo.py
>

When I do that
(rmcbride@strangiato:~/canonical/ubunet/homepage-teh-client-pkg$ python
/home/rmcbride/foo.py lp:~rmcbride/ubunet/homepage-teh-client-pkg)

I get

bzrlib.errors.NotBranchError: Not a branch:
"/home/rmcbride/canonical/ubunet/homepage-teh-client-pkg/lp:~rmcbride/ubunet/homepage-teh-client-pkg/".

When I leave it out I now get

Traceback (most recent call last):
   File "/home/rmcbride/foo.py", line 4, in <module>
     b = bzrlib.branch.Branch.open(sys.argv[1])
IndexError: list index out of range

On Thu, 2009-04-23 at 23:28 +0000, Rick McBride wrote:
> Robert Collins wrote:
> > On Thu, 2009-04-23 at 22:20 +0000, Rick McBride wrote:
> >> when I try that script on an affected branch, I get
> >>
> >> Traceback (most recent call last):
> >> File "/home/rmcbride/foo.py", line 3, in <module>
> >> b = bzrlib.branch.Branch.open(argv[1])
> >> NameError: name 'argv' is not defined
> >>
> >>
> >> I'm guessing there's a parameter I should be passing
> >
> > New version that should fix that.
> >
> > And yes, run as
> > python foo.py lp:BRANCHONLP
> >
> >
> > ** Attachment added: "foo.py"
> > http://launchpadlibrarian.net/25904342/foo.py
> >
>
>
> When I do that
> (rmcbride@strangiato:~/canonical/ubunet/homepage-teh-client-pkg$ python
> /home/rmcbride/foo.py lp:~rmcbride/ubunet/homepage-teh-client-pkg)
>
> I get
>
> bzrlib.errors.NotBranchError: Not a branch:
> "/home/rmcbride/canonical/ubunet/homepage-teh-client-pkg/lp:~rmcbride/ubunet/homepage-teh-client-pkg/".

Oh,I haven't loaded plugins.

use bzr
+ssh://bazaar.launchpad.net/~rmcbride/ubunet/homepage-teh-client-pkg

That seems to work more, but I do get the following message:

Missing inventories: set([])

Robert Collins wrote:
> On Thu, 2009-04-23 at 23:28 +0000, Rick McBride wrote:
>> Robert Collins wrote:
>>> On Thu, 2009-04-23 at 22:20 +0000, Rick McBride wrote:
>>>> when I try that script on an affected branch, I get
>>>>
>>>> Traceback (most recent call last):
>>>> File "/home/rmcbride/foo.py", line 3, in <module>
>>>> b = bzrlib.branch.Branch.open(argv[1])
>>>> NameError: name 'argv' is not defined

>>>>
>>>>

>>>> I'm guessing there's a parameter I should be passing
>>> New version that should fix that.
>>>
>>> And yes, run as
>>> python foo.py lp:BRANCHONLP
>>>
>>>
>>> ** Attachment added: "foo.py"
>>> http://launchpadlibrarian.net/25904342/foo.py
>>>
>>
>> When I do that
>> (rmcbride@strangiato:~/canonical/ubunet/homepage-teh-client-pkg$ python
>> /home/rmcbride/foo.py lp:~rmcbride/ubunet/homepage-teh-client-pkg)
>>
>> I get
>>
>> bzrlib.errors.NotBranchError: Not a branch:
>> "/home/rmcbride/canonical/ubunet/homepage-teh-client-pkg/lp:~rmcbride/ubunet/homepage-teh-client-pkg/".
>
> Oh,I haven't loaded plugins.
>
> use bzr
> +ssh://bazaar.launchpad.net/~rmcbride/ubunet/homepage-teh-client-pkg
>

On Thu, 2009-04-23 at 23:58 +0000, Rick McBride wrote:

> That seems to work more, but I do get the following message:
>
> Missing inventories: set([])

That means that your branch appears fine. You're not suffering *this
bug* on *that branch*.

-Rob

OK. I suspected as much. I'm afraid something else may be wrong then, as
Phil Fibiger reported the same error with that exact branch...

Robert Collins wrote:
> On Thu, 2009-04-23 at 23:58 +0000, Rick McBride wrote:
>
>> That seems to work more, but I do get the following message:
>>
>> Missing inventories: set([])
>
> That means that your branch appears fine. You're not suffering *this
> bug* on *that branch*.
>
> -Rob
>

spiv, care to double check my foo.py?

Robert, your script looks sane, but I haven't tested it...

Robert I tried the script but I keep getting this error:

Missing inventories: set([('<email address hidden>',), ('<email address hidden>',)])
Traceback (most recent call last):
  File "/home/guillermo/Desktop/foo(2).py", line 36, in <module>
    main()
  File "/home/guillermo/Desktop/foo(2).py", line 27, in main
    raw_r.inventories.insert_recort_stream(stream)
AttributeError: 'KnitVersionedFiles' object has no attribute 'insert_recort_stream'

Also tried to delete and re-push the branch with bzr.dev and hitted the error with 1.14rc2 clients.

There's a typo in the script:

L27:

      raw_r.inventories.insert_recort_stream(stream)

should be:

      raw_r.inventories.insert_record_stream(stream)

On Mon, 2009-04-27 at 13:24 +0000, Guillermo Gonzalez wrote:
> Robert I tried the script but I keep getting this error:
>
> Missing inventories: set([('<email address hidden>',), ('<email address hidden>',)])
> Traceback (most recent call last):
> File "/home/guillermo/Desktop/foo(2).py", line 36, in <module>
> main()
> File "/home/guillermo/Desktop/foo(2).py", line 27, in main
> raw_r.inventories.insert_recort_stream(stream)
> AttributeError: 'KnitVersionedFiles' object has no attribute 'insert_recort_stream'

s/recort/record :) Just a typo - might want to attach a new vesion.

-Rob

The script is producing the following output:

Missing inventories: set([('<email address hidden>',), ('<email address hidden>',), ('<email address hidden>',), ('<email address hidden>',), ('<email address hidden>',), ('<email address hidden>',), ('<email address hidden>',), ('<email address hidden>',), ('<email address hidden>',), ('<email address hidden>',)])

It also seem to contain a typo on line #27, insert_recort_stream should possibly be insert_record_stream . The above output was obtained after correcting said typo.

bzr versions 1.6->1.12 inclusive will continue to create branches with insufficient data. We can't tell the server to tell them to upload more data as we can for 1.13 and above. We should look at running the fix script or something similar after such clients write to a stacked branch.

Michael: why is this Fix Committed in Launchpad? As far as I know users still have no way to fix branches in the mirrored area of code hosting, even if they run the fix script on their hosted branch. (or on the mirror source for externally hosted branches).

I've updated the bug summary to try reflect the outstanding issue AIUI, which is that the fix script from Robert isn't enough to make branches on Launchpad usable because the fix doesn't get propagated from from the hosted (writeable) copy to the mirrored (read-only) copy. If there's a separate bug tracking that already please feel free to link to it from this bug.

I presume it's fix committed because we cherry picked the fix that asks older clients to upload more data, as per comment #15.

If something more needs to be done (and it looks like there is), I'm not sure a bug report is the best place for talking about it, especially one this long and colourful. Let's talk about it on IRC, and send an RT?

Lets talk, but we may need open the bug depending whether the outcome is
'more code' or 'fix all branches'.

-Rob

We talked. The summary is 'File an RT'