Knit.add_lines() doesn't allow filling in text ghost parents

What versions of bzr-svn / bzr are you running?

/opt/bzr$ bzr --version
Bazaar (bzr) 1.14.1
  Python interpreter: /usr/local/bin/python 2.5.2
  Python standard library: /usr/local/lib/python2.5
  bzrlib: /usr/local/lib/python2.5/site-packages/bzrlib
  Bazaar configuration: /home/reubenf/.bazaar
  Bazaar log file: /home/reubenf/.bzr.log

Copyright 2005, 2006, 2007, 2008, 2009 Canonical Ltd.
http://bazaar-vcs.org/

bzr comes with ABSOLUTELY NO WARRANTY. bzr is free software, and
you may use, modify and redistribute it under the terms of the GNU
General Public License version 2 or later.

/opt/bzr$ svn --version
svn, version 1.5.6 (r36142)
   compiled Mar 19 2009, 03:08:28

Copyright (C) 2000-2008 CollabNet.
Subversion is open source software, see http://subversion.tigris.org/
This product includes software developed by CollabNet (http://www.Collab.Net/).

What version of bzr-svn? (Check "bzr plugins".)

svn 0.5.3
    Support for Subversion branches

Any chance you can give 0.5.4 a try? I did fix at least one bug with a backtrace similar to yours. If 0.5.4 doesn't help, we can have a closer look at what's actually going wrong. Is the repository public?

Hi, with 0.5.4, we get a different (related?) error:

bzr svn-import --trees /opt/svn/repositories/benetech.org/Bookshare/
Initialising Subversion metadata cache in /root/.bazaar/svn-cache/13cce0d3-8e6b-44ce-8dc0-d6332f4951be
Using repository layout: trunk0
bzr: ERROR: bzrlib.errors.KnitCorrupt: Knit _KnitGraphIndex(CombinedGraphIndex(GraphIndex('file:///tmp/foobahh/Bookshare/.bzr/repository/indices/77f9da8f074fce0ac5772c1ca8f3c6ed.tix'), GraphIndex('file:///tmp/foobahh/Bookshare/.bzr/repository/indices/c0f11e5d47642e2fed6fd3df6ab36fd5.tix'), GraphIndex('file:///tmp/foobahh/Bookshare/.bzr/repository/indices/2d1e9e274509eb9ac36b13f6993a0a0a.tix'), GraphIndex('file:///tmp/foobahh/Bookshare/.bzr/repository/indices/ff643f97ad642a0a59e4a4e674bf390d.tix'), GraphIndex('file:///tmp/foobahh/Bookshare/.bzr/repository/indices/e5ccb74bb128a24085ad6a5935018f09.tix'))) corrupt: inconsistent details in add_records: (' 11134595 123', ((('xml2smillinkverifica-20090314122245-oww4lg1a6y0bhrkn-6', 'reubenf@fridge-20090314122250-f6ne0s1fv1c3jgzs'),), (('xml2smillinkverifica-20090314122245-oww4lg1a6y0bhrkn-6', 'reubenf@fridge-20090314122250-f6ne0s1fv1c3jgzs'),))) (' 15316748 2648', ((), ()))

Traceback (most recent call last):
  File "/usr/local/lib/python2.6/site-packages/bzrlib/commands.py", line 727, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/local/lib/python2.6/site-packages/bzrlib/commands.py", line 922, in run_bzr
    ret = run(*run_argv)
  File "/usr/local/lib/python2.6/site-packages/bzrlib/commands.py", line 559, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/local/lib/python2.6/site-packages/bzrlib/plugins/svn/commands.py", line 170, in run
    to_revnum=to_revnum, prefix=prefix)
  File "/usr/local/lib/python2.6/site-packages/bzrlib/plugins/svn/convert.py", line 288, in convert_repository
    inter.fetch(needed=revfinder.get_missing())
  File "/usr/local/lib/python2.6/site-packages/bzrlib/plugins/svn/fetch.py", line 1260, in fetch
    self._fetch_revisions(needed, pb, use_replay=use_replay)
  File "/usr/local/lib/python2.6/site-packages/bzrlib/plugins/svn/fetch.py", line 1184, in _fetch_revisions
    parent_revmeta)
  File "/usr/local/lib/python2.6/site-packages/bzrlib/plugins/svn/fetch.py", line 1128, in _fetch_revision_switch
    report_inventory_contents(reporter, parent_revnum, start_empty)
  File "/usr/local/lib/python2.6/site-packages/bzrlib/plugins/svn/errors.py", line 130, in convert
    return unbound(*args, **kwargs)
  File "/usr/local/lib/python2.6/site-packages/bzrlib/plugins/svn/fetch.py", line 912, in report_inventory_contents
    reporter.finish()
  File "/usr/local/lib/python2.6/site-packages/bzrlib/plugins/svn/fetch.py", line 393, in close
    return self._close()
  File "/usr/local/lib/python2.6/site-packages/bzrlib/plugins/svn/fetch.py", line 578, i...

Does this happen even if you try to import into a different shared repository (preferably an empty one) than you did with < 0.5.4 ?

Yes, it happens consistently regardless of the destination.

This is related to bug 336749, which we'll hopefully fix next week at UDS.

I *think* this should be fixable by removing "node_refs[:1] != keys[key][1][:1]" from the check in the line where this failure is occuring (this check is too strict).

With 0.6.1 I get a different error. By the way, is there a way to do an svn import without including the branches? I thought omitting --trees would work, but it still seems to pull branches.

bzr: ERROR: bzrlib.errors.KnitCorrupt: Knit _KnitGraphIndex(CombinedGraphIndex(GraphIndex('file:///opt/bzr/Bookshare/branches/product-development/.bzr/repository/indices/aa569ec94624924c6ebe0809f4452d36.tix'))) corrupt: inconsistent details in add_records: (' 789427252 2648', ((), ())) (' 790515602 123', ((('xml2smillinkverifica-20090314122245-oww4lg1a6y0bhrkn-6', 'reubenf@fridge-20090314122250-f6ne0s1fv1c3jgzs'),), (('xml2smillinkverifica-20090314122245-oww4lg1a6y0bhrkn-6', 'reubenf@fridge-20090314122250-f6ne0s1fv1c3jgzs'),)))

Traceback (most recent call last):
  File "/usr/local/lib/python2.6/site-packages/bzrlib/commands.py", line 729, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/local/lib/python2.6/site-packages/bzrlib/commands.py", line 924, in run_bzr
    ret = run(*run_argv)
  File "/usr/local/lib/python2.6/site-packages/bzrlib/commands.py", line 560, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/local/lib/python2.6/site-packages/bzrlib/plugins/svn/commands.py", line 177, in run
    to_revnum=to_revnum, prefix=prefix)
  File "/usr/local/lib/python2.6/site-packages/bzrlib/plugins/svn/convert.py", line 334, in convert_repository
    RepositoryConverter(*args, **kwargs)
  File "/usr/local/lib/python2.6/site-packages/bzrlib/plugins/svn/convert.py", line 285, in __init__
    target_branch.pull(source_branch, overwrite=True)
  File "/usr/local/lib/python2.6/site-packages/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)

What would the use case be for not having any trees or branches at all? That would basically give a useless repository as far as I can tell.

Maybe I misunderstand the option, but for the purposes of the initial import from svn I don't care about svn's branches, only the trunk (and maybe the tags). I'm looking at bzr primarily for the ability to do fairly painless branching and merging.

If you're only interested in one of the particular branches you can use the "bzr branch" command on trunk rather than svn-import, the purpose of the latter is explicitly to import multiple branches.

Fair enough. This appears to be calling the same code under the hood; i.e. I get the same / similar stacktrace that I did originally:

bzr: ERROR: exceptions.AssertionError:

Traceback (most recent call last):
  File "/usr/local/lib/python2.6/site-packages/bzrlib/commands.py", line 729, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/local/lib/python2.6/site-packages/bzrlib/commands.py", line 924, in run_bzr
    ret = run(*run_argv)
  File "/usr/local/lib/python2.6/site-packages/bzrlib/commands.py", line 560, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/local/lib/python2.6/site-packages/bzrlib/builtins.py", line 1143, in run
    source_branch=br_from)
  File "/usr/local/lib/python2.6/site-packages/bzrlib/plugins/svn/remote.py", line 120, in sprout
    return super(SvnRemoteAccess, self).sprout(*args, **kwargs)
  File "/usr/local/lib/python2.6/site-packages/bzrlib/bzrdir.py", line 1178, in sprout
    result_repo.fetch(source_repository, fetch_spec=fetch_spec)
  File "/usr/local/lib/python2.6/site-packages/bzrlib/repository.py", line 1545, in fetch
    find_ghosts=find_ghosts, fetch_spec=fetch_spec)
  File "/usr/local/lib/python2.6/site-packages/bzrlib/plugins/svn/fetch.py", line 1253, in fetch
    use_replay=self._use_replay)
  File "/usr/local/lib/python2.6/site-packages/bzrlib/plugins/svn/fetch.py", line 1184, in _fetch_revisions
    parent_revmeta)
  File "/usr/local/lib/python2.6/site-packages/bzrlib/plugins/svn/fetch.py", line 1129, in _fetch_revision_switch
    report_inventory_contents(reporter, parent_revnum, start_empty)
  File "/usr/local/lib/python2.6/site-packages/bzrlib/plugins/svn/errors.py", line 129, in convert
    return unbound(*args, **kwargs)
  File "/usr/local/lib/python2.6/site-packages/bzrlib/plugins/svn/fetch.py", line 900, in report_inventory_contents
    reporter.finish()

> I *think* this should be fixable by removing "node_refs[:1] != keys[key][1][:1]" from the check in the line where this failure is occuring (this check is too strict).

If that actually fixes the problem, then we probably want to look more closely at what is going on.

Namely, that should indicate that you are adding the same text *2 times* to the repository, only this time you are adding different content. (A different size delta, etc.)

Have you checked what is actually different here? I can think of ways that check might be too strict, but I can also think of ways that it would uncover bugs in the code calling add_lines().

We're adding the same text (with the same contents, etc) but with more parents than previously since we didn't know all of the parents previously. Should this not be possible?

It is a dangerous thing to be doing, because we don't have a way to propagate that sort of change. I don't have a great answer, but it is something that I think needs real discussion about what should happen...

Certainly bzr-svn is the only tool that can generate (file_id, revision_id) pairs when we don't know anything about the revision_id... (conversions from arch don't do it, general ghosts can't)
The small issues we've had in bzr are about a (file_id, revision_id) when Inventory[revision_id] doesn't claim the same text to be modified. We haven't encountered texts from ghosts before so that we know what to do in this case.

This issue can no longer occur with newer Bazaar formats as introduced in 2a. This is a fundamental problem with the old (pre-2.0 knits) format that is not easy to fix.

This issue can no longer occur with newer Bazaar formats as introduced in 2.0. This is a fundamental problem with the old (pre-2.0 knits) format that is not easy to fix.