bionic git-ubuntu checkout has '..git' file in subfolder that causes build failure

cc'ing @racb fyi since this is related to git-ubuntu

I don't follow.

git-ubuntu does escape the .git with a ..git intentionally, since as you say this is unrepresentable in git. It's done this way so it's entirely reversible.

The intention is that "git ubuntu build" (once fully implemented) would correctly unescape the tree before a build. For anyone building by hand, this unescaping is therefore also necessary by design.

So what's the bug here?

> So what's the bug here?

the bug is:

$ git ubuntu clone qemu
$ cd qemu
$ git checkout pkg/ubuntu/bionic-devel
$ sudo apt build-dep .
$ dpkg-buildpackage

that fails, due to the ..git file.

Or are you saying that using git-ubuntu implies people should no longer use dpkg-buildpackage (or debuild, etc)?

On Thu, Jul 16, 2020 at 05:21:16PM -0000, Dan Streetman wrote:
> > So what's the bug here?
>
> the bug is:
>
> $ git ubuntu clone qemu
> $ cd qemu
> $ git checkout pkg/ubuntu/bionic-devel
> $ sudo apt build-dep .
> $ dpkg-buildpackage
>
> that fails, due to the ..git file.

I understand that this fails, and that you expect this to work. However
as far as I'm aware there's no design of git-ubuntu possible where this
will work in all edge cases, so this is an unfortunate interaction and
not a bug in git-ubuntu or the imported qemu package.

> Or are you saying that using git-ubuntu implies people should no longer
> use dpkg-buildpackage (or debuild, etc)?

Where this edge case applies, correct. I don't know how to make this
possible in the general case. git won't let us (and it doesn't make
sense to) have an imported source package contain a .git directory. If
the build requires that, then it's not going to work.

The closest we can get is to this is to escape the .git in a way that
can be undone before calling dpkg-buildpackage, and that's already done
and working as expected. If you want some code to do the unescaping,
that's available in the git-ubuntu source.

Until "git ubuntu build" is fully implemented, you need to rename ..git
to .git before your build step if you want this to work.

Put another way: what behaviour do you want from git-ubuntu instead?

> this is an unfortunate interaction and not a bug in git-ubuntu

that's fine, i was only cc'ing you in case you thought it was something that should be fixed

> or the imported qemu package

well IMHO it is a bug to have the file in the orig tarball

> If the build requires that, then it's not going to work

well again IMHO any package that requires a working git repo is fundamentally broken

> what behaviour do you want from git-ubuntu instead

i'd suggest removing the .git file/dir instead of renaming it. Is there actually a package that contains a .git file/dir and requires it for build?

> well IMHO it is a bug to have the file in the orig tarball

Agreed, but aren't you tracking that in bug 1887535? What are you expecting to see changed that will, in your eyes, resolve _this_ bug?

> well again IMHO any package that requires a working git repo is fundamentally broken

Agreed. Unfortunately git-ubuntu still needs to present something for such a source package though, including (in the case that it's policy compliant) the capability of correctly building it whether it is buggy or not.

> i'd suggest removing the .git file/dir instead of renaming it. Is there actually a package that contains a .git file/dir and requires it for build?

This bug is an example of a source package that changes behaviour depending on whether the .git exists or not. What "git ubuntu build" needs to do (when implemented) is provide a build that matches what the source package "officially" does, even if buggy. For this to work, we can't irreversibly remove directories. That's why we escape .git rather than remove it.

Put another way, consider these two situations:

(a) if we always remove .git, then I can construct a source package that succeeds without git-ubuntu, but fails to build (or results in a build that is functionally different) with it.

(b) if we always escape .git, then I can only construct a source package that succeeds without git-ubuntu, but doesn't work until unescaping has been performed. When unescaping is performed, a build whose source came via git-ubuntu can always be identical to a build that is done without it.

I consider that (a) would make git-ubuntu fundamentally broken, so (b) is the least worst choice.

Further, I'd like one day for "git ubuntu build" to be all that is necessary - with no need for newcomers to use dpkg-buildpackage or "mk-sbuild"/sbuild directly, unless they want to dig deeper. In that world, the unescaping won't be an issue because "git ubuntu build" will just know to do it.

Correction:

> that succeeds without git-ubuntu, but doesn't work until unescaping has been

but doesn't work [with git ubuntu] until unescaping has been

> > well IMHO it is a bug to have the file in the orig tarball
>
> Agreed, but aren't you tracking that in bug 1887535? What are you expecting
> to see changed that will, in your eyes, resolve _this_ bug?

that bug is different, the configure and Makefile actually behave differently if there is a top-level '.git' directory

> This bug is an example of a source package that changes behaviour depending on
> whether the .git exists or not

this bug is about a file 'ui/keycodemapdb/.git', not a top-level .git file/dir, and the existence of that lower-level .git file does not affect the actual build (it only breaks dpkg-source's attempt to apply all patches and verify there are no local changes).

the other bug is about the existence (or not) of a top-level .git directory, which is not related to git-ubuntu's behavior of creating a '..git' file/dir. (and also, the build acting differently if a top-level .git exists or not is itself a bug)

> I consider that (a) would make git-ubuntu fundamentally broken, so (b) is the least worst choice.

yep, it's a difficult choice; possibly git-ubuntu could cause a warning to be output after a 'git checkout' and maybe 'git reset' if it detects there is a '..git' file/dir that is a result of such git-ubuntu escaping. It was confusing to me that the build failure happened, and I assume would be confusing to others using git-ubuntu but building with traditional tooling.

On Fri, Jul 17, 2020 at 03:43:51PM -0000, Dan Streetman wrote:
> this bug is about a file 'ui/keycodemapdb/.git', not a top-level .git
> file/dir, and the existence of that lower-level .git file does not
> affect the actual build (it only breaks dpkg-source's attempt to apply
> all patches and verify there are no local changes).
>
> the other bug is about the existence (or not) of a top-level .git
> directory, which is not related to git-ubuntu's behavior of creating a
> '..git' file/dir. (and also, the build acting differently if a top-level
> .git exists or not is itself a bug)

Ah - I see now, thanks. So this bug will be resolved if/when the qemu
source package no longer ships ui/keycodemapdb/.git?

> yep, it's a difficult choice; possibly git-ubuntu could cause a warning
> to be output after a 'git checkout' and maybe 'git reset' if it detects
> there is a '..git' file/dir that is a result of such git-ubuntu
> escaping. It was confusing to me that the build failure happened, and I
> assume would be confusing to others using git-ubuntu but building with
> traditional tooling.

This is a good idea, thanks. We already do something similar for empty
directory detection using git hooks that are set up by "git ubuntu
clone". I'd be happy to accept a bug and/or patch that does the same for
detection of /^\.\.+git/ (or whatever the exact unescaping rule is).

ui/keycodemapdb/.git was removed with https://launchpad.net/ubuntu/+source/qemu/1:2.12+dfsg-3ubuntu1

It seems like it only exists in qemu_2.11+dfsg.orig.tar.xz