neutron-specs repo is not synced with GitHub

[2014-07-07 12:54:13,293] ERROR com.googlesource.gerrit.plugins.replication.ReplicationQueue : Cannot replicate to <email address hidden>:openstack/neutron-specs.git
org.eclipse.jgit.errors.TransportException: <email address hidden>:openstack/neutron-specs.git: error occurred during unpacking on the remote end: index-pack abnormal exit
        at org.eclipse.jgit.transport.BasePackPushConnection.readStatusReport(BasePackPushConnection.java:315)
        at org.eclipse.jgit.transport.BasePackPushConnection.doPush(BasePackPushConnection.java:200)
        at org.eclipse.jgit.transport.BasePackPushConnection.push(BasePackPushConnection.java:152)
        at org.eclipse.jgit.transport.PushProcess.execute(PushProcess.java:165)
        at org.eclipse.jgit.transport.Transport.push(Transport.java:1173)
        at org.eclipse.jgit.transport.Transport.push(Transport.java:1219)
        at com.googlesource.gerrit.plugins.replication.PushOne.pushVia(PushOne.java:377)
        at com.googlesource.gerrit.plugins.replication.PushOne.runImpl(PushOne.java:351)
        at com.googlesource.gerrit.plugins.replication.PushOne.runPushOperation(PushOne.java:267)
        at com.googlesource.gerrit.plugins.replication.PushOne.access$000(PushOne.java:78)
        at com.googlesource.gerrit.plugins.replication.PushOne$1.call(PushOne.java:240)
        at com.googlesource.gerrit.plugins.replication.PushOne$1.call(PushOne.java:237)
        at com.google.gerrit.server.util.RequestScopePropagator$5.call(RequestScopePropagator.java:222)
        at com.google.gerrit.server.util.RequestScopePropagator$4.call(RequestScopePropagator.java:201)
        at com.google.gerrit.server.git.PerThreadRequestScope$Propagator$1.call(PerThreadRequestScope.java:75)
        at com.googlesource.gerrit.plugins.replication.PushOne.run(PushOne.java:237)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:364)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)

Deleting the project from GitHub and rerunning manage-projects does not seem to have helped, as Gerrit continues to throw the same error attempting to replicate to the empty project there.

Aha!

    fungi@review:~/neutron-specs.git$ git fsck
    Checking object directories: 100% (256/256), done.
    error in commit 5909192d566e994b2c2d29a54e5f283d8c2a51b5: invalid author/committer line - bad date
    Checking objects: 100% (4612/4612), done.

That seems to be the case...

    fungi@review:~/neutron-specs.git$ git show --pretty=fuller 5909192 | head
    commit 5909192d566e994b2c2d29a54e5f283d8c2a51b5
    Author: gustavo panizzo <email address hidden>
    AuthorDate: Wed May 7 17:40:11 2014 -0300
    Commit: gustavo panizzo <gfa>
    CommitDate: Thu Jan 1 00:00:00 1970 +0000

        Create multiple flat on a single physical BP

        Change-Id: Ie31ad1614d3be0858f2d78a37ed271642800a99b

It appears this was broken when patchset #2 of https://review.openstack.org/92704 was uploaded. Gerrit is trying to replicate that reference, but github.com is refusing it on the grounds that it is malformed.

Note that Gerrit UI shows proper CommitDate: May 23, 2014 00:15

I'm weighing possible solutions now... unfortunately any replacement of the broken commit will require database updates and a Gerrit shutdown to regenerate its lucene search index. I could just delete it from the on-disk repository, though that would likely cause issues in the UI.

It's worrisome that jgit will permit upload of commits which cgit considers to be malformed. Perhaps this could be prevented in the future with a hook script? Otherwise it's an easy-to-exploit denial of service against replication to GitHub. At least replication to other cgit servers seems to be unaffected... http://git.openstack.org/cgit/openstack/neutron-specs/commit/?id=5909192

Are all Gerrit refs being synchronized? If only branches are we can add a rule to prevent such commits from landing to any branch.

Oh, I see, it's just a second patchset, it hasn't landed yet.

Note that everything's ok on my local clone:

    yorik@ytaraday neutron-specs % git fetch gerrit refs/changes/04/92704/2
    <...>
    yorik@ytaraday neutron-specs % git show --pretty=fuller 5909192 | head
    commit 5909192d566e994b2c2d29a54e5f283d8c2a51b5 (HEAD)
    Author: gustavo panizzo <email address hidden>
    AuthorDate: Wed May 7 17:40:11 2014 -0300
    Commit: gustavo panizzo <gfa>
    CommitDate: Thu May 22 20:15:53 2014 +0000

        Create multiple flat on a single physical BP

        Change-Id: Ie31ad1614d3be0858f2d78a37ed271642800a99b

It looks like some issue with Git backend. You can probably delete this commit from Git repo on the server (e.g. delete ref and run gc) and replace it with the proper one (adding ref back afterwards).

It's actually not okay, it looks like newer Git versions just quietly make educated guesses when they run into this issue... I can reproduce it remotely with clone --mirror and fsck:

    fungi@dhole:~$ git clone --mirror git://git.openstack.org/openstack/neutron-specs.git
    Cloning into bare repository 'neutron-specs.git'...
    remote: Counting objects: 4632, done.
    remote: Compressing objects: 100% (1563/1563), done.
    remote: Total 4632 (delta 3105), reused 4540 (delta 3044)
    Receiving objects: 100% (4632/4632), 970.26 KiB | 816.00 KiB/s, done.
    Resolving deltas: 100% (3105/3105), done.
    Checking connectivity... done.
    fungi@dhole:~$ cd neutron-specs.git/
    fungi@dhole:~/neutron-specs.git$ git fsck
    Checking object directories: 100% (256/256), done.
    error in commit 5909192d566e994b2c2d29a54e5f283d8c2a51b5: invalid author/committer line - bad date
    Checking objects: 100% (4632/4632), done.

Yes, that works on my laptop as well.

I've found that Gerrit's CLI can delete patchsets. "ssh <...> gerrit --delete 92704,2" might fix the issue.
Of course we should notify the author of the patchset that it should be amended and reuploaded.

Unfortunately not...

    error: Patch set is not a draft.
    one or more approvals failed; review output above

Apparently it only works for draft patchsets, so I faked it out by switching it to a draft in the backend DB via gsql...

    update changes set status='d' where change_id='92704';
    update patch_sets set draft='Y' where change_id='92704';

Then deleted patchset #2 using the command you suggested and published the remaining patchset with...

    gerrit review --publish 92704,1

After that I was able to git review an updated patchset to 92704 without the original defect, and Gerrit made it the new patchset #2. The down side is that it lost comments associated with the old patchset #2, but it did at least preserve the code review votes.

I tried asking for suggestions in #gerrit yesterday but that place is a ghost town lately.

Right. I've found out that it has to be marked as draft by hands but forgot to post it here.

So the issue is resolved now, right? GitHub repo seems to have all recent commits.

Yes, this seems to have solved it. As a takeaway, if we can identify any specific filters we want to apply before commits are accepted into Gerrit at all, we can use a custom "ref-update" (not to be confused with "ref-updated") hook: https://review.openstack.org/Documentation/config-hooks.html#_ref_update