git clone fails with "fatal: Not a git repository", "git remote update failed."

I accidentally filed a dupe of this in bug https://bugs.launchpad.net/openstack-ci/+bug/1283038 ... copying my description here--

Attempting to `git clone` nova in logs/devstack-gate-setup-workspace-new.txt [1] resulted in:

+ git clone https://git.openstack.org/openstack/nova
Cloning into 'nova'...
error: RPC failed; result=56, HTTP code = 200
fatal: The remote end hung up unexpectedly
fatal: protocol error: bad pack header
+ cd nova
/home/jenkins/workspace/gate-tempest-dsvm-large-ops/devstack-gate/functions.sh: line 98: cd: nova: No such file or directory
+ git_remote_set_url origin https://git.openstack.org/openstack/nova
+ git remote set-url origin https://git.openstack.org/openstack/nova
fatal: Not a git repository (or any parent up to mount parent )
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).

Which ultimately led to:

+ echo 'git remote update failed.'
git remote update failed.

[1]: http://logs.openstack.org/99/73699/11/gate/gate-tempest-dsvm-neutron/5d7df22/logs/devstack-gate-setup-workspace-new.txt

After talking to fungi, it appears that the "git fatal: Couldn't find remote ref" is actually just a red herring (and is unfortunately normal behavior in the logs).

The salient error is

fatal: Not a git repository (or any parent up to mount parent )

which comes from using https:// to clone

this was lately showing up trying to clone nova and neutron repositories.

Another manifestation of this bug: http://logs.openstack.org/88/75188/1/check/check-tempest-dsvm-full/cafeca7/console.html

2014-02-21 13:04:57.782 | Started by user anonymous
2014-02-21 13:04:57.787 | [EnvInject] - Loading node environment variables.
2014-02-21 13:05:01.068 | Building remotely on devstack-precise-rax-iad-1496017 in workspace /home/jenkins/workspace/check-tempest-dsvm-full
2014-02-21 13:05:12.170 | [check-tempest-dsvm-full] $ /bin/bash -xe /tmp/hudson1116277386280346131.sh
2014-02-21 13:05:13.410 | + [[ ! -e devstack-gate ]]
2014-02-21 13:05:13.413 | + git clone git://git.openstack.org/openstack-infra/devstack-gate
2014-02-21 13:05:13.762 | Cloning into 'devstack-gate'...
2014-02-21 13:05:14.637 | [check-tempest-dsvm-full] $ /bin/bash -xe /tmp/hudson8798132986219882664.sh
2014-02-21 13:05:14.682 | + export PYTHONUNBUFFERED=true
2014-02-21 13:05:14.683 | + PYTHONUNBUFFERED=true
2014-02-21 13:05:14.683 | + export DEVSTACK_GATE_TIMEOUT=180
2014-02-21 13:05:14.683 | + DEVSTACK_GATE_TIMEOUT=180
2014-02-21 13:05:14.683 | + export DEVSTACK_GATE_TEMPEST=1
2014-02-21 13:05:14.683 | + DEVSTACK_GATE_TEMPEST=1
2014-02-21 13:05:14.683 | + export DEVSTACK_GATE_TEMPEST_FULL=1
2014-02-21 13:05:14.683 | + DEVSTACK_GATE_TEMPEST_FULL=1
2014-02-21 13:05:14.684 | + export BRANCH_OVERRIDE=default
2014-02-21 13:05:14.684 | + BRANCH_OVERRIDE=default
2014-02-21 13:05:14.684 | + '[' default '!=' default ']'
2014-02-21 13:05:14.684 | + cp devstack-gate/devstack-vm-gate-wrap.sh ./safe-devstack-vm-gate-wrap.sh
2014-02-21 13:05:14.685 | + ./safe-devstack-vm-gate-wrap.sh
2014-02-21 13:05:14.685 | Triggered by: https://review.openstack.org/75188 patchset 1
2014-02-21 13:05:14.685 | Pipeline: check
2014-02-21 13:05:14.686 | IP configuration of this host:
2014-02-21 13:05:14.686 | 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
2014-02-21 13:05:14.686 | link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2014-02-21 13:05:14.687 | inet 127.0.0.1/8 scope host lo
2014-02-21 13:05:14.687 | inet6 ::1/128 scope host
2014-02-21 13:05:14.687 | valid_lft forever preferred_lft forever
2014-02-21 13:05:14.687 | 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
2014-02-21 13:05:14.687 | link/ether bc:76:4e:20:40:a5 brd ff:ff:ff:ff:ff:ff
2014-02-21 13:05:14.687 | inet 162.242.239.57/24 brd 162.242.239.255 scope global eth0
2014-02-21 13:05:14.688 | inet6 2001:4802:7802:102:d43e:b81a:ff20:40a5/64 scope global
2014-02-21 13:05:14.688 | valid_lft forever preferred_lft forever
2014-02-21 13:05:14.688 | inet6 fe80::be76:4eff:fe20:40a5/64 scope link
2014-02-21 13:05:14.688 | valid_lft forever preferred_lft forever
2014-02-21 13:05:14.688 | 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
2014-02-21 13:05:14.688 | link/ether bc:76:4e:20:4e:69 brd ff:ff:ff:ff:ff:ff
2014-02-21 13:05:14.689 | inet 10.208.166.253/19 brd 10.208.191.255 scope global eth1
2014-02-21 13:05:14.689 | inet6 fe80::be76:4eff:fe20:4e69/64 scope link
2014-02-21 13:05:14.689 | valid_lft forever preferred_lft forever
2014-02-21 13:05:14.689 | ...

The underlying error is that some nodepool server images in some provider regions failed to pre-cache some repositories when they were rebuilt overnight (it's possible they attempted to clone them while the git servers were running a repack, which shouldn't cause a problem, but maybe it does). Because servers launched from these images were missing some repositories, DevStack attempted to reclone them when running in jobs leading to a fairly substantial denial of service against the git servers and causing many of those clone attempts to fail, breaking the jobs performing them.

The images in question have all been deleted and replaced at this point along with the nodes built from them. The scripts which perform git repository caching on nodepool images have also been improved so that they'll now retry once on a failure, perform some basic tests within each clone, and abort the image build if issues are encountered.

Related fix proposed to branch: master
Review: https://review.openstack.org/75542

Reviewed: https://review.openstack.org/75542
Committed: https://git.openstack.org/cgit/openstack-infra/elastic-recheck/commit/?id=0f89211d45c9eb1fbcd7ffafa79549753b7d38c0
Submitter: Jenkins
Branch: master

commit 0f89211d45c9eb1fbcd7ffafa79549753b7d38c0
Author: Joe Gordon <email address hidden>
Date: Fri Feb 21 15:53:11 2014 -0800

    Add fingerprint for bug 1282876

    This is an infra bug that can manifest itself in several ways, so this
    fingerprint won't catch all cases. But a fingerprint that only catches a
    subset is better then no fingerprint at all.

    Change-Id: I80d62f500407582b7bd5825943775d4cb86209df
    Related-Bug: #1282876

Related fix proposed to branch: master
Review: https://review.openstack.org/178220

Reviewed: https://review.openstack.org/178220
Committed: https://git.openstack.org/cgit/openstack-infra/elastic-recheck/commit/?id=a4485a992c630d300cd010b436b5fb9dc80db5fd
Submitter: Jenkins
Branch: master

commit a4485a992c630d300cd010b436b5fb9dc80db5fd
Author: Sean Dague <email address hidden>
Date: Tue Apr 28 10:45:01 2015 -0400

    Remove tripleo from signature

    The tripleo jobs fail a lot because they are talking to github,
    exclude that queue from this signature which is about our servers
    failing.

    Long term we should exclude everything from the tripleo queue.

    Change-Id: I45c4f86147a04522b026f2f677cab7eb72a867d5
    Related-Bug: #1282876

Reviewed: https://review.openstack.org/178608
Committed: https://git.openstack.org/cgit/openstack-infra/project-config/commit/?id=8039e854aae833f5e62c3f63d1fa5be508c8fcaf
Submitter: Jenkins
Branch: master

commit 8039e854aae833f5e62c3f63d1fa5be508c8fcaf
Author: Sean Dague <email address hidden>
Date: Wed Apr 29 06:56:55 2015 -0400

    fix git_timed retry logic

    git_timed retry logic actually only retried *if* we timed out, not if
    the command itself failed, as it would explicitly exit if the return
    code was anything except 124. This loops in all failure cases, which
    was the intent of the original fix.

    Related-Bug: #1282876

    Change-Id: I576df4832665beb21992ffa873da28a198018530

Related fix proposed to branch: master
Review: https://review.openstack.org/326739