bzr fails to do proxy authentication

There's some description of NTLM proxy auth here: http://squid.sourceforge.net/ntlm/client_proxy_protocol.html

I may be missing your point but if the proxy requires NTLM it's not going to be enough for bzr to just have the username/password in the HTTP_PROXY setting. bzr (or the library it uses) needs to know how to do the fairly complicated dance to open the connection.

However, if urllib2 can do this in other cases, but bzr's use of urllib2 doesn't work we may be able to fix it.

Can you give an example of some urllib2 code that does work in this case?

As a workaround you may be able to use ssh to Launchpad.

Hi, thanks for taking a look at this bug.

As I understand it, this particular proxy accepts both NTLM and Basic authentication. I can authenticate successfully using the HTTP_PROXY variable and I cited wget and urllib2 as examples. Something as simple as "urllib2.urlopen('http://www.google.com').read()" works here, but I can't get bzr to work. It appears that it is simply dropping the authentication settings and trying to connect to the proxy without a username/password.

Ah, I see you're right that it does apparently accept either of them, and so the NTLM thing may be a distraction. We may be able to just ignore it - though it may also be that the NTLM tcp connection dance means the behaviour is different even for Basic auth.

Do you have a traceback in .bzr.log showing where the error is raised from?

On Tue, 2009-08-11 at 00:21 +0000, Martin Pool wrote:
> Ah, I see you're right that it does apparently accept either of them,
> and so the NTLM thing may be a distraction. We may be able to just
> ignore it - though it may also be that the NTLM tcp connection dance
> means the behaviour is different even for Basic auth.
>
> Do you have a traceback in .bzr.log showing where the error is raised
> from?

The basic handshake should be unaltered:
request
denied, [auth schemes]
request w/basic
response

If NTLM is 'just working' with urlib (not urlib2) then we may be doing
something wrong. AFAIK though, we wouldn't interfere with NTLM if the
core supported it.

We might want to consider adopting urlib2 if it does have NTLM built in,
as that would be nice for windows users.

-Rob

1.13 is pretty old, I think you're encountering bug #366107 (fixed in bzr-1.15), where bzr wasn't falling back to using Basic when presented with *several* possible authentication schemes.

Hey, you can't call 1.13 "pretty old" when it's the latest available version in Jaunty! :)
I'm joking but I'm serious... No one asked for a backport?
Anyway, I have downloaded the latest version (1.17) from the PPA and proxy authentication is working fine now. Thanks!

Oh, and one more thing: if I just say "bzr branch lp:project" instead of giving the full URL, it doesn't use the proxy settings. I remember reading something about it being a limitation when I first opened this bug, but it's been such a long time that I don't remember what it was anymore. Could anybody clear that up for me?

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Bruno Gomes wrote:
> Oh, and one more thing: if I just say "bzr branch lp:project" instead of
> giving the full URL, it doesn't use the proxy settings. I remember
> reading something about it being a limitation when I first opened this
> bug, but it's been such a long time that I don't remember what it was
> anymore. Could anybody clear that up for me?
>

lp:project uses XMLRPC to resolve lp:project => http://....

However, it seems that python's xmlrpc library doesn't have native
support for proxies. So it will require a bunch of development on our
part to sneak in proxy support.

It is an open bug, though I don't have the number offhand.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkqBr9kACgkQJdeBCYSNAAO5ogCgp1ngpbDAq1iGL8sEwlGlOmvb
GUYAoLXaZQ7Q+nwu4xOBfXp0fHDqE0CE
=xttH
-----END PGP SIGNATURE-----

Yes, that was it... Thanks for the info!

Hi guys. I think I'm having this issue too. I can't authenticate to the proxy and I get:

C:\Program Files\Bazaar>bzr branch http://bazaar-vcs.org/bzr/bzr.dev
bzr: ERROR: Invalid http response for http://bazaar-vcs.org/bzr/bzr.dev/.bzr/branch-format: Unable to handle http code 407: Proxy Authentication Required ( The ISA Server requires authorization to fulfill the request. Access to the Web Proxy filter is denied. )

I have set the http_proxy environment variable and supplied my credentials but it still isn't working.

I'm using version 2.0.3 on Windows.

@Simone: We need some details about your proxy settings and the relevant
part of your .bzr.log after issuing a failing command where you
will specify -Dhttp and -Derror.

Make sure to replace your password with xxx if it appears. Be
aware that Authorization HTTP headers may also contain your
password slightly encrypted, you can replace the encrypted string
by xxx there.

For the record, this bug has been open for three different failures:
- fallback to Basic if possible (bug #366107 ),
- proxies for xmlrpc (bug #186920),
- support NTLM auth scheme

Only the later has still to be fixed.

Thanks Vincent, I will try and report back.

Here is the output of the supplied command:

C:\Program Files\Bazaar>bzr branch -Dhttp -Derror https://code.launchpad.net/~g
tg/gtg/trunk
bzr: ERROR: bzrlib.errors.ConnectionError: Connection error: Couldn't resolve ho
st 'code.launchpad.net' (11001, 'getaddrinfo failed')

Traceback (most recent call last):
  File "bzrlib\commands.pyo", line 842, in exception_to_return_code
  File "bzrlib\commands.pyo", line 1037, in run_bzr
  File "bzrlib\commands.pyo", line 654, in run_argv_aliases
  File "bzrlib\builtins.pyo", line 1202, in run
  File "bzrlib\bzrdir.pyo", line 959, in open_tree_or_branch
  File "bzrlib\bzrdir.pyo", line 844, in open
  File "bzrlib\bzrdir.pyo", line 874, in open_from_transport
  File "bzrlib\lazy_import.pyo", line 125, in __call__
  File "bzrlib\transport\__init__.pyo", line 1642, in do_catching_redirections
  File "bzrlib\bzrdir.pyo", line 861, in find_format
  File "bzrlib\bzrdir.pyo", line 1797, in find_format
  File "C:/Program Files/Bazaar/plugins\svn\format.py", line 101, in probe_trans
port
  File "C:/Program Files/Bazaar/plugins\svn\transport.py", line 102, in get_svn_
ra_transport
  File "bzrlib\transport\http\_urllib.pyo", line 79, in _perform
  File "urllib2.pyo", line 381, in open
  File "urllib2.pyo", line 399, in _open
  File "urllib2.pyo", line 360, in _call_chain
  File "bzrlib\transport\http\_urllib2_wrappers.pyo", line 706, in https_open
  File "bzrlib\transport\http\_urllib2_wrappers.pyo", line 612, in do_open
  File "bzrlib\transport\http\_urllib2_wrappers.pyo", line 515, in retry_or_rais
e
ConnectionError: Connection error: Couldn't resolve host 'code.launchpad.net' (1
1001, 'getaddrinfo failed')

@Simone: We need the associated .bzr.log ('bzr version' will tell you where it is)
since -Dhttp put the debug data there !

How you did setup (or not) the proxy is also relevant

The proxy is a corporate one, I have no control over it, but I guess it uses NTLM. Here's the log:

this is a debug log for diagnosing/reporting problems in bzr
you can delete or truncate this file, or include sections in
bug reports to https://bugs.launchpad.net/bzr/+filebug

ven 2010-01-22 16:30:11 +0100
0.031 bzr arguments: [u'branch', u'-Dhttp', u'-Derror', u'https://code.launchpad.net/~gtg/gtg/trunk']
0.046 looking for plugins in[...]/Application Data/bazaar/2.0/plugins
0.046 looking for plugins in C:/Program Files/Bazaar/plugins
0.140 encoding stdout as sys.stdout encoding 'cp437'
0.156 failed to import pycurl: No module named pycurl
0.156 failed to instantiate transport <bzrlib.registry._LazyObjectGetter object at 150b698, module='bzrlib.transport.http._pycurl' attribute='PyCurlTransport'> for 'https://code.launchpad.net/~gtg/gtg/trunk': DependencyNotPresent(Unable to import library "pycurl": No module named pycurl)
0.171 bzr-svn: using Subversion 1.5.6 ()
1.796 * About to connect() to code.launchpad.net:443
1.796 Traceback (most recent call last):
  File "bzrlib\commands.pyo", line 842, in exception_to_return_code
  File "bzrlib\commands.pyo", line 1037, in run_bzr
  File "bzrlib\commands.pyo", line 654, in run_argv_aliases
  File "bzrlib\builtins.pyo", line 1202, in run
  File "bzrlib\bzrdir.pyo", line 959, in open_tree_or_branch
  File "bzrlib\bzrdir.pyo", line 844, in open
  File "bzrlib\bzrdir.pyo", line 874, in open_from_transport
  File "bzrlib\lazy_import.pyo", line 125, in __call__
  File "bzrlib\transport\__init__.pyo", line 1642, in do_catching_redirections
  File "bzrlib\bzrdir.pyo", line 861, in find_format
  File "bzrlib\bzrdir.pyo", line 1797, in find_format
  File "C:/Program Files/Bazaar/plugins\svn\format.py", line 101, in probe_transport
  File "C:/Program Files/Bazaar/plugins\svn\transport.py", line 102, in get_svn_ra_transport
  File "bzrlib\transport\http\_urllib.pyo", line 79, in _perform
  File "urllib2.pyo", line 381, in open
  File "urllib2.pyo", line 399, in _open
  File "urllib2.pyo", line 360, in _call_chain
  File "bzrlib\transport\http\_urllib2_wrappers.pyo", line 706, in https_open
  File "bzrlib\transport\http\_urllib2_wrappers.pyo", line 612, in do_open
  File "bzrlib\transport\http\_urllib2_wrappers.pyo", line 515, in retry_or_raise
ConnectionError: Connection error: Couldn't resolve host 'code.launchpad.net' (11001, 'getaddrinfo failed')

1.796 return code 3

I'm launching bzr with environment variable

set http_proxy=http://username@domain:password@proxy_host:port

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Simone Busoli wrote:
> I'm launching bzr with environment variable
>
> set http_proxy=http://username@domain:password@proxy_host:port
>

If it is NTLM, I did find this:
http://sourceforge.net/projects/ntlmaps/

Which claims to be a bridge so that you can use NTLM authentication for
applications that aren't Microsoft based.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAktZ14oACgkQJdeBCYSNAANcsQCdGy0EjuOPKBcnH31c7i7bYJD2
sHcAn0GKUkNUHh2HSH2V4dR5j3gjNWod
=AiV6
-----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Simone Busoli wrote:
> I'm launching bzr with environment variable
>
> set http_proxy=http://username@domain:password@proxy_host:port
>

There also seems to be:
http://code.google.com/p/python-ntlm/

Which is a library for handling proxy authentication using NTLM. It
looks like it would be as simple as hooking that library into our
existing urllib2 authentication schemes.

Looking at that page, you may need
"http://domain\user:password@proxy_host:port".

I certainly don't know the specifics of what NTLM would expect. And it
looks like we would probably need to add the HTTPNtlmAuthHandler anyway.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAktZ2GAACgkQJdeBCYSNAAPmhwCgysu9uR6zvYaBh7bpwI7OxJVk
x6QAnAk3u5YkSb4uGl2ba/mo+sxKYH1U
=dT4G
-----END PGP SIGNATURE-----

> The proxy is a corporate one, I have no control over it, but I guess it uses NTLM.

Such proxies often accepts Basic authentication for clients that don't handle NTLM, so there is still hope.

Hmm, weird, the line:

    1.796 * About to connect() to code.launchpad.net:443

means that we don't take the proxy into account *at all*.

On the other hand:

    Connection error: Couldn't resolve host 'code.launchpad.net' (11001, 'getaddrinfo failed')

means we can't even resolve the address itself, can you do a 'ping code.launchpad.net' from that machine ?

The ping in fact fails. The network of the company I'm trying to connect from has very strict security policies, it looks like I cannot ping at all. It fails on name resolution.

CNTLM is also a nice lightweight proxy that allows apps which don't
understand NTLM to use NTLM proxies. I use it myself at work behind an
NTLM proxy with Bazaar successfully.

On 22/01/2010 1:00 PM, Simone Busoli wrote:
> The ping in fact fails. The network of the company I'm trying to connect
> from has very strict security policies, it looks like I cannot ping at
> all. It fails on name resolution.

So I'm guessing that only the proxy is able to resolve names.

Have you set "https_proxy" *note the 's'* env var?

I didn't try with https_proxy, I didn't know it even existed. I will try out and let you know, thanks a lot for your support so far and for the workarounds, I will try them.

This issue is still present in Bazaar. I'm using Windows, which is all set up to get through the ISA server proxy, so I don't know why Bazaar is failing. I get:

Unable to handle http code 407: Proxy Authentication Required ( The ISA Server requires authorization to fulfill the request. Access to the Web Proxy service is denied. )

Pretty much all internet software on Windows (Firefox etc.) gets through the ISA server fine with no additional configuration, so Bazaar should be able to as well. Currently this is making source code management with Launchpad/Bazaar unusable.

@max: Please file a new bug, this one already mention too many obsolete beliefs about bzr behavior
in old versions.

To summarize: most of the NTLM proxies allows Basic as a fallback scheme and bzr knows
how to handle Basic and does handle it.

So presumably you don't provide a valid user/password (Firefox etc. handle their own cache for such things
and there is no way to access them), you can either specify them in the env variable or use the
authentication.conf file to specify user or user and password.

I'm going to mark this bug as Fix Released, if you still have problems, please file a new one and attach
the relevant part of .bzr.log (bzr version will tell you where to find it) and be sure to anonymize it properly
by replacing the values in the Auth* headers (or use a scratch user/pass if you can). But don't delete the
headers, we need them to properly diagnose what is happening.

https://bugs.launchpad.net/bzr/+bug/802908 points out this is apparently not yet documented.