apt-key and add-apt-repository don't honor Acquire::http::Proxy

See also: https://bugs.launchpad.net/maas/+bug/1257082

Thank you for taking the time to report this bug and helping to make Ubuntu better.

I understand why you expected apt-key and add-apt-repository to use the proxy you defined with Acquire::http::proxy in /etc/apt. But I'm not sure this is the only interpretation.

I expect "Acquire::http::proxy" to define the proxy for the apt repository itself. But apt-key and add-apt-repository don't actually access apt repositories at all - they access other metadata sources to set apt up instead. When configuring an "apt proxy", I might have even denied access to everything except to apt repositories themselves, and so apt-key and add-apt-repository wouldn't work in this case anyway.

Instead, I'd expect http_proxy and https_proxy environment variables to be used instead for these tools. But I believe (this needs to be checked) that add-apt-repository will need https to access Launchpad, and both tools need keyserver access which can't easily be proxied (they access a port most proxies wouldn't allow).

> The long time this issue has been standing and has affected people

Any difficulty with using add-apt-repository and/or apt-key via a proxy - reasonable. But that Acquire::http::proxy didn't configure apt-key and add-apt-repository - I'm not convinced, for reasons above. This is the first bug that I'm aware of that has mentioned this.

I understand the need to for proxy support for these tools for environments where direct access cannot be permitted. Maybe in this case only a socks proxy would do.

In any case, I think further discussion is needed, and piecemeal fixes will exacerbate the problem by adding confusion. I think this needs to be a blueprint-level item to fix behind-firewall-proxy-access for standard server/Openstack deployments that covers all use cases (cloud images, apt, keyserver, utilities like add-apt-repository) in a standard way.

> (cloud images, apt, keyserver, utilities like add-apt-repository)

Looking at your other bug, ntp should be covered too - and anything else relevant.

Status changed to 'Confirmed' because the bug affects multiple users.

This is a very annoying bug that hits me very often. It's also the source of yet another place to configure the proxy server.

My usual work around is trying to patch all executions of apt-key and friends to use --keyserver-options http-proxy=$http_proxy.

It might be possible to patch /root/.gnupg/gpg.conf to include that setting, but I haven't tried that yet.

@Muelli,

using use --keyserver-options http-proxy=$http_proxy is nice idea. But I still has the issue.

gpg: keyserver timed out
gpg: keyserver receive failed: keyserver error

Regarding the gpg proxy setting, I found this document

https://www.gnupg.org/documentation/manuals/gnupg/GPG-Configuration-Options.html

http-proxy=value
This options is deprecated. Set the proxy to use for HTTP and HKP keyservers. This overrides any proxy defined in dirmngr.conf.

Will take a try.

Bill, just for some more information.
Also note "This options is deprecated".

The correct way to do it now is to configure dirmngr.conf (which is backgrounded when you invoke gpg).

This bug was fixed in the package software-properties - 0.96.24.25

---------------
software-properties (0.96.24.25) bionic; urgency=medium

  * ppa.py:
   - rework key retrieval, instead of using hkp & gnupg/dirmngr, use https
     & python's built in urllib.
   - thus, add-apt-key for PPAs observes https_proxy for key retrieval
   - simplify gnupg operations, depend on gpg package only, and use
     import/public key operations only.
   - fix unicode process output bugs, when operating in a non-UTF-8
     locale, thus enabling to import keys for my ppas in C locale.
   - avoid creating trustdb, or requiring any gpg-agent systemd socket to
     be activated
   - update tests to execute key addition fully with less things stubbed
     out with mock
   - stop using apt-key for installing keys
   - dirmngr is a heavy dependency and not used, and it is hard to pass
     proxy information to it when invoking gpg from a non-standard homedir
   - deprecate --keyserver option, making HTTPS access to
     keyserver.ubuntu.com required
   - LP: #1755192, LP: #1713962, LP: #1699086, LP: #1510220, LP: #1433761,
     LP: #1395321, LP: #1312267

 -- Dimitri John Ledkov <email address hidden> Mon, 02 Apr 2018 10:19:34 +0100

I am not sure this was fixed, or maybe I just hardly misunderstand that.

Under current Bionic (software-properties-common 0.96.24.32.3) I still can't retrieve keys via env-var defined `http_proxy`.

# echo $http_proxy
http://1.2.3.4:3128
# echo $https_proxy
http://1.2.3.4:3128

# apt-key adv --recv-keys AB1C1EF6EDB5746803FE13E00F6DD8135234BF2B
Executing: /tmp/apt-key-gpghome.tj9ezR01dk/gpg.1.sh --recv-keys AB1C1EF6EDB5746803FE13E00F6DD8135234BF2B
gpg: keyserver receive failed: Connection timed out

What it does (when stracing it) is

```
...
stat("/tmp/apt-key-gpghome.xI4xFLcUnH/S.dirmngr", {st_mode=S_IFSOCK|0700, st_size=0, ...}) = 0
connect(3, {sa_family=AF_UNIX, sun_path="/tmp/apt-key-gpghome.xI4xFLcUnH/S.dirmngr"}, 43) = 0
read(3, "# ", 1002) = 2
read(3, "Home: /tmp/apt-key-gpghome.xI4xF"..., 1000) = 37
read(3, "\n", 963) = 1
read(3, "# ", 1002) = 2
read(3, "Config: [none]", 1000) = 14
read(3, "\n", 986) = 1
read(3, "OK ", 1002) = 3
read(3, "Dirmngr 2.2.4 at your service", 999) = 29
read(3, "\n", 970) = 1
write(3, "GETINFO version", 15) = 15
write(3, "\n", 1) = 1
read(3, "D 2.2.4\n", 1002) = 8
read(3, "OK", 1002) = 2
read(3, "\n", 1000) = 1
write(3, "KEYSERVER --clear http://keyserv"..., 49) = 49
write(3, "\n", 1) = 1
read(3, "OK", 1002) = 2
read(3, "\n", 1000) = 1
write(3, "KS_GET -- 0xAB1C1EF6EDB5746803FE"..., 52) = 52
write(3, "\n", 1) = 1
read(3, "ERR 167805060 Connection timed o"..., 1002) = 44
```

dirmngr manpage makes me think that `http_proxy` env var is ignored by default and I need to explicitly add `honor-http-proxy` option to dirmngr, but the problem is that apt-key creates the homedir with a random name so I can't use any system-wide setting.

I know I can work this around with `--keyserver-options http-proxy=http://1.2.3.4:3128/` but this is not what I should do, the system should take care itself.

This is really a pain-in-the-ass issue for in-corporate-network automation developers; we just need to use standard tools for adding apt repositories, not working this around.

@miskokot

Probably best to open this up in a new bug report if you think there is an error. However, I can at least confirm that having http_proxy and https_proxy set does yield connections to my proxy with apt-add-repository

root@bionic:~# http_proxy=http://10.5.0.5:8000 https_proxy=http://10.5.0.5:8000 apt-add-repository ppa:canonical-kernel-team/ppa

Notice, that your example has you doing an 'echo $http...' to prove that things are set, however that doesn't show if the variable is exported (available to suprocesses).

I double-checked add-apt-repository and apt-key behavior on bionic (the same doesn't work on xenial as the fix was landed for bionic only).

add-apt-proxy:

➜ ~ sudo strace -f -e connect bash -c 'HTTP_PROXY=http://192.168.1.1:3128 HTTPS_PROXY=http://192.168.1.1:3128 add-apt-repository ppa:landscape/18.03'
strace: Process 20044 attached
# ...

connect(3, {sa_family=AF_INET, sin_port=htons(3128), sin_addr=inet_addr("192.168.1.1")}, 16) = 0

 More info: https://launchpad.net/~landscape/+archive/ubuntu/18.03
Press [ENTER] to continue or Ctrl-c to cancel adding it.

strace: Process 20070 attached
[pid 20070] connect(5, {sa_family=AF_INET, sin_port=htons(3128), sin_addr=inet_addr("192.168.1.1")}, 16) = 0
[pid 20070] connect(3, {sa_family=AF_INET, sin_port=htons(3128), sin_addr=inet_addr("192.168.1.1")}, 16) = 0
# ...
[pid 20080] connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.53")}, 16) = 0
0% [Connecting to ru.archive.ubuntu.com] [Connecting to security.ubuntu.com][pid 20079] connect(3, {sa_family=AF_INET6, sin6_port=htons(80), inet_pton(AF_INET6, "2001:67c:1360:8001::17", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28) = 0
# ...

# squid
1533649727.927 237 192.168.1.100 TCP_TUNNEL/200 5539 CONNECT launchpad.net:443 - HIER_DIRECT/91.189.89.223 -
1533649731.841 269 192.168.1.100 TCP_TUNNEL/200 5539 CONNECT launchpad.net:443 - HIER_DIRECT/91.189.89.223 -
1533649732.499 625 192.168.1.100 TCP_TUNNEL/200 4549 CONNECT keyserver.ubuntu.com:443 - HIER_DIRECT/91.189.89.49 -
1533649732.499 625 192.168.1.100 TCP_TUNNEL/200 4549 CONNECT keyserver.ubuntu.com:443 - HIER_DIRECT/91.189.89.49 -

with apt-key HTTP_PROXY and HTTPS_PROXY had no effect, only --keyserver-options affected it:

apt-key adv --keyserver-options 'http-proxy=http://192.168.1.1:3128' --keyserver keyserver.ubuntu.com --recv-keys 35F77D63B5CEC106C577ED856E85A86E4652B4E6

[pid 4385] connect(7, {sa_family=AF_INET, sin_port=htons(3128), sin_addr=inet_addr("192.168.1.1")}, 16) = -1 EINPROGRESS (Operation now in progress)

1533651214.446 148 192.168.1.100 TCP_MISS/200 1150 GET http://keyserver.ubuntu.com:11371/pks/lookup? - HIER_DIRECT/91.189.89.49 application/pgp-keys

This makes sense as the commit below has only changed software-properties not apt-key to use pycurl and https for key retrieval instead of HKP:

https://git.launchpad.net/ubuntu/+source/software-properties/commit/?id=f57935235ca0f52b32da7efe2a24cb26c7fc4573

+ return get_info_from_https(SKS_KEYSERVER % signing_key_fingerprint, accept_json=False)

Plus, the changelog actually makes the relevant note:

" - stop using apt-key for installing keys"

The apt-key part can be worked around on xenial by using curl directly before add-apt-repository:

HTTPS_PROXY=http://<your-https-proxy>:3128 curl 'https://keyserver.ubuntu.com/pks/lookup?op=get&options=mr&exact=on&search=0x6E85A86E4652B4E6' | sudo apt-key add -

It also seems that when running apt-key, dirmngr starts with --daemon and --homedir /tmp/apt-key-gpghome.[RANDOM STRING]. This results that the dirmngr.conf in ~/.gnupg is also not honored.

We also tried having the configuration-file in /etc/dirmngr/dirmngr.conf which is also not honored.

We are out of other options since we're running MAAS with Juju and trying to deploy the filebeat charm which has the following command for fetching the key.

"sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys D88E42B4"

The dirmngr.conf is read if you simply run dirmngr, but not when apt-key is starting it.

The difference for Xenial / Bionic

In Xenial apt-key spawns a process named gpg that honors the env. for http_proxy.

In Bionic the command “apt-key” spawns a process “dirmngr –daemon –homedir /tmp/apt-key-gpghome.[RANDOM STRING]” which do not honor the env. for http_proxy / https_proxy.

Because of the above (homedir set to /tmp/….) the configuration that could be added to the ~/.gnupg/dirmngr.conf is not read by dirmngr. When running just dirmngr in the terminal the config from ~/.gnupg/dirmngr.conf is read. We’ve also tried setting “GNUPGHOME=/home/ubuntu/.gnupg” in /etc/environment.

Added apt based on #15.

dpkg -S `which apt-key`
apt: /usr/bin/apt-key

apt-key adv is as the name says an advanced feature for direct access to gpg. Hence support for proxies lies with gpg. The adv command is deprecated, as is apt-key in general.

Sorry to ping everyone here, but I'm facing the same on bionic while it was okay in xenial.

The env variables HTTP_PROXY and HTTPS_PROXY are correctly exported, but it seems that bionic do not use them anymore.

What are the hints for possible fix? I see that the two affected packages are INVALID and Fix release, maybe it's another package responsability?

Thanks.

The variables are usually http_proxy and https_proxy, not uppercase versions. That said, gpg does not seem to honor either. It does, however, honor an option --keyserver-options http-proxy=, although it is deprecated.

The fix is to put .asc or .gpg files in trusted.gpg.d and not use apt-key.

{juliank, magnetik},

I checked add-apt-repository operation about a week ago while testing some (unrelated) automation changes on bionic with proxy environment variables and it was fine.

apt-key should not be used at all due to its deprecated functionality.

For your reference:

The manpage for apt-key mentions the following in a section about the
"add" command:
"Note: Instead of using this command a keyring should be placed directly in the /etc/apt/trusted.gpg.d/ directory with a descriptive name and either "gpg" or "asc" as file extension."

The support for /etc/apt/trusted.gpg.d/ goes back to 2010:
https://salsa.debian.org/nathanruiz-guest/apt/commit/c24f6ce22cd6720004addad2e3382b3caa6b1b7c

Debian is dropping apt-key usage as well based on what I see:
https://salsa.debian.org/live-team/live-build/merge_requests/11

Using "asc" in this directory is only supported as of apt 1.4 (on versions before that gpg --dearmor can be used to transform a key to the binary form).

https://salsa.debian.org/nathanruiz-guest/apt/commit/f77ea8235cafb258d1cb0b2b90e95aa36e5c4650
https://salsa.debian.org/nathanruiz-guest/apt/commit/2906182db398419a9c59a928b7ae73cf7c7aa307

this is still not working quite fully for add-apt-repository on a new clean install of 18.04.1.

it may be working for http only, but for https you still need to manually provide the https_proxy env var.