possible information leak due to setting docker registry insecure by default

Bug #1940547 reported by Sven Kieske
260
This bug affects 2 people
Affects Status Importance Assigned to Milestone
kolla-ansible
Fix Released
High
Radosław Piliszek
Ussuri
Opinion
High
Unassigned
Victoria
Opinion
High
Unassigned
Wallaby
Opinion
High
Unassigned
Xena
Fix Released
High
Radosław Piliszek

Bug Description

Hi,

during testing of openstack "train" release (but this applies to "master" as well, I checked) we realized the following, which applies afaik to latest docker, kolla and kolla-ansible versions:

if you configure a custom docker-registry via variable "docker_registry" in ansible/group_vars/all.yml (line 112 on current master branch)

then, the following logic is applied in line 115:

docker_registry_insecure: "{{ 'yes' if docker_registry else 'no' }}"

which translates into the service being deployed with the insecure registry flag being enabled:

https://review.opendev.org/c/openstack/kolla-ansible/+/575023/5/ansible/roles/baremetal/templates/docker_systemd_service.j2

so, per default, if you provide your own registry, kolla-ansible deploys all docker daemons with the insecure option "insecure-registries", which is defined as follows:

> With insecure registries enabled, Docker goes through the following steps:

> First, try using HTTPS.
> If HTTPS is available but the certificate is invalid, ignore the error about the certificate.
> If HTTPS is not available, fall back to HTTP.

Source: https://docs.docker.com/registry/insecure/#deploy-a-plain-http-registry

So, if, for some reason, HTTPS fails, docker will connect via plaintext protocol HTTP to the registry.
I did not check since which version docker introduced this behaviour, but I assume it's there for a long time.

Notice also, how docker discourages this option, even if you use self signed certs.

This might leak confidential information, for example, if you configure your registry (let's say gitlab) to redirect to an S3 Bucket, there are plaintext jwt authentication tokens contained in the URL which will be transfered via HTTP plaintext.

so this could leak confidental access to a registry/images, or in the worst case, to complete S3 Buckets, or other storage backends, which might contain other sensitive data.

This was introduced, it seems in: https://review.opendev.org/c/openstack/kolla-ansible/+/575023

Notice the description says "Option for enable SSL verification on docker registry"

but this is not really true.

to default to a secure state, this option should IMHO not be set as a default!

the default should be, to not set "insecure-registries".

You need to deploy a valid SSL Cert to your registry, which gets checked by docker (the cert chain must be in the local truststore, of course).

if you did this, there is no need for additional configuration like the above flag!

the above flag should imho only be used in test setup or on dev workstations (and even there maybe not, depending on your threat model!), when you can't provide a valid TLS Cert Chain.

it's way to easy to end up with an unencrypted, insecure channel, which transmits cleartext data over the internet.

luckily in our test setup we had additional security guards enabled, which prevented an information leak.

if you have any questions or feedback, please don't hesitate to contact me.

kind regards

Sven Kieske

Revision history for this message
Radosław Piliszek (yoctozepto) wrote :

Agreed, I think I have chatted about it before but it did not get much traction and nobody else seemed to care at the time. Now that you do, I feel more empowered to propose the change.

Changed in kolla-ansible:
status: New → Triaged
assignee: nobody → Radosław Piliszek (yoctozepto)
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (master)
Changed in kolla-ansible:
status: Triaged → In Progress
Revision history for this message
Radosław Piliszek (yoctozepto) wrote :

Please review the linked change.

summary: - possible information leak due to kolla-ansible deployment
+ possible information leak due to setting docker registry insecure by
+ default
Revision history for this message
Mark Goddard (mgoddard) wrote :

I agree we should change this default. I'm unsure about changing it in stable branches, however.

Revision history for this message
Sven Kieske (s-kieske) wrote :

Yeah I don't know if it's a good idea to backport this to stable.

Maybe a security notice does suffice? After all, this can be relatively easily be changed
by operators themselves.

Thanks for your fast responses!

Revision history for this message
Sven Kieske (s-kieske) wrote :

I noticed the same behaviour seems to be prevalent in k8s clusters deployed via magnum. As far as I can see, I'm not entirely sure if it's the default though:

https://review.opendev.org/c/openstack/magnum/+/775110

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (master)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/805449
Committed: https://opendev.org/openstack/kolla-ansible/commit/802f7c62186cff91cc4b104984330ab56218ae4e
Submitter: "Zuul (22348)"
Branch: master

commit 802f7c62186cff91cc4b104984330ab56218ae4e
Author: Radosław Piliszek <email address hidden>
Date: Fri Aug 20 18:23:56 2021 +0000

    Never make Docker registry insecure by default

    To follow best security practices and help fellow operators.

    More details inline and in the linked bug report.

    Closes-Bug: #1940547
    Change-Id: Ide9e9009a6e272f20a43319f27d257efdf315f68

Changed in kolla-ansible:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/808000

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/808001

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/808002

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on kolla-ansible (stable/wallaby)

Change abandoned by "Radosław Piliszek <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/808000
Reason: we agreed not to backport

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on kolla-ansible (stable/victoria)

Change abandoned by "Radosław Piliszek <email address hidden>" on branch: stable/victoria
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/808001
Reason: we agreed not to backport

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on kolla-ansible (stable/ussuri)

Change abandoned by "Radosław Piliszek <email address hidden>" on branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/808002
Reason: we agreed not to backport

Revision history for this message
Radosław Piliszek (yoctozepto) wrote :

we agreed not to backport

information type: Private Security → Public Security
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 13.0.0.0rc1

This issue was fixed in the openstack/kolla-ansible 13.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public Security information  
Everyone can see this security related information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.