kolla-ansible

SSL termination in external loadbalancer

Bug #1606143 reported by Nenad Radojevic
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
kolla-ansible
Fix Released
Wishlist
Unassigned

Bug Description

We encountered a bug while trying to access public endpoints over HTTPS.
Following setup:

external loadbalancer | internal haproxy | keystone

As you can see we put an external loadbalancer in front of the internal one. In our scenario the external loadbalancer terminates SSL and then redirects to the internal haproxy without SSL. The problem is that keystone always returns HTTP URLs instead of HTTPS.

The reason for this is that keystone simply doesn't know about HTTPS due to the fact that the internal haproxy deletes the X-Forwarded-Proto header. If you look closer at haproxy.cfg and take one step further, you will see that haproxy only adds the HTTPS header if the front connection was made via an SSL.

Fix --> Tell haproxy to delete the X-Forwarded-Proto header only in case of { ssl_fc }.

Nenad Radojevic (nradojevic)
Changed in kolla:
assignee: nobody → Nenad Radojevic (nradojevic)
zhubingbing (zhubingbing)
Changed in kolla:
status: New → Triaged
milestone: none → newton-rc2
zhubingbing (zhubingbing)
Changed in kolla:
importance: Undecided → Medium
Revision history for this message
Steven Dake (sdake) wrote :

While I couldn't actually confirm this because I don't have an external load balancer, the logic of the bug report looks sound. Hence I marked it confirmed - as in confirming it looks like a real problem.

Changed in kolla:
importance: Medium → High
status: Triaged → Confirmed
Steven Dake (sdake)
Changed in kolla:
milestone: newton-rc2 → ocata-1
Jeffrey Zhang (jeffrey4l)
Changed in kolla:
milestone: ocata-1 → ocata-2
Jeffrey Zhang (jeffrey4l)
Changed in kolla:
milestone: ocata-2 → ocata-3
Christian Berendt (berendt)
Changed in kolla:
assignee: Nenad Radojevic (nradojevic) → nobody
Jeffrey Zhang (jeffrey4l)
Changed in kolla:
milestone: ocata-3 → ocata-rc1
Jeffrey Zhang (jeffrey4l)
Changed in kolla:
milestone: ocata-rc1 → pike-1
Jeffrey Zhang (jeffrey4l)
Changed in kolla:
milestone: pike-2 → pike-3
Jeffrey Zhang (jeffrey4l)
Changed in kolla:
milestone: pike-3 → pike-rc1
Eduardo Gonzalez (egonzalez90)
Changed in kolla:
milestone: pike-rc1 → queens-1
Jeffrey Zhang (jeffrey4l)
Changed in kolla:
milestone: queens-2 → queens-3
Jeffrey Zhang (jeffrey4l)
Changed in kolla:
milestone: queens-3 → queens-rc1
Jeffrey Zhang (jeffrey4l)
Changed in kolla:
milestone: queens-rc1 → queens-rc2
Jeffrey Zhang (jeffrey4l)
Changed in kolla:
milestone: queens-rc2 → rocky-1
Revision history for this message
Jeffrey Zhang (jeffrey4l) wrote : Cleanup EOL bug report

This is an automated cleanup. This bug report has been closed because it
is older than 18 months and there is no open code change to fix this.
After this time it is unlikely that the circumstances which lead to
the observed issue can be reproduced.

If you can reproduce the bug, please:
* reopen the bug report (set to status "New")
* AND add the detailed steps to reproduce the issue (if applicable)
* AND leave a comment "CONFIRMED FOR: <RELEASE_NAME>"
  Only still supported release names are valid (OCATA, PIKE, QUEENS, ROCKY, ROCKY).
  Valid example: CONFIRMED FOR: OCATA

Changed in kolla:
importance: High → Undecided
status: Confirmed → Expired
Revision history for this message
Egon (erijpkema) wrote :

Confirmed for USSURI
Steps to reproduce:
<exact same as bug description>

curl -i https://uri.tld:5000
HTTP/1.1 300 MULTIPLE CHOICES
Date: Tue, 13 Oct 2020 12:09:25 GMT
Server: Apache
Content-Length: 272
Location: http://uri.tld:5000/v3/
Vary: X-Auth-Token
x-openstack-request-id: req-4ad68dfd-4cd4-4fa4-8f1d-39a43692ce56
Content-Type: application/json

{"versions": {"values": [{"id": "v3.14", "status": "stable", "updated": "2020-04-07T00:00:00Z", "links": [{"rel": "self", "href": "http://uri.tld:5000/v3/"}], "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}]}]}}

Changed in kolla:
status: Expired → New
Revision history for this message
Mark Goddard (mgoddard) wrote :

Are you able to propose a fix for this issue?

Revision history for this message
Egon (erijpkema) wrote :

I'm working on it. Just appending `if { ssl_fc }` to the

`http-request del-header X-Forwarded-Proto`

line doesn't seem to do the trick though...

Revision history for this message
Egon (erijpkema) wrote :

After some more investigation, it seems adding `if { ssl_fc }` does lead to the desired behaviour.
But only if the `X-Forwarded-Proto https` is set. (which makes sense).

So if you use nginx as a ssl terminating reverse proxy, you need to do something like this.

       location / {
            proxy_pass http://<floating_IP>:5000;
            proxy_set_header Host name.domain.tld:5001;
            proxy_set_header X-Forwarded-Proto https;
       }

Revision history for this message
Radosław Piliszek (yoctozepto) wrote :

Makes sense.

Mark Goddard (mgoddard)
affects: kolla → kolla-ansible
Changed in kolla-ansible:
milestone: rocky-2 → none
importance: Undecided → Wishlist
Revision history for this message
Mark Goddard (mgoddard) wrote :

Proposed fix: https://review.opendev.org/758361

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on kolla-ansible (master)

Change abandoned by "Egon Rijpkema <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/758361
Reason: When doing a rebase, I found that the "if { ssl_fc } check has been incorporated.

OpenStack Infra (hudson-openstack)
Changed in kolla-ansible:
status: New → In Progress
Revision history for this message
Egon (erijpkema) wrote :

Sorry for the confusion. But i'd actually still like to have this merged.

Revision history for this message
Tom Fifield (fifieldt) wrote :

Looking at ansible/roles/haproxy-config/templates/haproxy_single_service_split.cfg.j2, it looks like Egon's suggested fix is in the code. Marking as resolved.

Changed in kolla-ansible:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.