GET EC policy object includes connection:close header which causes external LB returns 502 bad gateway to client

It seems like this issue only happens on object in EC policy,

# EC policy

$ curl -i http://192.168.190.21/v1/AUTH_demo/ec64/a -X GET -H "X-Auth-Token: AUTH_tk383654937f2d4fa3a1cb26fbaaf82b40"
HTTP/1.1 200 OK
Content-Length: 0
X-Object-Meta-Mtime: 1491541780.000000
Accept-Ranges: bytes
Last-Modified: Fri, 07 Apr 2017 05:09:45 GMT
Connection: close
Etag: d41d8cd98f00b204e9800998ecf8427e
X-Timestamp: 1491541784.48274
Date: Fri, 07 Apr 2017 06:52:24 GMT
Content-Type: application/octet-stream
X-Trans-Id: tx476d99bf575f41e48e8ba-0058e73728
X-Openstack-Request-Id: tx476d99bf575f41e48e8ba-0058e73728

# Replica policy

$ curl -i http://192.168.190.21/v1/AUTH_demo/gw/20MB -I -H "X-Auth-Token: AUTH_tk383654937f2d4fa3a1cb26fbaaf82b40"
HTTP/1.1 200 OK
Content-Length: 20971520
Content-Type: application/octet-stream
Accept-Ranges: bytes
Last-Modified: Fri, 07 Apr 2017 06:51:27 GMT
Etag: 8f4e33f3dc3e414ff94e5fb6905cba8c
X-Timestamp: 1491547886.25414
X-Object-Meta-Mtime: 1491547877.000000
X-Trans-Id: tx63c97a2b94e34ae2a5830-0058e7372c
X-Openstack-Request-Id: tx63c97a2b94e34ae2a5830-0058e7372c
Date: Fri, 07 Apr 2017 06:52:28 GMT

We should be able to test for this in a functest, i'm worried there may be other backend headers that leak out.

FWIW no gateway should translate this response to 502 - or any response that returns connection: close at any time (in in fact that may not even be related to this bug).

However, any client or gateway that gets a swift EC GET response tho will see this header and should close the connection - although FIWI I have noticed you *can* pipeline EC get requests just fine - this isn't eventlet.wsgi server setting close and shutting down the socket - it's just a left over backend header. But the semantic is wrong.

We've always returned it too - since when Sam & Paul kicked around the first versions of EC GET on the feature branch till right now as best I can tell.

ZOMGBBQ - you know what it *might* be - if a client *sends* 'Connection: keep-alive' (which is redundant in http 1.1 but not invalid) the most insane thing comes back:

ubuntu@saio:~$ curl -H 'x-auth-token: AUTH_tkbda4caa25b4f4f60a50ec26207487741' http://saio:8080/v1/AUTH_test/ec-test/deleteme -H 'connection: keep-alive' -v
* Trying 127.0.0.1...
* Connected to saio (127.0.0.1) port 8080 (#0)
> GET /v1/AUTH_test/ec-test/deleteme HTTP/1.1
> Host: saio:8080
> User-Agent: curl/7.47.0
> Accept: */*
> x-auth-token: AUTH_tkbda4caa25b4f4f60a50ec26207487741
> connection: keep-alive
>
< HTTP/1.1 200 OK
< Content-Length: 3145728
< X-Object-Meta-Mtime: 1491470291.015581
< Accept-Ranges: bytes
< Last-Modified: Fri, 07 Apr 2017 08:01:02 GMT
< Connection: close <-*****close****
< Etag: d1dd210d6b1312cb342b56d02bd5e651
< X-Timestamp: 1491552061.98489
< Date: Fri, 07 Apr 2017 08:04:21 GMT
< Content-Type: application/octet-stream
< X-Trans-Id: tx11ec5191d0304934b174d-0058e74805
< X-Openstack-Request-Id: tx11ec5191d0304934b174d-0058e74805
< Connection: keep-alive <-*****keep-alive****
<
* Closing connection 0

That is interesting and could very well invalid, and almost definitely could cause problem for otherwise reasonably behaved clients...

without the raw dump curl seems to sort out the correct answer:

ubuntu@saio:~$ curl -H 'x-auth-token: AUTH_tkbda4caa25b4f4f60a50ec26207487741' http://saio:8080/v1/AUTH_test/ec-test/deleteme -H 'connection: keep-alive' -I
HTTP/1.1 200 OK
Content-Length: 3145728
Content-Type: application/octet-stream
Accept-Ranges: bytes
Last-Modified: Fri, 07 Apr 2017 08:01:02 GMT
Etag: d1dd210d6b1312cb342b56d02bd5e651
X-Timestamp: 1491552061.98489
X-Object-Meta-Mtime: 1491470291.015581
X-Trans-Id: tx3c00e2a73c444b4798050-0058e74869
X-Openstack-Request-Id: tx3c00e2a73c444b4798050-0058e74869
Date: Fri, 07 Apr 2017 08:06:01 GMT
Connection: keep-alive

swiftclient has a slightly different opinion

DEBUG:swiftclient:RESP HEADERS: {u'Content-Length': u'3145728', u'Content-Type': u'application/octet-stream', u'Accept-Ranges': u'bytes', u'Last-Modified': u'Fri, 07 Apr 2017 08:01:02 GMT', u'Connection': u'close, keep-alive', u'Etag': u'd1dd210d6b1312cb342b56d02bd5e651', u'X-Timestamp': u'1491552061.98489', u'X-Trans-Id': u'tx9bbeda8da5914148b6228-0058e7487e', u'Date': u'Fri, 07 Apr 2017 08:06:22 GMT', u'X-Object-Meta-Mtime': u'1491470291.015581', u'X-Openstack-Request-Id': u'tx9bbeda8da5914148b6228-0058e7487e'}

fascinating

EC GET responses (and only EC GET, not HEAD, not replicated policy) seem to have always incuded the connection: close header, logs below from 2.9.0 but I saw similar on Mitaka (2.7.1):

swift@vm-16:~/junk$ cd ../swift; git status
HEAD detached at 2.9.0
nothing to commit, working directory clean

Replicated policy:

swift@vm-16:~/junk$ curl -i http://localhost:8080/v1/AUTH_test/c2/10MB_file -X GET -H "X-Auth-Token: AUTH_tk5005e34fa7f84e9b85dd698467496a11"
HTTP/1.1 200 OK
Content-Length: 10485760
Content-Type: application/octet-stream
Accept-Ranges: bytes
Last-Modified: Fri, 07 Apr 2017 08:14:08 GMT
Etag: f1c9645dbc14efddc7d8a322685f26eb
X-Timestamp: 1491552847.85534
X-Object-Meta-Mtime: 1491498549.279092
X-Trans-Id: txf617173c4682472eacaed-0058e74b36
Date: Fri, 07 Apr 2017 08:17:58 GMT

EC policy:

swift@vm-16:~/junk$ curl -i http://localhost:8080/v1/AUTH_test/c1/10MB_file -X GET -H "X-Auth-Token: AUTH_tk5005e34fa7f84e9b85dd698467496a11"
HTTP/1.1 200 OK
Content-Length: 10485760
X-Object-Meta-Mtime: 1491498549.279092
Accept-Ranges: bytes
Last-Modified: Fri, 07 Apr 2017 08:14:16 GMT
Connection: close
Etag: f1c9645dbc14efddc7d8a322685f26eb
X-Timestamp: 1491552855.87341
Date: Fri, 07 Apr 2017 08:18:02 GMT
Content-Type: application/octet-stream
X-Trans-Id: txf6518213957f499cb39ba-0058e74b3a

EC HEAD:

swift@vm-16:~/swift$ curl -i http://localhost:8080/v1/AUTH_test/c1/10MB_file -I -H "X-Auth-Token: AUTH_tk5005e34fa7f84e9b85dd698467496a11"
HTTP/1.1 200 OK
Content-Length: 10485760
Content-Type: application/octet-stream
Accept-Ranges: bytes
Last-Modified: Fri, 07 Apr 2017 08:14:16 GMT
Etag: f1c9645dbc14efddc7d8a322685f26eb
X-Timestamp: 1491552855.87341
X-Object-Meta-Mtime: 1491498549.279092
X-Trans-Id: txff4b34d61f0b4323a8dc1-0058e74bae
Date: Fri, 07 Apr 2017 08:19:58 GMT

@Clay in your comment #4 you say that curl sorts out the connection headers, but actually I think you have issued a HEAD request by having the -I option:

"
curl -H 'x-auth-token: AUTH_tkbda4caa25b4f4f60a50ec26207487741' http://saio:8080/v1/AUTH_test/ec-test/deleteme -H 'connection: keep-alive' -I
HTTP/1.1 200 OK
Content-Length: 3145728
Content-Type: application/octet-stream
Accept-Ranges: bytes
Last-Modified: Fri, 07 Apr 2017 08:01:02 GMT
Etag: d1dd210d6b1312cb342b56d02bd5e651
X-Timestamp: 1491552061.98489
X-Object-Meta-Mtime: 1491470291.015581
X-Trans-Id: tx3c00e2a73c444b4798050-0058e74869
X-Openstack-Request-Id: tx3c00e2a73c444b4798050-0058e74869
Date: Fri, 07 Apr 2017 08:06:01 GMT
Connection: keep-alive
"

I see both Connection headers reported for a GET:

swift@vm-16:~/junk$ curl -i http://localhost:8080/v1/AUTH_test/c1/10MB_file -X GET -H "X-Auth-Token: AUTH_tk5005e34fa7f84e9b85dd698467496a11" -H "Connection: keep-alive"
HTTP/1.1 200 OK
Content-Length: 10485760
X-Object-Meta-Mtime: 1491498549.279092
Accept-Ranges: bytes
Last-Modified: Fri, 07 Apr 2017 08:14:16 GMT
Connection: close <----
Etag: f1c9645dbc14efddc7d8a322685f26eb
X-Timestamp: 1491552855.87341
Date: Fri, 07 Apr 2017 09:11:07 GMT
Content-Type: application/octet-stream
X-Trans-Id: tx72f5f1ca261c42a6a98a1-0058e757ab
Connection: keep-alive <----

the -I request is definitely a HEAD request, and need to remember *we don't leak backend headers on HEAD

Thank you for correcting me.

So we already have a function in the swift/proxy/controllers/base.py that strips the connection from the backend called update_headers. this is why HEAD and replication work.

We don't call it on the EC side, so maybe something like this is all we need to do.

Reviewed: https://review.openstack.org/456706
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=6c320b2908e5821e2f398e0dae8a67774e110603
Submitter: Jenkins
Branch: master

commit 6c320b2908e5821e2f398e0dae8a67774e110603
Author: Alistair Coles <email address hidden>
Date: Thu Apr 13 11:16:54 2017 +0100

    Stop including Connection header in EC GET responses

    Currently, EC GET responses from proxy to clients, unlike any other
    response, include a "Connection: close" header. If the client has sent
    a "Connection: keep-alive" header then eventlet.wsgi appends this to
    the client response, so clients can receive a response with both
    headers:

    Connection: close
    Connection: keep-alive

    This patch fixes the proxy EC GET path to remove any Connection header
    from it's response, but does not change the behaviour of eventlet.wsgi
    with respect to returning any client provided 'Connection: keep-alive'
    header.

    Change-Id: I43cd27c978edb4a1a587f031dbbee26e9acdc920
    Co-Authored-By: Matthew Oliver <email address hidden>
    Closes-Bug: #1680731

This issue was fixed in the openstack/swift 2.15.0 release.