Cannot authenticate to swift - image upload fails

Bug #1415795 reported by Bob Ball
36
This bug affects 7 people
Affects Status Importance Assigned to Milestone
OpenStack Object Storage (swift)
Confirmed
Undecided
Unassigned
devstack
Fix Released
Undecided
Jamie Lennox

Bug Description

Introduced by https://review.openstack.org/#/c/142967/

auth_host is now left at "keystonehost" which cannot be resolved.
Sample logs at http://dd6b71949550285df7dc-dda4e480e005aaa13ec303551d2d8155.r49.cf1.rackcdn.com/67/142967/8/4654/results.html

In particular note screen-s-proxy.txt.gz which errors with:
Unable to establish connection to http://keystonehost:35357

Confirmed by several people on https://review.openstack.org/#/c/142967/

Revision history for this message
Bob Ball (bob-ball) wrote :

auth_host was deprecated in https://github.com/openstack/python-keystoneclient/commit/708886b2c6bdd150c4d3dae16fb8807c19f7403c so on the surface the change @ https://review.openstack.org/#/c/142967/ is valid.

Somehow keystone is still using auth_host instead of auth_uri

Changed in swift:
status: New → Confirmed
Changed in devstack:
status: New → Confirmed
Bob Ball (bob-ball)
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to devstack (master)

Fix proposed to branch: master
Review: https://review.openstack.org/151151

Changed in devstack:
assignee: nobody → Bob Ball (bob-ball)
status: Confirmed → In Progress
Revision history for this message
Alistair Coles (alistair-coles) wrote :

@Bob could you post the contents of /etc/swift/proxy-server.conf as used by the CI env on which you observed the failed tests?

Curious that openstack Jenkins jobs are passing but you are seeing this fail. Could it be a mismatch between the change in https://review.openstack.org/#/c/142967/ and the keystonemiddleware and/or keystone versions??

Revision history for this message
Bob Ball (bob-ball) wrote :

Struggling to get my CI to collect those configuration files as well (watch this space), but the issue is:
 [filter:authtoken]
 log_name = swift
 signing_dir = /var/cache/swift
 cafile = /opt/stack/data/ca-bundle.pem
 project_domain_id = default
 project_name = service
 user_domain_id = default
 password = secretservice
 username = swift
 auth_url = http://192.168.33.1:35357
 auth_plugin = password
 paste.filter_factory = keystonemiddleware.auth_token:filter_factory
 auth_host = keystonehost
 auth_port = 35357
 auth_protocol = http
 auth_uri = http://192.168.33.1:5000
 admin_tenant_name = service
 admin_user = swift
 admin_password = password
 delay_auth_decision = 1
 cache = swift.cache
 include_service_catalog = False

devstack uncomments all of the commented-out settings from the sample proxy-server.conf, including this auth_host.

I've checked pip freeze against the gate (which passes) and keystone/keystonemiddlewere are the same.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on devstack (master)

Change abandoned by Bob Ball (<email address hidden>) on branch: master
Review: https://review.openstack.org/151151

Revision history for this message
Fabien Boucher (fabien-boucher) wrote :

Hi,
There is another bug opened here related to the same problem:
https://bugs.launchpad.net/devstack/+bug/1415242

And an associated fix:
https://review.openstack.org/#/c/151204/

Revision history for this message
Jamie Lennox (jamielennox) wrote :

So, what we were working towards here is that auth_token middleware should be configured via the global config file and not via the paste options. This relies on oslo.config populating the conf conf correctly. This is a problem because swift doesn't use oslo.config!!! I generally try to limit exclamation marks but WTF!

I will revert swift to not use the plugin based authentication methods as these are only configurable via oslo.config. However this is the way auth_token middleware is going to support v3 authentication and new authentication methods such as handling services via client certificate, and putting services users in there own domain.

Essentially is swift expecting to start using oslo.config or do we need to figure out a new way to let swift configure auth_token middleware?

Changed in devstack:
assignee: Bob Ball (bob-ball) → Jamie Lennox (jamielennox)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to swift (master)

Reviewed: https://review.openstack.org/150832
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=cec00660cbcb11086acd1906bcb0a206a846509c
Submitter: Jenkins
Branch: master

commit cec00660cbcb11086acd1906bcb0a206a846509c
Author: Bob Ball <email address hidden>
Date: Wed Jan 28 16:07:58 2015 +0000

    Remove deprecated config variables

    I1f8f5064ea8028af60f167df9b97e215cdadba44 deprecated auth_host etc but the default
    config still used them. Ieac26806bd420aa08fc79bbc6a11eb6a1c15c7df then switched
    devstack to using the new variables, but if the old variables still existed in the
    default config, some installations were broken (e.g. XenServer CI)

    Partial-bug: 1415795

    Change-Id: I7076fa03ab531cbb1114918f75113620b65590dc

Changed in devstack:
status: In Progress → Fix Released
Revision history for this message
Sumanth (avadhanisumanth1) wrote :

I am still getting getting Authentication error even after this fix

Revision history for this message
Abishek Subramanian (absubram) wrote :

I am also still hitting this bug when running devstack. For now to get around this in my VM I have backed out the keystone changes made by https://review.openstack.org/#/c/142967/
But as mentioned above, we do need the changes made by that review, so this is only a workaround.

Revision history for this message
Jamie Lennox (jamielennox) wrote :

I'm not sure why it's not mentioned in the bug above but does https://review.openstack.org/#/c/151506/ not at least take swift back to working as it did before? We will need to do some work in auth_token middleware to support swift with plugins - but this should continue to work for now.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to swift (feature/ec)

Fix proposed to branch: feature/ec
Review: https://review.openstack.org/154627

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: feature/ec
Review: https://review.openstack.org/155026

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on swift (feature/ec)

Change abandoned by paul luse (<email address hidden>) on branch: feature/ec
Review: https://review.openstack.org/155026
Reason: sam did this yesterday, duh

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to swift (feature/ec)
Download full text (19.5 KiB)

Reviewed: https://review.openstack.org/154627
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=e852ca94cc7efe6da77d8d72f4155ab151356fd4
Submitter: Jenkins
Branch: feature/ec

commit f96b8e412db73721660aa8247430db76c3149508
Author: Janie Richling <email address hidden>
Date: Mon Feb 9 18:16:25 2015 -0600

    Included sysmeta in the object info

    The cached info object dict did not include
    the sysmeta. This patch fixes that, and adds
    a unit test.

    Change-Id: I092200e76586af322ed4ff7d194a1034b1ca0433

commit b54532ca0581738c7c4622cd32e936088e9adf11
Author: OpenStack Proposal Bot <email address hidden>
Date: Fri Feb 6 06:10:34 2015 +0000

    Imported Translations from Transifex

    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure

    Change-Id: I4b8411f84784a2825a24946c51f4c08ad01febc2

commit b45b83fb00917b729253804443094d97d39ef78f
Author: John Dickinson <email address hidden>
Date: Thu Feb 5 11:01:02 2015 -0800

    Correct the config default for delay_auth_decision

    Updated proxy-server.conf-sample with the correct default. Also
    updated the note on the overview-auth doc page.

    Change-Id: I5cd62a7a118a28f7b58f47b8d8d4d963f6bc7347

commit 74563ece4782d16bf5a66db63d7c0c718a69db38
Author: Mahati Chamarthy <email address hidden>
Date: Sat Jan 31 01:40:22 2015 +0530

    Tests for the base class of storage nodes

    Change-Id: I8866e2360e30a239d5f6b4a5ed92344291184c2a

commit b5f8c594f98ec943b73633f7418695880b315b6c
Author: Nicolas Trangez <email address hidden>
Date: Wed Feb 4 16:54:46 2015 +0100

    Add `swift-scality-backend` to associated projects

    Change-Id: I7fd56c7cf5b7634224b8a2876258cf1f6be447f1

commit 2eba998a7ccde6a490d927f2489838e2954f712d
Author: John Dickinson <email address hidden>
Date: Tue Feb 3 22:12:03 2015 -0800

    added swift-ui browser to associated projects

    Change-Id: I23abd014b21d1a968fe7352f3915f02e3d4d47cf

commit d85371c31995c2d672f363e511708ba6e736fc34
Author: Alistair Coles <email address hidden>
Date: Tue Feb 3 18:24:41 2015 +0000

    Don't skip account acl functional tests

    The functional tests covering account acls are skipped
    if keystoneauth is in the pipeline, even if keystone auth
    is not being used. Devstack configures the pipeline to have
    both tempauth and keystoneauth, so these tests are always
    skipped.

    This patch changes the condition for skipping account acl
    to be based on tempauth being the auth service configured
    for the tests.

    Change-Id: I378ec6aa0ba52d37a33796057e59a9ebfcab2574

commit 5a0f8f14029121b35f03c2518347fafea8a4f28f
Author: Alistair Coles <email address hidden>
Date: Tue Feb 3 11:57:06 2015 +0000

    Update auth_token section in documentation

    Bring docs in line with changes to auth_token config
    defaults made in I7076fa03ab531cbb1114918f75113620b65590dc

    Change-Id: Ia21685ebd1f3ed7bdba9de2ebac9fdcce8495949

commit a270dca23989b5cf7b59cf1d0e285bc736717a44
Author: Takashi Kajinami <email address hidden>
Date: Fri Jan 23 20:16:20 2015 +0900
...

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.