tripleo

undercloud, swift returns 503 in undercloud install

Bug #1718454 reported by wes hayutin on 2017-09-20

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	tripleo	Fix Released	Critical	Christian Schwede	tripleo queens-1

Bug Description

2017-09-19 06:45:27 | 2017-09-19 06:45:27,928 INFO: Configuring Mistral workbooks
2017-09-19 06:46:03 | 2017-09-19 06:46:03,343 INFO: Mistral workbooks configured successfully
2017-09-19 06:46:35 | 2017-09-19 06:46:35,286 DEBUG: An exception occurred
2017-09-19 06:46:35 | Traceback (most recent call last):
2017-09-19 06:46:35 | File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1782, in install
2017-09-19 06:46:35 | _post_config(instack_env)
2017-09-19 06:46:35 | File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1699, in _post_config
2017-09-19 06:46:35 | _post_config_mistral(instack_env, mistral, swift)
2017-09-19 06:46:35 | File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1652, in _post_config_mistral
2017-09-19 06:46:35 | plans = [container["name"] for container in swift.get_account()[1]]
2017-09-19 06:46:35 | File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1729, in get_account
2017-09-19 06:46:35 | full_listing=full_listing, headers=headers)
2017-09-19 06:46:35 | File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1678, in _retry
2017-09-19 06:46:35 | service_token=self.service_token, **kwargs)
2017-09-19 06:46:35 | File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 769, in get_account
2017-09-19 06:46:35 | raise ClientException.from_response(resp, 'Account GET failed', body)
2017-09-19 06:46:35 | ClientException: Account GET failed: http://192.168.24.1:8080/v1/AUTH_f0d0444566d64bbc9c432f87c35bae2e?format=json 503 Service Unavailable [first 60 chars of response] <html><h1>Service Unavailable</h1><p>The server is currently
2017-09-19 06:46:35 | 2017-09-19 06:46:35,288 ERROR:
2017-09-19 06:46:35 | #############################################################################
2017-09-19 06:46:35 | Undercloud install failed.
2017-09-19 06:46:35 |
2017-09-19 06:46:35 | Reason: Account GET failed: http://192.168.24.1:8080/v1/AUTH_f0d0444566d64bbc9c432f87c35bae2e?format=json 503 Service Unavailable [first 60 chars of response] <html><h1>Service Unavailable</h1><p>The server is currently
2017-09-19 06:46:35 |
2017-09-19 06:46:35 | See the previous output for details about what went wrong. The full install
2017-09-19 06:46:35 | log can be found at /home/jenkins/.instack/install-undercloud.log.
2017-09-19 06:46:35 |
2017-09-19 06:46:35 | #############################################################################

http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-scenario003-multinode-oooq/5f0e976/logs/undercloud/home/jenkins/undercloud_install.log.txt.gz#_2017-09-19_06_46_35

http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-ha-oooq/59af5e0/logs/undercloud/home/jenkins/undercloud_install.log.txt.gz#_2017-09-19_06_42_35

http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-scenario004-multinode-oooq/cc80716/logs/undercloud/home/jenkins/undercloud_install.log.txt.gz#_2017-09-19_06_49_23

Tags:

Revision history for this message

Alex Schultz (alex-schultz) wrote on 2017-09-20:

Updated title, the error points to swift being unavailable (not keystone)

summary:

- undercloud, keystone auth fails in undercloud install
+ undercloud, swift returns 503 in undercloud install

Revision history for this message

Thomas Herve (therve) wrote on 2017-09-21:

I think that's the error from the logs:

proxy-server: ERROR Insufficient Storage 192.168.24.1:6002/1

Not sure what changed.

Revision history for this message

Christian Schwede (cschwede) wrote on 2017-09-21:

The error message is a bit misleading, there is plenty of storage available according to http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-scenario003-multinode-oooq/5f0e976/logs/undercloud/var/log/host_info.txt.gz:

+ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 38G 3.0G 33G 9% /
devtmpfs 3.9G 0 3.9G 0% /dev
tmpfs 3.9G 148K 3.9G 1% /dev/shm
tmpfs 3.9G 428K 3.9G 1% /run
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/xvdd 506K 506K 0 100% /mnt/config
tmpfs 798M 0 798M 0% /run/user/1000
/dev/xvde2 72G 7.8G 60G 12% /opt

The error reported by the proxy-server propagated from the backend account-server:

Sep 19 06:46:03 centos-7-2-node-rax-dfw-10988205 account-server: 192.168.24.1 - - [19/Sep/2017:06:46:03 +0000] "HEAD /1/47/AUTH_f0d0444566d64bbc9c432f87c35bae2e" 507 - "HEAD http://192.168.24.1:8080/v1/AUTH_f0d0444566d64bbc9c432f87c35bae2e" "tx5f5cc4e06d714a658a458-0059c0bd2b" "proxy-server 1836" 0.0007 "-" 3218 -

The account-server only returns a 507 if the mount_check is set to True, which is not the case on the undercloud after puppet run. However, it is the default if the config file has not been updated by puppet yet, so I'm wondering if the config file was not synced to disk yet?

Revision history for this message

Christian Schwede (cschwede) wrote on 2017-09-21:

Just for completeness - this is where it happens: https://github.com/openstack/swift/blob/master/swift/account/server.py#L172-L173

Changed in tripleo:
assignee:	nobody → Christian Schwede (cschwede)

OpenStack Infra (hudson-openstack) on 2017-09-21

Changed in tripleo:
status:	Triaged → In Progress

Revision history for this message

Christian Schwede (cschwede) wrote on 2017-09-21:

Swift now checks if the storage directory truely exists, and no longer creates this if not. This is related to a recently merged change: https://github.com/openstack/swift/commit/30fd4e

This will affect more CI issues pretty as soon as this change lands in the rpms.

Patch proposed to fix this: https://review.openstack.org/506187

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-09-22: Fix merged to instack-undercloud (master)

Reviewed: https://review.openstack.org/506187
Committed: https://git.openstack.org/cgit/openstack/instack-undercloud/commit/?id=d899ac3deb80baf908eca9aa9fa655c60fbc42d3
Submitter: Jenkins
Branch: master

commit d899ac3deb80baf908eca9aa9fa655c60fbc42d3
Author: Christian Schwede <email address hidden>
Date: Thu Sep 21 15:54:22 2017 +0200

Fix Swift deployment due to missing directory

    A recent change in Swift will now check if the storage directories are
    existing, and return an error if not. This requires that the storage
    directory required by Swift is created in advance.

Closes-Bug: 1718454
Change-Id: I3b42dbc7185ee1c6f9f848ff0a491160f081c4eb