OpenStack Swift Proxy Charm

Broken dependency when setting swift-proxy on Bionic

Bug #1817055 reported by Pedro Guimarães
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Swift Proxy Charm
Invalid
Undecided
Unassigned
python-eventlet (Ubuntu)
Fix Released
Medium
Unassigned
Bionic
Won't Fix
High
James Page
Cosmic
Won't Fix
Medium
James Page
Disco
Fix Released
Medium
Unassigned

Bug Description

[Impact]
Swift proxy server will sometimes fail to start due to racy-ness in monkey patching by eventlet and daemon startup. Swift dies, no object storage.

[Test Case]
Deploy swift using the reference bundles for deploying OpenStack with Juju and the OpenStack charms.
Depending on the number of cores being used the stacktrace in the original bug report will be seen and swift-proxy will fail to startup.

[Regression Potential]
The workaround fix to ensure that the eventlet hub is correctly initialised has been in the upstream codebase since 2017 so low potential for regression.

[Original Bug Report]
Charm options:
  swift-proxy:
    charm: cs:swift-proxy
    num_units: 3
    bindings:
      "": *oam-space
      public: *public-space
      admin: *admin-space
      internal: *internal-space
      swift-storage: *swift-cluster-space
    options:
      openstack-origin: *openstack-origin
      vip: *swift-proxy-vip
      region: *openstack-region
      zone-assignment: manual
      partition-power: *partition-power
      workers: *workers
    to:
    - lxd:0
    - lxd:1
    - lxd:2

Setting up swift-proxy on Bionic causes swift-proxy.service to fail at start with:

Feb 21 11:16:07 juju-29125a-0-lxd-15 systemd[1]: Stopped LSB: Swift proxy server.
Feb 21 11:16:07 juju-29125a-0-lxd-15 systemd[1]: swift-proxy.service: Failed to reset devices.list: Operation not permitted
Feb 21 11:16:07 juju-29125a-0-lxd-15 systemd[1]: Starting LSB: Swift proxy server...
Feb 21 11:16:07 juju-29125a-0-lxd-15 swift-proxy[41939]: * Starting Swift Swift proxy server swift-init proxy-server
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: Starting proxy-server...(/etc/swift/proxy-server.conf)
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: Traceback (most recent call last):
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/bin/swift-proxy-server", line 23, in <module>
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: sys.exit(run_wsgi(conf_file, 'proxy-server', **options))
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/swift/common/wsgi.py", line 912, in run_wsgi
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: loadapp(conf_path, global_conf=global_conf)
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/swift/common/wsgi.py", line 390, in loadapp
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: ctx = loadcontext(loadwsgi.APP, conf_file, global_conf=global_conf)
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/swift/common/wsgi.py", line 374, in loadcontext
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: global_conf=global_conf)
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 296, in loadcontext
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: global_conf=global_conf)
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 320, in _loadconfig
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: return loader.get_context(object_type, name, global_conf)
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/swift/common/wsgi.py", line 66, in get_context
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: object_type, name=name, global_conf=global_conf)
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 450, in get_context
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: global_additions=global_additions)
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 562, in _pipeline_app_context
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: for name in pipeline[:-1]]
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/swift/common/wsgi.py", line 66, in get_context
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: object_type, name=name, global_conf=global_conf)
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 458, in get_context
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: section)
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 517, in _context_from_explicit
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: value = import_string(found_expr)
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 22, in import_string
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: return pkg_resources.EntryPoint.parse("x=" + s).load(False)
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2324, in load
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: return self.resolve()
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2330, in resolve
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: module = __import__(self.module_name, fromlist=['__name__'], level=0)
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 221, in <module>
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: from keystoneauth1 import adapter
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 16, in <module>
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: from keystoneauth1 import session
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 40, in <module>
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: import osprofiler.web as osprofiler_web
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/osprofiler/web.py", line 20, in <module>
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: from osprofiler import profiler
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 26, in <module>
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: from osprofiler import notifier
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/osprofiler/notifier.py", line 16, in <module>
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: from osprofiler.drivers import base
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/osprofiler/drivers/__init__.py", line 4, in <module>
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: from osprofiler.drivers import loginsight # noqa
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/osprofiler/drivers/loginsight.py", line 24, in <module>
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: from oslo_concurrency.lockutils import synchronized
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 27, in <module>
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: import fasteners
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/fasteners/__init__.py", line 23, in <module>
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: from fasteners.lock import locked # noqa
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/fasteners/lock.py", line 24, in <module>
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: from fasteners import _utils
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/fasteners/_utils.py", line 22, in <module>
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: from monotonic import monotonic as now # noqa
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/monotonic.py", line 164, in <module>
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: raise RuntimeError('no suitable implementation for this system')
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: RuntimeError: no suitable implementation for this system
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: ...fail!
Feb 21 11:16:08 juju-29125a-0-lxd-15 systemd[1]: Started LSB: Swift proxy server.

Looking at errors for monotonic lib, I can find other references around github:
https://github.com/nameko/nameko/issues/533
https://github.com/eventlet/eventlet/issues/401

Swift-proxy indeed depends on eventlet:
root@juju-29125a-0-lxd-15:~# dpkg -l | grep eventlet
ii python-eventlet 0.20.0-4 all concurrent networking library - Python 2.x

And then the issue described at: https://github.com/eventlet/eventlet/issues/401 applies.

Moving back to Xenial seems to stabilize swift proxy.

See original description
Revision history for this message
James Page (james-page) wrote :

Distro issue; needs pick of:

  https://github.com/eventlet/eventlet/commit/b756447bab51046dfc6f1e0e299cc997ab343701

back to bionic and cosmic - 0.24.0 has the required commit (as in disco).

Changed in charm-swift-proxy:
status: New → Invalid
Changed in python-eventlet (Ubuntu Disco):
status: New → Fix Released
Changed in python-eventlet (Ubuntu Cosmic):
status: New → Triaged
Changed in python-eventlet (Ubuntu Bionic):
status: New → Triaged
importance: Undecided → High
Changed in python-eventlet (Ubuntu Cosmic):
importance: Undecided → Medium
Revision history for this message
James Page (james-page) wrote :

Patched packages for bionic/cosmic uploaded to:

  https://launchpad.net/~james-page/+archive/ubuntu/bug1817055

If you can please try out the proposed updates; I'll start the SRU process.

Changed in python-eventlet (Ubuntu Bionic):
assignee: nobody → James Page (james-page)
Changed in python-eventlet (Ubuntu Cosmic):
assignee: nobody → James Page (james-page)
Changed in python-eventlet (Ubuntu Bionic):
status: Triaged → In Progress
Changed in python-eventlet (Ubuntu Cosmic):
status: Triaged → In Progress
Revision history for this message
Pedro Guimarães (pguimaraes) wrote :

The problem disappears when switching to your PPA. However, I still see some trace on journalctl for swift-proxy: https://pastebin.ubuntu.com/p/RFzk7qRQ6J/

The process marks:
● swift-proxy.service - LSB: Swift proxy server
   Loaded: loaded (/etc/init.d/swift-proxy; generated)
   Active: active (exited) since Sat 2019-02-23 12:03:14 UTC; 1min 4s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 54857 ExecStop=/etc/init.d/swift-proxy stop (code=exited, status=0/SUCCESS)
  Process: 54865 ExecStart=/etc/init.d/swift-proxy start (code=exited, status=0/SUCCESS)

Feb 23 12:03:14 juju-b57129-2-lxd-17 swift-proxy[54865]: pid = self.spawn(conf_file, **kwargs)
Feb 23 12:03:14 juju-b57129-2-lxd-17 swift-proxy[54865]: File "/usr/lib/python2.7/dist-packages/swift/common/manager.py", line 669, in spawn
Feb 23 12:03:14 juju-b57129-2-lxd-17 swift-proxy[54865]: proc = subprocess.Popen(args, stdout=re_out, stderr=re_err)
Feb 23 12:03:14 juju-b57129-2-lxd-17 swift-proxy[54865]: File "/usr/lib/python2.7/subprocess.py", line 394, in __init__
Feb 23 12:03:14 juju-b57129-2-lxd-17 swift-proxy[54865]: errread, errwrite)
Feb 23 12:03:14 juju-b57129-2-lxd-17 swift-proxy[54865]: File "/usr/lib/python2.7/subprocess.py", line 938, in _execute_child
Feb 23 12:03:14 juju-b57129-2-lxd-17 swift-proxy[54865]: self.pid = os.fork()
Feb 23 12:03:14 juju-b57129-2-lxd-17 swift-proxy[54865]: OSError: [Errno 11] Resource temporarily unavailable
Feb 23 12:03:14 juju-b57129-2-lxd-17 swift-proxy[54865]: ...fail!
Feb 23 12:03:14 juju-b57129-2-lxd-17 systemd[1]: Started LSB: Swift proxy server.

Although it marks as Started, I fail to connect to swift with: https://pastebin.canonical.com/p/m23rCwsTjS/

The VIP is correctly allocated to one swift-proxy instance.

I dont see any process referring to swift-proxy on my lxd:
root 823 0.0 0.0 21768 3524 ? Ss 11:36 0:00 bash /lib/systemd/system/jujud-unit-swift-proxy-3/exec-start.sh
root 827 0.0 0.0 3617772 75964 ? Sl 11:36 0:07 /var/lib/juju/tools/unit-swift-proxy-3/jujud unit --data-dir /var/lib/juju --unit-name swift-proxy/3 --debug
root 16347 0.0 0.0 21768 3512 ? Ss 11:39 0:00 bash /lib/systemd/system/jujud-unit-hacluster-swift-5/exec-start.sh
root 16362 0.0 0.0 3543528 70764 ? Sl 11:39 0:02 /var/lib/juju/tools/unit-hacluster-swift-5/jujud unit --data-dir /var/lib/juju --unit-name hacluster-swift/5 --debug
ubuntu 70915 0.0 0.0 13160 1076 pts/0 S+ 12:14 0:00 grep --color=auto swift

Revision history for this message
Pedro Guimarães (pguimaraes) wrote :

To keep it registered, I've set up swift-proxy on Bionic containers (same configs as described on the main text of this bug).

After deployed, I've ran:
juju run --application swift-proxy "add-apt-repository ppa:james-page/bug1817055"
juju run --application swift-proxy "apt update"
juju run --application swift-proxy "apt install python-eventlet=0.20.0-4ubuntu0.18.04.1~ubuntu18.04.1~ppa201902221457"
juju run --application swift-proxy "apt -y install python-eventlet=0.20.0-4ubuntu0.18.04.1~ubuntu18.04.1~ppa201902221457"
juju run --application swift-proxy "systemctl restart swift-proxy.service"

Revision history for this message
Pedro Guimarães (pguimaraes) wrote :

Also, despite those issues, charm is stating "Ready/Active" on its status

Revision history for this message
Pedro Guimarães (pguimaraes) wrote :

Well, I've found out why charm is not complaining: systemctl is marking as SUCCESS despite failures on journalctl.

Also, running an strace on this, I see:
close(3) = 0
stat("/etc/swift/proxy-server.conf", {st_mode=S_IFREG|0644, st_size=2797, ...}) = 0
openat(AT_FDCWD, "/var/run/swift", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = -1 ENOENT (No such file or directory)
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0
write(1, "Starting proxy-server...(/etc/sw"..., 55Starting proxy-server...(/etc/swift/proxy-server.conf)
) = 55
pipe([3, 4]) = 0
fcntl(3, F_GETFD) = 0
fcntl(3, F_SETFD, FD_CLOEXEC) = 0
fcntl(4, F_GETFD) = 0
fcntl(4, F_SETFD, FD_CLOEXEC) = 0
pipe([5, 6]) = 0
fcntl(5, F_GETFD) = 0
fcntl(5, F_SETFD, FD_CLOEXEC) = 0
fcntl(6, F_GETFD) = 0
fcntl(6, F_SETFD, FD_CLOEXEC) = 0
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f3707c16a10) = -1 EAGAIN (Resource temporarily unavailable)

So, my guess is that control nodes are overloaded with containers and swift cannot spawn any more workers.
I moved swift-proxy to swift-storage nodes and it worked.

Revision history for this message
James Page (james-page) wrote :

Patched versions of eventlet for bionic and cosmic uploaded for SRU Team review.

description: updated
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Pedro, or anyone else affected,

Accepted python-eventlet into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-eventlet/0.20.0-5ubuntu0.18.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in python-eventlet (Ubuntu Cosmic):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-cosmic
Revision history for this message
Brian Murray (brian-murray) wrote :

Hello Pedro, or anyone else affected,

Accepted python-eventlet into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-eventlet/0.20.0-4ubuntu0.18.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in python-eventlet (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed-bionic
Mathew Hodson (mhodson)
Changed in python-eventlet (Ubuntu Disco):
importance: Undecided → Medium
Revision history for this message
Brian Murray (brian-murray) wrote : [python-eventlet/cosmic] verification still needed

The fix for this bug has been awaiting testing feedback in the -proposed repository for cosmic for more than 90 days. Please test this fix and update the bug appropriately with the results. In the event that the fix for this bug is still not verified 15 days from now, the package will be removed from the -proposed repository.

tags: added: removal-candidate
Revision history for this message
Steve Langasek (vorlon) wrote : Proposed package removed from archive

The version of python-eventlet in the proposed pocket of Cosmic that was purported to fix this bug report has been removed because the bugs that were to be fixed by the upload were not verified in a timely (105 days) fashion.

tags: removed: verification-needed-cosmic
Changed in python-eventlet (Ubuntu Cosmic):
status: Fix Committed → Won't Fix
tags: removed: verification-needed-bionic
Revision history for this message
Steve Langasek (vorlon) wrote :

The version of python-eventlet in the proposed pocket of Bionic that was purported to fix this bug report has been removed because the bugs that were to be fixed by the upload were not verified in a timely (105 days) fashion.

Changed in python-eventlet (Ubuntu Bionic):
status: Fix Committed → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.