xattr import error on devstack caused by cffi 1.0beta (swift-object-server fails to start in devstack which kills the gate)

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwid2l0aCBvcGVuKHNlbGYuc291cmNlZmlsZW5hbWUsIFxcXCJ3XFxcIikgYXMgZnA6XCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0MzA4NTM1MzQyNTV9

Might be this? http://morepypy.blogspot.ch/2015/05/cffi-10-beta-1.html

http://logs.openstack.org/27/169827/10/check/check-tempest-dsvm-full/506c93a/logs/pip-freeze.txt.gz

cffi==0.9.2

xattr==0.7.5

diffs for pip and apt-get from good to bad: http://paste.openstack.org/show/215061 http://paste.openstack.org/show/215062

It's not the referenced commit in Swift (https://github.com/openstack/swift/commit/215cd551df8be066edafd2a1e16d0bd143ec214b). All that one does is get a little fancier with tarball parsing.

Swift stores all sorts of things in xattrs, including object name, md5sum, upload time, content type, user metadata, and system metadata. All these things get jammed into a single dictionary, pickled, and saved in xattrs. If xattr isn't working, Swift won't work.

Besides, the referenced commit only affects bulk uploads (i.e. multiple files in an .tar, .tar.gz, or .tar.bz2), which Glance doesn't use.

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.openstack.org/180276
Reason: Alright, thanks Samuel.

Description of the traceback:

* swift/obj/diskfile.py imports the xattr module
* xattr is Python binding of the libxattr library, it is implemented with cffi
* if available, cffi uses the compiled module. For example, on my DevStack it's the file /usr/local/lib/python2.7/dist-packages/xattr/_cffi__x282d9483x6ee6f75.so (xattr 0.7.5, cffi 0.9.2)
* the compiled module is missing, cffi tries to write a C code to compile it as a library, but it fails to write the file, probably because the directory doesn't exist: /usr/local/lib/python2.7/dist-packages/xattr/__pycache__/_cffi__x282d9483x6ee6f75.c
* (cffi tries to create the directory, but ignores any OS error)

The "_cffi__(...).so" library is created at the installation of the cffi module.

There is no wheel package for xattr. DevStack builds a local wheel package for xattr. The wheel package contains the .so library, but it's not installed in the path expected by cffi. cffi looks in the .../xattr/__pycache__/ directory, whereas it is written in .../xattr/.

Indeed, I don't see how Sam's patch could affect what happens during the module import.

Ok so this is the second symptom from https://bitbucket.org/cffi/cffi/issue/109/enable-sane-packaging-for-cffi
a workaround would be to hook into the wheel build for xattr and move the .so

Or alternatively to depend on cffi 1 beta and and patch xattr to use the proper packaging support 1.0 has (http://morepypy.blogspot.ch/2015/05/cffi-10-beta-1.html)

https://review.openstack.org/#/c/180341/ was approved to quickly unblock gates.

I wrote a different workaround https://review.openstack.org/#/c/180354/

Change abandoned by Joe Gordon (<email address hidden>) on branch: master
Review: https://review.openstack.org/180330

Reviewed: https://review.openstack.org/180549
Committed: https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=5a59ac7d43bb10a5bbc912b94edea19e1009b675
Submitter: Jenkins
Branch: master

commit 5a59ac7d43bb10a5bbc912b94edea19e1009b675
Author: Sean Dague <email address hidden>
Date: Wed May 6 09:48:54 2015 -0400

    create a more generic work around for cffi & wheels

    This is an attempt to fix the cffi vs. wheels bug in a more generic
    way by just ensuring that pip has installed cffi with a pip understood
    version into the venv before we try to do any builds.

    Related-Bug: #1451992

    Change-Id: Ibc58668c53933033405b40f79b0e9ffc73a01a6f

Related fix proposed to branch: stable/kilo
Review: https://review.openstack.org/181374

Reviewed: https://review.openstack.org/181374
Committed: https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=3f97104828b4c51ff016c31225ee5ee8aeb4d489
Submitter: Jenkins
Branch: stable/kilo

commit 3f97104828b4c51ff016c31225ee5ee8aeb4d489
Author: Sean Dague <email address hidden>
Date: Wed May 6 09:48:54 2015 -0400

    create a more generic work around for cffi & wheels

    This is an attempt to fix the cffi vs. wheels bug in a more generic
    way by just ensuring that pip has installed cffi with a pip understood
    version into the venv before we try to do any builds.

    Related-Bug: #1451992

    Change-Id: Ibc58668c53933033405b40f79b0e9ffc73a01a6f
    (cherry picked from commit 5a59ac7d43bb10a5bbc912b94edea19e1009b675)

Change abandoned by Victor Stinner (<email address hidden>) on branch: master
Review: https://review.openstack.org/180354
Reason: https://review.openstack.org/#/c/180549/ was better and is better than my change (no hardcoded version).