[MIR] python-pyeclib, liberasurecode, jerasure, gf-complete

Bug #1482227 reported by James Page
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
gf-complete (Ubuntu)
Fix Released
High
Unassigned
jerasure (Ubuntu)
Fix Released
High
Unassigned
liberasurecode (Ubuntu)
Fix Released
High
Unassigned
python-pyeclib (Ubuntu)
Fix Released
High
Matthias Klose

Bug Description

[Overview]

Swift 2.3.0 has experimental support for erasure coded storage pools; even though this is optional, the integration is fairly well laced into the storage pool code, and the project are working towards making this production grade.

This MIR requests that the required dependency be included in main, so we can move to the newer Swift versions (getting other new version goodness as well).

Its also worth noting that CPU optimization is currently a build time feature in this dependency chain - as as a result is disable to meet the minimum CPU feature baseline in Debian, so the algorithms are not optimized in any way so won't be as performant as they should be on modern CPU architectures.

>> pyeclib <<

[Availability]
In universe and Debian

[Rationale]
New dependency for OpenStack Swift to support erasure coded pools of object data.

[Security]
No security history.

[Quality assurance]
Package builds OK, unit tests executed as part of package build

[Dependencies]
All in main or on this MIR request.

[Standards compliance]
OK

[Maintenance]
ubuntu-server/ubuntu-openstack

[Background information]
pyeclib provides a python wrap around liberasurecode - versions must be exactly matched

>> jerasure <<

[Availability]
In universe and Debian

[Rationale]
New dependency for OpenStack Swift to support erasure coded pools of object data.

[Security]
No security history.

[Quality assurance]
Package builds OK

[Dependencies]
All in main or on this MIR request.

[Standards compliance]
OK

[Maintenance]
ubuntu-server/ubuntu-openstack

>> gf-complete <<

[Availability]
In universe and Debian

[Rationale]
New dependency for OpenStack Swift to support erasure coded pools of object data.

[Security]
No security history.

[Quality assurance]
Package builds OK

[Dependencies]
All in main or on this MIR request.

[Standards compliance]
OK

[Maintenance]
ubuntu-server/ubuntu-openstack

>> liberasurecode <<

[Availability]
In universe and Debian

[Rationale]
New dependency for OpenStack Swift to support erasure coded pools of object data.

[Security]
No security history.

[Quality assurance]
Package builds OK, unit tests executed as part of package build.

[Dependencies]
All in main or on this MIR request.

[Standards compliance]
OK

[Maintenance]
ubuntu-server/ubuntu-openstack

James Page (james-page)
summary: - [MIR] python-pyeclib, liberasurecode, libjerasure, libgf-complete
+ [MIR] python-pyeclib, liberasurecode, jerasure, gf-complete
description: updated
description: updated
description: updated
James Page (james-page)
description: updated
description: updated
Changed in gf-complete (Ubuntu):
milestone: none → ubuntu-15.08
Changed in jerasure (Ubuntu):
milestone: none → ubuntu-15.08
Changed in liberasurecode (Ubuntu):
milestone: none → ubuntu-15.08
Changed in python-pyeclib (Ubuntu):
milestone: none → ubuntu-15.08
Changed in gf-complete (Ubuntu):
importance: Undecided → High
Changed in jerasure (Ubuntu):
importance: Undecided → High
Changed in liberasurecode (Ubuntu):
importance: Undecided → High
Changed in python-pyeclib (Ubuntu):
importance: Undecided → High
Revision history for this message
James Page (james-page) wrote :

Un-subbing mir team until this stuff actually works in a meaningful way.

Revision history for this message
James Page (james-page) wrote :

After resolution of the backend loading issues in liberasurecode, I think this is now ready for review.

description: updated
Revision history for this message
Matthias Klose (doko) wrote :
Download full text (6.2 KiB)

python-pyeclib ftbfs on powerpc:

======================================================================
ERROR: test_codes (test.test_pyeclib_c.TestPyECLib)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/«PKGBUILDDIR»/test/test_pyeclib_c.py", line 425, in test_codes
    size_str, self.iterations)
  File "/«PKGBUILDDIR»/test/test_pyeclib_c.py", line 183, in time_decode
    len(fragments[0]))
ECInvalidFragmentMetadata: pyeclib_c_decode ERROR: Fragment integrity check failed. Please inspect syslog for liberasurecode error report.

======================================================================
ERROR: test_xor_code (test.test_pyeclib_c.TestPyECLib)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/«PKGBUILDDIR»/test/test_pyeclib_c.py", line 313, in test_xor_code
    self.iterations)
  File "/«PKGBUILDDIR»/test/test_pyeclib_c.py", line 183, in time_decode
    len(fragments[0]))
ECInvalidFragmentMetadata: pyeclib_c_decode ERROR: Fragment integrity check failed. Please inspect syslog for liberasurecode error report.

======================================================================
ERROR: test_get_metadata_formatted (test.test_pyeclib_api.TestPyECLibDriver)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/«PKGBUILDDIR»/test/test_pyeclib_api.py", line 287, in test_get_metadata_formatted
    "inline_crc32")
  File "/«PKGBUILDDIR»/test/test_pyeclib_api.py", line 248, in check_metadata_formatted
    metadata = pyeclib_driver.get_metadata(fragment, 1)
  File "/«PKGBUILDDIR»/pyeclib/ec_iface.py", line 275, in get_metadata
    return self.ec_lib_reference.get_metadata(fragment, formatted)
  File "/«PKGBUILDDIR»/pyeclib/core.py", line 129, in get_metadata
    formatted)
ECInvalidParameter: pyeclib_c_get_metadata ERROR: Invalid arguments. Please inspect syslog for liberasurecode error report.

======================================================================
ERROR: test_liberasurecode_insufficient_frags_error (test.test_pyeclib_api.TestPyECLibDriver)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/«PKGBUILDDIR»/test/test_pyeclib_api.py", line 531, in test_liberasurecode_insufficient_frags_error
    [fragments[0]], [1, 2, 3, 4, 5, 6])
  File "/usr/lib/python2.7/unittest/case.py", line 473, in assertRaises
    callableObj(*args, **kwargs)
  File "/«PKGBUILDDIR»/pyeclib/ec_iface.py", line 238, in reconstruct
    available_fragment_payloads, missing_fragment_indexes)
  File "/«PKGBUILDDIR»/pyeclib/core.py", line 98, in reconstruct
    "Invalid fragment payload in ECPyECLibDriver.reconstruct")
ECDriverError: Invalid fragment payload in ECPyECLibDriver.reconstruct

======================================================================
ERROR: test_rs (test.test_pyeclib_api.TestPyECLibDriver)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/«PKGBUILDDIR»/test/test_pyecli...

Read more...

Changed in python-pyeclib (Ubuntu):
status: New → Incomplete
Revision history for this message
James Page (james-page) wrote :

powerpc tests now fixed.

Changed in python-pyeclib (Ubuntu):
status: Incomplete → New
Revision history for this message
Michael Terry (mterry) wrote :

Doko, I'll take your comment as interest in reviewing these. :) Let me know if you're too busy.

Changed in python-pyeclib (Ubuntu):
assignee: nobody → Matthias Klose (doko)
Revision history for this message
Matthias Klose (doko) wrote :

gf-incomplete:

 - tests are disabled. please enable
 - the home page says: "GF-Complete and Jerasure versions 2.0 and later are no longer supported."
 - really ship 1.0, not the current 2.0?

please also fix:

 - add Multi-Arch: same attributes to both binary packages
 - enable parallel build (--parallel)
 - update homepage in control: http://web.eecs.utk.edu/~plank/plank/www/software.html
   or http://jerasure.org/ ?

Changed in gf-complete (Ubuntu):
status: New → Incomplete
Revision history for this message
Matthias Klose (doko) wrote :

jerasure:

 - tests are disabled; please enable
 - please update to new home page / watch file

please also fix:

 - enable parallel build (--parallel)
 - add Multi-Arch: same attribute

looks fine otherwise.

Changed in jerasure (Ubuntu):
status: New → Incomplete
Revision history for this message
Matthias Klose (doko) wrote :

liberasurecode:

 - liberasurecode1 has a hard-coded dependency on a shared library, please remove
 - there is a newer 1.0.9 release, update?

please also fix:

 - enable parallel build
 - add Multi-Arch: same attributes
 - library is underlinked, -ldl missing

Changed in liberasurecode (Ubuntu):
status: New → Incomplete
Revision history for this message
Matthias Klose (doko) wrote :

python-pyeclib itself looks fine.

Changed in python-pyeclib (Ubuntu):
status: New → Triaged
Revision history for this message
Matthias Klose (doko) wrote :

liberasurecode:

 libjerasure2 is a plugin (supposed to be used as a default)

Changed in liberasurecode (Ubuntu):
status: Incomplete → Triaged
Revision history for this message
James Page (james-page) wrote :

jerasure and gf-complete have been updated inline with feedback above.

Deferring moving to new releases until upstream re-align on versioning and make things a bit more consistent.

Changed in gf-complete (Ubuntu):
status: Incomplete → New
Changed in jerasure (Ubuntu):
status: Incomplete → New
Revision history for this message
Matthias Klose (doko) wrote :

gf-complete and jerasure look ok now.

Override component to main
gf-complete 1.0.2-1ubuntu1 in wily: universe/misc -> main
gf-complete-tools 1.0.2-1ubuntu1 in wily amd64: universe/libs/extra/100% -> main
gf-complete-tools 1.0.2-1ubuntu1 in wily arm64: universe/libs/extra/100% -> main
gf-complete-tools 1.0.2-1ubuntu1 in wily armhf: universe/libs/extra/100% -> main
gf-complete-tools 1.0.2-1ubuntu1 in wily i386: universe/libs/extra/100% -> main
gf-complete-tools 1.0.2-1ubuntu1 in wily powerpc: universe/libs/extra/100% -> main
gf-complete-tools 1.0.2-1ubuntu1 in wily ppc64el: universe/libs/extra/100% -> main
libgf-complete-dev 1.0.2-1ubuntu1 in wily amd64: universe/libdevel/extra/100% -> main
libgf-complete-dev 1.0.2-1ubuntu1 in wily arm64: universe/libdevel/extra/100% -> main
libgf-complete-dev 1.0.2-1ubuntu1 in wily armhf: universe/libdevel/extra/100% -> main
libgf-complete-dev 1.0.2-1ubuntu1 in wily i386: universe/libdevel/extra/100% -> main
libgf-complete-dev 1.0.2-1ubuntu1 in wily powerpc: universe/libdevel/extra/100% -> main
libgf-complete-dev 1.0.2-1ubuntu1 in wily ppc64el: universe/libdevel/extra/100% -> main
libgf-complete1 1.0.2-1ubuntu1 in wily amd64: universe/libs/extra/100% -> main
libgf-complete1 1.0.2-1ubuntu1 in wily arm64: universe/libs/extra/100% -> main
libgf-complete1 1.0.2-1ubuntu1 in wily armhf: universe/libs/extra/100% -> main
libgf-complete1 1.0.2-1ubuntu1 in wily i386: universe/libs/extra/100% -> main
libgf-complete1 1.0.2-1ubuntu1 in wily powerpc: universe/libs/extra/100% -> main
libgf-complete1 1.0.2-1ubuntu1 in wily ppc64el: universe/libs/extra/100% -> main
19 publications overridden.

Override component to main
jerasure 2.0.0-2ubuntu1 in wily: universe/misc -> main
libjerasure-dev 2.0.0-2ubuntu1 in wily amd64: universe/libdevel/extra/100% -> main
libjerasure-dev 2.0.0-2ubuntu1 in wily arm64: universe/libdevel/extra/100% -> main
libjerasure-dev 2.0.0-2ubuntu1 in wily armhf: universe/libdevel/extra/100% -> main
libjerasure-dev 2.0.0-2ubuntu1 in wily i386: universe/libdevel/extra/100% -> main
libjerasure-dev 2.0.0-2ubuntu1 in wily powerpc: universe/libdevel/extra/100% -> main
libjerasure-dev 2.0.0-2ubuntu1 in wily ppc64el: universe/libdevel/extra/100% -> main
libjerasure2 2.0.0-2ubuntu1 in wily amd64: universe/libs/extra/100% -> main
libjerasure2 2.0.0-2ubuntu1 in wily arm64: universe/libs/extra/100% -> main
libjerasure2 2.0.0-2ubuntu1 in wily armhf: universe/libs/extra/100% -> main
libjerasure2 2.0.0-2ubuntu1 in wily i386: universe/libs/extra/100% -> main
libjerasure2 2.0.0-2ubuntu1 in wily powerpc: universe/libs/extra/100% -> main
libjerasure2 2.0.0-2ubuntu1 in wily ppc64el: universe/libs/extra/100% -> main
13 publications overridden.

Changed in gf-complete (Ubuntu):
status: New → Fix Released
Changed in jerasure (Ubuntu):
status: New → Fix Released
Revision history for this message
Matthias Klose (doko) wrote :

Override component to main
liberasurecode 1.0.8-3 in wily: universe/misc -> main
liberasurecode-dev 1.0.8-3 in wily amd64: universe/libdevel/extra/100% -> main
liberasurecode-dev 1.0.8-3 in wily arm64: universe/libdevel/extra/100% -> main
liberasurecode-dev 1.0.8-3 in wily armhf: universe/libdevel/extra/100% -> main
liberasurecode-dev 1.0.8-3 in wily i386: universe/libdevel/extra/100% -> main
liberasurecode-dev 1.0.8-3 in wily powerpc: universe/libdevel/extra/100% -> main
liberasurecode-dev 1.0.8-3 in wily ppc64el: universe/libdevel/extra/100% -> main
liberasurecode1 1.0.8-3 in wily amd64: universe/libs/extra/100% -> main
liberasurecode1 1.0.8-3 in wily arm64: universe/libs/extra/100% -> main
liberasurecode1 1.0.8-3 in wily armhf: universe/libs/extra/100% -> main
liberasurecode1 1.0.8-3 in wily i386: universe/libs/extra/100% -> main
liberasurecode1 1.0.8-3 in wily powerpc: universe/libs/extra/100% -> main
liberasurecode1 1.0.8-3 in wily ppc64el: universe/libs/extra/100% -> main
13 publications overridden.

Changed in liberasurecode (Ubuntu):
status: Triaged → Fix Released
Revision history for this message
Matthias Klose (doko) wrote :

Override component to main
python-pyeclib 1.0.8-3 in wily: universe/misc -> main
python-pyeclib 1.0.8-3 in wily amd64: universe/python/optional/100% -> main
python-pyeclib 1.0.8-3 in wily arm64: universe/python/optional/100% -> main
python-pyeclib 1.0.8-3 in wily armhf: universe/python/optional/100% -> main
python-pyeclib 1.0.8-3 in wily i386: universe/python/optional/100% -> main
python-pyeclib 1.0.8-3 in wily powerpc: universe/python/optional/100% -> main
python-pyeclib 1.0.8-3 in wily ppc64el: universe/python/optional/100% -> main
python3-pyeclib 1.0.8-3 in wily amd64: universe/python/optional/100% -> main
python3-pyeclib 1.0.8-3 in wily arm64: universe/python/optional/100% -> main
python3-pyeclib 1.0.8-3 in wily armhf: universe/python/optional/100% -> main
python3-pyeclib 1.0.8-3 in wily i386: universe/python/optional/100% -> main
python3-pyeclib 1.0.8-3 in wily powerpc: universe/python/optional/100% -> main
python3-pyeclib 1.0.8-3 in wily ppc64el: universe/python/optional/100% -> main
13 publications overridden.

Changed in python-pyeclib (Ubuntu):
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.