Cinder

recompiled pycparser 2.14 breaks Cinder db sync and Nova UTs

Bug #1629726 reported by Matt Riedemann
48
This bug affects 9 people
Affects Status Importance Assigned to Milestone
Cinder
Invalid
Undecided
Unassigned
Designate
Fix Released
Critical
Unassigned
Ironic
Fix Released
Critical
Unassigned
Magnum
Fix Released
Critical
Unassigned
OpenStack Heat
Fix Released
Undecided
Unassigned
OpenStack Shared File Systems Service (Manila)
Fix Released
Critical
Unassigned
OpenStack-Ansible
Fix Released
Undecided
Unassigned
Rally
Fix Released
Undecided
monika
ec2-api
Fix Released
Critical
Unassigned
gce-api
Fix Released
Critical
Unassigned
networking-cisco
Fix Released
Critical
Unassigned

Bug Description

http://logs.openstack.org/76/380876/1/check/gate-grenade-dsvm-ubuntu-xenial/3d5e102/logs/grenade.sh.txt.gz#_2016-10-02_23_32_34_069

2016-10-02 23:32:34.069 | + lib/cinder:init_cinder:421 : /usr/local/bin/cinder-manage --config-file /etc/cinder/cinder.conf db sync
2016-10-02 23:32:34.691 | Traceback (most recent call last):
2016-10-02 23:32:34.691 | File "/usr/local/bin/cinder-manage", line 6, in <module>
2016-10-02 23:32:34.691 | from cinder.cmd.manage import main
2016-10-02 23:32:34.691 | File "/opt/stack/old/cinder/cinder/cmd/manage.py", line 77, in <module>
2016-10-02 23:32:34.691 | from cinder import db
2016-10-02 23:32:34.691 | File "/opt/stack/old/cinder/cinder/db/__init__.py", line 20, in <module>
2016-10-02 23:32:34.691 | from cinder.db.api import * # noqa
2016-10-02 23:32:34.691 | File "/opt/stack/old/cinder/cinder/db/api.py", line 43, in <module>
2016-10-02 23:32:34.691 | from cinder.api import common
2016-10-02 23:32:34.691 | File "/opt/stack/old/cinder/cinder/api/common.py", line 30, in <module>
2016-10-02 23:32:34.691 | from cinder import utils
2016-10-02 23:32:34.691 | File "/opt/stack/old/cinder/cinder/utils.py", line 40, in <module>
2016-10-02 23:32:34.691 | from os_brick import encryptors
2016-10-02 23:32:34.691 | File "/usr/local/lib/python2.7/dist-packages/os_brick/encryptors/__init__.py", line 16, in <module>
2016-10-02 23:32:34.691 | from os_brick.encryptors import nop
2016-10-02 23:32:34.691 | File "/usr/local/lib/python2.7/dist-packages/os_brick/encryptors/nop.py", line 16, in <module>
2016-10-02 23:32:34.691 | from os_brick.encryptors import base
2016-10-02 23:32:34.691 | File "/usr/local/lib/python2.7/dist-packages/os_brick/encryptors/base.py", line 19, in <module>
2016-10-02 23:32:34.691 | from os_brick import executor
2016-10-02 23:32:34.691 | File "/usr/local/lib/python2.7/dist-packages/os_brick/executor.py", line 21, in <module>
2016-10-02 23:32:34.691 | from os_brick.privileged import rootwrap as priv_rootwrap
2016-10-02 23:32:34.691 | File "/usr/local/lib/python2.7/dist-packages/os_brick/privileged/__init__.py", line 13, in <module>
2016-10-02 23:32:34.691 | from oslo_privsep import capabilities as c
2016-10-02 23:32:34.691 | File "/usr/local/lib/python2.7/dist-packages/oslo_privsep/capabilities.py", line 73, in <module>
2016-10-02 23:32:34.691 | ffi.cdef(CDEF)
2016-10-02 23:32:34.692 | File "/usr/local/lib/python2.7/dist-packages/cffi/api.py", line 105, in cdef
2016-10-02 23:32:34.692 | self._cdef(csource, override=override, packed=packed)
2016-10-02 23:32:34.692 | File "/usr/local/lib/python2.7/dist-packages/cffi/api.py", line 119, in _cdef
2016-10-02 23:32:34.692 | self._parser.parse(csource, override=override, **options)
2016-10-02 23:32:34.692 | File "/usr/local/lib/python2.7/dist-packages/cffi/cparser.py", line 299, in parse
2016-10-02 23:32:34.692 | self._internal_parse(csource)
2016-10-02 23:32:34.692 | File "/usr/local/lib/python2.7/dist-packages/cffi/cparser.py", line 304, in _internal_parse
2016-10-02 23:32:34.692 | ast, macros, csource = self._parse(csource)
2016-10-02 23:32:34.692 | File "/usr/local/lib/python2.7/dist-packages/cffi/cparser.py", line 260, in _parse
2016-10-02 23:32:34.692 | ast = _get_parser().parse(csource)
2016-10-02 23:32:34.692 | File "/usr/local/lib/python2.7/dist-packages/cffi/cparser.py", line 40, in _get_parser
2016-10-02 23:32:34.692 | _parser_cache = pycparser.CParser()
2016-10-02 23:32:34.692 | File "/usr/local/lib/python2.7/dist-packages/pycparser/c_parser.py", line 87, in __init__
2016-10-02 23:32:34.692 | outputdir=taboutputdir)
2016-10-02 23:32:34.692 | File "/usr/local/lib/python2.7/dist-packages/pycparser/c_lexer.py", line 66, in build
2016-10-02 23:32:34.692 | self.lexer = lex.lex(object=self, **kwargs)
2016-10-02 23:32:34.692 | File "/usr/local/lib/python2.7/dist-packages/pycparser/ply/lex.py", line 911, in lex
2016-10-02 23:32:34.692 | lexobj.readtab(lextab, ldict)
2016-10-02 23:32:34.692 | File "/usr/local/lib/python2.7/dist-packages/pycparser/ply/lex.py", line 233, in readtab
2016-10-02 23:32:34.692 | titem.append((re.compile(pat, lextab._lexreflags | re.VERBOSE), _names_to_funcs(func_name, fdict)))
2016-10-02 23:32:34.692 | File "/usr/lib/python2.7/re.py", line 194, in compile
2016-10-02 23:32:34.692 | return _compile(pattern, flags)
2016-10-02 23:32:34.692 | File "/usr/lib/python2.7/re.py", line 249, in _compile
2016-10-02 23:32:34.692 | p = sre_compile.compile(pattern, flags)
2016-10-02 23:32:34.692 | File "/usr/lib/python2.7/sre_compile.py", line 583, in compile
2016-10-02 23:32:34.692 | "sorry, but this version only supports 100 named groups"
2016-10-02 23:32:34.692 | AssertionError: sorry, but this version only supports 100 named groups

Looks like this is due to a newly recompiled version of pycparser 2.14 as of Oct 2nd.

The upstream issue is:

https://github.com/eliben/pycparser/issues/147

Matt Riedemann (mriedem)
summary: - 2016-10-02 23:32:34.069 | + lib/cinder:init_cinder:421 :
- /usr/local/bin/cinder-manage --config-file /etc/cinder/cinder.conf db
- sync 2016-10-02 23:32:34.691 | Traceback (most recent call last):
- 2016-10-02 23:32:34.691 | File "/usr/local/bin/cinder-manage", line 6,
- in <module> 2016-10-02 23:32:34.691 | from cinder.cmd.manage import
- main 2016-10-02 23:32:34.691 | File
- "/opt/stack/old/cinder/cinder/cmd/manage.py", line 77, in <module>
- 2016-10-02 23:32:34.691 | from cinder import db 2016-10-02
- 23:32:34.691 | File "/opt/stack/old/cinder/cinder/db/__init__.py",
- line 20, in <module> 2016-10-02 23:32:34.691 | from cinder.db.api
- import * # noqa 2016-10-02 23:32:34.691 | File
- "/opt/stack/old/cinder/cinder/db/api.py", line 43, in <module>
- 2016-10-02 23:32:34.691 | from cinder.api import common 2016-10-02
- 23:32:34.691 | File "/opt/stack/old/cinder/cinder/api/common.py", line
- 30, in <module> 2016-10-02 23:32:34.691 | from cinder import utils
- 2016-10-02 23:32:34.691 | File
- "/opt/stack/old/cinder/cinder/utils.py", line 40, in <module> 2016-10-02
- 23:32:34.691 | from os_brick import encryptors 2016-10-02
- 23:32:34.691 | File "/usr/local/lib/python2.7/dist-
- packages/os_brick/encryptors/__init__.py", line 16, in <module>
- 2016-10-02 23:32:34.691 | from os_brick.encryptors import nop
- 2016-10-02 23:32:34.691 | File "/usr/local/lib/python2.7/dist-
- packages/os_brick/encryptors/nop.py", line 16, in <module> 2016-10-02
- 23:32:34.691 | from os_brick.encryptors import base 2016-10-02
- 23:32:34.691 | File "/usr/local/lib/python2.7/dist-
- packages/os_brick/encryptors/base.py", line 19, in <module> 2016-10-02
- 23:32:34.691 | from os_brick import executor 2016-10-02 23:32:34.691
- | File "/usr/local/lib/python2.7/dist-packages/os_brick/executor.py",
- line 21, in <module> 2016-10-02 23:32:34.691 | from
- os_brick.privileged import rootwrap as priv_rootwrap 2016-10-02
- 23:32:34.691 | File "/usr/local/lib/python2.7/dist-
- packages/os_brick/privileged/__init__.py", line 13, in <module>
- 2016-10-02 23:32:34.691 | from oslo_privsep import capabilities as c
- 2016-10-02 23:32:34.691 | File "/usr/local/lib/python2.7/dist-
- packages/oslo_privsep/capabilities.py", line 73, in <module> 2016-10-02
- 23:32:34.691 | ffi.cdef(CDEF) 2016-10-02 23:32:34.692 | File
- "/usr/local/lib/python2.7/dist-packages/cffi/api.py", line 105, in cdef
- 2016-10-02 23:32:34.692 | self._cdef(csource, override=override,
- packed=packed) 2016-10-02 23:32:34.692 | File
- "/usr/local/lib/python2.7/dist-packages/cffi/api.py", line 119, in _cdef
- 2016-10-02 23:32:34.692 | self._parser.parse(csource,
- override=override, **options) 2016-10-02 23:32:34.692 | File
- "/usr/local/lib/python2.7/dist-packages/cffi/cparser.py", line 299, in
- parse 2016-10-02 23:32:34.692 | self._internal_parse(csource)
- 2016-10-02 23:32:34.692 | File "/usr/local/lib/python2.7/dist-
- packages/cffi/cparser.py", line 304, in _internal_parse 2016-10-02
- 23:32:34.692 | ast, macros, csource = self._parse(csource)
- 2016-10-02 23:32:34.692 | File "/usr/local/lib/python2.7/dist-
- packages/cffi/cparser.py", line 260, in _parse 2016-10-02 23:32:34.692 |
- ast = _get_parser().parse(csource) 2016-10-02 23:32:34.692 | File
- "/usr/local/lib/python2.7/dist-packages/cffi/cparser.py", line 40, in
- _get_parser 2016-10-02 23:32:34.692 | _parser_cache =
- pycparser.CParser() 2016-10-02 23:32:34.692 | File
- "/usr/local/lib/python2.7/dist-packages/pycparser/c_parser.py", line 87,
- in __init__ 2016-10-02 23:32:34.692 | outputdir=taboutputdir)
- 2016-10-02 23:32:34.692 | File "/usr/local/lib/python2.7/dist-
- packages/pycparser/c_lexer.py", line 66, in build 2016-10-02
- 23:32:34.692 | self.lexer = lex.lex(object=self, **kwargs)
- 2016-10-02 23:32:34.692 | File "/usr/local/lib/python2.7/dist-
- packages/pycparser/ply/lex.py", line 911, in lex 2016-10-02 23:32:34.692
- | lexobj.readtab(lextab, ldict) 2016-10-02 23:32:34.692 | File
- "/usr/local/lib/python2.7/dist-packages/pycparser/ply/lex.py", line 233,
- in readtab 2016-10-02 23:32:34.692 | titem.append((re.compile(pat,
- lextab._lexreflags | re.VERBOSE), _names_to_funcs(func_name, fdict)))
- 2016-10-02 23:32:34.692 | File "/usr/lib/python2.7/re.py", line 194,
- in compile 2016-10-02 23:32:34.692 | return _compile(pattern, flags)
- 2016-10-02 23:32:34.692 | File "/usr/lib/python2.7/re.py", line 249,
- in _compile 2016-10-02 23:32:34.692 | p =
- sre_compile.compile(pattern, flags) 2016-10-02 23:32:34.692 | File
- "/usr/lib/python2.7/sre_compile.py", line 583, in compile 2016-10-02
- 23:32:34.692 | "sorry, but this version only supports 100 named
- groups" 2016-10-02 23:32:34.692 | AssertionError: sorry, but this
- version only supports 100 named groups
+ cinder-manage db sync fails with recompiled pycparser 2.14
Changed in cinder:
status: New → Confirmed
Dmitry Tantsur (divius)
Changed in ironic:
status: New → Confirmed
importance: Undecided → Critical
Revision history for this message
Sylvain Bauza (sylvain-bauza) wrote :

Most of our gate jobs are impacted (Nova UTs, Tempest tests, Grenade checks) by a new respin of pycparser that raises an AssertionError when imported.

Note that not only Nova but a lot of OpenStack projects are impacted :
http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%20\%22AssertionError:%20sorry,%20but%20this%20version%20only%20supports%20100%20named%20groups\%22&from=24h

summary: - cinder-manage db sync fails with recompiled pycparser 2.14
+ recompiled pycparser 2.14 breaks Cinder db sync and Nova UTs
Changed in nova:
status: New → Confirmed
importance: Undecided → Critical
Revision history for this message
Sylvain Bauza (sylvain-bauza) wrote :

https://review.openstack.org/#/c/381031/ first decouples Swift from testing upper-constraints modifications as it prevents us to land https://review.openstack.org/#/c/381018/1 because Swift doesn't actually use upper-requirements.txt :-)

Rabi Mishra (rabi)
Changed in heat:
status: New → Confirmed
Amrith Kumar (amrith)
Changed in trove:
importance: Undecided → Critical
status: New → Confirmed
Tom Barron (tpb)
Changed in manila:
status: New → Confirmed
Revision history for this message
Sylvain Bauza (sylvain-bauza) wrote :

Now that https://review.openstack.org/#/c/381031/ is merged, we can discuss on how to modify upper-constraints for not using the current pycparser wheel version. Discussion in #openstack-requirements

Graham Hayes (grahamhayes)
Changed in designate:
status: New → Confirmed
importance: Undecided → Critical
Oleksiy Butenko (obutenko)
Changed in rally:
status: New → Confirmed
Andrey Pavlov (apavlov-e)
Changed in ec2-api:
importance: Undecided → Critical
status: New → Confirmed
Changed in gce-api:
importance: Undecided → Critical
status: New → Confirmed
Revision history for this message
Michal Ptacek (michalx-ptacek) wrote :

I successfully workarounded the problem by using older version of pycparser

2:51:32.203 2016-10-03 11:51:32.188 | + :pre_test_hook:L0: sudo sed -i.bak s/pycparser===2.14/pycparser===2.13/g /opt/stack/new/requirements/upper-constraints.txt
12:51:32.204 2016-10-03 11:51:32.190 | + :pre_test_hook:L1: sudo sed -i.bak2 s/cffi===1.8.3/cffi===1.8.2/g /opt/stack/new/requirements/upper-constraints.txt

Spyros Trigazis (strigazi)
Changed in magnum:
importance: Undecided → Critical
status: New → Confirmed
Sam Betts (sambetts)
Changed in networking-cisco:
importance: Undecided → Critical
status: New → Confirmed
Revision history for this message
Matt Riedemann (mriedem) wrote :

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22AssertionError%3A%20sorry%2C%20but%20this%20version%20only%20supports%20100%20named%20groups%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d

Revision history for this message
Sam Betts (sambetts) wrote :

Seems like upper-constraints patch has been merged to avoid this issue until its repaired: https://review.openstack.org/#/c/381011/

Revision history for this message
Ihar Hrachyshka (ihar-hrachyshka) wrote :

Should be fixed now.

Amrith Kumar (amrith)
Changed in trove:
status: Confirmed → Fix Committed
Dmitry Tantsur (divius)
Changed in ironic:
status: Confirmed → Fix Released
Sean McGinnis (sean-mcginnis)
Changed in cinder:
status: Confirmed → Invalid
Andrey Pavlov (apavlov-e)
Changed in ec2-api:
status: Confirmed → Fix Released
Changed in gce-api:
status: Confirmed → Fix Released
Graham Hayes (grahamhayes)
Changed in designate:
status: Confirmed → Fix Released
Revision history for this message
Matt Riedemann (mriedem) wrote :

Yes this should be fixed now upstream. I've reverted the block on 2.14 in upper-constraints on master.

no longer affects: nova
Goutham Pacha Ravi (gouthamr)
Changed in manila:
importance: Undecided → Critical
Revision history for this message
Jean-Philippe Evrard (jean-philippe-evrard) wrote :

Fix released for openstack-ansible on https://review.openstack.org/#/c/381007/

Changed in openstack-ansible:
status: New → Fix Released
Rabi Mishra (rabi)
Changed in heat:
status: Confirmed → Fix Released
Spyros Trigazis (strigazi)
Changed in magnum:
status: Confirmed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to rally (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/385360

Amrith Kumar (amrith)
no longer affects: trove
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to rally (master)

Reviewed: https://review.openstack.org/385360
Committed: https://git.openstack.org/cgit/openstack/rally/commit/?id=31a8f05a650d5268eb1537e15b1c71a3b090c241
Submitter: Jenkins
Branch: master

commit 31a8f05a650d5268eb1537e15b1c71a3b090c241
Author: Jesse Pretorius <email address hidden>
Date: Wed Oct 12 10:21:01 2016 +0100

    Remove pycparser restriction from requirements

    The assertion error mentioned to justify the restriction is
    not actually due to this version of the python package, but
    instead due to an improperly prepared wheel which was
    uploaded to pypi. This wheel has now been removed and the
    package works properly.

    Reference: https://github.com/eliben/pycparser/issues/147

    This patch removes the restriction so that Rally's
    requirements are back in line with the OpenStack upper
    constraint for pycparser.

    Change-Id: I66ef46ad4c1511f39817270f59c7c1511645f2eb
    Related-Bug: #1629830
    Related-Bug: #1629726

monika (monikaparkar25)
Changed in manila:
assignee: nobody → monika (monikaparkar25)
Changed in rally:
assignee: nobody → monika (monikaparkar25)
Revision history for this message
Jeremy Liu (liujiong) wrote :

The bug repeated in pycparser 2.15

Revision history for this message
Jay Faulkner (jason-oldos) wrote :

Just saw this breakage again in the Ironic gate: http://logs.openstack.org/57/377957/10/gate/gate-tempest-dsvm-ironic-multitenant-network/0649016/logs/devstacklog.txt.gz#_2016-10-18_15_18_55_545

Changed in ironic:
status: Fix Released → Confirmed
Revision history for this message
Dmitry Tantsur (divius) wrote :

Seems fixed again?

Changed in ironic:
status: Confirmed → Fix Released
Andriy Kurilin (andreykurilin)
Changed in rally:
status: Confirmed → Fix Released
Rajat Sharma (tajar29)
Changed in networking-cisco:
assignee: nobody → Rajat Sharma (tajar29)
Rajat Sharma (tajar29)
Changed in networking-cisco:
assignee: Rajat Sharma (tajar29) → nobody
Sam Betts (sambetts)
Changed in networking-cisco:
status: Confirmed → Fix Released
Goutham Pacha Ravi (gouthamr)
Changed in manila:
assignee: monika (monikaparkar25) → nobody
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.