jmespath/netaddr SyntaxWarning: "is" with a literal when installing ansible.

Bug #1877792 reported by Edward Vielmetti
72
This bug affects 11 people
Affects Status Importance Assigned to Milestone
ansible (Ubuntu)
Invalid
Undecided
Unassigned
Bionic
Won't Fix
Undecided
Unassigned
Focal
Won't Fix
Undecided
Unassigned
Hirsute
Won't Fix
Undecided
Unassigned
Impish
Invalid
Undecided
Unassigned
python-jmespath (Debian)
Fix Released
Unknown
python-jmespath (Ubuntu)
Fix Released
Undecided
Unassigned
Bionic
Fix Released
Undecided
Eric Desrochers
Focal
Fix Released
Undecided
Eric Desrochers
Hirsute
Fix Released
Undecided
Unassigned
Impish
Fix Released
Undecided
Unassigned
python-netaddr (Ubuntu)
Confirmed
Undecided
Unassigned
Bionic
In Progress
Undecided
Unassigned
Focal
Fix Released
Undecided
Eric Desrochers
Hirsute
Confirmed
Undecided
Unassigned
Impish
Won't Fix
Undecided
Unassigned

Bug Description

[IMPACT]

* Allow broader Ops/Eng team to consume Ansible from our packages instead of the upstream PyPi repositories in order to fix this warning.

* Sanitize package installation:

Setting up python3-jmespath (0.9.4-2) ...
/usr/lib/python3/dist-packages/jmespath/visitor.py:32: SyntaxWarning: "is" with a literal. Did you mean "=="?

Setting up python3-netaddr (0.7.19-3) ...
/usr/lib/python3/dist-packages/netaddr/strategy/__init__.py:189: SyntaxWarning: "is not" with a literal. Did you mean "!="?
  if word_sep is not '':

jmespath and netaddr are 2 runtime dependencies of ansible.

The syntaxwarning while purely cosmetic, may mislead user and their trust in using a sane and tested package.

[TEST CASE]

* Install ansible with syntax fixed of netaddr and jmespath with py3.8
* Install ansible with syntax fixed of netaddr and jmespath with py2

It shouldn't report any syntaxwarning after both fixes.

Extra test case:

** Scenario #1 **

$ pull-lp-source python-jmespath

$ cd python-jmespath*

$ python3-coverage run jmespath/visitor.py
jmespath/visitor.py:32: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if x is 0 or x is 1:
jmespath/visitor.py:32: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if x is 0 or x is 1:
jmespath/visitor.py:34: SyntaxWarning: "is" with a literal. Did you mean "=="?
  elif y is 0 or y is 1:
jmespath/visitor.py:34: SyntaxWarning: "is" with a literal. Did you mean "=="?
  elif y is 0 or y is 1:
jmespath/visitor.py:260: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if original_result is 0:

$ quilt push -a
Applying patch 0001-satisfy-python38-syntaxwarning.patch
patching file jmespath/visitor.py

$ python3-coverage run jmespath/visitor.py

** Scenario #2 **

$ pull-lp-source python-jmespath

$ cd python-jmespath*

$ nosetests3
/tmp/jmes/f/python-jmespath-0.9.4/jmespath/visitor.py:32: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if x is 0 or x is 1:
/tmp/jmes/f/python-jmespath-0.9.4/jmespath/visitor.py:32: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if x is 0 or x is 1:
/tmp/jmes/f/python-jmespath-0.9.4/jmespath/visitor.py:34: SyntaxWarning: "is" with a literal. Did you mean "=="?
  elif y is 0 or y is 1:
/tmp/jmes/f/python-jmespath-0.9.4/jmespath/visitor.py:34: SyntaxWarning: "is" with a literal. Did you mean "=="?
  elif y is 0 or y is 1:
/tmp/jmes/f/python-jmespath-0.9.4/jmespath/visitor.py:260: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if original_result is 0:
.........................................
----------------------------------------------------------------------
Ran 41 tests in 0.039s

OK

$ quilt push -a
Applying patch 0001-satisfy-python38-syntaxwarning.patch
patching file jmespath/visitor.py

Now at patch 0001-satisfy-python38-syntaxwarning.patch

$ nosetests3
.........................................
----------------------------------------------------------------------
Ran 41 tests in 0.021s

OK

$ nosetests
.........................................
----------------------------------------------------------------------
Ran 41 tests in 0.016s

OK

** Scenario #3 **
This could be reproduced outside python-jmespath as follows (a little script I have written to compare result between py2 and py3 :

# reproducer_test.py
----
def reproducer(x,y):
    if x is 0 or x is 1:
        return y is True or y is False
    elif y is 0 or y is 1:
        return x is True or x is False

print(reproducer(1,0))
print(reproducer(0,1))
---

$ python2 reproducer_test.py
False
False

$ python3 -W ignore reproducer_test.py
False
False

$ python3 reproducer_test.py
test.py:2: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if x is 0 or x is 1:
test.py:2: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if x is 0 or x is 1:
test.py:4: SyntaxWarning: "is" with a literal. Did you mean "=="?
  elif y is 0 or y is 1:
test.py:4: SyntaxWarning: "is" with a literal. Did you mean "=="?
  elif y is 0 or y is 1:
False
False

With the fix in a small scale (Again, using my reproducer_test.py):

# reproducer_test.py
----
def reproducer(x,y):
# if x is 0 or x is 1:
     if type(x) is int and (x == 0 or x == 1):
        return y is True or y is False
# elif y is 0 or y is 1:
     elif type(y) is int and (y == 0 or y == 1):
        return x is True or x is False

print(reproducer(1,0))
print(reproducer(0,1))
---

# python2 reproducer_test.py
False
False

$ python3 reproducer_test.py
False
False

The above proves that it maintains compatibility in py2 while fixing the "SyntaxWarning "in py3.

[WHERE PROBLEM COULD OCCURS]

Risk: Low

* The fix maintain backward compatibility tested with python3-coverage/python-coverage, and python3-nose/python-nose manually and didn't report any problem.

This package lack of test suite and coverage for things that could have been easily avoided with proper test in place. There may or may not be other places in other code path generating syntax warning, as this is not well tested I'm afraid.

In general, looking upstream:

* No observed regression caused by this fix in upstream issues nor in the Ubuntu release where the fix has landed (Focal/Hirsute).

* No other py3.8 commits (either fix and/or regression fix) found in the project git log.

[OTHER INFORMATIONS]

Upstream issue:
https://github.com/jmespath/jmespath.py/issues/187

Upstream commit:
https://github.com/jmespath/jmespath.py/commit/56263b84cdb0feb7c8d54e426ec472f4dd0de44f

[ORIGINAL DESCRIPTIONS]
As reported upstream

https://github.com/jmespath/jmespath.py/issues/201
Problems installing jmespath in Ubuntu 20.04 LTS

Setting up python3-jmespath (0.9.4-2) ...
/usr/lib/python3/dist-packages/jmespath/visitor.py:32: SyntaxWarning: "is" with a literal. Did you mean "=="?

Fixed upstream in https://github.com/jmespath/jmespath.py/commit/56263b84cdb0feb7c8d54e426ec472f4dd0de44f

Impact: this causes applications like Ansible that have dependencies on python-jmespath to send out warnings. Caused by language changes in Python 3.8.

System is Ubuntu 20.04 LTS on a Pi 4.

emv@pinnatus:~$ apt-cache policy python3-jmespath
python3-jmespath:
  Installed: 0.9.4-2
  Candidate: 0.9.4-2
  Version table:
 *** 0.9.4-2 500
        500 http://ports.ubuntu.com/ubuntu-ports focal/main arm64 Packages
        100 /var/lib/dpkg/status

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in ansible (Ubuntu):
status: New → Confirmed
Changed in python-jmespath (Ubuntu):
status: New → Confirmed
Revision history for this message
Antti Palsola (antti.palsola) wrote :

I confirm this. Running any Ansible playbook displays the following warning on every task which is quite annoying:

/usr/lib/python3/dist-packages/jmespath/visitor.py:32: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if x is 0 or x is 1:
/usr/lib/python3/dist-packages/jmespath/visitor.py:32: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if x is 0 or x is 1:
/usr/lib/python3/dist-packages/jmespath/visitor.py:34: SyntaxWarning: "is" with a literal. Did you mean "=="?
  elif y is 0 or y is 1:
/usr/lib/python3/dist-packages/jmespath/visitor.py:34: SyntaxWarning: "is" with a literal. Did you mean "=="?
  elif y is 0 or y is 1:
/usr/lib/python3/dist-packages/jmespath/visitor.py:260: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if original_result is 0:

The system is Ubuntu 20.04 LTS on amd64.

$ apt-cache policy ansible python3-jmespath
ansible:
  Installed: 2.9.6+dfsg-1
  Candidate: 2.9.6+dfsg-1
  Version table:
 *** 2.9.6+dfsg-1 500
        500 http://www.nic.funet.fi/pub/mirrors/archive.ubuntu.com focal/universe amd64 Packages
        500 http://www.nic.funet.fi/pub/mirrors/archive.ubuntu.com focal/universe i386 Packages
        100 /var/lib/dpkg/status
python3-jmespath:
  Installed: 0.9.4-2
  Candidate: 0.9.4-2
  Version table:
 *** 0.9.4-2 500
        500 http://www.nic.funet.fi/pub/mirrors/archive.ubuntu.com focal/main amd64 Packages
        500 http://www.nic.funet.fi/pub/mirrors/archive.ubuntu.com focal/main i386 Packages
        100 /var/lib/dpkg/status

Revision history for this message
Edward Vielmetti (edward-vielmetti) wrote :

It looks like https://launchpad.net/ubuntu/+source/python-jmespath/0.9.5-1 addresses this issue, from Groovy and Sid.

Revision history for this message
Kai Kasurinen (kai-kasurinen) wrote :

python-jmespath (0.9.5-1) unstable; urgency=medium

  [ TANIGUCHI Takaki ]
  * add CI config
  * New upstream version 0.9.5
    - Fix SyntaxWarning (Closes: #953534)
  * Set upstream metadata fields: Bug-Database, Bug-Submit, Repository,
    Repository-Browse.
  * Update standards version to 4.4.1, no changes needed.

 -- TANIGUCHI Takaki <email address hidden> Wed, 01 Apr 2020 16:35:10 +0900

Changed in ansible (Ubuntu):
status: Confirmed → Invalid
Changed in python-jmespath (Ubuntu):
status: Confirmed → Fix Released
Changed in python-jmespath (Debian):
status: Unknown → Fix Released
Revision history for this message
Gilbert Standen (gstanden) wrote :

Hi, I am also seeing this error when running the following install in a Ubuntu 20.04 focal fossa on amd64 running in an LXC Linux Container:

The command run is:

COMMAND=/bin/lxc-attach -n nsa -- sudo apt-get -y install bind9 isc-dhcp-server bind9utils dnsutils openssh-server man awscli sshpass

During installation of those package, the following output is observed:

Setting up python3-jmespath (0.9.4-2) ...
/usr/lib/python3/dist-packages/jmespath/visitor.py:32: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if x is 0 or x is 1:
/usr/lib/python3/dist-packages/jmespath/visitor.py:32: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if x is 0 or x is 1:
/usr/lib/python3/dist-packages/jmespath/visitor.py:34: SyntaxWarning: "is" with a literal. Did you mean "=="?
  elif y is 0 or y is 1:
/usr/lib/python3/dist-packages/jmespath/visitor.py:34: SyntaxWarning: "is" with a literal. Did you mean "=="?
  elif y is 0 or y is 1:
/usr/lib/python3/dist-packages/jmespath/visitor.py:260: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if original_result is 0:

The warnings do not seem to affect operation of the system or the installation of the packages in any observed way so far.

Revision history for this message
Gilbert Standen (gstanden) wrote :

Version information for python-jmespath in the Ubuntu 20.04 LXC Linux container:

root@afns1:/# dpkg -l | grep jmespath
ii python3-jmespath 0.9.4-2 all JSON Matching Expressions (Python 3)

Eric Desrochers (slashd)
Changed in python-jmespath (Ubuntu Hirsute):
status: New → Fix Released
Eric Desrochers (slashd)
description: updated
description: updated
description: updated
tags: added: seg sts
Changed in python-jmespath (Ubuntu Focal):
status: New → Confirmed
Changed in python-jmespath (Ubuntu Bionic):
status: New → Confirmed
Eric Desrochers (slashd)
description: updated
description: updated
description: updated
description: updated
Changed in ansible (Ubuntu Bionic):
status: New → Won't Fix
Changed in ansible (Ubuntu Focal):
status: New → Won't Fix
Changed in ansible (Ubuntu Hirsute):
status: New → Won't Fix
Eric Desrochers (slashd)
description: updated
Eric Desrochers (slashd)
description: updated
Eric Desrochers (slashd)
description: updated
Eric Desrochers (slashd)
description: updated
description: updated
Eric Desrochers (slashd)
Changed in python-jmespath (Ubuntu Focal):
assignee: nobody → Eric Desrochers (slashd)
status: Confirmed → In Progress
Changed in python-jmespath (Ubuntu Bionic):
assignee: nobody → Eric Desrochers (slashd)
status: Confirmed → In Progress
tags: added: sts-sponsors-halves
Revision history for this message
Eric Desrochers (slashd) wrote :

Bionic debdiff

Revision history for this message
Eric Desrochers (slashd) wrote :

Focal debdiff

Revision history for this message
Łukasz Zemczak (sil2100) wrote :

While reviewing this SRU I'm trying to measure the severity of the current situation. Since this seems to be a warning, does it actually impact users in any way besides printing the warning? Are there any consequences of it? Are any of the reverse-dependencies impacted, requiring this fix?

Eric Desrochers (slashd)
Changed in python-netaddr (Ubuntu Focal):
assignee: nobody → Eric Desrochers (slashd)
status: New → In Progress
Changed in python-netaddr (Ubuntu Bionic):
assignee: nobody → Eric Desrochers (slashd)
status: New → In Progress
summary: - jmespath SyntaxWarning: "is" with a literal.
+ jmespath/netaddr SyntaxWarning: "is" with a literal when installing
+ ansible.
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in python-netaddr (Ubuntu Hirsute):
status: New → Confirmed
Changed in python-netaddr (Ubuntu Impish):
status: New → Confirmed
Changed in python-netaddr (Ubuntu):
status: New → Confirmed
Eric Desrochers (slashd)
description: updated
description: updated
tags: added: patch
Revision history for this message
Eric Desrochers (slashd) wrote :

To answer #10 from Lukasz.

python-netaddr so as python-jmespath (both ansible runtime dependencies) spew syntaxwarning at ansible installation time (or their respective package).

It is purely cosmetic, but we hope fixing these bug will make user gain confidence in the Ansible package found in the Ubuntu archive, instead of users diverting by using upstream ansible packages from PyPi or other third-party repositories.

Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Okay, thanks for the explanation! Normally these kind of cosmetic bugs are not really SRU material by themselves - the SRU policy is quite strict about what kind of fixes are to be backported to stable series. That being said, we can look into special circumstances like this one here.

With both the packages being in main and never before getting any SRU updates, I think we can do a one-off SRU for such a low priority bug. I think it's important for users to have confidence in our packages, and if such warnings make them switch from our official repositories to pip, I think our LTSes would merit from such a fix.

Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello Edward, or anyone else affected,

Accepted python-jmespath into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-jmespath/0.9.4-2ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in python-jmespath (Ubuntu Focal):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-focal
Changed in python-jmespath (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed-bionic
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Hello Edward, or anyone else affected,

Accepted python-jmespath into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-jmespath/0.9.3-1ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Hello Edward, or anyone else affected,

Accepted python-netaddr into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-netaddr/0.7.19-3ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in python-netaddr (Ubuntu Focal):
status: In Progress → Fix Committed
Eric Desrochers (slashd)
Changed in python-netaddr (Ubuntu Bionic):
assignee: Eric Desrochers (slashd) → nobody
Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (python-jmespath/0.9.4-2ubuntu1)

All autopkgtests for the newly accepted python-jmespath (0.9.4-2ubuntu1) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

azure-cli/2.0.81+ds-4ubuntu0.2 (s390x, amd64, arm64, ppc64el, armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#python-jmespath

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (python-netaddr/0.7.19-3ubuntu1)

All autopkgtests for the newly accepted python-netaddr (0.7.19-3ubuntu1) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

horizon/3:18.3.4-0ubuntu1 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#python-netaddr

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Eric Desrochers (slashd) wrote :

The regressions observed in azure-cli for Focal/20.04LTS have nothing to do with this particular SRU change.

I have reported a separate bug for this:
https://bugs.launchpad.net/ubuntu/+source/azure-cli/+bug/1948595

One unit test is failing as it misses an argument (--subscription) in "az cloud list" cli.
From my preliminary research, it seems like this argument isn't globally exposed.

- Eric

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-jmespath - 0.9.4-2ubuntu1

---------------
python-jmespath (0.9.4-2ubuntu1) focal; urgency=medium

  * d/p/0001-satisfy-python38-syntaxwarning.patch:
    - Python 3.8 raises a syntax warning when using
      is to compare integers or string literals.
      (LP: #1877792)

 -- Eric Desrochers <email address hidden> Fri, 08 Oct 2021 08:56:43 -0400

Changed in python-jmespath (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-jmespath - 0.9.3-1ubuntu1.1

---------------
python-jmespath (0.9.3-1ubuntu1.1) bionic; urgency=medium

  * d/p/0001-satisfy-python38-syntaxwarning.patch:
    - Python 3.8 raises a syntax warning when using
      is to compare integers or string literals.
      (LP: #1877792)

 -- Eric Desrochers <email address hidden> Fri, 08 Oct 2021 08:50:18 -0400

Changed in python-jmespath (Ubuntu Bionic):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-netaddr - 0.7.19-3ubuntu1

---------------
python-netaddr (0.7.19-3ubuntu1) focal; urgency=medium

  * d/p/fix-python-38-syntaxwarning.patch: (LP: #1877792)
    - Fix Python 3.8 SyntaxWarning on using `is not` with a string
    literal.

 -- Eric Desrochers <email address hidden> Mon, 18 Oct 2021 11:25:44 -0400

Changed in python-netaddr (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Eric Desrochers (slashd) wrote :

[VERIFICATION]

The ansible (and the fixed packages listed in the bugs) installation has been tested by impacted users and the outcomes were as expected.

- Eric

tags: added: verification-done-bionic verification-done-focal
removed: verification-needed-bionic verification-needed-focal
Revision history for this message
Rolf Leggewie (r0lf) wrote :

Awesome, thank you very much, @slashd!

Revision history for this message
Brian Murray (brian-murray) wrote :

Ubuntu 21.10 (Impish Indri) has reached end of life, so this bug will not be fixed for that specific release.

Changed in python-netaddr (Ubuntu Impish):
status: Confirmed → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.