[SRU] Designate DNS – Secondary zone is failed to be created and gets into the ERROR status

Bug #1941988 reported by Arkady Shtempler
30
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Designate
Fix Released
High
Unassigned
OpenStack Designate Charm
Invalid
Undecided
Unassigned
dnspython (Ubuntu)
Fix Released
Undecided
Unassigned
Focal
Fix Released
Medium
Brett Milford
dnspython3 (Ubuntu)
Invalid
Undecided
Unassigned
Focal
Invalid
Undecided
Unassigned

Bug Description

[Impact]

* Designate logs stack traces as per the original description:

  dns.query.xfr() w/ timeout=not None and lifetime=None (default):
  `TypeError: '>' not supported between instances of 'float' and 'NoneType'`

* Backport will improve the robustness of Designate on Ubuntu.

[Test Case]

 * Synthetic reproducer for dnspython alone in comment #11.

 * See below for OpenStack Designate included:

### Scenario ###
1) Start remote BIND
Use some external host as a DNS server to tor AXFR the Zone from.
Note: BIND9 on RHEL is provided in attached doc.

2) From your setup check that AXFR is possible
dig @10.9.95.132 debuntu.foo axfr

3) Create a Secondary zone with:
openstack zone create debuntu.foo. --type SECONDARY --master 10.9.95.132

### Expected result ###
Secondary zone is created and ACTIVE

### Actual Result ###
Secondary zone gets into the ERROR statuses + Error in log

journalctl -f -u devstack@designate-mdns
Aug 25 12:56:43 seal08 designate-mdns[116065]: INFO designate.dnsutils [None req-0fcfe0f2-b8cb-4eec-ae42-9ea88e089c69 None None] Doing AXFR for debuntu.foo. from {'zone_id': 'ccadec69-4723-4c24-987f-7cb1587cff2c', 'host': '10.35.64.8', 'port': 53, 'id': 'ba2932b2-3f81-4ad5-985c-86ce081fe62d', 'created_at': datetime.datetime(2021, 8, 25, 9, 56, 44), 'updated_at': None, 'version': 1}
Aug 25 12:56:43 seal08 designate-mdns[116065]: ERROR designate.dnsutils [None req-0fcfe0f2-b8cb-4eec-ae42-9ea88e089c69 None None] Problem doing AXFR debuntu.foo. from {'zone_id': 'ccadec69-4723-4c24-987f-7cb1587cff2c', 'host': '10.35.64.8', 'port': 53, 'id': 'ba2932b2-3f81-4ad5-985c-86ce081fe62d', 'created_at': datetime.datetime(2021, 8, 25, 9, 56, 44), 'updated_at': None, 'version': 1}. Trying next server.: TypeError: '>' not supported between instances of 'float' and 'NoneType'
Aug 25 12:56:43 seal08 designate-mdns[116065]: ERROR designate.dnsutils Traceback (most recent call last):
Aug 25 12:56:43 seal08 designate-mdns[116065]: ERROR designate.dnsutils   File "/opt/stack/designate/designate/dnsutils.py", line 358, in do_axfr
Aug 25 12:56:43 seal08 designate-mdns[116065]: ERROR designate.dnsutils     raw_zone = dns.zone.from_xfr(xfr, relativize=False)
Aug 25 12:56:43 seal08 designate-mdns[116065]: ERROR designate.dnsutils   File "/usr/local/lib/python3.8/dist-packages/dns/zone.py", line 1106, in from_xfr
Aug 25 12:56:43 seal08 designate-mdns[116065]: ERROR designate.dnsutils     for r in xfr:
Aug 25 12:56:43 seal08 designate-mdns[116065]: ERROR designate.dnsutils   File "/usr/local/lib/python3.8/dist-packages/dns/query.py", line 611, in xfr
Aug 25 12:56:43 seal08 designate-mdns[116065]: ERROR designate.dnsutils     if mexpiration is None or mexpiration > expiration:
Aug 25 12:56:43 seal08 designate-mdns[116065]: ERROR designate.dnsutils TypeError: '>' not supported between instances of 'float' and 'NoneType'
Aug 25 12:56:43 seal08 designate-mdns[116065]: ERROR designate.dnsutils
Aug 25 12:56:43 seal08 designate-mdns[116065]: WARNING designate.mdns.xfr [None req-0fcfe0f2-b8cb-4eec-ae42-9ea88e089c69 None None] XFR failed for debuntu.foo.. No servers in [{'zone_id': 'ccadec69-4723-4c24-987f-7cb1587cff2c', 'host': '10.35.64.8', 'port': 53, 'id': 'ba2932b2-3f81-4ad5-985c-86ce081fe62d', 'created_at': datetime.datetime(2021, 8, 25, 9, 56, 44), 'updated_at': None, 'version': 1}] was reached.: designate.exceptions.XFRFailure: XFR failed for debuntu.foo.. No servers in [{'zone_id': 'ccadec69-4723-4c24-987f-7cb1587cff2c', 'host': '10.35.64.8', 'port': 53, 'id': 'ba2932b2-3f81-4ad5-985c-86ce081fe62d', 'created_at': datetime.datetime(2021, 8, 25, 9, 56, 44), 'updated_at': None, 'version': 1}] was reached.
Aug 25 12:56:43 seal08 designate-mdns[116064]: DEBUG designate.service [None req-b13130f1-3391-4bbc-b026-22f6ffefa71e None None] Handling UDP Request from: 10.35.64.8:38424 {{(pid=116064) _dns_handle_udp /opt/stack/designate/designate/service.py:317}}
Aug 25 12:56:44 seal08 designate-mdns[116064]: DEBUG designate.storage.impl_sqlalchemy [None req-97b992fe-cf4f-4c84-bc3d-88556d9b9abc None None] Fetched zone <Zone id:'ccadec69-4723-4c24-987f-7cb1587cff2c' type:'SECONDARY' name:'debuntu.foo.' pool_id:'794ccc2c-d751-44fe-b57f-8894c9f5c842' serial:'1' action:'CREATE' status:'PENDING'> {{(pid=116064) _find_zones /opt/stack/designate/designate/storage/impl_sqlalchemy/__init__.py:252}}
Aug 25 12:56:44 seal08 designate-mdns[116058]: DEBUG designate.service [None req-df0d9bc9-e4d6-43ff-b108-05b4005b4357 None None] Handling TCP Request from: 10.35.64.8:49715 {{(pid=116058) _dns_handle_tcp /opt/stack/designate/designate/service.py:203}}
Aug 25 12:56:44 seal08 designate-mdns[116058]: DEBUG designate.storage.impl_sqlalchemy [None req-e1f94cdb-a9dc-434c-907c-54769d7375b8 None None] Fetched zone <Zone id:'ccadec69-4723-4c24-987f-7cb1587cff2c' type:'SECONDARY' name:'debuntu.foo.' pool_id:'794ccc2c-d751-44fe-b57f-8894c9f5c842' serial:'1' action:'CREATE' status:'PENDING'> {{(pid=116058) _find_zones /opt/stack/designate/designate/storage/impl_sqlalchemy/__init__.py:252}}

### Note ###
Michael has already found the reason.
From:Michael Johnson
Aug 26, 2021, 8:13 PM (3 days ago)
to me

Ok, so this is a bug in the "dnspython" library designate uses.

Version 1.16 which you had installed has:
        if mexpiration is None or mexpiration > expiration:
            mexpiration = expiration
Which is broken.
Version 2.0.0rc1 has: (https://github.com/rthalley/dnspython/blob/v2.0.0rc1/dns/query.py#L925)
            if mexpiration is None or \
               (expiration is not None and mexpiration > expiration):
                mexpiration = expiration
Which works.

[Where problems could occur]

* This small patch improves the robustness of python library code.
  Issues should be self evident in errors logged.

[Other Info]

* Jammy+ have the fix. Focal/Bionic don't have it.

Revision history for this message
Arkady Shtempler (ashtempl) wrote :
description: updated
Changed in designate:
status: New → Confirmed
importance: Undecided → High
affects: designate → dnspython3 (Ubuntu)
affects: dnspython3 (Ubuntu) → designate
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in dnspython3 (Ubuntu):
status: New → Confirmed
affects: ubuntu → dnspython3 (Ubuntu)
Changed in dnspython3 (Ubuntu):
status: New → Confirmed
Revision history for this message
Tobias Urdin (tobias-urdin) wrote :

dnspython (python3-dns) package on RHEL (and thus CentOS Stream 8) is patched with version 1.15.0-11
https://bugzilla.redhat.com/show_bug.cgi?id=2075187

Revision history for this message
Tobias Urdin (tobias-urdin) wrote :

CentOS Stream 9 is not affected as it ships a version >= 2.0.0

Revision history for this message
Michael Johnson (johnsom) wrote :

upper-constraints was updated in yoga to 2.1.0, so yoga forward this issue is resolved.

Changed in designate:
status: Confirmed → Fix Released
summary: - Designate DNS – Secondary zone is failed to be created and gets into the
- ERROR status
+ [SRU] Designate DNS – Secondary zone is failed to be created and gets
+ into the ERROR status
Changed in dnspython (Ubuntu Focal):
assignee: nobody → Brett Milford (brettmilford)
Changed in dnspython3 (Ubuntu Focal):
assignee: nobody → Brett Milford (brettmilford)
Revision history for this message
Brett Milford (brettmilford) wrote :
description: updated
tags: added: sru-needed sts
Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "lp1941988-dnspython.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
tags: added: sts-sponsor-mfo
Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

Hi Brett, Tiago,

Thanks for the debdiff!

I reviewed it and fixed a few bits (please check below),
then tested and sponsored it to Focal.

Question: it seems Bionic is affected; shoud it be fixed?

Fixes:
- Reword changelog
- Add DEP-3 headers [1] in .patch (Origin/Bug-Ubuntu)
- Run update-maintainer (pkg switched to '-ubuntuN')

[1] https://dep-team.pages.debian.net/deps/dep3/

Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

Fix notes:
---

Fix commit is a trivial None check, still upstream.

It's in Jammy+. Fix Focal only; asked about Bionic.

 $ git log --oneline --patch -1 9fbf9b223dc26
 9fbf9b223dc2 When doing xfr, do not compare with expiration if it is None. [Issue #390]
 ...
 @@ -607,7 +607,8 @@ def xfr(where, zone, rdtype=dns.rdatatype.AXFR, rdclass=dns.rdataclass.IN,
 ...
 - if mexpiration is None or mexpiration > expiration:
 + if mexpiration is None or \
 + (expiration is not None and mexpiration > expiration):
 ...

 ~/git/dnspython$ git describe --contains 9fbf9b223dc2
 v2.0.0rc1~360

Also marking dnspython3 Invalid; python3 is in dnspython nowadays.

 $ apt-cache show python3-dnspython | grep ^Source:
 Source: dnspython

 $ rmadison -a source dnspython
  dnspython | 1.11.1-1build1 | trusty | source
  dnspython | 1.12.0-1 | xenial | source
  dnspython | 1.15.0-1 | bionic | source
  dnspython | 1.16.0-1build1 | focal | source
  dnspython | 2.1.0-1ubuntu1 | jammy | source
  dnspython | 2.2.1-2 | kinetic | source

 $ rmadison -a source dnspython3
  dnspython3 | 1.11.1-1 | trusty/universe | source
  dnspython3 | 1.12.0-0ubuntu3 | xenial | source

Changed in dnspython3 (Ubuntu):
status: Confirmed → Invalid
Changed in dnspython3 (Ubuntu Focal):
status: New → Invalid
assignee: Brett Milford (brettmilford) → nobody
Changed in dnspython (Ubuntu Focal):
status: New → In Progress
Changed in dnspython (Ubuntu):
status: New → Fix Released
Changed in dnspython (Ubuntu Focal):
importance: Undecided → Medium
Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

Test Steps for dnspython only (no OpenStack Designate required)
----------

1) Setup bind9 for 'example.tld':

 $ sudo apt install bind9

 - /etc/bind/named.conf.options: add in 'options':

 listen-on port 12753 { 127.0.0.1/32; };
 allow-query { any; };
 allow-transfer { any; };

 - /etc/bind/named.conf.local: add:

 zone "example.tld" IN {
   type master;
   file "/etc/bind/example.tld.db";
 };

 - /etc/bind/example.tlb.db: create:

 $TTL 5m
 @ IN SOA ns.example.tld. email.example.tld. 90 4h 15m 8h 4m
 @ IN NS ns.example.tld.
 ns IN A 1.1.1.1
 test IN A 1.2.3.4

 $ sudo systemctl restart named.service

 $ systemctl status named.service | grep Active:
      Active: active (running) since ...

 $ journalctl -u named.service | grep -e example.tld -e 'all zones loaded'
 ... named[3668]: zone example.tld/IN: loaded serial 90
 ... named[3668]: all zones loaded

2) Test the basics and AXFR:

Basics:

 $ dig +noall +authority @127.0.0.1 -p 12753 example.tld
 example.tld. 240 IN SOA ns.example.tld. email.example.tld. 90 14400 900 28800 240

 $ dig +noall +answer @127.0.0.1 -p 12753 ns.example.tld
 ns.example.tld. 300 IN A 1.1.1.1

 $ dig +noall +answer @127.0.0.1 -p 12753 test.example.tld
 test.example.tld. 300 IN A 1.2.3.4

AXFR:

 $ dig +noall +answer @127.0.0.1 -p 12753 example.tld axfr
 example.tld. 300 IN SOA ns.example.tld. email.example.tld. 90 14400 900 28800 240
 example.tld. 300 IN NS ns.example.tld.
 ns.example.tld. 300 IN A 1.1.1.1
 test.example.tld. 300 IN A 1.2.3.4
 example.tld. 300 IN SOA ns.example.tld. email.example.tld. 90 14400 900 28800 240

3) Test AXFR with dnspython:

Check same answers with python:

 $ sudo apt install python3-dnspython

 $ python3 -q

 import dns.query
 import dns.zone

 axfr = dns.zone.from_xfr(dns.query.xfr(where='127.0.0.1', port=12753, zone='example.tld', rdtype=dns.rdatatype.AXFR))
 for node in axfr.nodes.keys():
     print(axfr.nodes[node].to_text(node))

 @ 300 IN SOA ns email 90 14400 900 28800 240
 @ 300 IN NS ns
 ns 300 IN A 1.1.1.1
 test 300 IN A 1.2.3.4

Now set dns.query.xfr(lifetime=None, timeout=not None) to hit the bug:

 axfr = dns.zone.from_xfr(dns.query.xfr(where='127.0.0.1', port=12753, zone='example.tld', rdtype=dns.rdatatype.AXFR, lifetime=None, timeout=30))
 Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "/usr/lib/python3/dist-packages/dns/zone.py", line 1106, in from_xfr
     for r in xfr:
   File "/usr/lib/python3/dist-packages/dns/query.py", line 611, in xfr
     if mexpiration is None or mexpiration > expiration:
 TypeError: '>' not supported between instances of 'float' and 'NoneType'

As reported:

 Aug 25 12:56:43 seal08 designate-mdns[116065]: ERROR designate.dnsutils TypeError: '>' not supported between instances of 'float' and 'NoneType'

4) With the patch applied, the issue doesn't happen.

description: updated
Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

The updated debdiff, for reference purposes.

Revision history for this message
Robie Basak (racb) wrote : Please test proposed package

Hello Arkady, or anyone else affected,

Accepted dnspython into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dnspython/1.16.0-1ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in dnspython (Ubuntu Focal):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-focal
Revision history for this message
Robie Basak (racb) wrote :

Unsubscribing ~ubuntu-sponsors as I see nothing that needs sponsoring for Ubuntu.

Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (dnspython/1.16.0-1ubuntu1)

All autopkgtests for the newly accepted dnspython (1.16.0-1ubuntu1) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

recon-ng/5.0.1-1 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#dnspython

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

The autopkgtest regression report for recon-ng/5.0.1-1 (arm64)
was due to a connection timeout and passed with a rerun (thanks, ~teward!)

Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

Verification done on focal-proposed with python3-dnspython only.

Setup:
-----

$ sudo apt install -y bind9

$ FILE=/tmp/bind-options
$ cat <<EOF >"$FILE"
listen-on port 12753 { 127.0.0.1/32; };
allow-query { any; };
allow-transfer { any; };
EOF

$ sudo sed -i "/^options {/ r $FILE" /etc/bind/named.conf.options

$ cat <<EOF | sudo tee -a /etc/bind/named.conf.local
zone "example.tld" IN {
  type master;
  file "/etc/bind/example.tld.db";
};
EOF

$ cat <<"EOF" | sudo tee -a /etc/bind/example.tld.db
$TTL 5m
@ IN SOA ns.example.tld. email.example.tld. 90 4h 15m 8h 4m
@ IN NS ns.example.tld.
ns IN A 1.1.1.1
test IN A 1.2.3.4
EOF

$ sudo systemctl restart named.service

Check:
-----

$ dig +noall +authority @127.0.0.1 -p 12753 example.tld
example.tld. 240 IN SOA ns.example.tld. email.example.tld. 90 14400 900 28800 240
$ dig +noall +answer @127.0.0.1 -p 12753 ns.example.tld
ns.example.tld. 300 IN A 1.1.1.1
$ dig +noall +answer @127.0.0.1 -p 12753 test.example.tld
test.example.tld. 300 IN A 1.2.3.4

$ dig +noall +answer @127.0.0.1 -p 12753 example.tld axfr
example.tld. 300 IN SOA ns.example.tld. email.example.tld. 90 14400 900 28800 240
example.tld. 300 IN NS ns.example.tld.
ns.example.tld. 300 IN A 1.1.1.1
test.example.tld. 300 IN A 1.2.3.4
example.tld. 300 IN SOA ns.example.tld. email.example.tld. 90 14400 900 28800 240

Before: (fails)
------

$ sudo apt install -y python3-dnspython
$ dpkg -s python3-dnspython | grep Version:
Version: 1.16.0-1build1

$ python3 -q -c \
"import dns.query
import dns.zone
axfr = dns.zone.from_xfr(dns.query.xfr(where='127.0.0.1', port=12753, zone='example.tld', rdtype=dns.rdatatype.AXFR, lifetime=None, timeout=30))
for node in axfr.nodes.keys() : print(axfr.nodes[node].to_text(node))"
TypeError: '>' not supported between instances of 'float' and 'NoneType'

After: (works)
-----

$ sudo add-apt-repository 'deb http://archive.ubuntu.com/ubuntu focal-proposed main'
$ sudo apt install -y python3-dnspython
$ dpkg -s python3-dnspython | grep Version:
Version: 1.16.0-1ubuntu1

$ python3 -q -c \
"import dns.query
import dns.zone
axfr = dns.zone.from_xfr(dns.query.xfr(where='127.0.0.1', port=12753, zone='example.tld', rdtype=dns.rdatatype.AXFR, lifetime=None, timeout=30))
for node in axfr.nodes.keys() : print(axfr.nodes[node].to_text(node))"
@ 300 IN SOA ns email 90 14400 900 28800 240
@ 300 IN NS ns
ns 300 IN A 1.1.1.1
test 300 IN A 1.2.3.4

tags: added: verification-done verification-done-focal
removed: verification-needed verification-needed-focal
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package dnspython - 1.16.0-1ubuntu1

---------------
dnspython (1.16.0-1ubuntu1) focal; urgency=medium

  * Fix dns.query.xfr(lifetime=None, timeout=not None) (LP: #1941988)
    - d/p/lp1941988-When-doing-xfr-do-not-compare-with-expiration.patch

 -- Brett Milford <email address hidden> Thu, 30 Jun 2022 08:40:05 +0000

Changed in dnspython (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of the Stable Release Update for dnspython has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

Apparently no changes are needed to the charm; marking its track as Invalid.
Please feel free to correct/note if required.

Changed in charm-designate:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.