Admin deleting servers or ports leaves orphaned DNS records

I can reproduce this and I think the error is in the neutron code, not in designate.

I don't think it is an new RFE. IMO it sounds more like a bug in our existing implementation of integration with Designate. So let's track it as a bug for now.

if its ok i can do the change needed for this
or at least i can help if needed

Fix proposed to branch: master
Review: https://review.opendev.org/728385

@hamza: I posted a WIP patch for neutron, please have a look or feel free to submit your own solution. I'll also try to create a scenario test, note that you can also reproduce with just a port that has DNS records attached, doesn't need to be a server, that also takes Nova out of the loop.

Related fix proposed to branch: master
Review: https://review.opendev.org/728409

@Jens its looks like it will solve the problem, thanks

Reviewed: https://review.opendev.org/728385
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=622714b63e08feaba4e81d218541319d2ffada30
Submitter: Zuul
Branch: master

commit 622714b63e08feaba4e81d218541319d2ffada30
Author: Jens Harbott <email address hidden>
Date: Fri May 15 08:43:18 2020 +0000

    Optionally use admin powers when deleting DNS records

    This resolves a bug that causes stale records to be kept in place when
    an admin deletes a port, server or floating IP that was created in some
    project other than the admin project.

    Change-Id: I7cbb0e87a7e87f23ccf5d8750835b4785693473a
    Closes-Bug: #1875981

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/740675

Fix proposed to branch: stable/train
Review: https://review.opendev.org/740676

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/740677

Reviewed: https://review.opendev.org/740675
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c172235f585be07bc01df9de470405c20d062bec
Submitter: Zuul
Branch: stable/ussuri

commit c172235f585be07bc01df9de470405c20d062bec
Author: Jens Harbott <email address hidden>
Date: Fri May 15 08:43:18 2020 +0000

    Optionally use admin powers when deleting DNS records

    This resolves a bug that causes stale records to be kept in place when
    an admin deletes a port, server or floating IP that was created in some
    project other than the admin project.

    Change-Id: I7cbb0e87a7e87f23ccf5d8750835b4785693473a
    Closes-Bug: #1875981
    (cherry picked from commit 622714b63e08feaba4e81d218541319d2ffada30)

Reviewed: https://review.opendev.org/740676
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=b47a2684b7ab611f7e7ba3ff31f8a82cffe66724
Submitter: Zuul
Branch: stable/train

commit b47a2684b7ab611f7e7ba3ff31f8a82cffe66724
Author: Jens Harbott <email address hidden>
Date: Fri May 15 08:43:18 2020 +0000

    Optionally use admin powers when deleting DNS records

    This resolves a bug that causes stale records to be kept in place when
    an admin deletes a port, server or floating IP that was created in some
    project other than the admin project.

    Change-Id: I7cbb0e87a7e87f23ccf5d8750835b4785693473a
    Closes-Bug: #1875981
    (cherry picked from commit 622714b63e08feaba4e81d218541319d2ffada30)

Reviewed: https://review.opendev.org/740677
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=72085466b546ed97c79a5ff4ecbafc52c947a49f
Submitter: Zuul
Branch: stable/stein

commit 72085466b546ed97c79a5ff4ecbafc52c947a49f
Author: Jens Harbott <email address hidden>
Date: Fri May 15 08:43:18 2020 +0000

    Optionally use admin powers when deleting DNS records

    This resolves a bug that causes stale records to be kept in place when
    an admin deletes a port, server or floating IP that was created in some
    project other than the admin project.

    Change-Id: I7cbb0e87a7e87f23ccf5d8750835b4785693473a
    Closes-Bug: #1875981
    (cherry picked from commit 622714b63e08feaba4e81d218541319d2ffada30)

Reviewed: https://review.opendev.org/728409
Committed: https://git.openstack.org/cgit/openstack/neutron-tempest-plugin/commit/?id=9e101bfcbecae77aa76d0acdecf5e99fd34a6fba
Submitter: Zuul
Branch: master

commit 9e101bfcbecae77aa76d0acdecf5e99fd34a6fba
Author: Dr. Jens Harbott <email address hidden>
Date: Fri May 15 10:10:26 2020 +0000

    Verify admin deletion not to fail

    This is the test for the scenario described in bug [0], where an admin
    deletes resources from a different project, resulting in orphaned DNS
    records.

    [0] https://launchpad.net/bugs/1875981

    Change-Id: Ibc12a80fad28bb54b0416de7dfe14ef67e4420ef
    Related-Bug: 1875981
    Depends-On: https://review.opendev.org/740675
    Depends-On: https://review.opendev.org/740676
    Depends-On: https://review.opendev.org/740677

Now that fix breaks something else:

with integration with Designate enabled, suppose the required dns domain/zone to create records in was not there to begin with, or was deleted out of band.

Then e.g. adding floating ip to server works (just the record is not created in Designate), with only exception logged.
But now it is impossible for the user to delete such VM right away w/o removing the fip:
- first the ordinary client tried, which raises NotFound
- then the "all-projects" client is tried, but the user lacks authorisation to actually call the API with "all projects", so client returns Forbidden, and error is returned to the API caller.
- the VM in Nova goes to ERROR state.

It seems we have to translate Forbidden from Designate to the same DNSDomainNotFound in Neutron.

We actually caught this in downstream when the

neutron_tempest_plugin.scenario.test_internal_dns.InternalDNSTest.test_dns_domain_and_name

test started failing on our CI wich designate integration enabled (after we synced the cherry-picked ussuri patch).
Btw, the test uses domain name that is technically impossible to create in Designate...

Related fix proposed to branch: master
Review: https://review.opendev.org/753268

We are also facing another issue, which is in a way related.

If a PTR record was created for a Floating IP as a managed record, e.g. via:
'openstack ptr record create' than on release of Floating IP the PTR is not removed.

$ openstack ptr record set qa-de-1:33962041-4359-4d8a-91c8-269b05370120 newtestfipptr.cloud.com.
+-------------+----------------------------------------------+
| Field | Value |
+-------------+----------------------------------------------+
| action | CREATE |
| address | 10.236.37.138 |
| description | None |
| id | qa-de-1:33962041-4359-4d8a-91c8-269b05370120 |
| ptrdname | newtestfipptr.cloud.com. |
| status | PENDING |
| ttl | 3600 |
+-------------+----------------------------------------------+

$ openstack ptr record list | grep 33962041-4359-4d8a-91c8-269b05370120
| qa-de-1:33962041-4359-4d8a-91c8-269b05370120 | newtestfipptr.cloud.com. | | 3600 |

Now remove the Floating IP:

$ openstack floating ip delete 33962041-4359-4d8a-91c8-269b05370120

The PTR is left in the respective reverse zone:

$ openstack recordset list 352d8538-cf23-4323-93e1-98ddb0aeca65 | grep newtestfipptr
| 81eb2ead-b777-457e-b832-9a15c2756618 | 138.37.236.10.in-addr.arpa. | PTR | newtestfipptr.cloud.com. | ACTIVE | NONE |

@Dmitry: This is a different issue altogether. Currently using the "openstack ptr record" functionality is not compatible with using the neutron dns-integration. You should use just one of them, not both concurrently.

@j-harbott
yes, you're right, that is another issue. I could not find the relevant bug, will open a new one.