Designate tooz error

Bug #1872205 reported by Viorel-Cosmin Miron
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Designate
Confirmed
Undecided
Unassigned
kolla-ansible
Fix Released
Medium
Radosław Piliszek
Stein
Fix Released
Medium
Mark Goddard
Train
Fix Released
Medium
Mark Goddard
Ussuri
Fix Released
Medium
Radosław Piliszek
tooz
Invalid
Undecided
Unassigned

Bug Description

**Bug Report**

What happened: I have recently deployed designate and in kibana can extensively find this WARNING:
Failed to start Coordinator:: tooz.ToozError: Can not start a driver which has not been stopped
2020-04-11 14:36:05.366 22 ERROR designate.coordination Traceback (most recent call last):
2020-04-11 14:36:05.366 22 ERROR designate.coordination File "/var/lib/kolla/venv/lib/python3.6/site-packages/designate/coordination.py", line 77, in start
2020-04-11 14:36:05.366 22 ERROR designate.coordination self._coordinator.start()
2020-04-11 14:36:05.366 22 ERROR designate.coordination File "/var/lib/kolla/venv/lib/python3.6/site-packages/tooz/coordination.py", line 690, in start

What you expected to happen: Have Designate working as designed.

How to reproduce it (minimal and precise): Use the normal train release and deploy with kolla-ansible. create a default pool and assign it to designate-sink.conf.

**Environment**:
(venv) root@compute-1:/home/kolla-ansible_setup# sh diag.sh
##### System Identification #####
PRETTY_NAME="Ubuntu 18.04.4 LTS"
VERSION_ID="18.04"
##### Docker Version #####
Docker version 19.03.8, build afacb8b7f0
##### Docker Info #####
Client:
 Debug Mode: false

Server:
 Containers: 104
  Running: 99
  Paused: 0
  Stopped: 5
 Images: 283
 Server Version: 19.03.8
 Storage Driver: overlay2
  Backing Filesystem: <unknown>
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan kuryr macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.15.0-91-generic
 Operating System: Ubuntu 18.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 125.7GiB
 Name: compute-1
 ID: QMPD:TNCC:PZXR:4LG7:PXSZ:QIMB:N5Y4:ZX7G:PWN7:JVKK:M6JV:T6CG
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Cluster Store: etcd://172.22.0.1:2379
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: API is accessible on http://172.22.0.1:2375 without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/engine/security/security/#docker-daemon-attack-surface
WARNING: No swap limit support
##### Ansible Version #####
ansible 2.9.6
  config file = /home/kolla-ansible_setup/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /root/venv/lib/python3.6/site-packages/ansible
  executable location = /root/venv/bin/ansible
  python version = 3.6.9 (default, Nov 7 2019, 10:44:02) [GCC 8.3.0]
##### List installed packages, including editables #####
Package Version
---------------------- ----------
alembic 1.4.2
ansible 2.9.6
appdirs 1.4.3
attrs 19.3.0
Babel 2.8.0
bcrypt 3.1.7
cachetools 4.1.0
certifi 2020.4.5.1
cffi 1.14.0
chardet 3.0.4
click 7.1.1
cliff 3.1.0
cliff-tablib 2.0
cmd2 0.8.9
colorama 0.4.3
cryptography 2.9
debtcollector 2.0.1
decorator 4.4.2
distlib 0.3.0
docker 4.2.0
dogpile.cache 0.9.0
entrypoints 0.3
extras 1.0.0
fasteners 0.15
filelock 3.0.12
fixtures 3.0.0
flake8 3.7.9
futurist 2.1.1
gitdb 4.0.2
gitdb2 4.0.2
GitPython 3.1.0
google-auth 1.13.1
idna 2.9
importlib-metadata 1.6.0
importlib-resources 1.4.0
iso8601 0.1.12
Jinja2 2.11.1
jmespath 0.9.5
jsonpatch 1.25
jsonpath-rw 1.4.0
jsonpath-rw-ext 1.2.2
jsonpickle 1.3
jsonpointer 2.0
jsonschema 3.2.0
keystoneauth1 4.0.0
kolla 9.0.1
kolla-ansible 9.0.1
kolla-cli 9.0.0
linecache2 1.0.0
Mako 1.1.2
MarkupSafe 1.1.1
mccabe 0.6.1
monotonic 1.5
msgpack 1.0.0
munch 2.5.0
murano-pkg-check 0.3.0
netaddr 0.7.19
netifaces 0.10.9
networkx 2.4
oauthlib 3.1.0
openstacksdk 0.46.0
os-client-config 2.1.0
os-service-types 1.7.0
osc-lib 2.0.0
oslo.concurrency 4.0.2
oslo.config 8.0.2
oslo.context 3.0.2
oslo.db 8.1.0
oslo.i18n 4.0.1
oslo.log 4.1.1
oslo.serialization 3.1.1
oslo.utils 4.1.1
osprofiler 3.1.0
packaging 20.3
pankoclient 1.0.1
paramiko 2.7.1
pbr 5.4.5
pip 20.0.2
pip-review 1.1.0
pkg-resources 0.0.0
ply 3.11
prettytable 0.7.2
pyasn1 0.4.8
pyasn1-modules 0.2.8
pycodestyle 2.5.0
pycparser 2.20
pyflakes 2.2.0
pyghmi 1.5.13
pyinotify 0.9.6
PyNaCl 1.3.0
pyOpenSSL 19.1.0
pyparsing 2.4.7
pyperclip 1.8.0
pyrsistent 0.16.0
python-barbicanclient 4.10.0
python-cinderclient 7.0.0
python-dateutil 2.8.1
python-designateclient 4.0.0
python-editor 1.0.4
python-glanceclient 3.1.0
python-heatclient 2.0.0
python-keystoneclient 3.22.0
python-magnumclient 3.0.0
python-mimeparse 1.6.0
python-mistralclient 4.0.1
python-neutronclient 7.1.1
python-novaclient 16.0.0
python-octaviaclient 2.0.1
python-openstackclient 5.2.0
python-subunit 1.4.0
python-swiftclient 3.9.0
python-troveclient 3.3.1
python-zunclient 4.0.0
pytz 2019.3
PyYAML 5.3.1
requests 2.23.0
requests-oauthlib 1.3.0
requestsexceptions 1.4.0
rfc3986 1.4.0
rsa 4.0
semantic-version 2.8.4
setuptools 46.1.3
simplejson 3.17.0
six 1.14.0
smmap 3.0.1
SQLAlchemy 1.3.16
sqlalchemy-migrate 0.13.0
sqlparse 0.3.1
stevedore 1.32.0
tablib 1.1.0
Tempita 0.5.2
testresources 2.0.1
testscenarios 0.5.0
testtools 2.4.0
traceback2 1.4.0
ujson 2.0.3
unittest2 1.1.0
urllib3 1.25.8
virtualenv 20.0.17
warlock 1.3.3
wcwidth 0.1.9
WebOb 1.8.6
websocket-client 0.57.0
wheel 0.34.2
wrapt 1.12.1
yaql 1.1.3
zipp 3.1.0
##### Globals.yml file #####
kolla_base_distro: "ubuntu"
kolla_install_type: "source"
openstack_release: "train"
enable_openstack_core: "yes"
enable_glance: "{{ enable_openstack_core | bool }}"
enable_haproxy: "yes"
enable_keepalived: "{{ enable_haproxy | bool }}"
enable_keystone: "{{ enable_openstack_core | bool }}"
enable_mariadb: "yes"
enable_memcached: "yes"
enable_neutron: "{{ enable_openstack_core | bool }}"
enable_nova: "{{ enable_openstack_core | bool }}"
enable_rabbitmq: "{{ 'yes' if om_rpc_transport == 'rabbit' or om_notify_transport == 'rabbit' else 'no' }}"
enable_barbican: "yes"
enable_ceilometer: "yes"
enable_central_logging: "yes"
enable_ceph: "no"
enable_chrony: "yes"
enable_cinder: "yes"
enable_cinder_backup: "yes"
enable_cloudkitty: "yes"
enable_designate: "yes"
enable_destroy_images: "yes"
enable_elasticsearch: "{{ 'yes' if enable_central_logging | bool or enable_osprofiler | bool or enable_skydive | bool or enable_monasca | bool else 'no' }}"
enable_elasticsearch_curator: "yes"
enable_etcd: "yes"
enable_fluentd: "yes"
enable_gnocchi: "yes"
enable_grafana: "yes"
enable_heat: "{{ enable_openstack_core | bool }}"
enable_horizon: "{{ enable_openstack_core | bool }}"
enable_horizon_cloudkitty: "{{ enable_cloudkitty | bool }}"
enable_horizon_designate: "{{ enable_designate | bool }}"
enable_horizon_heat: "{{ enable_heat | bool }}"
enable_horizon_ironic: "{{ enable_ironic | bool }}"
enable_horizon_magnum: "{{ enable_magnum | bool }}"
enable_horizon_mistral: "{{ enable_mistral | bool }}"
enable_horizon_neutron_vpnaas: "{{ enable_neutron_vpnaas | bool }}"
enable_horizon_octavia: "{{ enable_octavia | bool }}"
enable_horizon_trove: "{{ enable_trove | bool }}"
enable_horizon_zun: "{{ enable_zun | bool }}"
enable_influxdb: "{{ enable_monasca | bool or (enable_cloudkitty | bool and cloudkitty_storage_backend == 'influxdb') }}"
enable_ironic: "yes"
enable_ironic_neutron_agent: "{{ enable_neutron | bool and enable_ironic | bool }}"
enable_ironic_pxe_uefi: "yes"
enable_kibana: "{{ 'yes' if enable_central_logging | bool or enable_monasca | bool else 'no' }}"
enable_kuryr: "yes"
enable_magnum: "yes"
enable_manila_backend_cephfs_native: "yes"
enable_mariabackup: "yes"
enable_mistral: "yes"
enable_neutron_vpnaas: "yes"
enable_neutron_qos: "yes"
enable_neutron_provider_networks: "yes"
enable_neutron_segments: "yes"
enable_neutron_sfc: "yes"
enable_neutron_metering: "yes"
enable_neutron_port_forwarding: "yes"
enable_nova_serialconsole_proxy: "yes"
enable_nova_ssh: "yes"
enable_octavia: "yes"
enable_openvswitch: "{{ enable_neutron | bool and neutron_plugin_agent != 'linuxbridge' }}"
enable_osprofiler: "yes"
enable_panko: "yes"
enable_placement: "{{ enable_nova | bool or enable_zun | bool }}"
enable_skydive: "yes"
enable_trove: "yes"
enable_trove_singletenant: "yes"
enable_zun: "yes"
enable_ceph_rgw_keystone: "yes"

Revision history for this message
Viorel-Cosmin Miron (uhl-hosting) wrote :
Revision history for this message
Radosław Piliszek (yoctozepto) wrote :

This looks more like programming bug. Is there no other context there?

Please attach generated designate config.

I added designate as target of bug to ask for help with debugging (I don't know designate internals).

Revision history for this message
Viorel-Cosmin Miron (uhl-hosting) wrote :

Any idea what config from these would do?

(venv) root@compute-1:/etc/kolla# tree designate-*
designate-api
├── config.json
└── designate.conf
designate-backend-bind9
├── config.json
├── designate.conf
├── named.conf
├── rndc.conf
└── rndc.key
designate-central
├── config.json
└── designate.conf
designate-mdns
├── config.json
└── designate.conf
designate-producer
├── config.json
└── designate.conf
designate-sink
├── config.json
└── designate.conf
designate-worker
├── config.json
├── designate.conf
├── pools.yaml
├── rndc.conf
└── rndc.key

I have only deisgate-sink.conf altered according to the docs.

(venv) root@compute-1:/etc/kolla# cat config/designate/designate-sink.conf
[handler:nova_fixed]
zone_id = 9fd48cac-51c7-4de7-a35c-acdb176d1c07
[handler:neutron_floatingip]
zone_id = 9fd48cac-51c7-4de7-a35c-acdb176d1c07
(venv) root@compute-1:/etc/kolla# openstack zone list
+--------------------------------------+------------------+---------+------------+--------+--------+
| id | name | type | serial | status | action |
+--------------------------------------+------------------+---------+------------+--------+--------+
| 9fd48cac-51c7-4de7-a35c-acdb176d1c07 | dns.uhlhost.net. | PRIMARY | 1586585345 | ACTIVE | NONE |
+--------------------------------------+------------------+---------+------------+--------+--------+
(venv) root@compute-1:/etc/kolla#

Revision history for this message
Erik Olof Gunnar Andersson (eandersson) wrote :

What is the driver used? Looks like etcd3?
https://docs.openstack.org/tooz/rocky/user/compatibility.html

The implementation is using create_group / join_group which probably raises NotImplemented errors.
https://github.com/openstack/designate/blob/master/designate/coordination.py#L89

I am wondering if maybe the coordination driver is started successfully, but the backend does not support what Designate is trying to do, making Designate think that it wasn't connected successfully.

Revision history for this message
Viorel-Cosmin Miron (uhl-hosting) wrote :

Here using Train release with:

(etcd)[etcd@compute-1 /]$ etcd --version
etcd Version: 3.2.17
Git SHA: Not provided (use ./build instead of go build)
Go Version: go1.10
Go OS/Arch: linux/amd64
(etcd)[etcd@compute-1 /]$

Deployed via Kolla-Ansible .

Revision history for this message
Erik Olof Gunnar Andersson (eandersson) wrote :

I believe the only supported coordinators for Designate are Redis, Zookeeper and Memcached.

Revision history for this message
Viorel-Cosmin Miron (uhl-hosting) wrote :

Kolla has a line in globals.yml that says:

designate_coordination_backend: "{{ 'redis' if enable_redis|bool else 'etcd' if enable_etcd|bool else '' }}"

It seems i had etcd also enabled and redis disabled. Still if so, they will need to update this line.

Revision history for this message
Erik Olof Gunnar Andersson (eandersson) wrote :
Revision history for this message
Radosław Piliszek (yoctozepto) wrote :

Ack, Kolla Ansible got this line in https://review.opendev.org/682095 (before there was just no coordination in Designate). I've done some further analysis in https://bugs.launchpad.net/kolla-ansible/+bug/1840070/comments/13

I will propose a proper fix on k-a side.

As for Designate, I guess two things need to be done: one is fixing logic for start/stop (it looks like it could fail due to something else and get in this weirdness), the other one is checking tooz features of selected backend to fail early with proper message (also message in docs for that it needs group membership in tooz).

As for tooz... I guess this is hardly a bug per se, I would switch it to wishlist so that maybe some day one implements this for etcd. Maybe. Considering other shortcomings of it, it needs a bit more work to be wide-scale production-ready.

Changed in kolla-ansible:
importance: Undecided → Medium
status: New → Triaged
assignee: nobody → Radosław Piliszek (yoctozepto)
milestone: none → 10.0.0
Changed in designate:
status: New → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (master)

Fix proposed to branch: master
Review: https://review.opendev.org/719583

Changed in kolla-ansible:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (master)

Reviewed: https://review.opendev.org/719583
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=3c234603a9c8e443dc3d38989e3e78147757d1da
Submitter: Zuul
Branch: master

commit 3c234603a9c8e443dc3d38989e3e78147757d1da
Author: Radosław Piliszek <email address hidden>
Date: Mon Apr 13 17:33:02 2020 +0200

    Fix Designate not to use etcd coordination backend

    etcd via tooz does not support group membership required by
    Designate coordination.
    The best k-a can do is not to configure etcd in Designate.

    Change-Id: I2f64f928e730355142ac369d8868cf9f65ca357e
    Closes-bug: #1872205
    Related-bug: #1840070

Changed in kolla-ansible:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/723253

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/723255

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/train)

Reviewed: https://review.opendev.org/723253
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=8c7afb73bcc1dc5e8982660b7c3774686f658356
Submitter: Zuul
Branch: stable/train

commit 8c7afb73bcc1dc5e8982660b7c3774686f658356
Author: Radosław Piliszek <email address hidden>
Date: Mon Apr 13 17:33:02 2020 +0200

    Fix Designate not to use etcd coordination backend

    etcd via tooz does not support group membership required by
    Designate coordination.
    The best k-a can do is not to configure etcd in Designate.

    Change-Id: I2f64f928e730355142ac369d8868cf9f65ca357e
    Closes-bug: #1872205
    Related-bug: #1840070
    (cherry picked from commit 3c234603a9c8e443dc3d38989e3e78147757d1da)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/stein)

Reviewed: https://review.opendev.org/723255
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=66a2f493589bd737d8a4302ff27c0c430371ee68
Submitter: Zuul
Branch: stable/stein

commit 66a2f493589bd737d8a4302ff27c0c430371ee68
Author: Radosław Piliszek <email address hidden>
Date: Mon Apr 13 17:33:02 2020 +0200

    Fix Designate not to use etcd coordination backend

    etcd via tooz does not support group membership required by
    Designate coordination.
    The best k-a can do is not to configure etcd in Designate.

    Change-Id: I2f64f928e730355142ac369d8868cf9f65ca357e
    Closes-bug: #1872205
    Related-bug: #1840070
    (cherry picked from commit 3c234603a9c8e443dc3d38989e3e78147757d1da)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to designate (master)

Reviewed: https://review.opendev.org/719114
Committed: https://git.openstack.org/cgit/openstack/designate/commit/?id=7ff2f3956d3c543bbeaf5aafaf214965a1f44df3
Submitter: Zuul
Branch: master

commit 7ff2f3956d3c543bbeaf5aafaf214965a1f44df3
Author: Erik Olof Gunnar Andersson <email address hidden>
Date: Fri Apr 10 15:42:29 2020 -0700

    Simplify Coordination implementation

    This patch attempts to simplify the coordination
    implementation. It also removes some unnecessary
    exception handling that could hide valid issues, e.g.
    the use of unsupported drivers.

    - Use inbuilt heartbeat.
    - Fixed bug with etcd3 driver requiring name to be encoded.
    - Only use grouping if it is used by the service.
    - Remove unnecessary exception handling.

    Closes-Bug: #1872858
    Related-Bug: #1872205
    Change-Id: Ic659c52bf3adca2e97c8b669fe4fa1b5ed992e90

Revision history for this message
Takashi Kajinami (kajinamit) wrote :

It seems this was an issue with configuration made by kolla-ansible (or the requirement in Designate ?)

I'll close this bug from tooz side. Please reopen this if I misunderstand the problem.

Changed in python-tooz:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.