MTC openstack command failed when standby controller is down

Issue w/ openstack regression after the rebase to Ussuri; maybe related to the rebase. Marking as stx.4.0 gating for now until further investigation by the distro.openstack team.

Uploaded log file doesn't match with bug description. Log file for date 20200720, ALL_NODES_20200720.091540, but issue description, happens at 20200722 and on such command history in bash.log

[2020-07-22 15:16:36,912] 314 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.144.1:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-swact controller-1'

Please ensure log file correct.

Will reproduce depends on bug description.

issue reproduced

controller-0:~$ openstack --os-username 'admin' --os-password 'Local.123' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne image list
Internal Server Error (HTTP 500)
controller-0:~$
controller-0:~$ openstack --os-username 'admin' --os-password 'Local.123' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server list
Internal Server Error (HTTP 500)
controller-0:~$
controller-0:~$ openstack --os-username 'admin' --os-password 'Local.123' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne flavor list
Internal Server Error (HTTP 500)
controller-0:~$
controller-0:~$ openstack --os-username 'admin' --os-password 'Local.123' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne volume list

Internal Server Error (HTTP 500)
controller-0:~$
controller-0:~$ openstack --os-username 'admin' --os-password 'Local.123' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne endpoint list

Internal Server Error (HTTP 500)
controller-0:~$
controller-0:~$
controller-0:~$ system --os-username 'admin' --os-password 'Local.123' --os-project-name admin --os-auth-url http://192.188.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-list
+----+--------------+-------------+----------------+-------------+--------------+
| id | hostname | personality | administrative | operational | availability |
+----+--------------+-------------+----------------+-------------+--------------+
| 1 | controller-0 | controller | unlocked | enabled | available |
| 2 | controller-1 | controller | unlocked | disabled | offline |
| 3 | compute-0 | worker | unlocked | enabled | available |
| 4 | compute-1 | worker | unlocked | enabled | available |
+----+--------------+-------------+----------------+-------------+--------------+
controller-0:~$

There is just one mariadb-server pod on standby controller work well. So if standby controller is down, no mariadb service, which keystone-api service doesn't work. Then "openstack server list" fail

controller-0:~$ kubectl get pods -n openstack | grep mariadb
mariadb-ingress-797fcf4f55-9pb4f 1/1 Running 0 44m
mariadb-ingress-797fcf4f55-wd58l 1/1 Running 0 15m
mariadb-ingress-error-pages-6755f56fbf-k4rrm 1/1 Running 0 32m
mariadb-server-0 1/1 Running 1 6m17s
mariadb-server-1 0/1 Running 5 35m
controller-0:~$ system --os-username 'admin' --os-password 'Local.123' --os-project-name admin --os-auth-url http://192.188.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-list
+----+--------------+-------------+----------------+-------------+--------------+
| id | hostname | personality | administrative | operational | availability |
+----+--------------+-------------+----------------+-------------+--------------+
| 1 | controller-0 | controller | unlocked | enabled | available |
| 2 | controller-1 | controller | unlocked | enabled | available |
| 3 | compute-0 | worker | unlocked | enabled | available |
| 4 | compute-1 | worker | unlocked | enabled | available |
+----+--------------+-------------+----------------+-------------+--------------+
controller-0:~$

a bit progress made today: the same test case could pass on Duplex, while it failed on multi-nodes deployment. The major diff is that "garbd" is enabled on multi-nodes while not there on duplex.

Need further debugging.

suggest to continue addressing this LP in stx.4.0 maintenance releases.

Fix proposed to branch: master
Review: https://review.opendev.org/744486

Reviewed: https://review.opendev.org/744486
Committed: https://git.openstack.org/cgit/starlingx/openstack-armada-app/commit/?id=2f664927c4d990c2151677da5be2b0d0ac355121
Submitter: Zuul
Branch: master

commit 2f664927c4d990c2151677da5be2b0d0ac355121
Author: Martin, Chen <email address hidden>
Date: Mon Aug 3 21:40:02 2020 +0800

    Update mariadb-server suspect_timeout to default value to align
    with garbd's suspect_timeout

    In openstack-helm-infra, it launch evs.suspect_timeout=PT30S
    for mariadb-server in configmap, mariadb-etc. This setting is
    for three mariadb-server pod deployment, every mariadb-server
    with same setting suspect_timeout=30s. But after change to two
    mariadb-server and one garbd arbitrator. Setting in configmap
    mariadb-etc evs.suspect_timeout=PT30S, only takes effect for 2
    mariadb-server, for garbd arbitrator, it use galera default
    setting evs.suspect_timeout=PT5S. If mariadb-server-1 exit
    abnormal, after 5s, garbd arbitrator suspects mariadb-server-1
    is dead, but as not reach 30s, mariadb-server-0 thinks mariadb-server-1
    is not dead. In this state, quorum fail, garbd arbitrator and
    mariadb-server-0 both set to none primary component, service
    down.
    For fix solution, set value.conf.data.config_override to override
    wsrep_provider_option in mariadb helm chart, which makes garbd
    arbitrator and mariadb-server launch with same setting for
    "evs.suspect_timeout=PT5S", default value. By this way, mariadb
    server recovery time will also improve. To update setting for
    "evs.suspect_timeout", it should both update override for mariadb
    and garbd helm chart.

    Setting for "gmcast.listen_addr=tcp://0.0.0.0:<port>", takes
    effect for both ipv4 and ipv6. So keeps such setting.

    Reference link for wsrep option and galera cluster quorum
    https://mariadb.com/kb/en/wsrep_provider_options/
    https://galeracluster.com/library/documentation/weighted-quorum.html

    Closes-Bug: 1888546

    Change-Id: I06983cf0d91d4d9aa88f352e64b1e6571b816ec6
    Signed-off-by: Martin, Chen <email address hidden>

Fix proposed to branch: r/stx.4.0
Review: https://review.opendev.org/747120

Fix proposed to branch: r/stx.3.0
Review: https://review.opendev.org/747124

Fix proposed to branch: r/stx.3.0
Review: https://review.opendev.org/747125

Reviewed: https://review.opendev.org/747120
Committed: https://git.openstack.org/cgit/starlingx/openstack-armada-app/commit/?id=4340d9e8e6f3e3e39afe167d1d21268923b65482
Submitter: Zuul
Branch: r/stx.4.0

commit 4340d9e8e6f3e3e39afe167d1d21268923b65482
Author: Martin, Chen <email address hidden>
Date: Thu Aug 20 16:11:02 2020 +0800

    Update mariadb-server suspect_timeout to default value to align
    with garbd's suspect_timeout

    In openstack-helm-infra, it launch evs.suspect_timeout=PT30S
    for mariadb-server in configmap, mariadb-etc. This setting is
    for three mariadb-server pod deployment, every mariadb-server
    with same setting suspect_timeout=30s. But after change to two
    mariadb-server and one garbd arbitrator. Setting in configmap
    mariadb-etc evs.suspect_timeout=PT30S, only takes effect for 2
    mariadb-server, for garbd arbitrator, it use galera default
    setting evs.suspect_timeout=PT5S. If mariadb-server-1 exit
    abnormal, after 5s, garbd arbitrator suspects mariadb-server-1
    is dead, but as not reach 30s, mariadb-server-0 thinks mariadb-server-1
    is not dead. In this state, quorum fail, garbd arbitrator and
    mariadb-server-0 both set to none primary component, service
    down.
    For fix solution, set value.conf.data.config_override to override
    wsrep_provider_option in mariadb helm chart, which makes garbd
    arbitrator and mariadb-server launch with same setting for
    "evs.suspect_timeout=PT5S", default value. By this way, mariadb
    server recovery time will also improve. To update setting for
    "evs.suspect_timeout", it should both update override for mariadb
    and garbd helm chart.

    Setting for "gmcast.listen_addr=tcp://0.0.0.0:<port>", takes
    effect for both ipv4 and ipv6. So keeps such setting.

    Reference link for wsrep option and galera cluster quorum
    https://mariadb.com/kb/en/wsrep_provider_options/
    https://galeracluster.com/library/documentation/weighted-quorum.html

    Closes-Bug: 1888546

    Change-Id: Ie26fd33488616b13bc16102dd92d1be2143c82ce
    Signed-off-by: Martin, Chen <email address hidden>

Adding the stx.3.0 label since the dev prime is proposing to fix this in stx.3.0 as well

Reviewed: https://review.opendev.org/747124
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=245023894acfb163b4ed73ccded72914550d982c
Submitter: Zuul
Branch: r/stx.3.0

commit 245023894acfb163b4ed73ccded72914550d982c
Author: Martin, Chen <email address hidden>
Date: Thu Aug 20 16:26:50 2020 +0800

    Update mariadb-server suspect_timeout to default value to align
    with garbd's suspect_timeout

    In openstack-helm-infra, it launch evs.suspect_timeout=PT30S
    for mariadb-server in configmap, mariadb-etc. This setting is
    for three mariadb-server pod deployment, every mariadb-server
    with same setting suspect_timeout=30s. But after change to two
    mariadb-server and one garbd arbitrator. Setting in configmap
    mariadb-etc evs.suspect_timeout=PT30S, only takes effect for 2
    mariadb-server, for garbd arbitrator, it use galera default
    setting evs.suspect_timeout=PT5S. If mariadb-server-1 exit
    abnormal, after 5s, garbd arbitrator suspects mariadb-server-1
    is dead, but as not reach 30s, mariadb-server-0 thinks mariadb-server-1
    is not dead. In this state, quorum fail, garbd arbitrator and
    mariadb-server-0 both set to none primary component, service
    down.
    For fix solution, set value.conf.data.config_override to override
    wsrep_provider_option in mariadb helm chart, which makes garbd
    arbitrator and mariadb-server launch with same setting for
    "evs.suspect_timeout=PT5S", default value. By this way, mariadb
    server recovery time will also improve. To update setting for
    "evs.suspect_timeout", it should both update override for mariadb
    and garbd helm chart.

    Setting for "gmcast.listen_addr=tcp://0.0.0.0:<port>", takes
    effect for both ipv4 and ipv6. So keeps such setting.

    Reference link for wsrep option and galera cluster quorum
    https://mariadb.com/kb/en/wsrep_provider_options/
    https://galeracluster.com/library/documentation/weighted-quorum.html

    Closes-Bug: 1888546

    Change-Id: I92af77fab929c9f598b7dc41543db6ad6238f812
    Signed-off-by: Martin, Chen <email address hidden>

Reviewed: https://review.opendev.org/747125
Committed: https://git.openstack.org/cgit/starlingx/openstack-armada-app/commit/?id=ef4745c7fd583a1a44e8dcb2631a7634150b44b6
Submitter: Zuul
Branch: r/stx.3.0

commit ef4745c7fd583a1a44e8dcb2631a7634150b44b6
Author: Martin, Chen <email address hidden>
Date: Thu Aug 20 16:33:26 2020 +0800

    Update mariadb-server suspect_timeout to default value to align
    with garbd's suspect_timeout

    In openstack-helm-infra, it launch evs.suspect_timeout=PT30S
    for mariadb-server in configmap, mariadb-etc. This setting is
    for three mariadb-server pod deployment, every mariadb-server
    with same setting suspect_timeout=30s. But after change to two
    mariadb-server and one garbd arbitrator. Setting in configmap
    mariadb-etc evs.suspect_timeout=PT30S, only takes effect for 2
    mariadb-server, for garbd arbitrator, it use galera default
    setting evs.suspect_timeout=PT5S. If mariadb-server-1 exit
    abnormal, after 5s, garbd arbitrator suspects mariadb-server-1
    is dead, but as not reach 30s, mariadb-server-0 thinks mariadb-server-1
    is not dead. In this state, quorum fail, garbd arbitrator and
    mariadb-server-0 both set to none primary component, service
    down.
    For fix solution, set value.conf.data.config_override to override
    wsrep_provider_option in mariadb helm chart, which makes garbd
    arbitrator and mariadb-server launch with same setting for
    "evs.suspect_timeout=PT5S", default value. By this way, mariadb
    server recovery time will also improve. To update setting for
    "evs.suspect_timeout", it should both update override for mariadb
    and garbd helm chart.

    Setting for "gmcast.listen_addr=tcp://0.0.0.0:<port>", takes
    effect for both ipv4 and ipv6. So keeps such setting.

    Reference link for wsrep option and galera cluster quorum
    https://mariadb.com/kb/en/wsrep_provider_options/
    https://galeracluster.com/library/documentation/weighted-quorum.html

    Closes-Bug: 1888546

    Depends-on: https://review.opendev.org/#/c/747093/

    Change-Id: Ie648745b5670fffb3a513a312e0b085cfea4544a
    Signed-off-by: Martin, Chen <email address hidden>