kolla-ansible

Grafana takes longer to get online than expected by kolla-ansible

Bug #1769962 reported by Allan Krueger
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla-ansible
Fix Released
Medium
Dr. Jens Harbott

Bug Description

When deploying a cluster, sometimes the Grafana container fails after the 10 connection retries.

The error reports a 503, but if I go to the grafana endpoint a little after it is online and working.

I can then run "kolla-ansible reconfigure", and it has success in the container.

A suggestion would be to increase the number of connection retries for the grafana container.

Im using kolla-ansinble from git, but I experienced the same problem with kolla from pip.

Install type is "source", distro is "centos", openstack is "queens".

The kolla-images where pushed from the repos.

Revision history for this message
João Feteira (joaofet) wrote :

I also came across this. For me changing the retry tinmer to 30 does the trick and the deployment is always sucessful.

Running kolla-ansible pip install rocky version

Revision history for this message
nissekalle (nissekalle) wrote :

This happens the same with Rocky version and kolla-ansible 7.0.1 binary

By rerunning deploy it gets up and running and works.

Revision history for this message
Dr. Jens Harbott (j-harbott) wrote :

Issue sometimes happens in CI, too, see e.g. https://522d8c65037ee2f8378e-e8ad39bee4b03d74ca06a7b7508191a6.ssl.cf1.rackcdn.com/820074/1/check/kolla-ansible-centos8s-source-prometheus-efk/4c38ff2/primary/logs/docker_logs/grafana.txt

2021-12-02T00:01:20.332705401Z t=2021-12-02T00:01:20+0000 lvl=info msg="migrations completed" logger=migrator performed=346 skipped=0 duration=50.878447223s

Default wait time is 20s, patch proposed in https://review.opendev.org/c/openstack/kolla-ansible/+/820400

Changed in kolla-ansible:
status: New → In Progress
assignee: nobody → Dr. Jens Harbott (j-harbott)
Radosław Piliszek (yoctozepto)
Changed in kolla-ansible:
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/xena)

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/821907

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (master)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/820400
Committed: https://opendev.org/openstack/kolla-ansible/commit/f8f34e0c471c37c4f9e39135bbcf6501f00440aa
Submitter: "Zuul (22348)"
Branch: master

commit f8f34e0c471c37c4f9e39135bbcf6501f00440aa
Author: Dr. Jens Harbott <email address hidden>
Date: Fri Dec 3 19:28:29 2021 +0100

    Bump timeout for grafana startup

    The initial migrations when starting grafana for the first time may
    sometimes take much longer than 20s, we have seen samples up to near
    60s. Allow 120s to have some margin. Also make the timeout parameters
    configurable.

    Closes-Bug: 1769962
    Signed-off-by: Dr. Jens Harbott <email address hidden>
    Change-Id: If9186d8aa65150c492657550064789e211dbb570

Changed in kolla-ansible:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/822018

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/822019

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/821907
Committed: https://opendev.org/openstack/kolla-ansible/commit/f6ff1a0a5899f9ce31a3bc8058aa21ebd16e1dd9
Submitter: "Zuul (22348)"
Branch: stable/xena

commit f6ff1a0a5899f9ce31a3bc8058aa21ebd16e1dd9
Author: Dr. Jens Harbott <email address hidden>
Date: Fri Dec 3 19:28:29 2021 +0100

    Bump timeout for grafana startup

    The initial migrations when starting grafana for the first time may
    sometimes take much longer than 20s, we have seen samples up to near
    60s. Allow 120s to have some margin. Also make the timeout parameters
    configurable.

    Closes-Bug: 1769962
    Signed-off-by: Dr. Jens Harbott <email address hidden>
    Change-Id: If9186d8aa65150c492657550064789e211dbb570
    (cherry picked from commit f8f34e0c471c37c4f9e39135bbcf6501f00440aa)

tags: added: in-stable-xena
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/822018
Committed: https://opendev.org/openstack/kolla-ansible/commit/d7ebe7c24061f706e57d352f5d91abcd0435fd85
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit d7ebe7c24061f706e57d352f5d91abcd0435fd85
Author: Dr. Jens Harbott <email address hidden>
Date: Fri Dec 3 19:28:29 2021 +0100

    Bump timeout for grafana startup

    The initial migrations when starting grafana for the first time may
    sometimes take much longer than 20s, we have seen samples up to near
    60s. Allow 120s to have some margin. Also make the timeout parameters
    configurable.

    Closes-Bug: 1769962
    Signed-off-by: Dr. Jens Harbott <email address hidden>
    Change-Id: If9186d8aa65150c492657550064789e211dbb570
    (cherry picked from commit f8f34e0c471c37c4f9e39135bbcf6501f00440aa)

tags: added: in-stable-wallaby
tags: added: in-stable-victoria
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/victoria)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/822019
Committed: https://opendev.org/openstack/kolla-ansible/commit/8c63d9360d6127d59ae4daf1784e5349dda71df3
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 8c63d9360d6127d59ae4daf1784e5349dda71df3
Author: Dr. Jens Harbott <email address hidden>
Date: Fri Dec 3 19:28:29 2021 +0100

    Bump timeout for grafana startup

    The initial migrations when starting grafana for the first time may
    sometimes take much longer than 20s, we have seen samples up to near
    60s. Allow 120s to have some margin. Also make the timeout parameters
    configurable.

    Closes-Bug: 1769962
    Signed-off-by: Dr. Jens Harbott <email address hidden>
    Change-Id: If9186d8aa65150c492657550064789e211dbb570
    (cherry picked from commit f8f34e0c471c37c4f9e39135bbcf6501f00440aa)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 11.2.0

This issue was fixed in the openstack/kolla-ansible 11.2.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 12.3.0

This issue was fixed in the openstack/kolla-ansible 12.3.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 13.0.1

This issue was fixed in the openstack/kolla-ansible 13.0.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 14.0.0.0rc1

This issue was fixed in the openstack/kolla-ansible 14.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.