Mistral on gates is old and does not have the required patches

Is this happening on other patches too? Because the failure linked here is actually related to the code being tested in the change. The cron_triggers call is added in https://review.openstack.org/#/c/469608 so this specific failure can't be happening anywhere else. This is CI doing its job. :-)

Although this is two changes that added more Mistral calls in undercloud post-config that have run into these spurious failures, so there's definitely a pattern.

The error is shown by the patch but it is not related to it.

As seen here: http://logs.openstack.org/08/469608/13/check/gate-tripleo-ci-centos-7-undercloud-oooq/ce4f18f/logs/undercloud/var/log/mistral/api.log.txt.gz#_2017-08-31_19_59_43_964

It is caused by https://bugs.launchpad.net/mistral/+bug/1711069

We are missing

https://review.openstack.org/#/c/493550/ (master)
or
https://review.openstack.org/#/c/498358/ (backported to pike)

I have no idea why the patch is not present I've seen promotions yesterday and two days ago. this should have landed on ci. Does anyone have an idea on why is that not so?

Looking at https://trunk.rdoproject.org/centos7-master/current-passed-ci/openstack-mistral-5.0.0-0.20170823122251.1a8837b.el7.centos.src.rpm it does contain the patch needed.

and the tell tale that the version running on ci doesn't have is the line 105 from mistral/utils/openstack/keystone.py

<from the log files listed above>
 File "/usr/lib/python2.7/site-packages/mistral/utils/openstack/keystone.py", line 105, in _admin_client
    trust_id=trust_id

on a patched file line 105 is blank and that line moved to line 111

This is not unique to mistral and has happened before http://lists.openstack.org/pipermail/openstack-dev/2017-March/113529.html

Yeah I suspect this is the same NVR issue we hit last release ref http://lists.openstack.org/pipermail/openstack-dev/2017-March/113529.html

Looking at the logs confirms it:

http://logs.openstack.org/08/469608/13/check/gate-tripleo-ci-centos-7-undercloud-oooq/80a6ddc/logs/rpm-qa.txt.gz

Shows the wrong version of mistral: openstack-mistral-api-5.0.0-1.el7.noarch

We're not using the current-tripleo version which is https://trunk.rdoproject.org/centos7-master/current-tripleo/openstack-mistral-api-5.0.0-0.20170823122251.1a8837b.el7.centos.noarch.rpm

The repos are configured here:

http://logs.openstack.org/08/469608/13/check/gate-tripleo-ci-centos-7-undercloud-oooq/80a6ddc/logs/undercloud/etc/yum.repos.d/

We're using this repo:

http://mirror.dfw.rax.openstack.org:8080/buildlogs.centos/centos/7/cloud/x86_64/openstack-pike/

Instead of this one, because the stable/pike NVR is newer:

http://mirror.dfw.rax.openstack.org:8080/rdo/centos7/21/a2/21a252272f31950155e4a8086561214b2743c1ae_774c92ec/

We never really reached a satisfactory conclusion last time, AFAIK the only solution is/was to cut a new release for everything from master, so the NVR is newer than stable/pike. Unless we blacklist everything from the pike-testing repo which is built by delorean, but that may prove fragile?

As per the suggestion above we are releasing a bugfix for pike https://review.openstack.org/#/c/500522/ it still will have to pass the promotion pipeline but it is a start (right now the promotion has failed for the last 11 days)

Ok so I can confirm that installing via quickstart with tripleo-ci-master does indeed give you a ton of things from delorean-pike-testing which should instead be from delorean, I'm looking at adding a blacklist to our repo config so we can exclude all of what we know should be installed from delorean instead of delorean-pike-testing (which should really only provide the non openstack deps AFAIK?)

we have a new release of Mistral so it should help: https://review.openstack.org/#/c/500522/

I guess we probably need a promotion to pick up the new release, as this shows we're still using pike mistral:

http://logs.openstack.org/80/499580/2/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/b25296d/logs/rpm-qa.txt.gz

I'll push a patch adding a blacklist for openstack* to delorean-pike-testing and hopefully that will help (at least as a stopgap but perhaps we should always blacklist things we know shouldn't be used from the -testing repo?)

Fix proposed to branch: master
Review: https://review.openstack.org/502102

@shardy thanks! The promotion failed today due to the rdo infra being down

Reviewed: https://review.openstack.org/502102
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart/commit/?id=adf3bbd53c7642dd1f4fc7f6c87c030fe81d14e9
Submitter: Jenkins
Branch: master

commit adf3bbd53c7642dd1f4fc7f6c87c030fe81d14e9
Author: Steven Hardy <email address hidden>
Date: Fri Sep 8 16:33:31 2017 +0100

    Set priorities for all delorean repos, not only -current

    Currently we're incorrectly installing from delorean-pike-testing
    for a bunch of packages which we should install from the delorean
    repo - this aligns priorities with those we always set in tripleo.sh[0]
    so that the delorean repo should take priority.

    [0]
    https://github.com/openstack-infra/tripleo-ci/blob/3ef026abc3bde8e6f23f51a72c21f74fbd07f7e1/scripts/tripleo.sh#L324-L371

    Change-Id: Ide4557e57fbe6257d163a233a7cf5e4aca0c558f
    Partial-Bug: #1714361

Also see related patch https://review.openstack.org/#/c/502360/