DB check appears to not be working right

Bug #1698900 reported by Jesse Pretorius
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Identity (keystone)
Fix Released
High
Lance Bragstad

Bug Description

Using current master of keystone, executing a keystone-manage db_sync --check seems to always return the RC of 2 regardless of the steps previously confirmed. This happens until the contract is done, then it returns 0.

Steps to reproduce:

root@keystone1:/# mysqladmin drop keystone; mysql create keystone

root@keystone1:/# keystone-manage db_sync --check; echo $?
2

root@keystone1:/# keystone-manage db_sync --expand; echo $?
0

root@keystone1:/# keystone-manage db_sync --check; echo $?
2

root@keystone1:/# keystone-manage db_sync --migrate; echo $?
0

root@keystone1:/# keystone-manage db_sync --check; echo $?
2

root@keystone1:/# keystone-manage db_sync --contract; echo $?
0

root@keystone1:/# keystone-manage db_sync --check; echo $?
0

Not getting the right return codes or advise from the check can spell disaster for automation that uses it, or humans following the documented migration process.

Tags: sql
Revision history for this message
Lance Bragstad (lbragstad) wrote :

Looks like the `--check` functionality was added in Pike, so this won't need to back ported. Marking this as High since automation projects can't trust the information we provide.

Changed in keystone:
status: New → Confirmed
importance: Undecided → High
tags: added: sql
Revision history for this message
Lance Bragstad (lbragstad) wrote :

One of the issues here is caused by a try/except when collecting version information for each repository [0]. The logic within the try/except is attempting to assign the version of each repository (expand, migrate, and contract) to a variable. The issue is that with a new installation, not all of those repositories might exist yet, depending on what commands you've issued.

For example, if you run `keystone-manage db_sync --expand` on a fresh database, the legacy migrations will be run and the expand migrations will be run. The data_migration_repo and contract repositories will not exist yet, since those repositories haven't been invoked yet. This causes the `--check` to fail with inaccurate information after the first `--expand` is run because the migration.exception.DbMigrationError is caught and we assume it to mean the legacy repository doesn't exist, which isn't true.

One possible solution is to break that try/except into multiple blocks for each repository and handle them separately.

[0] https://github.com/openstack/keystone/blob/45265c0ddffa399fcb10bed6eac98069decdf910/keystone/cmd/cli.py#L460-L469

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (master)

Fix proposed to branch: master
Review: https://review.openstack.org/475472

Changed in keystone:
assignee: nobody → Lance Bragstad (lbragstad)
status: Confirmed → In Progress
Changed in keystone:
milestone: none → pike-3
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (master)

Reviewed: https://review.openstack.org/475472
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=2a2f8535e2858400d68af74a8d813ca278f7a1b6
Submitter: Jenkins
Branch: master

commit 2a2f8535e2858400d68af74a8d813ca278f7a1b6
Author: Lance Bragstad <email address hidden>
Date: Mon Jun 19 17:41:13 2017 +0000

    Improve handling of database migration checks

    The `--check` subcommand is suppose to provide useful information
    and status codes depending on the state of the keystone database.
    Operators and automation use this information to determine what their
    next step is in a rolling upgrade. The current logic is broken
    becuase it doesn't account for new installations that might be
    relying on this information.

    This change breaks that case into multiple try/except statements and
    handles each appropriately so that the status code and logging
    information is accurate for operators and automation using this
    information for upgrading a new keystone database.

    Change-Id: I331fa663a99f79ea9a79a75e4ae07c45278556bf
    Closes-Bug: 1698900

Changed in keystone:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/keystone 12.0.0.0b3

This issue was fixed in the openstack/keystone 12.0.0.0b3 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.