Cinder

Online database schema migrations to Xena always fail

Bug #1942077 reported by Radosław Piliszek on 2021-08-30

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Cinder	Fix Released	Critical	Rajat Dhasmana	Cinder xena-rc1 "19.0.0.0rc1"

Bug Description

It seems to be some recent issue.
Wallaby deploys fine and upgrade to Xena fails.

Stack trace:

2021-08-30 10:44:07.227 16 ERROR cinder.cmd.manage [req-1452bc60-5198-4e3c-a6bf-00a81d52994e - - - - -] Error attempting to run volume_use_quota_online_data_migration: AttributeError: 'NoneType' object has no attribute 'startswith'
2021-08-30 10:44:07.227 16 ERROR cinder.cmd.manage Traceback (most recent call last):
2021-08-30 10:44:07.227 16 ERROR cinder.cmd.manage File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/cmd/manage.py", line 238, in _run_migration
2021-08-30 10:44:07.227 16 ERROR cinder.cmd.manage found, done = migration_meth(ctxt, count)
2021-08-30 10:44:07.227 16 ERROR cinder.cmd.manage File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/db/api.py", line 1947, in volume_use_quota_online_data_migration
2021-08-30 10:44:07.227 16 ERROR cinder.cmd.manage IMPL.volume_use_quota_online_data_migration(context, max_count)
2021-08-30 10:44:07.227 16 ERROR cinder.cmd.manage File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1010, in wrapper
2021-08-30 10:44:07.227 16 ERROR cinder.cmd.manage return fn(*args, **kwargs)
2021-08-30 10:44:07.227 16 ERROR cinder.cmd.manage File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/db/sqlalchemy/api.py", line 7401, in volume_use_quota_online_data_migration
2021-08-30 10:44:07.227 16 ERROR cinder.cmd.manage return use_quota_online_data_migration(context, max_count, 'Volume',
2021-08-30 10:44:07.227 16 ERROR cinder.cmd.manage File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1010, in wrapper
2021-08-30 10:44:07.227 16 ERROR cinder.cmd.manage return fn(*args, **kwargs)
2021-08-30 10:44:07.227 16 ERROR cinder.cmd.manage File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/db/sqlalchemy/api.py", line 7436, in use_quota_online_data_migration
2021-08-30 10:44:07.227 16 ERROR cinder.cmd.manage resource.use_quota = calculate_use_quota(resource)
2021-08-30 10:44:07.227 16 ERROR cinder.cmd.manage File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/db/sqlalchemy/api.py", line 7398, in calculate_use_quota
2021-08-30 10:44:07.227 16 ERROR cinder.cmd.manage return not (volume.migration_status.startswith('target:') or
2021-08-30 10:44:07.227 16 ERROR cinder.cmd.manage AttributeError: 'NoneType' object has no attribute 'startswith'
2021-08-30 10:44:07.227 16 ERROR cinder.cmd.manage
2021-08-30 10:44:07.275 16 ERROR cinder.cmd.manage [req-1452bc60-5198-4e3c-a6bf-00a81d52994e - - - - -] Error attempting to run snapshot_use_quota_online_data_migration: TypeError: cannot unpack non-iterable NoneType object
2021-08-30 10:44:07.275 16 ERROR cinder.cmd.manage Traceback (most recent call last):
2021-08-30 10:44:07.275 16 ERROR cinder.cmd.manage File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/cmd/manage.py", line 238, in _run_migration
2021-08-30 10:44:07.275 16 ERROR cinder.cmd.manage found, done = migration_meth(ctxt, count)
2021-08-30 10:44:07.275 16 ERROR cinder.cmd.manage TypeError: cannot unpack non-iterable NoneType object
2021-08-30 10:44:07.275 16 ERROR cinder.cmd.manage

Tags:

Sofia Enriquez (lsofia-enriquez) on 2021-09-01

Changed in cinder:
importance:	Undecided → Medium
tags:	added: db schema

Sofia Enriquez (lsofia-enriquez) on 2021-09-01

Changed in cinder:
importance:	Medium → High

Revision history for this message

Eric Harney (eharney) wrote on 2021-09-01:

volume_use_quota_online_data_migation() may need to handling added for the case where volume.migration_status is None.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-09-02: Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/807083

Changed in cinder:
status:	New → In Progress

Rajat Dhasmana (whoami-rajat) on 2021-09-02

Changed in cinder:
assignee:	nobody → Rajat Dhasmana (whoami-rajat)

Brian Rosmaita (brian-rosmaita) on 2021-09-08

Changed in cinder:
importance:	High → Critical
milestone:	none → xena-rc1

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-09-15: Fix merged to cinder (master)

Reviewed: https://review.opendev.org/c/openstack/cinder/+/807083
Committed: https://opendev.org/openstack/cinder/commit/53c3b192ec3327c5426810e227abf9fe25540ae1
Submitter: "Zuul (22348)"
Branch: master

commit 53c3b192ec3327c5426810e227abf9fe25540ae1
Author: Rajat Dhasmana <email address hidden>
Date: Thu Sep 2 05:08:25 2021 -0400

Fix: Online migration for volume_use_quota_online_data_migration

    In the calculate_use_quota method, added a check for the case when
    there is no migration i.e. volume.migration_status is None and
    return True in that case since we should count that volume in quotas.

Also changed the property from admin_metadata to volume_admin_metadata
since we are dealing with ORM objects and not OVO.

It also fixes another issue where the db/api layer wasn't returning
the result of total and updated records.

Closes-Bug: #1942077
Change-Id: I640e039833001a91b23bfe0e6bb9c4e083d012a3

Changed in cinder:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-09-17: Fix included in openstack/cinder 19.0.0.0rc1

This issue was fixed in the openstack/cinder 19.0.0.0rc1 release candidate.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.