kolla-ansible

cinder's rbd backend's image_upload_use_cinder_backend=True causes ERRORs and WARNINGs without extra config on user's side since Victoria

Bug #1991516 reported by Radosław Piliszek on 2022-10-03

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	kolla-ansible	Fix Released	Medium	Radosław Piliszek

Bug Description

When uploading volume to glance image, cinder looks at the backend's image_upload_use_cinder_backend config knob to decide whether to try link the glance image to a cloned volume made by cinder, i.e. by doing all work locally and only updating glance's locations for the image (when the knob is set to True). However, after all [1], [2] and [3], which happens since Victoria, this option requires further config from user (using volume type with image_service:store_id property (aka extra spec) set to the desired glance store (even if there is only one cinder store configured). This shows both in cinder-volume and glance-api logs as WARNINGs:

cinder-volume:
ERROR Failed to register image volume location cinder://ABCD .: cinder.exception.Invalid: HTTP 400 Bad Request: Invalid location
... (a juicy stack trace here) ...
WARNING Registration of image volume URI cinder://ABCD to image XYZ failed

glance-api
WARNING Invalid location uri cinder://ABCD

If it actually worked out-of-the-box, the image would have the cinder://-prefixed uri. This could be confusing as a default as image_upload_use_cinder_backend=True is not the upstream default. Furthermore, this entails image management dependent on cinder unlike the usual premise that glance is self-contained. Finally, it would actually break such images as there is no recommendation (nor support) to configure glance to be able to use cinder's pool of volumes (i.e. these images would only be usable by cinder and nova which have the cinder's ceph key, other services as well as downloading the image by the user would be broken).

My suggestion is to remove this config knob from Kolla Ansible's defaults because it seems to have been added accidentally (also do a codesearch on image_upload_use_cinder_backend and it seems not too popular - only Kolla Ansible seems to be enforcing it on users nowadays). Upgrade note would be included of course.

I have set this bug to Medium importance because it creates enough noise to consider it higher impacting than Low. Plus, as I mention above, a change in upstream support would cause the behaviour to change drastically (and "non-defaultly" so to speak).

[1] https://review.opendev.org/c/openstack/kolla-ansible/+/708114
[2] https://review.opendev.org/c/openstack/glance_store/+/746556
[3] https://review.opendev.org/c/openstack/cinder/+/661676/

Tags:

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-10-03: Fix proposed to kolla-ansible (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/860093

Changed in kolla-ansible:
status:	Triaged → In Progress

Revision history for this message

Radosław Piliszek (yoctozepto) wrote on 2022-10-03:

Yet another point against this overridden default - it forces users to use show_multiple_locations in Glance. The reason why the original fix disabling show_multiple_locations had to be reverted back then. [1]

[1] https://review.opendev.org/c/openstack/kolla-ansible/+/509792

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-10-10: Fix merged to kolla-ansible (master)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/860093
Committed: https://opendev.org/openstack/kolla-ansible/commit/a4b4043308f2caa4df2318648bfbfbab4e7902dc
Submitter: "Zuul (22348)"
Branch: master

commit a4b4043308f2caa4df2318648bfbfbab4e7902dc
Author: Radosław Piliszek <email address hidden>
Date: Mon Oct 3 15:58:32 2022 +0200

Fix image from volume upload ERRORs and WARNINGs with Ceph RBD

By resetting image_upload_use_cinder_backend to upstream default.

    When uploading volume to glance image, cinder looks at the backend's
    image_upload_use_cinder_backend config knob to decide whether to try link
    the glance image to a cloned volume made by cinder, i.e. by doing all work
    locally and only updating glance's locations for the image (when the knob
    is set to True). However, after all [1], [2] and [3], which happens since
    Victoria, this option requires further config from user (using volume type
    with image_service:store_id property (aka extra spec) set to the desired
    glance store (even if there is only one cinder store configured).

Please read the bug report as to why the option removal is the
best option (TL;DR it is the most compatible approach).

    [1] https://review.opendev.org/c/openstack/kolla-ansible/+/708114
    [2] https://review.opendev.org/c/openstack/glance_store/+/746556
    [3] https://review.opendev.org/c/openstack/cinder/+/661676

Closes-Bug: #1991516
Change-Id: Ife87ee0241d907a0c407eb21811a354ed1734408

Changed in kolla-ansible:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-10-10: Fix proposed to kolla-ansible (stable/yoga)

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/860834

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-10-10: Fix proposed to kolla-ansible (stable/xena)

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/860835

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-10-10: Fix proposed to kolla-ansible (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/860836

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-10-28: Fix merged to kolla-ansible (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/860834
Committed: https://opendev.org/openstack/kolla-ansible/commit/ec6fbf72cddcba5575466cf988eb5322b114a8b9
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit ec6fbf72cddcba5575466cf988eb5322b114a8b9
Author: Radosław Piliszek <email address hidden>
Date: Mon Oct 3 15:58:32 2022 +0200

Fix image from volume upload ERRORs and WARNINGs with Ceph RBD

By resetting image_upload_use_cinder_backend to upstream default.

    When uploading volume to glance image, cinder looks at the backend's
    image_upload_use_cinder_backend config knob to decide whether to try link
    the glance image to a cloned volume made by cinder, i.e. by doing all work
    locally and only updating glance's locations for the image (when the knob
    is set to True). However, after all [1], [2] and [3], which happens since
    Victoria, this option requires further config from user (using volume type
    with image_service:store_id property (aka extra spec) set to the desired
    glance store (even if there is only one cinder store configured).

Please read the bug report as to why the option removal is the
best option (TL;DR it is the most compatible approach).

    [1] https://review.opendev.org/c/openstack/kolla-ansible/+/708114
    [2] https://review.opendev.org/c/openstack/glance_store/+/746556
    [3] https://review.opendev.org/c/openstack/cinder/+/661676

    Closes-Bug: #1991516
    Change-Id: Ife87ee0241d907a0c407eb21811a354ed1734408
    (cherry picked from commit a4b4043308f2caa4df2318648bfbfbab4e7902dc)

tags:

added: in-stable-yoga

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-10-28: Fix merged to kolla-ansible (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/860835
Committed: https://opendev.org/openstack/kolla-ansible/commit/bca9fa7a47c701ee80889786aff2e7f6b222fc57
Submitter: "Zuul (22348)"
Branch: stable/xena

commit bca9fa7a47c701ee80889786aff2e7f6b222fc57
Author: Radosław Piliszek <email address hidden>
Date: Mon Oct 3 15:58:32 2022 +0200

Fix image from volume upload ERRORs and WARNINGs with Ceph RBD

By resetting image_upload_use_cinder_backend to upstream default.

    When uploading volume to glance image, cinder looks at the backend's
    image_upload_use_cinder_backend config knob to decide whether to try link
    the glance image to a cloned volume made by cinder, i.e. by doing all work
    locally and only updating glance's locations for the image (when the knob
    is set to True). However, after all [1], [2] and [3], which happens since
    Victoria, this option requires further config from user (using volume type
    with image_service:store_id property (aka extra spec) set to the desired
    glance store (even if there is only one cinder store configured).

Please read the bug report as to why the option removal is the
best option (TL;DR it is the most compatible approach).

    [1] https://review.opendev.org/c/openstack/kolla-ansible/+/708114
    [2] https://review.opendev.org/c/openstack/glance_store/+/746556
    [3] https://review.opendev.org/c/openstack/cinder/+/661676

    Closes-Bug: #1991516
    Change-Id: Ife87ee0241d907a0c407eb21811a354ed1734408
    (cherry picked from commit a4b4043308f2caa4df2318648bfbfbab4e7902dc)

tags:	added: in-stable-xena
tags:	added: in-stable-wallaby

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-10-28: Fix merged to kolla-ansible (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/860836
Committed: https://opendev.org/openstack/kolla-ansible/commit/07398b038de8149158ca83993c1d21d91022d4b0
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 07398b038de8149158ca83993c1d21d91022d4b0
Author: Radosław Piliszek <email address hidden>
Date: Mon Oct 3 15:58:32 2022 +0200

Fix image from volume upload ERRORs and WARNINGs with Ceph RBD

By resetting image_upload_use_cinder_backend to upstream default.

    When uploading volume to glance image, cinder looks at the backend's
    image_upload_use_cinder_backend config knob to decide whether to try link
    the glance image to a cloned volume made by cinder, i.e. by doing all work
    locally and only updating glance's locations for the image (when the knob
    is set to True). However, after all [1], [2] and [3], which happens since
    Victoria, this option requires further config from user (using volume type
    with image_service:store_id property (aka extra spec) set to the desired
    glance store (even if there is only one cinder store configured).

Please read the bug report as to why the option removal is the
best option (TL;DR it is the most compatible approach).

    [1] https://review.opendev.org/c/openstack/kolla-ansible/+/708114
    [2] https://review.opendev.org/c/openstack/glance_store/+/746556
    [3] https://review.opendev.org/c/openstack/cinder/+/661676

    Closes-Bug: #1991516
    Change-Id: Ife87ee0241d907a0c407eb21811a354ed1734408
    (cherry picked from commit a4b4043308f2caa4df2318648bfbfbab4e7902dc)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-11-07: Fix included in openstack/kolla-ansible 13.6.0

#10

This issue was fixed in the openstack/kolla-ansible 13.6.0 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-11-07: Fix included in openstack/kolla-ansible 14.6.0

#11

This issue was fixed in the openstack/kolla-ansible 14.6.0 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-12-09: Fix included in openstack/kolla-ansible 15.0.0.0rc1

#12

This issue was fixed in the openstack/kolla-ansible 15.0.0.0rc1 release candidate.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-08-31: Fix included in openstack/kolla-ansible wallaby-eol

#13

This issue was fixed in the openstack/kolla-ansible wallaby-eol release.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.