Creating volume backup fails when iscsi_tcp module is not inserted when using LVM backend

Bug #1863094 reported by Simon Andrejko
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla-ansible
Fix Released
Medium
Radosław Piliszek
Rocky
Fix Released
Medium
Radosław Piliszek
Stein
Fix Released
Medium
Radosław Piliszek
Train
Fix Released
Medium
Radosław Piliszek
Ussuri
Fix Released
Medium
Radosław Piliszek

Bug Description

Description
===========
Backup of created cinder volume with backend set as LVM fails in case iscsi_tcp kernel module is not inserted on storage node. During backup iscsi do discovery and requests iscsi_tcp module. cinder_backup container could not insert such module and fails.

Steps to reproduce
==================
Actions:

1.
$ openstack volume create --image cirros-0.3.5 --size 3 test_volume
-> Volume is created.

2.
$ openstack volume backup create --name test_backup test_volume
-> Backup of the volume fails.

Expected result
===============
After step (2) backup of the volume should be created successfully.

Logs
====
2020-02-12 23:44:20.717 39 DEBUG oslo_concurrency.processutils [-] u'iscsiadm -m discovery -t sendtargets -I default -p 9.9.9.11:3260' failed. Not Retrying. execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:457

2020-02-12 23:44:20.718 39 DEBUG oslo.privsep.daemon [-] privsep: Exception during request[140264042973136]: Unexpected error while running command.
Command: iscsiadm -m discovery -t sendtargets -I default -p 9.9.9.11:3260
Exit code: 21
Stdout: u''
Stderr: u"libkmod: kmod_module_insert_module: could not find module by name='iscsi_tcp'\niscsiadm: Could not insert module tcp. Kmod error -38\niscsiadm: Could not load transport tcp.Dropping interface default.\n" loop /usr/lib/python2.7
/site-packages/oslo_privsep/daemon.py:449
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 445, in loop
    reply = self._process_cmd(*msg)
  File "/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 428, in _process_cmd
    ret = func(*f_args, **f_kwargs)
  File "/usr/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 209, in _wrap
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/os_brick/privileged/rootwrap.py", line 194, in execute_root
    return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
  File "/usr/lib/python2.7/site-packages/os_brick/privileged/rootwrap.py", line 143, in custom_execute
    on_completion=on_completion, *cmd, **kwargs)
  File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 424, in execute
    cmd=sanitized_cmd)
ProcessExecutionError: Unexpected error while running command.
Command: iscsiadm -m discovery -t sendtargets -I default -p 9.9.9.11:3260
Exit code: 21
Stdout: u''
Stderr: u"libkmod: kmod_module_insert_module: could not find module by name='iscsi_tcp'\niscsiadm: Could not insert module tcp. Kmod error -38\niscsiadm: Could not load transport tcp.Dropping interface default.\n"

Additional
==========
Root cause is probably that volume /lib/modules is not mounted in cinder_backup.
After inserting iscsi_tcp module ($ modprobe iscsi_tcp) creating of backup worked.

Best Regards,
Šimon

Revision history for this message
Radosław Piliszek (yoctozepto) wrote :

Cinder Backup does not have LVM backend... do you mean you have Cinder Volume backend set as LVM and Cinder Backup uses something else?

Changed in kolla-ansible:
status: New → Incomplete
description: updated
description: updated
Revision history for this message
Radosław Piliszek (yoctozepto) wrote :

If only every bug report was as precise. :-) Thank you very much for the clarifications.

Changed in kolla-ansible:
status: Incomplete → New
Changed in kolla-ansible:
status: New → Triaged
Chason Chan (chen-xing)
Changed in kolla-ansible:
assignee: nobody → Chason Chan (chen-xing)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (master)

Fix proposed to branch: master
Review: https://review.opendev.org/707776

Changed in kolla-ansible:
assignee: Chason Chan (chen-xing) → Radosław Piliszek (yoctozepto)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.opendev.org/707838

Changed in kolla-ansible:
assignee: Radosław Piliszek (yoctozepto) → Chason Chan (chen-xing)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on kolla-ansible (master)

Change abandoned by Chason Chan (<email address hidden>) on branch: master
Review: https://review.opendev.org/707838

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (master)

Reviewed: https://review.opendev.org/707776
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=eff22e7c56ba35870475dd92d9d89fde1b48de4c
Submitter: Zuul
Branch: master

commit eff22e7c56ba35870475dd92d9d89fde1b48de4c
Author: Radosław Piliszek <email address hidden>
Date: Fri Feb 14 08:49:55 2020 +0100

    Fix Cinder Backup access to kernel modules (iscsi_tcp issue)

    It looks like the only missing part was the actual mount of
    /lib/modules

    Now Cinder Backup volumes differ from Cinder Volume volumes only
    by /etc/target which is not relevant (Cinder Backup does not
    provide a target).

    Change-Id: Iccf4298c4f9306eb0a95b6712815778555ef44fc
    Closes-bug: #1863094

Changed in kolla-ansible:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/707863

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/707866

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/707867

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/stein)

Reviewed: https://review.opendev.org/707866
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=52302eb5301ba7c1848ae56a5d0a21d947c10ceb
Submitter: Zuul
Branch: stable/stein

commit 52302eb5301ba7c1848ae56a5d0a21d947c10ceb
Author: Radosław Piliszek <email address hidden>
Date: Fri Feb 14 08:49:55 2020 +0100

    Fix Cinder Backup access to kernel modules (iscsi_tcp issue)

    It looks like the only missing part was the actual mount of
    /lib/modules

    Now Cinder Backup volumes differ from Cinder Volume volumes only
    by /etc/target which is not relevant (Cinder Backup does not
    provide a target).

    Change-Id: Iccf4298c4f9306eb0a95b6712815778555ef44fc
    Closes-bug: #1863094
    (cherry picked from commit eff22e7c56ba35870475dd92d9d89fde1b48de4c)
    (cherry picked from commit 26d8fbcb6eae251fae3fac89c82b843e68bc049d)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/train)

Reviewed: https://review.opendev.org/707863
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=26d8fbcb6eae251fae3fac89c82b843e68bc049d
Submitter: Zuul
Branch: stable/train

commit 26d8fbcb6eae251fae3fac89c82b843e68bc049d
Author: Radosław Piliszek <email address hidden>
Date: Fri Feb 14 08:49:55 2020 +0100

    Fix Cinder Backup access to kernel modules (iscsi_tcp issue)

    It looks like the only missing part was the actual mount of
    /lib/modules

    Now Cinder Backup volumes differ from Cinder Volume volumes only
    by /etc/target which is not relevant (Cinder Backup does not
    provide a target).

    Change-Id: Iccf4298c4f9306eb0a95b6712815778555ef44fc
    Closes-bug: #1863094
    (cherry picked from commit eff22e7c56ba35870475dd92d9d89fde1b48de4c)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/rocky)

Reviewed: https://review.opendev.org/707867
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=fe7f8ce320cd2c9fc726015f819dabebe021ca7a
Submitter: Zuul
Branch: stable/rocky

commit fe7f8ce320cd2c9fc726015f819dabebe021ca7a
Author: Radosław Piliszek <email address hidden>
Date: Fri Feb 14 08:49:55 2020 +0100

    Fix Cinder Backup access to kernel modules (iscsi_tcp issue)

    It looks like the only missing part was the actual mount of
    /lib/modules

    Now Cinder Backup volumes differ from Cinder Volume volumes only
    by /etc/target which is not relevant (Cinder Backup does not
    provide a target).

    Change-Id: Iccf4298c4f9306eb0a95b6712815778555ef44fc
    Closes-bug: #1863094
    (cherry picked from commit eff22e7c56ba35870475dd92d9d89fde1b48de4c)
    (cherry picked from commit 26d8fbcb6eae251fae3fac89c82b843e68bc049d)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 7.2.1

This issue was fixed in the openstack/kolla-ansible 7.2.1 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.