OpenStack Compute (nova)

unable to atach-volume on different host (iscsiadm has volume-00000006, nova has vol-00000006)

Bug #712469 reported by justinc111
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Invalid
Undecided
Unassigned

Bug Description

I'm trying to euca-attach-volume.
Setup consists of one controller (IP 172.16.120.229) and one compute node (IP 172.16.120.228).
nova-volumes is running on compute node (ubuntu03c).
I can attach volume to guest running on compute node.
I cann't attach volume to guest running on controller node (euca-attach-volume is executed on controller (ubuntu04a) too).

ubuntu04a, nova-compute.log:
2011-02-03 12:53:05,244 ERROR nova.root [-] Exception during message handling
(nova.root): TRACE: Traceback (most recent call last):
(nova.root): TRACE: File "/usr/lib/pymodules/python2.6/nova/rpc.py", line 192, in receive
(nova.root): TRACE: rval = node_func(context=ctxt, **node_args)
(nova.root): TRACE: File "/usr/lib/pymodules/python2.6/nova/compute/manager.py", line 92, in decorated_function
(nova.root): TRACE: function(self, context, instance_id, *args, **kwargs)
(nova.root): TRACE: File "/usr/lib/pymodules/python2.6/nova/compute/manager.py", line 527, in attach_volume
(nova.root): TRACE: volume_id)
(nova.root): TRACE: File "/usr/lib/pymodules/python2.6/nova/volume/manager.py", line 159, in setup_compute_volume
(nova.root): TRACE: path = self.driver.discover_volume(volume_ref)
(nova.root): TRACE: File "/usr/lib/pymodules/python2.6/nova/volume/driver.py", line 311, in discover_volume
(nova.root): TRACE: volume['host'])
(nova.root): TRACE: File "/usr/lib/pymodules/python2.6/nova/volume/driver.py", line 300, in _get_name_and_portal
(nova.root): TRACE: "sendtargets -p %s" % host)
(nova.root): TRACE: File "/usr/lib/pymodules/python2.6/nova/utils.py", line 147, in execute
(nova.root): TRACE: cmd=cmd)
(nova.root): TRACE: ProcessExecutionError: Unexpected error while running command.
(nova.root): TRACE: Command: sudo iscsiadm -m discovery -t sendtargets -p ubuntu03c
(nova.root): TRACE: Exit code: 255
(nova.root): TRACE: Stdout: ''
(nova.root): TRACE: Stderr: 'iscsiadm: Cannot resolve host ubuntu03c. getaddrinfo error: [Name or service not known]\n\niscsiadm: cannot resolve host name ubuntu03c\niscsiadm: Could not perform SendTargets discovery.\n'
(nova.root): TRACE:

I just edited /etc/hosts. Next attempt:

011-02-03 13:11:48,818 AUDIT nova.compute.manager [YLNJR-8B-ZM3QE2JC-PR jc xproj] instance 83: attaching volume 6 to \/dev\/hdb
2011-02-03 13:11:48,915 ERROR nova.root [-] Exception during message handling
(nova.root): TRACE: Traceback (most recent call last):
(nova.root): TRACE: File "/usr/lib/pymodules/python2.6/nova/rpc.py", line 192, in receive
(nova.root): TRACE: rval = node_func(context=ctxt, **node_args)
(nova.root): TRACE: File "/usr/lib/pymodules/python2.6/nova/compute/manager.py", line 92, in decorated_function
(nova.root): TRACE: function(self, context, instance_id, *args, **kwargs)
(nova.root): TRACE: File "/usr/lib/pymodules/python2.6/nova/compute/manager.py", line 527, in attach_volume
(nova.root): TRACE: volume_id)
(nova.root): TRACE: File "/usr/lib/pymodules/python2.6/nova/volume/manager.py", line 159, in setup_compute_volume
(nova.root): TRACE: path = self.driver.discover_volume(volume_ref)
(nova.root): TRACE: File "/usr/lib/pymodules/python2.6/nova/volume/driver.py", line 311, in discover_volume
(nova.root): TRACE: volume['host'])
(nova.root): TRACE: File "/usr/lib/pymodules/python2.6/nova/volume/driver.py", line 305, in _get_name_and_portal
(nova.root): TRACE: iscsi_portal = location.split(",")[0]
(nova.root): TRACE: UnboundLocalError: local variable 'location' referenced before assignment
(nova.root): TRACE:

I guees restart of some iscsi services should help.
It didn't, so I restarted both hosts, and trace still occurs.

Some commands:

root@ubuntu04a:~# dpkg --list | grep nova-
ii nova-api 2011.1~bzr642-0ubuntu0ppa1~maverick1 OpenStack Compute - Nova - API frontend
ii nova-common 2011.1~bzr642-0ubuntu0ppa1~maverick1 OpenStack Compute - Nova - common files
ii nova-compute 2011.1~bzr642-0ubuntu0ppa1~maverick1 OpenStack Compute - Nova - compute node
ii nova-doc 2011.1~bzr635-0ubuntu0ppa1~maverick1 OpenStack Compute - Nova - documetation
ii nova-network 2011.1~bzr642-0ubuntu0ppa1~maverick1 OpenStack Compute - Nova - Network thingamajig
ii nova-objectstore 2011.1~bzr642-0ubuntu0ppa1~maverick1 OpenStack Compute - Nova - object store
ii nova-scheduler 2011.1~bzr642-0ubuntu0ppa1~maverick1 OpenStack Compute - Nova - Scheduler
ii nova-volume 2011.1~bzr642-0ubuntu0ppa1~maverick1 OpenStack Compute - Nova - storage

root@ubuntu04a:~# euca-describe-instances
RESERVATION r-2k6s17yl xproj default
INSTANCE i-00000059 ami-iv78p74r 10.0.0.27 10.0.0.27 running mykey (xproj, ubuntu03c) 0 m1.small 2011-02-03 13:05:25 nova
RESERVATION r-nd9pt10v xproj default
INSTANCE i-00000058 ami-iv78p74r 10.0.0.26 10.0.0.26 running mykey (xproj, ubuntu04a) 0 m1.small 2011-02-03 13:05:13 nova

root@ubuntu04a:~# euca-describe-volumes
...
VOLUME vol-00000006 2 nova available (jc, ubuntu03c, None, None) 2011-02-01 10:06:45
VOLUME vol-00000007 1 nova in-use (jc, ubuntu03c, i-0000004f[ubuntu03c], \/dev\/hdb) 2011-02-01 14:50:32

root@ubuntu04a:~# euca-attach-volume -i i-00000058 -d /dev/hdb vol-00000006

root@ubuntu04a:~# iscsiadm -m discovery -t sendtargets -p ubuntu03c
172.16.120.228:3260,1 iqn.2010-10.org.openstack:volume-00000006
192.168.122.1:3260,1 iqn.2010-10.org.openstack:volume-00000006
172.16.120.228:3260,1 iqn.2010-10.org.openstack:volume-00000007
192.168.122.1:3260,1 iqn.2010-10.org.openstack:volume-00000007

Revision history for this message
Vish Ishaya (vishvananda) wrote :

you need to set --iscsi_ip_prefix=192.168.122 for it to know which ip to use to connect to the target.

Revision history for this message
justinc111 (justin-cinkelj) wrote : Re: [Bug 712469] Re: unable to atach-volume on different host (iscsiadm has volume-00000006, nova has vol-00000006)
Download full text (7.6 KiB)

Tried to add --iscsi_ip_prefix=192.168.122 to /etc/nova/nova.conf.
In nova-compute.log, after 2 minutes:
(nova.root): TRACE: Stderr: 'iscsiadm: Could not login to [iface: default, target: iqn.2010-10.org.openstack:volume-00000006, portal: 192.168.122.1,3260]: \niscsiadm: initiator reported error (8 - connection timed out)\n'

Changed prefix to --iscsi_ip_prefix=172.16.120, restart services and
euca-attach-volume -i i-00000058 -d /dev/hdb vol-00000006
root@ubuntu04a:/etc/nova# euca-describe-volumes
VOLUME vol-00000006 2 nova in-use (jc, ubuntu03c, i-00000058[ubuntu04a], \/dev\/hdb) 2011-02-01 10:06:45
So volume is attached now.

After ssh-ing to VM, there is no /dev/hdb, but I have /sys/block/vdb
# cat /sys/block/vdb/dev
252:16
# mknod -m 0660 252 16 /dev/vdb
# fdisk -l /dev/vdb
Disk /dev/vdb: 21.4 GB, 21474836480 bytes
16 heads, 63 sectors/track, 41610 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes
Disk /dev/vdb doesn't contain a valid partition table

I guess fdisk should report size of 2 G. I will try to create and use new volume, on new /dev/loop.

Thank you for your help.

----- Original Message -----
From: "Vish Ishaya" <email address hidden>
To: "justin cinkelj" <email address hidden>
Sent: Thursday, February 3, 2011 3:24:59 PM
Subject: [Bug 712469] Re: unable to atach-volume on different host (iscsiadm has volume-00000006, nova has vol-00000006)

you need to set --iscsi_ip_prefix=192.168.122 for it to know which ip to
use to connect to the target.

--
You received this bug notification because you are a direct subscriber
of the bug.
https://bugs.launchpad.net/bugs/712469

Title:
  unable to atach-volume on different host (iscsiadm has
  volume-00000006, nova has vol-00000006)

Status in OpenStack Compute (Nova):
  New

Bug description:
  I'm trying to euca-attach-volume.
  Setup consists of one controller (IP 172.16.120.229) and one compute node (IP 172.16.120.228).
  nova-volumes is running on compute node (ubuntu03c).
  I can attach volume to guest running on compute node.
  I cann't attach volume to guest running on controller node (euca-attach-volume is executed on controller (ubuntu04a) too).

  ubuntu04a, nova-compute.log:
  2011-02-03 12:53:05,244 ERROR nova.root [-] Exception during message handling
  (nova.root): TRACE: Traceback (most recent call last):
  (nova.root): TRACE: File "/usr/lib/pymodules/python2.6/nova/rpc.py", line 192, in receive
  (nova.root): TRACE: rval = node_func(context=ctxt, **node_args)
  (nova.root): TRACE: File "/usr/lib/pymodules/python2.6/nova/compute/manager.py", line 92, in decorated_function
  (nova.root): TRACE: function(self, context, instance_id, *args, **kwargs)
  (nova.root): TRACE: File "/usr/lib/pymodules/python2.6/nova/compute/manager.py", line 527, in attach_volume
  (nova.root): TRACE: volume_id)
  (nova.root): TRACE: File "/usr/lib/pymodules/python2.6/nova/volume/manager.py", line 159, in setup_compute_volume
  (nova.root): TRACE: path = self.driver.discover_volume(volume_ref)
  (nova.root): TRACE: File "/usr/lib/pymodules/python2.6/nova/volume/driver.py", line 311, in discover_volume
  (nova.root): TRACE: volume['host'])
  (nova.root): ...

Read more...

Revision history for this message
justinc111 (justin-cinkelj) wrote :
Download full text (6.8 KiB)

Update:
Volume was detached and re-attached to same instance.
Inside VM, there are additional messages in dmesg (there were none during first attach), mentioning
vdc: unknown partition table
Major/minor numbers are now 252:32, so I created second node /dev/vdc
fdisk reports size of 2 G.

----- Original Message -----
From: "Vish Ishaya" <email address hidden>
To: "justin cinkelj" <email address hidden>
Sent: Thursday, February 3, 2011 3:24:59 PM
Subject: [Bug 712469] Re: unable to atach-volume on different host (iscsiadm has volume-00000006, nova has vol-00000006)

you need to set --iscsi_ip_prefix=192.168.122 for it to know which ip to
use to connect to the target.

--
You received this bug notification because you are a direct subscriber
of the bug.
https://bugs.launchpad.net/bugs/712469

Title:
  unable to atach-volume on different host (iscsiadm has
  volume-00000006, nova has vol-00000006)

Status in OpenStack Compute (Nova):
  New

Bug description:
  I'm trying to euca-attach-volume.
  Setup consists of one controller (IP 172.16.120.229) and one compute node (IP 172.16.120.228).
  nova-volumes is running on compute node (ubuntu03c).
  I can attach volume to guest running on compute node.
  I cann't attach volume to guest running on controller node (euca-attach-volume is executed on controller (ubuntu04a) too).

  ubuntu04a, nova-compute.log:
  2011-02-03 12:53:05,244 ERROR nova.root [-] Exception during message handling
  (nova.root): TRACE: Traceback (most recent call last):
  (nova.root): TRACE: File "/usr/lib/pymodules/python2.6/nova/rpc.py", line 192, in receive
  (nova.root): TRACE: rval = node_func(context=ctxt, **node_args)
  (nova.root): TRACE: File "/usr/lib/pymodules/python2.6/nova/compute/manager.py", line 92, in decorated_function
  (nova.root): TRACE: function(self, context, instance_id, *args, **kwargs)
  (nova.root): TRACE: File "/usr/lib/pymodules/python2.6/nova/compute/manager.py", line 527, in attach_volume
  (nova.root): TRACE: volume_id)
  (nova.root): TRACE: File "/usr/lib/pymodules/python2.6/nova/volume/manager.py", line 159, in setup_compute_volume
  (nova.root): TRACE: path = self.driver.discover_volume(volume_ref)
  (nova.root): TRACE: File "/usr/lib/pymodules/python2.6/nova/volume/driver.py", line 311, in discover_volume
  (nova.root): TRACE: volume['host'])
  (nova.root): TRACE: File "/usr/lib/pymodules/python2.6/nova/volume/driver.py", line 300, in _get_name_and_portal
  (nova.root): TRACE: "sendtargets -p %s" % host)
  (nova.root): TRACE: File "/usr/lib/pymodules/python2.6/nova/utils.py", line 147, in execute
  (nova.root): TRACE: cmd=cmd)
  (nova.root): TRACE: ProcessExecutionError: Unexpected error while running command.
  (nova.root): TRACE: Command: sudo iscsiadm -m discovery -t sendtargets -p ubuntu03c
  (nova.root): TRACE: Exit code: 255
  (nova.root): TRACE: Stdout: ''
  (nova.root): TRACE: Stderr: 'iscsiadm: Cannot resolve host ubuntu03c. getaddrinfo error: [Name or service not known]\n\niscsiadm: cannot resolve host name ubuntu03c\niscsiadm: Could not perform SendTargets discovery.\n'
  (nova.root): TRACE:

  I just edited /etc/hosts. Next attempt:

  0...

Read more...

Revision history for this message
Thierry Carrez (ttx) wrote :

If I read this correctly, the issue is now solved (by usage of a correct --iscsi_ip_prefix flag). Closing as Invalid, please reopen if I read this story wrong.

Changed in nova:
status: New → Invalid
Revision history for this message
justinc111 (justin-cinkelj) wrote :

Yes, the issue is solved.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.