Unable to create 2+ VMs at the same time using XenServer

Bug #795187 reported by Brian Lamar
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Invalid
Undecided
Unassigned

Bug Description

I am able to create 1 VM, wait until it has completed creation, and then create another.

If I attempt to create 2 at the same time I end up with what is shown in the logs here:

http://paste.openstack.org/show/1601/

After the last line in that paste, the nova-compute process hangs. If you attempt to restart the nova-compute process via:

service nova-compute stop
service nova-compute start

...the process ends up in an *unkillable* disk-wait state as shown here:

http://paste.openstack.org/show/1602/

Revision history for this message
Brian Lamar (blamar) wrote :

Attempts to kill process are futile:

http://paste.openstack.org/show/1603/

Revision history for this message
Thierry Carrez (ttx) wrote :

This sounds rather bad -- can someone else using XenServer confirm this ?

Changed in nova:
importance: Undecided → High
status: New → Incomplete
Revision history for this message
Brian Lamar (blamar) wrote :

Update with kernel messages:

http://paste.openstack.org/show/1628/

Revision history for this message
Chris Behrens (cbehrens) wrote :

I can build a number of instances at once with XenServer. I had a merge to fix an issue with parallel xapi calls maybe a month or 2 ago, and after then everything's been fine.

This is some sort of kernel bug according to the pastes above. Notice this line:

"
 kernel:[50922.735291] general protection fault: 0000 [#1] SMP
"

Due to the crash caught in the kernel, it has left some processes in a infinite disk wait state and thus they can't be killed. Since the kernel is hosed, all you can do is reboot.

What version of XenServer is this? What kernel?

Revision history for this message
Brian Lamar (blamar) wrote :

Latest XenServer 5.6 SP2 running Debian Squeeze in dom0:

root@compute-10-13-1-39:~# uname -a
Linux compute-10-13-1-39 2.6.32-5-amd64 #1 SMP Mon Mar 7 21:35:22 UTC 2011 x86_64 GNU/Linux

Revision history for this message
Ewan Mellor (ewanmellor) wrote :

Brian, presumably you don't mean Squeeze in dom0. Do you mean squeeze in domU?

If you are using unmodified XenServer and you're getting that kernel trace, then that's something that we will look at immediately. Please reproduce the bug, then run xen-bugtool in dom0, or use XenCenter > Tools > Get Server Status Report. I will escalate immediately to XS Engineering.

Feel free to send the bug report directly to me, if this is not the right forum for it.

Thanks.

Revision history for this message
Brian Lamar (blamar) wrote :

Yup, I meant domU. I'll try to get a report to you as soon as I can. Thanks for the assistance.

Revision history for this message
Brian Lamar (blamar) wrote :

I wasn't certain what to collect...so I collected everything. Let me know if I can help out, turn on debugging, or do anything else that might help expose the error! Thanks Ewan.

Revision history for this message
Antony Messerli (antonym) wrote :

I haven't seen issues running multiple builds but I'm also using --xenapi_inject_image=false and dropping the image directly to the XenServer host instead of injecting through the domU. I'd suspect the builds could be colliding somehow when mounting the VHDs to the compute worker.

Thierry Carrez (ttx)
Changed in nova:
importance: High → Undecided
Revision history for this message
Thierry Carrez (ttx) wrote :

What's the status for this one ? Is it still happening with current code ?

Revision history for this message
Paul Voccio (pvo) wrote :

I am unable to reproduce this in trunk.

Revision history for this message
Thierry Carrez (ttx) wrote :

Closing based on last comment -- please reopen if issue is still present.

Changed in nova:
status: Incomplete → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.