kvm disabled in bios (was: Unable to start EUC instances - no supported architecture for os type 'hvm')

On Thu, Oct 15, 2009 at 04:33:47PM -0000, Joseph Salisbury wrote:
>
> I'm having an issue starting an instance. I see errors stating "disk
> not found" and "no domain with matching name found". I'm also seeing an
> error saying "no supported architecture for os type 'hvm'":
>
> Snippet from nc.log:
> [Wed Oct 14 14:57:00 2009][001691][EUCAINFO ] currently running/booting: i-39A80763
> [Wed Oct 14 14:57:00 2009][001691][EUCAERROR ] libvirt: internal error no supported architecture for os type 'hvm' (code=1)
> [Wed Oct 14 14:57:00 2009][001691][EUCAFATAL ] hypervisor failed to start domain

I've seen this as well. The current workaround is to restart libvirt and
then eucalyptus-nc. The theory is that the eucalyptus starts and
connects to libvirt when the latter is not ready yet.

  status confirmed
  importance medium

--
Mathias Gug
Ubuntu Developer http://www.ubuntu.com

Restarting libvirt and eucalyptus-nc did not seem to work around the issue. Just to make sure I restarted things properly here is what I did on the Node Controller:

eqluser@granada:~$ sudo /etc/init.d/libvirt-bin restart
 * Restarting libvirt management daemon /usr/sbin/libvirtd [ OK ]

eqluser@granada:~$ sudo /etc/init.d/eucalyptus-nc restart
 * Restarting Eucalyptus Node Service [ OK ]

I also tried stopping libvirt, stopping eucalyptus-nc. Then started libvird, waited some time, then started eucalyptus-nc.

On Thu, Oct 15, 2009 at 06:05:28PM -0000, Joseph Salisbury wrote:
> I also tried stopping libvirt, stopping eucalyptus-nc. Then started libvird, waited some time, then started eucalyptus-nc.
>

What is the content of /var/log/eucalyptus/euca_test_nc.log?

--
Mathias Gug
Ubuntu Developer http://www.ubuntu.com

Contents of /var/log/eucalyptus/euca_test_nc.log:

This is perl, v5.10.0 built for i486-linux-gnu-thread-multi

Copyright 1987-2007, Larry Wall

Perl may be copied only under the terms of either the Artistic License or the
GNU General Public License, which may be found in the Perl 5 source kit.

Complete documentation for Perl, including FAQ lists, should be found on
this system using "man perl" or "perldoc perl". If you have access to the
Internet, point your browser at http://www.perl.org/, the Perl Home Page.

total_memory=8258
nr_cores=4

On Thu, Oct 15, 2009 at 06:49:54PM -0000, Joseph Salisbury wrote:
> Contents of /var/log/eucalyptus/euca_test_nc.log:
>
> This is perl, v5.10.0 built for i486-linux-gnu-thread-multi
>
> Copyright 1987-2007, Larry Wall
>
> Perl may be copied only under the terms of either the Artistic License or the
> GNU General Public License, which may be found in the Perl 5 source kit.
>
> Complete documentation for Perl, including FAQ lists, should be found on
> this system using "man perl" or "perldoc perl". If you have access to the
> Internet, point your browser at http://www.perl.org/, the Perl Home Page.
>
> total_memory=8258
> nr_cores=4
>

Ok. So this is *not* the issue I saw. If the NC wasn't able to connect
to libvirtd, you'd see an error in this file.

--
Mathias Gug
Ubuntu Developer http://www.ubuntu.com

Hmm. I wonder if this could be related to the bios. I'll check it to ensure the correct settings are enabled for KVM. Could an error like this occur if the bios was not setup correctly?

Joseph Salisbury wrote:
> Hmm. I wonder if this could be related to the bios. I'll check it to
> ensure the correct settings are enabled for KVM. Could an error like
> this occur if the bios was not setup correctly?

Yes, indeed.

I think I found the issue:

kern.log:Oct 13 15:53:45 granada kernel: [ 7.832803] kvm: disabled by bios

I will correct the bios and re-test.

On Thu, Oct 15, 2009 at 08:11:57PM -0000, Joseph Salisbury wrote:
> Hmm. I wonder if this could be related to the bios. I'll check it to
> ensure the correct settings are enabled for KVM. Could an error like
> this occur if the bios was not setup correctly?
>

Yes. Another option is to check dmesg output. If VT extensions are not
enabled in the BIOS, there should be an error message in there.

--
Mathias Gug
Ubuntu Developer http://www.ubuntu.com

I like your option. If a test case was automated, a check could be performed first :-) As long as that error is always reported by dmesg when the bios is not enabled :-)

I'm copying this bug over to qemu-kvm.

Mathias hit this same problem last week.

I'm going to add a few more checks to /usr/bin/kvm-ok in the qemu-kvm package.
 1) check that /dev/kvm exists
 2) check the permissions on /dev/kvm
 3) check dmesg for kvm disabled messages

I see this sort of thing with KVM fairly frequently, and usually point people to run kvm-ok to get a hint. We should update these hints.

:-Dustin

On Thu, Oct 15, 2009 at 08:47:51PM -0000, Dustin Kirkland wrote:
> Mathias hit this same problem last week.

Right. I ran into the same issue last week when I got some hardware
where the VT extensions were disabled. However the reported error from
the NC (hvm not supported) doesn't *always* mean that the BIOS has
disabled VT extensions.

I've run into this issue on some configuration that had already been
running a UEC infrastructure before. Upon reboot the NC would fail with
the same error message - this is the situation I described in my first
comment.

> I'm going to add a few more checks to /usr/bin/kvm-ok in the qemu-kvm package.
> 1) check that /dev/kvm exists
> 2) check the permissions on /dev/kvm
> 3) check dmesg for kvm disabled messages
>
> I see this sort of thing with KVM fairly frequently, and usually point
> people to run kvm-ok to get a hint. We should update these hints.
>

Sounds like a good option.

--
Mathias Gug
Ubuntu Developer http://www.ubuntu.com

Mathias-

Oh, sorry for closing the eucalyptus task. You can reopen it to cover
your issue, if you like.

Thanks,
:-Dustin

I checked the bios and found that VT extensions were not enabled. I enabled them and was able to start my instance :-)

However, I ran into an issuing the ssh command listed in the test case:

$ ssh -i mykey.priv ubuntu@$IPADDR
Permission denied (publickey).

I'll investigate this further.

I plan on performing an rsync to the latest build and re-running the EUC-003(ServerEConfig) test case from start to finish to ensure no issues.

Thanks for all the assistance.

One issue per bug, Joe...

:-Dustin

Attaching patch for SRU review.

This will be uploaded with two other fixes, once approved.

:-Dustin

Accepted qemu-kvm into karmic-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

I can give this a test today. I will post feedback shortly.

Here are the results from my testing:

Test 1: VT Enabled in BIOS(On a machine that supports VT):

$ kvm-ok
INFO: Your CPU supports KVM extensions
INFO: /dev/kvm exists
KVM acceleration can be used

Test 2a: (run this on a system that does not support VT)

joe@jts-ubuntu:/etc/apt$ kvm-ok
INFO: Your CPU does not support KVM extensions
INFO: /dev/kvm does not exist
HINT: sudo modprobe kvm
KVM acceleration can NOT be used

Test 2b: (sudo modprobe -r kvm, and then re-run kvm-ok and see that /dev/kvm is gone)
Note: I had to remove the kvm_intel module on my system before I could remove kvm.

$ kvm-ok
INFO: Your CPU supports KVM extensions
INFO: /dev/kvm does not exist
HINT: sudo modprobe kvm
KVM acceleration can NOT be used

Test 2c: (reboot into BIOS, disable VT, and re-run kvm-ok)

$ kvm-ok
INFO: Your CPU supports KVM extensions
INFO: /dev/kvm does not exist
HINT: sudo modprobe kvm
INFO: KVM is disable by your BIOS
HINT: Enter your BIOS setup and enable Virtualization Technology (VT)
KVM acceleration can NOT be used

All of the tests worked as expected. Only two minor things that could be changed:

1. The text when VT Extensions are disabled in the bios could be updated to say "KVM is disabled" instead of "KVM is disable".
2. Maybe not print the text "HINT: sudo modprobe kvm in the cases" in the cases when it would not work anyway, such as in test 2a and 2c.

Great testing, Joe!

Martin, I agree with Joe's two cosmetic suggestions. Shall I fix
those, and re-upload to -proposed?

Attaching a debdiff that addresses these two concerns.

Joe, prior to getting this into proposed, could you re-test with:
 * http://rookery.canonical.com/~kirkland/kvm-ok

:-Dustin

Dustin, thanks a lot for that fix! This will save tons of grieves to lots of people down the road. I know *I* lost a couple of hours before figuring out that VT extensions where disabled in the BIOS ... :D

I tested the script at:

 * http://rookery.canonical.com/~kirkland/kvm-ok

However, there seems to be an issue with test 2c. Test 2c is running the script with VT Extensions disabled in the bios. Here is the output for that case:

$ ./kvm-ok.new

INFO: Your CPU supports KVM extensions
INFO: /dev/kvm does not exist
HINT: sudo modprobe kvm
KVM acceleration can NOT be used

The script checks for /dev/kvm before checking dmesg for the bios message. This causes verdict to be called with an arg of 1 and the script exits. The user does not get the suggestion to enable KVM in the bios.

Maybe swap execution order of the /dev/kvm check and the dmesg check?

Joe,

Good call.

Try this one:

http://rookery.canonical.com/~kirkland/kvm-ok

:-Dustin

I tested the latest version at:

http://rookery.canonical.com/~kirkland/kvm-ok

All the tests were successful. The following is the output:

Test 1: VT Enabled in BIOS(On a machine that supports VT):
$ ./kvm-ok.new
INFO: Your CPU supports KVM extensions
INFO: /dev/kvm exists
KVM acceleration can be used

Test 2a: (run this on a system that does not support VT)
joe@jts-ubuntu:~$ kvm-ok.new
INFO: Your CPU does not support KVM extensions
KVM acceleration can NOT be used

Test 2b: (sudo modprobe -r kvm, and then re-run kvm-ok and see that /dev/kvm is gone)
$ ./kvm-ok.new
INFO: Your CPU supports KVM extensions
INFO: /dev/kvm does not exist
HINT: sudo modprobe kvm
KVM acceleration can NOT be used

Test 2c: (reboot into BIOS, disable VT, and re-run kvm-ok)
$ ./kvm-ok.new
INFO: Your CPU supports KVM extensions
INFO: KVM is disabled by your BIOS
HINT: Enter your BIOS setup and enable Virtualization Technology (VT)
KVM acceleration can NOT be used

Dustin, fine for me to update again. Please upload with a new changelog and use -v<version in karmic final> so that the source.changes covers both versions in -proposed (this will make SRU bug tracking much easier). Thanks!

Thanks, Martin.

Joe,

Can you test 0.11.0-0ubuntu6.3 now in karmic-proposed?

:-Dustin

FWIW, I tested the new /usr/bin/kvm-ok and it seems to work well.

Hopefully Joe can run his little barrage of tests and confirm the new behavior as provided in the updated package.

:-Dustin

I re-ran all of the tests using the latest in karmic-proposed. All tests passed:

$ dpkg -l | grep qemu
ii qemu-kvm 0.11.0-0ubuntu6.3 Full

virtualization on i386 and amd64 hardwa

Test 1: VT Enabled in BIOS(On a machine that supports VT):
$ kvm-ok
INFO: Your CPU supports KVM extensions
INFO: /dev/kvm exists
KVM acceleration can be used

Test 2a: (run this on a system that does not support VT)
joe@jts-ubuntu:/etc/apt$ kvm-ok
INFO: Your CPU does not support KVM extensions
KVM acceleration can NOT be used

Test 2b: (sudo modprobe -r kvm, and then re-run kvm-ok and see that /dev/kvm is gone)
$ kvm-ok
INFO: Your CPU supports KVM extensions
INFO: /dev/kvm does not exist
HINT: sudo modprobe kvm
KVM acceleration can NOT be used

Test 2c: (reboot into BIOS, disable VT, and re-run kvm-ok)
$ kvm-ok
INFO: Your CPU supports KVM extensions
INFO: KVM is disabled by your BIOS
HINT: Enter your BIOS setup and enable Virtualization Technology (VT)
KVM acceleration can NOT be used

Thanks,

Joe

This bug was fixed in the package qemu-kvm - 0.11.0-0ubuntu6.3

---------------
qemu-kvm (0.11.0-0ubuntu6.3) karmic-security; urgency=low

  * SECURITY UPDATE: linux <= 2.6.25 guests (e.g. hardy) with virtio
    networking are subject to DoS by qemu-kvm application crash;
    the crash can be remotely triggered by a malicious user flooding any
    open network port (LP: #458521)
    - debian/patches/12_whitelist_host_virtio_networking_features.patch:
      fix accounting of virtio networking features available to make
      available to the guests
    - CVE-2009-XXXX
  * debian/kvm-ok: check for other common reasons why KVM might not be
    usable, LP: #452323
  * debian/control: build-depend on libcurl devel, to allow booting from
    ISOs over http, LP: #453441

 -- Dustin Kirkland <email address hidden> Thu, 29 Oct 2009 11:36:18 -0500

Copied karmic-proposed to lucid.

This bug was fixed in the package qemu-kvm - 0.11.0-0ubuntu6.3

---------------
qemu-kvm (0.11.0-0ubuntu6.3) karmic-security; urgency=low

  * SECURITY UPDATE: linux <= 2.6.25 guests (e.g. hardy) with virtio
    networking are subject to DoS by qemu-kvm application crash;
    the crash can be remotely triggered by a malicious user flooding any
    open network port (LP: #458521)
    - debian/patches/12_whitelist_host_virtio_networking_features.patch:
      fix accounting of virtio networking features available to make
      available to the guests
    - CVE-2009-XXXX
  * debian/kvm-ok: check for other common reasons why KVM might not be
    usable, LP: #452323
  * debian/control: build-depend on libcurl devel, to allow booting from
    ISOs over http, LP: #453441

 -- Dustin Kirkland <email address hidden> Thu, 29 Oct 2009 11:36:18 -0500

I am having the same i ssue when i try to run on a virtualized node.
----------------------------------------------------------------------------------------------
[EUCAERROR ] libvirt: internal error no supported architecture for os type 'hvm' (code=1)
[EUCAFATAL ] hypervisor failed to start domain
[EUCAERROR ] libvirt: Domain not found: no domain with matching name 'i-3ACA0721' (code=42)
[EUCAINFO ] vrun(): [rm -rf /var/lib/eucalyptus/instances/admin/i-3ACA0721/]
[EUCAINFO ] stopping the network (vlan=10)
----------------------------------------------------------------------------------------------

Now my question is when i installed the UEC Node, it said that hardware acceleration will not be available. It didnt mention that my attempts in running the VM in the node will be futile.. Or is there a way to bypass this? if its not possible, it shouldnt have allowed me to install it in the first place.

Attached the screenshot.

I am having the same issue, and after several hours of research spread across several days, I've come to this bug.

I agree with mohanv in that it should not have let me setup the node if it won't work with UEC.

If kvm requires hardware supported virtual machines hvm, (as indicated here: http://open.eucalyptus.com/forum/problem-node-registration), then I would like to request that the node installation process either:

A) Prevents you from installing and gives you an explanation (you hardware is not compatible: hvm required for kvm)
OR
B) The node installation asks if you would like to use xen.

Xen is not an option.

If KVM is disabled in BIOS, you will be told this by the MOTD the first time you log in.

Hi,

I am confused whether my CPU supports KVM or not? Can anyone please help

Ubuntu 11.04
# kvm-ok
INFO: Your CPU does not support KVM extensions
INFO: /dev/kvm does not exist
HINT: sudo modprobe kvm_
INFO: Your CPU supports KVM extensions
FAIL: Unknown virtualization extension:
KVM acceleration can NOT be used

# cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 15
model : 2
model name : Intel(R) Xeon(TM) CPU 2.80GHz
stepping : 9
cpu MHz : 2798.828
cache size : 512 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 0
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe pebs bts cid xtpr
bogomips : 5597.65
clflush size : 64
cache_alignment : 128
address sizes : 36 bits physical, 32 bits virtual
power management:

processor : 1
vendor_id : GenuineIntel
cpu family : 15
model : 2
model name : Intel(R) Xeon(TM) CPU 2.80GHz
stepping : 9
cpu MHz : 2798.828
cache size : 512 KB
physical id : 3
siblings : 2
core id : 0
cpu cores : 1
apicid : 6
initial apicid : 6
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe pebs bts cid xtpr
bogomips : 5598.76
clflush size : 64
cache_alignment : 128
address sizes : 36 bits physical, 32 bits virtual
power management:

processor : 2
vendor_id : GenuineIntel
cpu family : 15
model : 2
model name : Intel(R) Xeon(TM) CPU 2.80GHz
stepping : 9
cpu MHz : 2798.828
cache size : 512 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 1
apicid : 1
initial apicid : 1
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe pebs bts cid xtpr
bogomips : 5598.76
clflush size : 64
cache_alignment : 128
address sizes : 36 bits physical, 32 bits virtual
power management:

processor : 3
vendor_id : GenuineIntel
cpu family : 15
model : 2
model name : Intel(R) Xeon(TM) CPU 2.80GHz
stepping : 9
cpu MHz : 2798.828
cache size : 512 KB
physical id : 3
siblings : 2
core id : 0
cpu cores : 1
apicid : 7
initial apicid : 7
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
...

Quoting Rishi Singhal (<email address hidden>):
> Hi,
>
> I am confused whether my CPU supports KVM or not? Can anyone please help
>
> Ubuntu 11.04
> # kvm-ok
> INFO: Your CPU does not support KVM extensions
> INFO: /dev/kvm does not exist
> HINT: sudo modprobe kvm

Try this command ('sudo modprobe kvm') and then run kvm-ok
again.

This is a bug in the kvm-ok utility. I'll try to get it fixed.