[OSSA 2015-018] IP, MAC, and DHCP spoofing rules can by bypassed by changing device_owner (CVE-2015-5240)

Bug #1489111 reported by Kevin Benton
268
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Security Advisory
Fix Released
Undecided
Tristan Cacqueray
neutron
Fix Released
Undecided
Tristan Cacqueray
Juno
Fix Released
Undecided
Unassigned

Bug Description

The anti-IP spoofing rules, anti-MAC spoofing rules, and anti-DHCP spoofing rules can be bypassed by changing the device_owner field of a compute node's port to something that starts with 'network:'.

Steps to reproduce:

Create a port on the target network:

neutron port-create some_network

Start a repeated update of the device_owner field to immediately change it back after nova sets it to 'compute:<whatever>' on VM attachment. (This has to be done quickly because the owner has to be set to 'network:something' before the L2 agent wires up the security group rules.)

watch neutron port-update <port-uuid-from-above> --device-owner network:hello

Then boot the VM with the port UUID:

nova boot test --nic port-id=<port-uuid-from-above> --flavor m1.tiny --image cirros-0.3.4-x86_64-uec

This VM will now have no iptables rules applied because it will be treated as a network owned port (e.g. router interface, DHCP interface, etc).

CVE References

Revision history for this message
Kevin Benton (kevinbenton) wrote :

Hi,

I have a attached a patch that fixes the problem.

It enables a regex match in the policy engine and then changes the default policy to prevent users from using the 'network:' prefix in the device_owner field unless they own the network, have the advanced services role, or are an admin.

Revision history for this message
Kevin Benton (kevinbenton) wrote :

The core issue here is that the L2 agent trusts any devices whose owner field starts with 'network:'. This is necessary because the DHCP port and the router ports can't have DHCP spoofing and IP spoofing rules (respectively).

Here is the logic that skips 'network:' owned ports for security groups.

https://github.com/openstack/neutron/blob/d66f0e27919a29682d7c65e4f9ce1f9c7b278542/neutron/api/rpc/handlers/securitygroups_rpc.py#L83

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

Changed in ossa:
status: New → Incomplete
description: updated
Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

Beside having an instance unprotected by iptables, what can the user do with it ?
e.g., can he spoof other tenants' ip ?

Revision history for this message
Kevin Benton (kevinbenton) wrote :

Yes, the big risk here is with shared networks. The attacker can spoof other VMs IPs, or give out incorrect information via spoofed DHCP responses to get other VMs to send traffic to it.

Revision history for this message
Kevin Benton (kevinbenton) wrote :

Had to update the patch to handle a changed response code in a unit test.

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

Assuming this affect all l2 agents and all supported release, here is the impact description:

Title: Neutron firewall rules bypass through port update
Reporter: Kevin Benton (Mirantis)
Products: Neutron
Affects: versions through 2014.2.3 and 2015.1 versions through 2015.1.1

Description:
Kevin Benton from Mirantis reported a vulnerability in Neutron. By changing the port's device owner of an instance right after it is created, an authenticated user may prevent firewall rules to be applied, potentially resulting in anti spoofing rules to be disabled. All Neutron setups are affected.

Changed in ossa:
status: Incomplete → Confirmed
assignee: nobody → Tristan Cacqueray (tristan-cacqueray)
Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

Thanks Kevin for the update, please let me know if the above description is accurate.

@neutron-coresec, is the proposed patch going to be easily backportable ? Not sure what's the rule for policy.json changes...

Anyway, please follow this guide to properly format proposed patch: https://security.openstack.org/#how-to-propose-and-review-a-security-patch

Revision history for this message
Salvatore Orlando (salvatore-orlando) wrote :

This scenario will likely enable an attacker on a shared network to hijack DHCP traffic as mentioned by Kevin.
IP spoofing could also be possible since the attacker VMs would be able to intercept ARP requests and fabricate ARP replies

It is not possible to do any sort of mitigation using security groups, so this attack is very practical and probably deserves an A in the OSSA taxonomy

Revision history for this message
Kevin Benton (kevinbenton) wrote :

Yes, that description sounds correct. I will format the patch appropriately.

Revision history for this message
Aaron Rosen (arosen) wrote :

Patch looks good to me. +2,

The shared network flag feature in neutron is such a hack. Thank goodness we have proper RBAC now thanks to the support you added kevin :)

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

Assuming this affect all the way down to Juno, can you also attach backports for stable/juno and stable/kilo branches as well ?

Changed in ossa:
status: Confirmed → Triaged
Revision history for this message
Jeremy Stanley (fungi) wrote :

Minor improvement to the wording:

By changing the device owner of an instance's port right after it is created, an authenticated user may prevent application of firewall rules and so avoid IP anti-spoofing controls.

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

Thanks for the review, here is the updated impact description:

Title: Neutron firewall rules bypass through port update
Reporter: Kevin Benton (Mirantis)
Products: Neutron
Affects: versions through 2014.2.3 and 2015.1 versions through 2015.1.1

Description:
Kevin Benton from Mirantis reported a vulnerability in Neutron. By changing the device owner of an instance's port right after it is created, an authenticated user may prevent application of firewall rules and so avoid IP anti-spoofing controls. All Neutron setups are affected.

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

CVE have been requested.

@neutron-coresec, please review proposed patch.

Changed in ossa:
status: Triaged → In Progress
summary: IP, MAC, and DHCP spoofing rules can by bypassed by changing
- device_owner
+ device_owner (CVE-2015-5240)
Revision history for this message
Salvatore Orlando (salvatore-orlando) wrote : Re: IP, MAC, and DHCP spoofing rules can by bypassed by changing device_owner (CVE-2015-5240)

I verified the patch fixes the security issue. I ran several tests trying to find other the way to exploit the same issue, for instance by re-using system created network ports. Nova does not allow one to boot instances on such ports as nova will return a 409 when attempting to boot on a port with an already-populated device_owner attribute.

The patch is also resilient to white space padding (eg: " network:" is accepted but won't cause the port_security_enabled attribute go to False).

I have minor observations on the code changes:
- the policy change works but it seems weird as it's an or condition and is acting as a "and". Perhaps the policy engine is not operating as expected here. This is unrelated to this patch however.
- Having to prepend ~ to say it's a regex is fine by me. Alternatively one could treat everything as a regex, adding automatically ^ and $ for doing exact matches. But then you'll have to document that regex evaluation will always match on the whole string.

On the CVE it is worth probably stating that as far as we know only affects the ML2 plugin. The logic where the vulnerability is introduced is specific indeed to that plugin. Other plugins might be affected too if they adopted similar logic for determining whether spoofing protections should be installed, but I think it's up to the plugin maintainers to verify that and fix if necessary.

Summarizing: the patch looks good to me, and there's no need of further revisions.

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

Thanks Salvatore for this prompt review.

About the description, since we only support plugins within the neutron code base, the affects can also be reduced to:
"All Neutron setups using the ML2 plugin"

I'd like to send the advance notification today with this disclosure date/time:
2015-09-03, 1500UTC

Though ?

Revision history for this message
Kevin Benton (kevinbenton) wrote :

"All Neutron setups using the ML2 plugin or a plugin that relies on the security groups AMQP API"

Any plugins that have agents calling this method will be affected:
https://github.com/openstack/neutron/blob/4f802232771d051d0fe076e6458674a995c6bc14/neutron/api/rpc/handlers/securitygroups_rpc.py#L79-L84

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

pre-OSSA sent with last suggestion from Kevin. Thanks everyone :-)

Changed in ossa:
status: In Progress → Fix Committed
Revision history for this message
Kevin Benton (kevinbenton) wrote :

There is a mistake in the patches, sending updated versions shortly.

Revision history for this message
Jeremy Stanley (fungi) wrote :

That's unfortunate. Should we send updated patches to the downstream stakeholders and extend the disclosure date?

Revision history for this message
Kevin Benton (kevinbenton) wrote :

Yes, we will need to send the updated patches. I'm not sure if the disclosure date needs to be adjusted.

Revision history for this message
Kevin Benton (kevinbenton) wrote :

difference between this and the previous version is 'rule:admin_or_network_owner' instead of 'role:admin_or_network_owner' in policy.json

Revision history for this message
Kevin Benton (kevinbenton) wrote :
Revision history for this message
Kevin Benton (kevinbenton) wrote :
Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

Ouch, well then we need to extend the disclosure date. If those new patch are approved by thursday it can be:

2015-09-08, 1500UTC

Revision history for this message
Kevin Benton (kevinbenton) wrote :
Revision history for this message
Salvatore Orlando (salvatore-orlando) wrote :

I verified the updated patches with Kevin.
I think this explains why I was observing the weird behaviour noted in the review with:
"the policy change works but it seems weird as it's an or condition and is acting as a "and". Perhaps the policy engine is not operating as expected here. This is unrelated to this patch however."

Basically the typo was causing the policy engine to enforce a different kind of check (which always failed afaict). If we merged that patch we would have fixed the security issue, but introduced a functional defect.

The difference between the two patches is minimal, so there's not much else to review.

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

The new patch have been sent, please do not push them until the new disclosure date:
2015-09-08, 1500UTC

information type: Private Security → Public Security
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/221342

Changed in neutron:
assignee: nobody → Tristan Cacqueray (tristan-cacqueray)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/221344

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/juno)

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/221345

summary: - IP, MAC, and DHCP spoofing rules can by bypassed by changing
- device_owner (CVE-2015-5240)
+ [OSSA 2015-018] IP, MAC, and DHCP spoofing rules can by bypassed by
+ changing device_owner (CVE-2015-5240)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/221342
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=bbca973986fdc99eae9d1b2545e8246c0b2be2e2
Submitter: Jenkins
Branch: master

commit bbca973986fdc99eae9d1b2545e8246c0b2be2e2
Author: Kevin Benton <email address hidden>
Date: Tue Aug 25 22:03:27 2015 -0700

    Stop device_owner from being set to 'network:*'

    This patch adjusts the FieldCheck class in the policy engine to
    allow a regex rule. It then leverages that to prevent users from
    setting the device_owner field to anything that starts with
    'network:' on networks which they do not own.

    This policy adjustment is necessary because any ports with a
    device_owner that starts with 'network:' will not have any security
    group rules applied because it is assumed they are trusted network
    devices (e.g. router ports, DHCP ports, etc). These security rules
    include the anti-spoofing protection for DHCP, IPv6 ICMP messages,
    and IP headers.

    Without this policy adjustment, tenants can abuse this trust when
    connected to a shared network with other tenants by setting their
    VM port's device_owner field to 'network:<anything>' and hijack other
    tenants' traffic via DHCP spoofing or MAC/IP spoofing.

    Closes-Bug: #1489111
    Change-Id: Ia64cf16142e0e4be44b5b0ed72c8e00792d770f9

Changed in neutron:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/juno)

Reviewed: https://review.openstack.org/221345
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ebb0c5403b347ae1a841f578971ed1a3e3b6de53
Submitter: Jenkins
Branch: stable/juno

commit ebb0c5403b347ae1a841f578971ed1a3e3b6de53
Author: Kevin Benton <email address hidden>
Date: Tue Aug 25 22:03:27 2015 -0700

    Stop device_owner from being set to 'network:*'

    This patch adjusts the FieldCheck class in the policy engine to
    allow a regex rule. It then leverages that to prevent users from
    setting the device_owner field to anything that starts with
    'network:' on networks which they do not own.

    This policy adjustment is necessary because any ports with a
    device_owner that starts with 'network:' will not have any security
    group rules applied because it is assumed they are trusted network
    devices (e.g. router ports, DHCP ports, etc). These security rules
    include the anti-spoofing protection for DHCP, IPv6 ICMP messages,
    and IP headers.

    Without this policy adjustment, tenants can abuse this trust when
    connected to a shared network with other tenants by setting their
    VM port's device_owner field to 'network:<anything>' and hijack other
    tenants' traffic via DHCP spoofing or MAC/IP spoofing.

    Conflicts:
     etc/policy.json
     neutron/api/v2/attributes.py
     neutron/tests/etc/policy.json
     neutron/tests/unit/test_policy.py

    Closes-Bug: #1489111
    Change-Id: Ia64cf16142e0e4be44b5b0ed72c8e00792d770f9
    (cherry picked from commit 959a2f28cbbfc309381ea9ffb55090da6fb9c78f)

tags: added: in-stable-juno
tags: added: in-stable-kilo
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/kilo)

Reviewed: https://review.openstack.org/221344
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=767cea23de44a963c6793ffe30ea5c6827d27a38
Submitter: Jenkins
Branch: stable/kilo

commit 767cea23de44a963c6793ffe30ea5c6827d27a38
Author: Kevin Benton <email address hidden>
Date: Tue Aug 25 22:03:27 2015 -0700

    Stop device_owner from being set to 'network:*'

    This patch adjusts the FieldCheck class in the policy engine to
    allow a regex rule. It then leverages that to prevent users from
    setting the device_owner field to anything that starts with
    'network:' on networks which they do not own.

    This policy adjustment is necessary because any ports with a
    device_owner that starts with 'network:' will not have any security
    group rules applied because it is assumed they are trusted network
    devices (e.g. router ports, DHCP ports, etc). These security rules
    include the anti-spoofing protection for DHCP, IPv6 ICMP messages,
    and IP headers.

    Without this policy adjustment, tenants can abuse this trust when
    connected to a shared network with other tenants by setting their
    VM port's device_owner field to 'network:<anything>' and hijack other
    tenants' traffic via DHCP spoofing or MAC/IP spoofing.

    Closes-Bug: #1489111
    Change-Id: Ia64cf16142e0e4be44b5b0ed72c8e00792d770f9
    (cherry picked from commit 959a2f28cbbfc309381ea9ffb55090da6fb9c78f)

Changed in ossa:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (feature/pecan)

Fix proposed to branch: feature/pecan
Review: https://review.openstack.org/224334

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: feature/pecan
Review: https://review.openstack.org/224357

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (feature/pecan)
Download full text (73.6 KiB)

Reviewed: https://review.openstack.org/224357
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=fdc3431ccd219accf6a795079d9b67b8656eed8e
Submitter: Jenkins
Branch: feature/pecan

commit fe236bdaadb949661a0bfb9b62ddbe432b4cf5f1
Author: Miguel Angel Ajo <email address hidden>
Date: Thu Sep 3 15:40:12 2015 +0200

    No network devices on network attached qos policies

    Network devices, like internal router legs, or dhcp ports
    should not be affected by bandwidth limiting rules.

    This patch disables application of network attached policies
    to network/neutron owned ports.

    Closes-bug: #1486039
    DocImpact

    Change-Id: I75d80227f1e6c4b3f5fa7762b8dc3b0c0f1abd46

commit db4a06f7caa20a4c7879b58b20e95b223ed8eeaf
Author: Ken'ichi Ohmichi <email address hidden>
Date: Wed Sep 16 10:04:32 2015 +0000

    Use tempest-lib's token_client

    Now tempest-lib provides token_client modules as library and the
    interface is stable. So neutron repogitory doesn't need to contain
    these modules.
    This patch makes neutron use tempest-lib's token_client and removes
    the own modules for the maintenance.

    Change-Id: Ieff7eb003f6e8257d83368dbc80e332aa66a156c

commit 78aed58edbe6eb8a71339c7add491fe9de9a0546
Author: Jakub Libosvar <email address hidden>
Date: Thu Aug 13 09:08:20 2015 +0000

    Fix establishing UDP connection

    Previously, in establish_connection() for UDP protocol data were sent
    but never read on peer socket. That lead to successful read on peer side
    if this connection was filtered. Having constant testing string masked
    this issue as we can't distinguish to which test of connectivity data
    belong.

    This patch makes unique data string per test_connectivity() and
    also makes establish_connection() to create an ASSURED entry in
    conntrack table. Finally, in last test after firewall filter was
    removed, connection is re-established in order to avoid troubles with
    terminated processes or TCP continuing sending packets which weren't
    successfully delivered.

    Closes-Bug: 1478847
    Change-Id: I2920d587d8df8d96dc1c752c28f48ba495f3cf0f

commit e6292fcdd6262434a7b713ad8802db6bc8a6d3dc
Author: YAMAMOTO Takashi <email address hidden>
Date: Wed Sep 16 13:20:51 2015 +0900

    ovsdb: Fix a few docstring

    Change-Id: I53e1e21655b28fe5da60e58aeeb7cbbd103ae014

commit c22949a4449d96a67caa616290cf76b67b182917
Author: fumihiko kakuma <email address hidden>
Date: Wed Sep 16 11:52:59 2015 +0900

    Remove requirements.txt for the ofagent mechanism driver

    It is no longer used.

    Related-Blueprint: core-vendor-decomposition
    https://blueprints.launchpad.net/neutron/+spec/core-vendor-decomposition

    Change-Id: Ib31fb3febf8968e50d86dd66e1e6e1ea2313f8ac

commit d1d4de19d85f961d388c91e70f31b3bafec418c5
Author: Kevin Benton <email address hidden>
Date: Thu Sep 3 20:25:57 2015 -0700

    Always return iterables in L3 get_candidates

    The caller of this function expects iterables.

    Closes-Bug: #1494996
    Change-Id: I3d103e63f4e127a77268502415c0ddb0d804b54a

commit 1ad6ac448067306...

tags: added: in-feature-pecan
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (feature/pecan)

Change abandoned by Doug Wiegley (<email address hidden>) on branch: feature/pecan
Review: https://review.openstack.org/224334

Thierry Carrez (ttx)
Changed in neutron:
milestone: none → liberty-rc1
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in neutron:
milestone: liberty-rc1 → 7.0.0
Revision history for this message
Perry (panxia6679) wrote :

I won't be able to reproduce the problem on the Juno with steps in Bug Description. I am wondering whether there is any incorrect step. Can anyone help correct me? Thanks.

1) create a neutron security group
neutron security-group-create --description permissive unlocked11
neutron security-group-rule-create --protocol tcp --port-range-min 22 --port-range-max 22 --direction ingress unlocked11
neutron security-group-rule-create --protocol tcp --port-range-min 28 --port-range-max 50000 --direction egress unlocked11
neutron security-group-rule-create --protocol udp --port-range-min 1 --port-range-max 1000 --direction ingress unlocked11

2) create a port with security group created above
neutron port-create --security-group unlocked11 <network>

3) update port immediately after step 2)
watch neutron port-update <port uuid above> --device-owner network:hello

4) boot a vm
nova boot --key-name perry --nic port-id= <port uuid above> --flavor Standard --image <image> test

I found that if VM got ERROR with the following message.
{"message": "Build of instance f5b5090c-4141-44a8-8130-95692c5b1199 aborted: Failed to allocate the network(s), not rescheduling.", "code": 500, "details": " File \"/usr/lib/python2.6/site-packages/nova/compute/manager.py\", line 2079, in _do_build_and_run_instance filter_properties File \"/usr/lib/python2.6/site-packages/nova/compute/manager.py\", line 2213, in _build_and_run_instance reason=msg)

In addition, if I update the port device owner after several seconds after nova boot, the the VM's iptables looks consistent with port security group rules which is the same as normal VM without updating port. There is no any thing missed.

Thanks.

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

Hi Perry, just a quick check, are you using 2014.2.4 ? Or make sure you don't have the fix (changeId: Ia64cf16142e0e4be44b5b0ed72c8e00792d770f9) in order to reproduce that bug...

description: updated
Revision history for this message
Perry (panxia6679) wrote :

Hi Tristan, thanks for your reply. I double checked servers and there is no such fix (changeId: Ia64cf16142e0e4be44b5b0ed72c8e00792d770f9) applied. The code base is 2014.2.2.

Revision history for this message
Kevin Benton (kevinbenton) wrote :

Can you see why the nova instance failed to build? It may be easier to create the port, start the update loop and then attach it to a running VM.

Revision history for this message
Perry (panxia6679) wrote :
Download full text (5.1 KiB)

Hi Kevin,

compute log:

2015-11-27 03:17:29.480 6812 ERROR nova.compute.manager [-] [instance: 0b98dcb2-d343-4350-b65a-6615a8c45a20] Instance failed to spawn
2015-11-27 03:17:29.480 6812 TRACE nova.compute.manager [instance: 0b98dcb2-d343-4350-b65a-6615a8c45a20] Traceback (most recent call last):
2015-11-27 03:17:29.480 6812 TRACE nova.compute.manager [instance: 0b98dcb2-d343-4350-b65a-6615a8c45a20] File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2310, in _build_resources
2015-11-27 03:17:29.480 6812 TRACE nova.compute.manager [instance: 0b98dcb2-d343-4350-b65a-6615a8c45a20] yield resources
2015-11-27 03:17:29.480 6812 TRACE nova.compute.manager [instance: 0b98dcb2-d343-4350-b65a-6615a8c45a20] File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2172, in _build_and_run_instance
2015-11-27 03:17:29.480 6812 TRACE nova.compute.manager [instance: 0b98dcb2-d343-4350-b65a-6615a8c45a20] block_device_info=block_device_info)
2015-11-27 03:17:29.480 6812 TRACE nova.compute.manager [instance: 0b98dcb2-d343-4350-b65a-6615a8c45a20] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2661, in spawn
2015-11-27 03:17:29.480 6812 TRACE nova.compute.manager [instance: 0b98dcb2-d343-4350-b65a-6615a8c45a20] block_device_info, disk_info=disk_info)
2015-11-27 03:17:29.480 6812 TRACE nova.compute.manager [instance: 0b98dcb2-d343-4350-b65a-6615a8c45a20] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 4485, in _create_domain_and_network
2015-11-27 03:17:29.480 6812 TRACE nova.compute.manager [instance: 0b98dcb2-d343-4350-b65a-6615a8c45a20] raise exception.VirtualInterfaceCreateException()
2015-11-27 03:17:29.480 6812 TRACE nova.compute.manager [instance: 0b98dcb2-d343-4350-b65a-6615a8c45a20] VirtualInterfaceCreateException: Virtual Interface creation failed
2015-11-27 03:17:29.480 6812 TRACE nova.compute.manager [instance: 0b98dcb2-d343-4350-b65a-6615a8c45a20]
2015-11-27 03:17:29.490 6812 AUDIT nova.compute.manager [req-4de7d5b4-69c9-4e62-971c-c67ff32f418c None] [instance: 0b98dcb2-d343-4350-b65a-6615a8c45a20] Terminating instance
2015-11-27 03:17:29.512 6812 WARNING nova.virt.libvirt.driver [-] [instance: 0b98dcb2-d343-4350-b65a-6615a8c45a20] During wait destroy, instance disappeared.
2015-11-27 03:17:29.880 6812 ERROR nova.compute.manager [-] [instance: 0b98dcb2-d343-4350-b65a-6615a8c45a20] Failed to allocate network(s)
2015-11-27 03:17:29.880 6812 TRACE nova.compute.manager [instance: 0b98dcb2-d343-4350-b65a-6615a8c45a20] Traceback (most recent call last):
2015-11-27 03:17:29.880 6812 TRACE nova.compute.manager [instance: 0b98dcb2-d343-4350-b65a-6615a8c45a20] File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2172, in _build_and_run_instance
2015-11-27 03:17:29.880 6812 TRACE nova.compute.manager [instance: 0b98dcb2-d343-4350-b65a-6615a8c45a20] block_device_info=block_device_info)
2015-11-27 03:17:29.880 6812 TRACE nova.compute.manager [instance: 0b98dcb2-d343-4350-b65a-6615a8c45a20] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2661, in spawn
2015-11-27 03:17:29.880 6812 TRACE nova.compute.manager ...

Read more...

Revision history for this message
Perry (panxia6679) wrote :
Download full text (3.2 KiB)

Looks my problem can be workarounded by https://ask.openstack.org/en/question/57342/spawning-new-instances-fails-to-allocate-the-network/. After updating vif_plugging_timeout=0 and vif_plugging_is_fatal=False, the VM could be created with steps (loop to update port) and able to ping and I observed that there is nothing related to that corresponding VM in compute node iptables. I also observed some error during vm's rebooting which looks separate problem.

ci-info: +-------+--------------+--------------+---------------+-----------+-------+
2015-11-27 06:34:02,362 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [3/15s]: request error [[Errno 113] No route to host]
2015-11-27 06:34:05,363 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [6/15s]: request error [[Errno 113] No route to host]
2015-11-27 06:34:09,366 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [10/15s]: request error [[Errno 113] No route to host]
2015-11-27 06:34:12,367 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [13/15s]: request error [[Errno 113] No route to host]
2015-11-27 06:34:13,370 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [14/15s]: request error [[Errno 115] Operation now in progress]
2015-11-27 06:34:14,371 - DataSourceEc2.py[CRITICAL]: Giving up on md from ['http://169.254.169.254/2009-04-04/meta-data/instance-id'] after 15 seconds
2015-11-27 06:34:14,371 - util.py[WARNING]: No instance datasource found! Likely bad things to come!
Starting cloud-init: Can not apply stage config, no datasource found! Likely bad things to come!
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/bin/cloud-init", line 315, in main_modules
    init.fetch()
  File "/usr/lib/python2.6/site-packages/cloudinit/stages.py", line 307, in fetch
    return self._get_data_source()
  File "/usr/lib/python2.6/site-packages/cloudinit/stages.py", line 235, in _get_data_source
    pkg_list)
  File "/usr/lib/python2.6/site-packages/cloudinit/sources/__init__.py", line 250, in find_source
    raise DataSourceNotFoundException(msg)
DataSourceNotFoundException: Did not find any data source, searched classes: ()
------------------------------------------------------------
Starting cloud-init: Can not apply stage final, no datasource found! Likely bad things to come!
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/bin/cloud-init", line 315, in main_modules
    init.fetch()
  File "/usr/lib/python2.6/site-packages/cloudinit/stages.py", line 307, in fetch
    return self._get_data_source()
  File "/usr/lib/python2.6/site-packages/cloudinit/stages.py", line 235, in _get_data_source
    pkg_list)
  File "/usr/lib/python2.6/site-packages/cloudinit/sources/__init__.py", line 250, in find_source
    raise DataSourceNotFoundException(msg)
DataSourceNotFoundException: Did not find any data source, searched classes: ()
---------------------...

Read more...

Revision history for this message
Hao Jun Wang (shhjwang) wrote :

I am using Liberty and the patch has been installed. I can ping and ssh with the default security group in devstack through the reproduce steps. I think maybe it should not be permitted. Or there are some incorrect steps. Can anyone help me? Thanks very much.

I am using "demo" user.
My steps:
(1) neutron port-create <net-id> --name port1
(2) watch neutron port-update port1 --device-owner network:hello
step one and two are in one script.

(3) nova boot test --nic port-id=<port-id> --flavor m1.tiny --image cirros-0.3.4-x86_64-uec

I can see that instance test is active. I can ping and ssh it.
"neutron port-show port1" indicates that the "device_owner" is "network:hello" and "port_security_enabled" is "True"

I have set "vif_plugging_timeout=0" and "vif_plugging_is_fatal=False" in nova.conf to ensure that instance can be successfully booted.

The created port id is c05720c0-0259-441c-8c48-f650a4b01202
And I found the following lines in "ip addr".

32: qbrc05720c0-02: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
33: qvoc05720c0-02: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovs-system state UP
34: qvbc05720c0-02: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master qbrc05720c0-02
35: tapc05720c0-02: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master qbrc05720c0-02 state

But when I just execute the command "nova boot test2 --nic net-id=<net-id> --flavor m1.tiny --image cirros-0.3.4-x86_64-uec",
I cannot ping or ssh the instance "test2".

To post a comment you must log in.
This report contains Public Security information  
Everyone can see this security related information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.