Sometimes vms on same fixed network cannot ping others if floating address present

This system has one controller node and two compute nodes.

I think I misunderstood the problem. I recently saw that vms cannot access the other vms 10.0... addresses in the same cluster if the source and target are on different compute nodes. I think that was the real source of "randomness" I was seeing. The floating address was probably a red herring. This was using multi-host and running nova-network on each compute node. Is this a bug? If not, how are a set of vms in the cluster supposed to communicate?

I have not seen this before, but it is possible your traffic is getting snatted between nodes. you could try setting --dmz_cidr to 10.0.0.0/8
and see if that makes a difference
On Jan 27, 2012, at 7:25 AM, David Kranz wrote:

> I think I misunderstood the problem. I recently saw that vms cannot
> access the other vms 10.0... addresses in the same cluster if the source
> and target are on different compute nodes. I think that was the real
> source of "randomness" I was seeing. The floating address was probably a
> red herring. This was using multi-host and running nova-network on each
> compute node. Is this a bug? If not, how are a set of vms in the cluster
> supposed to communicate?
>
> --
> You received this bug notification because you are subscribed to
> OpenStack Compute (nova).
> https://bugs.launchpad.net/bugs/920493
>
> Title:
> Sometimes vms on same fixed network cannot ping others if floating
> address present
>
> Status in OpenStack Compute (Nova):
> New
>
> Bug description:
> It is my understanding, and observation, that vms allocated from the
> same --fixed_range network can ping the fixed addresses of other such
> vms. Also, vms can be pinged by fixed address from the compute node on
> which they are running. A user reported that when a floating ip was
> assigned to a vm, she could not access that vm by fixed address, but
> could by the floating address. I was able to reproduce this problem
> but not in a consistent way. It seemed more likely to happen when the
> floating ip address was assigned before the vm finished booting but I
> could not create a reliable pattern to show this. When the vm's fixed
> address was not accessible to other vms, it was still OK from the
> compute node hosting the vm. This system is based on diablo-stable and
> this nova.conf:
>
> --use_deprecated_auth
> --dhcpbridge_flagfile=/etc/nova/nova.conf
> --dhcpbridge=/usr/bin/nova-dhcpbridge
> --sql_connection=mysql://nova:notnova@172.18.0.131/nova
> --s3_host=172.18.0.131
> --rabbit_host=172.18.0.131
> --glance_api_servers=172.18.0.131:9292
> --logdir=/var/log/nova
> --state_path=/var/lib/nova
> --lock_path=/var/lock/nova
> --verbose
> --ec2_url=http://172.18.0.131:8773/services/Cloud
> --dmz_cidr=172.18.0.131/32
> --fixed_range=10.0.0.0/24
> --network_size=256
> --image_service=nova.image.glance.GlanceImageService
> --bridge_interface=eth1
> --flat_network_bridge=br100
> --flat_interface=eth1
> --network_manager=nova.network.manager.FlatDHCPManager
> --force_dhcp_release
> --public_interface=eth0
> --multi_host=1
> --osapi_host=172.18.0.131
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nova/+bug/920493/+subscriptions

You are probably right. I recently removed a dmz_cidr because it prevented a vm from accessing the address of the controller node. I will try this again.

Actually that did not seem to have any effect. How many people are using multi_host? I would change the title of this ticket but don't know how.

A lot of people. Let me check in our environment and see if I have the same issue.

On Jan 27, 2012, at 10:59 AM, David Kranz wrote:

> Actually that did not seem to have any effect. How many people are using
> multi_host? I would change the title of this ticket but don't know how.
>
> --
> You received this bug notification because you are subscribed to
> OpenStack Compute (nova).
> https://bugs.launchpad.net/bugs/920493
>
> Title:
> Sometimes vms on same fixed network cannot ping others if floating
> address present
>
> Status in OpenStack Compute (Nova):
> New
>
> Bug description:
> It is my understanding, and observation, that vms allocated from the
> same --fixed_range network can ping the fixed addresses of other such
> vms. Also, vms can be pinged by fixed address from the compute node on
> which they are running. A user reported that when a floating ip was
> assigned to a vm, she could not access that vm by fixed address, but
> could by the floating address. I was able to reproduce this problem
> but not in a consistent way. It seemed more likely to happen when the
> floating ip address was assigned before the vm finished booting but I
> could not create a reliable pattern to show this. When the vm's fixed
> address was not accessible to other vms, it was still OK from the
> compute node hosting the vm. This system is based on diablo-stable and
> this nova.conf:
>
> --use_deprecated_auth
> --dhcpbridge_flagfile=/etc/nova/nova.conf
> --dhcpbridge=/usr/bin/nova-dhcpbridge
> --sql_connection=mysql://nova:notnova@172.18.0.131/nova
> --s3_host=172.18.0.131
> --rabbit_host=172.18.0.131
> --glance_api_servers=172.18.0.131:9292
> --logdir=/var/log/nova
> --state_path=/var/lib/nova
> --lock_path=/var/lock/nova
> --verbose
> --ec2_url=http://172.18.0.131:8773/services/Cloud
> --dmz_cidr=172.18.0.131/32
> --fixed_range=10.0.0.0/24
> --network_size=256
> --image_service=nova.image.glance.GlanceImageService
> --bridge_interface=eth1
> --flat_network_bridge=br100
> --flat_interface=eth1
> --network_manager=nova.network.manager.FlatDHCPManager
> --force_dhcp_release
> --public_interface=eth0
> --multi_host=1
> --osapi_host=172.18.0.131
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nova/+bug/920493/+subscriptions

I just tested this in our lab and it works fine. My guess is that your router isn't set up to allow the 10.x range traffic. Perhaps you have port security on? It has to allow traffic across the ports that br100 is bridging into.

Vish

On Jan 27, 2012, at 10:59 AM, David Kranz wrote:

> Actually that did not seem to have any effect. How many people are using
> multi_host? I would change the title of this ticket but don't know how.
>
> --
> You received this bug notification because you are subscribed to
> OpenStack Compute (nova).
> https://bugs.launchpad.net/bugs/920493
>
> Title:
> Sometimes vms on same fixed network cannot ping others if floating
> address present
>
> Status in OpenStack Compute (Nova):
> New
>
> Bug description:
> It is my understanding, and observation, that vms allocated from the
> same --fixed_range network can ping the fixed addresses of other such
> vms. Also, vms can be pinged by fixed address from the compute node on
> which they are running. A user reported that when a floating ip was
> assigned to a vm, she could not access that vm by fixed address, but
> could by the floating address. I was able to reproduce this problem
> but not in a consistent way. It seemed more likely to happen when the
> floating ip address was assigned before the vm finished booting but I
> could not create a reliable pattern to show this. When the vm's fixed
> address was not accessible to other vms, it was still OK from the
> compute node hosting the vm. This system is based on diablo-stable and
> this nova.conf:
>
> --use_deprecated_auth
> --dhcpbridge_flagfile=/etc/nova/nova.conf
> --dhcpbridge=/usr/bin/nova-dhcpbridge
> --sql_connection=mysql://nova:notnova@172.18.0.131/nova
> --s3_host=172.18.0.131
> --rabbit_host=172.18.0.131
> --glance_api_servers=172.18.0.131:9292
> --logdir=/var/log/nova
> --state_path=/var/lib/nova
> --lock_path=/var/lock/nova
> --verbose
> --ec2_url=http://172.18.0.131:8773/services/Cloud
> --dmz_cidr=172.18.0.131/32
> --fixed_range=10.0.0.0/24
> --network_size=256
> --image_service=nova.image.glance.GlanceImageService
> --bridge_interface=eth1
> --flat_network_bridge=br100
> --flat_interface=eth1
> --network_manager=nova.network.manager.FlatDHCPManager
> --force_dhcp_release
> --public_interface=eth0
> --multi_host=1
> --osapi_host=172.18.0.131
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nova/+bug/920493/+subscriptions

Thanks, Vish. I will check it out.

The machines were not configured to bring up eth1. Sorry for the trouble.