OpenStack Compute (nova)

Using FlatManager, Cannot ping or ssh to the instance

Bug #723488 reported by Tushar Patil
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Invalid
Undecided
Unassigned

Bug Description

I have setup nova on single machine.

1) created the br100 interface on the same machine (10.2.3.154)

a)brctl addbr br100
b)brctl addif br100 eth0
c) ifconfig br100 172.16.0.1 broadcast 172.16.0.255 netmask 255.255.255.0

2) Output of ifconfig command
-------------------------
root@ubuntu-volume-01:/home/tpatil# ifconfig
br100 Link encap:Ethernet HWaddr b8:ac:6f:93:48:11
          inet addr:172.16.0.1 Bcast:172.16.0.255 Mask:255.255.255.0
          inet6 addr: fe80::baac:6fff:fe93:4811/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:138 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:24916 (24.9 KB) TX bytes:368 (368.0 B)

eth0 Link encap:Ethernet HWaddr b8:ac:6f:93:48:11
          inet addr:192.168.0.6 Bcast:192.168.0.255 Mask:255.255.255.0
          inet6 addr: fe80::baac:6fff:fe93:4811/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:620683 errors:0 dropped:0 overruns:0 frame:0
          TX packets:39883 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:67866252 (67.8 MB) TX bytes:2569182 (2.5 MB)
          Interrupt:16 Memory:da000000-da012800

eth1 Link encap:Ethernet HWaddr b8:ac:6f:93:48:12
          inet addr:10.2.3.154 Bcast:10.2.3.255 Mask:255.255.255.0
          inet6 addr: fe80::baac:6fff:fe93:4812/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:1279604 errors:0 dropped:0 overruns:0 frame:0
          TX packets:681876 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:247677217 (247.6 MB) TX bytes:62583469 (62.5 MB)
          Interrupt:17 Memory:dc000000-dc012800

lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:16436 Metric:1
          RX packets:209631 errors:0 dropped:0 overruns:0 frame:0
          TX packets:209631 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:53337867 (53.3 MB) TX bytes:53337867 (53.3 MB)

vnet0 Link encap:Ethernet HWaddr fe:16:3e:4a:ad:02
          inet6 addr: fe80::fc16:3eff:fe4a:ad02/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:18 errors:0 dropped:0 overruns:0 frame:0
          TX packets:28 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:4332 (4.3 KB) TX bytes:6380 (6.3 KB)
--------------------------------------------------------------------------------------------------------
3) Create network using nova-manage
 $/usr/local/bin/nova-manage --flagfile=/etc/nova/nova.conf network create 172.16.0.0/24 1 256

4) nova.conf
--daemonize=1
--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/home/openstack/nova/bin/nova-dhcpbridge
--logdir=/var/log/nova
--state_path=/var/lib/nova
--networks_path=/var/lib/nova/networks
--sql_connection=mysql://root:nova@10.2.3.154/nova
--scheduler_driver=nova.scheduler.simple.SimpleScheduler
--s3_host=10.2.3.154
--cc_host=10.2.3.154
--rabbit_host=10.2.3.154
--ec2_url=http://10.2.3.154:8773/services/Cloud
--network_manager=nova.network.manager.FlatManager
--fixed_range=172.16.0.0/24
--network_size=32
--routing_source_ip=10.2.3.154
--verbose
--libvirt_type=kvm
--vlan_interface=eth0
--public_interface=eth1
--default_log_levels=amqplib=INFO, sqlalchemy=INFO, eventlet.wsgi.server=DEBUG
--use_ipv6=False
--ec2_prefix=http
--ec2_host=10.2.3.154
--ec2_port=8773
--ec2_suffix=services/Cloud
--allow_project_net_traffic=true

5) Run a new instance
$euca-run-instances -k flat -n 1 -t m1.tiny ami-tty

6) New instance launched is in the running state
$root@ubuntu-volume-01:/home/tpatil# euca-describe-instances
RESERVATION r-7m0lu20j admin default
INSTANCE i-00000005 ami-tty 172.16.0.5 172.16.0.5 running flat (admin, ubuntu-volume-01) 0 m1.tiny 2011-02-23 00:52:55 nova

7) Security groups
$root@ubuntu-volume-01:/home/tpatil# euca-describe-groups
GROUP admin default default
PERMISSION admin default ALLOWS icmp -1 -1 FROM CIDR 0.0.0.0/0
PERMISSION admin default ALLOWS tcp 22 22 FROM CIDR 0.0.0.0/0

8) Iptables output
root@ubuntu-volume-01:/home/tpatil# iptables -n -L
Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination
nova-local all -- 0.0.0.0/0 0.0.0.0/0

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
nova-local all -- 0.0.0.0/0 0.0.0.0/0

Chain nova-fallback (1 references)
target prot opt source destination
DROP all -- 0.0.0.0/0 0.0.0.0/0

Chain nova-inst-5 (1 references)
target prot opt source destination
DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
nova-sg-1 all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT udp -- 172.16.0.1 0.0.0.0/0 udp spt:67 dpt:68
ACCEPT all -- 172.16.0.0/24 0.0.0.0/0
nova-fallback all -- 0.0.0.0/0 0.0.0.0/0

Chain nova-local (2 references)
target prot opt source destination
nova-inst-5 all -- 0.0.0.0/0 172.16.0.5

Chain nova-sg-1 (1 references)
target prot opt source destination
ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:22

9) libvirt.xml
<domain type='kvm'>
    <name>instance-00000005</name>
    <memory>524288</memory>
    <os>
            <type>hvm</type>
            <kernel>/var/lib/nova/instances/instance-00000005/kernel</kernel>
                <cmdline>root=/dev/vda console=ttyS0</cmdline>
                <initrd>/var/lib/nova/instances/instance-00000005/ramdisk</initrd>
    </os>
    <features>
        <acpi/>
    </features>
    <vcpu>1</vcpu>
    <devices>
        <disk type='file'>
            <driver type='qcow2'/>
            <source file='/var/lib/nova/instances/instance-00000005/disk'/>
            <target dev='vda' bus='virtio'/>
        </disk>
        <interface type='bridge'>
            <source bridge='br100'/>
            <mac address='02:16:3e:4a:ad:02'/>
            <!-- <model type='virtio'/> CANT RUN virtio network right now -->
            <filterref filter="nova-instance-instance-00000005">
                <parameter name="IP" value="172.16.0.5" />
                <parameter name="DHCPSERVER" value="172.16.0.1" />
                <parameter name="PROJNET" value="172.16.0.0" />
<parameter name="PROJMASK" value="255.255.255.0" />

            </filterref>
        </interface>

        <!-- The order is significant here. File must be defined first -->
        <serial type="file">
            <source path='/var/lib/nova/instances/instance-00000005/console.log'/>
            <target port='1'/>
        </serial>

        <console type='pty' tty='/dev/pts/2'>
            <source path='/dev/pts/2'/>
            <target port='0'/>
        </console>
:q!
        <serial type='pty'>
            <source path='/dev/pts/2'/>
            <target port='0'/>
        </serial>

    </devices>
</domain>

10) tcpdump output
root@ubuntu-volume-01:/var/lib/nova/instances/instance-00000005# tcpdump -i br100 port 67 or port 68
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br100, link-type EN10MB (Ethernet), capture size 96 bytes
17:13:09.119573 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 02:16:3e:4a:ad:02 (oui Unknown), length 280
17:13:12.122724 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 02:16:3e:4a:ad:02 (oui Unknown), length 280
17:13:15.125840 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 02:16:3e:4a:ad:02 (oui Unknown), length 280
17:13:38.180931 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 02:16:3e:4a:ad:02 (oui Unknown), length 280
17:13:41.184208 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 02:16:3e:4a:ad:02 (oui Unknown), length 280
17:13:44.187426 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 02:16:3e:4a:ad:02 (oui Unknown), length 280

I cannot ping or ssh to the instance after the instance is launched successfully using --network_manager==nova.network.manager.FlatManager

I feel that even if the network model is flat the vm instance is requesting ip address from dhcp instead of static ip address (/etcnetwork/interfaces is properly injected). Please refer to libvirt.xml in #point9 above.

Revision history for this message
Vish Ishaya (vishvananda) wrote : Re: [Bug 723488] [NEW] Using FlatManager, Cannot ping or ssh to the instance
Download full text (19.2 KiB)

Generally networking does not work if you have an interface with an ip and a bridge bridged into that interface. You generally have to move the ip to the bridge. I'm not sure if that is causing your issue, but it might be.

On Feb 22, 2011, at 5:21 PM, Tushar Patil wrote:

> Public bug reported:
>
> I have setup nova on single machine.
>
> 1) created the br100 interface on the same machine (10.2.3.154)
>
> a)brctl addbr br100
> b)brctl addif br100 eth0
> c) ifconfig br100 172.16.0.1 broadcast 172.16.0.255 netmask 255.255.255.0
>
> 2) Output of ifconfig command
> -------------------------
> root@ubuntu-volume-01:/home/tpatil# ifconfig
> br100 Link encap:Ethernet HWaddr b8:ac:6f:93:48:11
> inet addr:172.16.0.1 Bcast:172.16.0.255 Mask:255.255.255.0
> inet6 addr: fe80::baac:6fff:fe93:4811/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:138 errors:0 dropped:0 overruns:0 frame:0
> TX packets:4 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:0
> RX bytes:24916 (24.9 KB) TX bytes:368 (368.0 B)
>
> eth0 Link encap:Ethernet HWaddr b8:ac:6f:93:48:11
> inet addr:192.168.0.6 Bcast:192.168.0.255 Mask:255.255.255.0
> inet6 addr: fe80::baac:6fff:fe93:4811/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:620683 errors:0 dropped:0 overruns:0 frame:0
> TX packets:39883 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:67866252 (67.8 MB) TX bytes:2569182 (2.5 MB)
> Interrupt:16 Memory:da000000-da012800
>
> eth1 Link encap:Ethernet HWaddr b8:ac:6f:93:48:12
> inet addr:10.2.3.154 Bcast:10.2.3.255 Mask:255.255.255.0
> inet6 addr: fe80::baac:6fff:fe93:4812/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:1279604 errors:0 dropped:0 overruns:0 frame:0
> TX packets:681876 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:247677217 (247.6 MB) TX bytes:62583469 (62.5 MB)
> Interrupt:17 Memory:dc000000-dc012800
>
> lo Link encap:Local Loopback
> inet addr:127.0.0.1 Mask:255.0.0.0
> inet6 addr: ::1/128 Scope:Host
> UP LOOPBACK RUNNING MTU:16436 Metric:1
> RX packets:209631 errors:0 dropped:0 overruns:0 frame:0
> TX packets:209631 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:0
> RX bytes:53337867 (53.3 MB) TX bytes:53337867 (53.3 MB)
>
> vnet0 Link encap:Ethernet HWaddr fe:16:3e:4a:ad:02
> inet6 addr: fe80::fc16:3eff:fe4a:ad02/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:18 errors:0 dropped:0 overruns:0 frame:0
> TX packets:28 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:500
> RX bytes:4332 (4.3 KB) TX bytes:6380 (6.3 KB)
> --------------------------------------------------------------------------------------------------------
> 3) Create network using no...

Revision history for this message
Christian Berendt (berendt) wrote :

Check if the field "injected" in table "networks" is set to "1". If this field is "0" there will be no network configuration injected into new instances.

Take a look into the nova-compute logfile, you'll find there some logging messages about the injection.

            if key:
                LOG.info(_('instance %(inst_name)s: injecting key into'
                        ' image %(img_id)s') % locals())
            if net:
                LOG.info(_('instance %(inst_name)s: injecting net into'
                        ' image %(img_id)s') % locals())

I don't think that the filterref defintion in your libvirt.xml file has something in common with the DHCPREQUEST. It's only a filter rule for the nwfilter in libvirt (take a look here: http://libvirt.org/formatnwfilter.html).

Try to login through the serial console after spawning the instance and check the network configuration.

Thierry Carrez (ttx)
Changed in nova:
status: New → Incomplete
Revision history for this message
Tushar Patil (tpatil) wrote :

The field "injected" in table "networks" is set to 1 as I am using FlatManager.
I have used http://images.ansolabs.com/tty.tgz images for spawning the instance.
I will try with other images too and see If I get same results.

>>Try to login through the serial console after spawning the instance and check the network configuration.
Is the serial console information is same as we get from the euca-get-console-output command?
Any help or direction on how to login through serial console is much appreciated.

Revision history for this message
Tushar Patil (tpatil) wrote :

In my test environment following VM images does not work with FlatManager.

1) ami-tiny image ( http://c2477062.cdn.cloudfiles.rackspacecloud.com/images.tgz)
2) ami-tty image ( http://images.ansolabs.com/tty.tgz)

I tried with another image which I have built using vmbuilder command and it works fine now with FlatManager.

I do made changes to the network configuration (bridge creation) as Vishy suggested
brctl addbr br100
brctl addif br100 eth0
ifconfig br100 10.0.0.1 broadcast 10.0.0.0 netmask 255.255.255.0 up
ifconfig eth0 0.0.0.0 up

Now I can ping as well SSH to the VM instance.

Revision history for this message
Thierry Carrez (ttx) wrote :

Those images rely on DHCP to get their IP address... Network injection doesn't work since it's not debian-based (no /etc/network/interfaces in the image).

Revision history for this message
Christian Berendt (berendt) wrote :

You could check if my modified injection code is working for you: https://code.launchpad.net/~berendt/nova/bp_injection/+merge/50737

Revision history for this message
Tushar Patil (tpatil) wrote :

After using proper VM Image, I can ping and SSH to the VM instance.
Marking as invalid.

Changed in nova:
status: Incomplete → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.