nova-network sometimes crashes with bad state

Hi,

I am also seeing this error running an ubuntu image or the ttylinux image using FlatDHCP...

2011-02-15 16:49:52,956 ERROR nova.root [-] Exception during message handling
550 (nova.root): TRACE: Traceback (most recent call last):
551 (nova.root): TRACE: File "/usr/lib/pymodules/python2.6/nova/rpc.py", line 192, in receive
552 (nova.root): TRACE: rval = node_func(context=ctxt, **node_args)
553 (nova.root): TRACE: File "/usr/lib/pymodules/python2.6/nova/network/manager.py", line 418, in allocate_fixed_ip
554 (nova.root): TRACE: self.driver.update_dhcp(context, network_ref['id'])
555 (nova.root): TRACE: TypeError: 'NoneType' object is unsubscriptable
556 (nova.root): TRACE:
557 2011-02-15 16:49:52,957 ERROR nova.rpc [-] Returning exception 'NoneType' object is unsubscriptable to caller
558 2011-02-15 16:49:52,957 ERROR nova.rpc [-] ['Traceback (most recent call last):\n', ' File "/usr/lib/pymodules/python2.6/nova/rpc.py", li ne 192, in receive\n rval = node_func(context=ctxt, **node_args)\n', ' File "/usr/lib/pymodules/python2.6/nova/network/manager.py", li ne 418, in allocate_fixed_ip\n self.driver.update_dhcp(context, network_ref[\'id\'])\n', "TypeError: 'NoneType' object is unsubscriptab le\n"]

Apparently in some cases db.network_get_associated_fixed_ips returns a FixedIp with fixed_ip_ref['instance']=None... but I don't understand this code enough to tell if that's a normal use case that should be supported in get_dhcp_hosts, or if we should find the root cause.

i have similiar

(nova.root): TRACE: Traceback (most recent call last):
(nova.root): TRACE: File "/usr/bin/nova-network", line 44, in <module>
(nova.root): TRACE: service.serve()
(nova.root): TRACE: File "/usr/lib/pymodules/python2.6/nova/service.py", line 231, in serve
(nova.root): TRACE: x.start()
(nova.root): TRACE: File "/usr/lib/pymodules/python2.6/nova/service.py", line 81, in start
(nova.root): TRACE: self.manager.init_host()
(nova.root): TRACE: File "/usr/lib/pymodules/python2.6/nova/network/manager.py", line 467, in init_host
(nova.root): TRACE: super(VlanManager, self).init_host()
(nova.root): TRACE: File "/usr/lib/pymodules/python2.6/nova/network/manager.py", line 125, in init_host
(nova.root): TRACE: self._on_set_network_host(ctxt, network['id'])
(nova.root): TRACE: File "/usr/lib/pymodules/python2.6/nova/network/manager.py", line 568, in _on_set_network_host
(nova.root): TRACE: network_ref)
(nova.root): TRACE: File "/usr/lib/pymodules/python2.6/nova/network/linux_net.py", line 169, in ensure_vlan_bridge
(nova.root): TRACE: interface = ensure_vlan(vlan_num)
(nova.root): TRACE: File "/usr/lib/pymodules/python2.6/nova/network/linux_net.py", line 178, in ensure_vlan
(nova.root): TRACE: _execute("sudo vconfig set_name_type VLAN_PLUS_VID_NO_PAD")
(nova.root): TRACE: File "/usr/lib/pymodules/python2.6/nova/network/linux_net.py", line 327, in _execute
(nova.root): TRACE: return utils.execute(cmd, *args, **kwargs)
(nova.root): TRACE: File "/usr/lib/pymodules/python2.6/nova/utils.py", line 147, in execute
(nova.root): TRACE: cmd=cmd)
(nova.root): TRACE: ProcessExecutionError: Unexpected error while running command.
(nova.root): TRACE: Command: sudo vconfig set_name_type VLAN_PLUS_VID_NO_PAD
(nova.root): TRACE: Exit code: 1
(nova.root): TRACE: Stdout: ''
(nova.root): TRACE: Stderr: 'sudo: no tty present and no askpass program specified\n'

Hi Edina, folks,

I think the issue Edina reported is different from the original issue by Narayan and Kost.
If 'requiretty' is set in /etc/sudores and nova-network was executed from init script on boot time,
we'll see the message which Edina got:

  "(nova.root): TRACE: Stderr: 'sudo: no tty present and no askpass program"

So, could you check your /etc/sudoers, Edina?

from man sudoers
       requiretty If set, sudo will only run when the user is logged in
                       to a real tty. When this flag is set, sudo can only be
                       run from a login session and not via other means such
                       as cron(8) or cgi-bin scripts. This flag is off by
                       default.

Yes, the issue from Edina should be split to another bug, since it's not the same issue.

Hi,

BTW, are these issues(Narayan's one and Kost one) still reproducible in trunk?
Then, I would like to know more information.

Especially, I'm wondering if 'nova-manage create network' was executed
successfully before starting nova-network and if so, fixed_range, num_networks and network_size.
Also results of 'nova-manage network list' and 'nova-manage fixed list' assuming the issue is reproducible
using trunk ppa...

I never had the ability to reproduce this bug on demand. It appeared
that there was some bad content in either the network database or
network-bound rabbitmq messages that would cause the code to
traceback, that would eventually flush itself out.

I'm in the process of upgrading our version of nova now; that should
be done in the next few days hopefully. We will see if it still occurs
then. Though, even in that case, I wouldn't necessarily be convinced
that the issue was gone.
 -nld

On Mon, Apr 4, 2011 at 12:52 PM, Masanori Itoh
<email address hidden> wrote:
> Hi,
>
> BTW, are these issues(Narayan's one and Kost one) still reproducible in trunk?
> Then, I would like to know more information.
>
> Especially, I'm wondering if 'nova-manage create network' was executed
> successfully before starting nova-network and if so, fixed_range, num_networks and network_size.
> Also results of 'nova-manage network list' and 'nova-manage fixed list' assuming the issue is reproducible
> using trunk ppa...
>
> --
> You received this bug notification because you are a direct subscriber
> of the bug.
> https://bugs.launchpad.net/bugs/719004
>
> Title:
>  nova-network crashes with bad data
>
> To unsubscribe from this bug, go to:
> https://bugs.launchpad.net/nova/+bug/719004/+subscribe
>

I just encountered this earlier tonight. It definitely is related to data in the networks table and fixed_ips not matching correctly. I've been screwing around with different network managers and deleting and recreating networks and this was the end result.

To fix it, I deleted everything from the networks and fixed_ips tables and then recreated the network with nova-manage and everything was fine.

Mark

That is interesting; when I had the problem, I added exception
handling for the error (and ignored it) and eventually the system
righted itself. I got the impression that there was some bad temporary
state.
 -nld

On Wed, Apr 20, 2011 at 3:05 AM, Mark Nelson <email address hidden> wrote:
> I just encountered this earlier tonight.  It definitely is related to
> data in the networks table and fixed_ips not matching correctly.  I've
> been screwing around with different network managers and deleting and
> recreating networks and this was the end result.
>
> To fix it, I deleted everything from the networks and fixed_ips tables
> and then recreated the network with nova-manage and everything was fine.
>
> Mark
>
> --
> You received this bug notification because you are a direct subscriber
> of the bug.
> https://bugs.launchpad.net/bugs/719004
>
> Title:
>  nova-network crashes with bad data
>
> To unsubscribe from this bug, go to:
> https://bugs.launchpad.net/nova/+bug/719004/+subscribe
>

I feel like this bug report is too generic. Are there still situation where nova-network crashes? If so, we need to create a bug for the individual cases and not for the general case. Going through bugs looking for things to fix right now, and I'm not certain this is relevant anymore. Can anyone provide recent tracebacks with specific situations in which nova-network crashes?

regarding 'sudo: no tty present and no askpass program specified\n':

it seems you are not running with root account. Devstack's solution is to run:

( umask 226 && echo "stack ALL=(ALL) NOPASSWD:ALL" \
        > /etc/sudoers.d/50_stack_sh )

As this bug is generic-ish, and is relating to quite old code, I propose marking this one as 'invalid' and encouraging the poster to resubmit a new bug with specific errors happening with newer code.