neutron unable to create veth devices on utopic

Bug #1364828 reported by James Page
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
iproute2 (Debian)
Fix Released
Unknown
iproute2 (Ubuntu)
Fix Released
High
James Page
Utopic
Fix Released
High
James Page
neutron (Ubuntu)
Fix Released
High
Unassigned
Utopic
Fix Released
High
Unassigned

Bug Description

Deploying OpenStack Juno on Utopic results in a non-functional cloud (at least with neutron).

On gateway and compute nodes, neutron can't create veth devices AFAICT resulting in missing interfaces etc.. and generally broken instance connectivity.

ProblemType: Bug
DistroRelease: Ubuntu 14.10
Package: neutron-common 1:2014.2~b2-0ubuntu1
ProcVersionSignature: User Name 3.16.0-11.16-generic 3.16.1
Uname: Linux 3.16.0-11-generic x86_64
ApportVersion: 2.14.7-0ubuntu1
Architecture: amd64
Date: Wed Sep 3 08:39:35 2014
Ec2AMI: ami-0000005d
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.medium
Ec2Kernel: aki-00000002
Ec2Ramdisk: ari-00000002
PackageArchitecture: all
SourcePackage: neutron
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.neutron.api.paste.ini: [deleted]
modified.conffile..etc.neutron.fwaas.driver.ini: [deleted]
modified.conffile..etc.neutron.l3.agent.ini: [deleted]
modified.conffile..etc.neutron.neutron.conf: [deleted]
modified.conffile..etc.neutron.policy.json: [deleted]
modified.conffile..etc.neutron.rootwrap.conf: [deleted]
modified.conffile..etc.neutron.rootwrap.d.debug.filters: [deleted]
modified.conffile..etc.neutron.rootwrap.d.iptables.firewall.filters: [deleted]
modified.conffile..etc.neutron.rootwrap.d.l3.filters: [deleted]
modified.conffile..etc.neutron.rootwrap.d.vpnaas.filters: [deleted]
modified.conffile..etc.neutron.vpn.agent.ini: [deleted]
modified.conffile..etc.sudoers.d.neutron.sudoers: [deleted]
---
AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access /dev/snd/: No such file or directory
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.7-0ubuntu1
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
CRDA: Error: [Errno 2] No such file or directory
DistroRelease: Ubuntu 14.10
Ec2AMI: ami-0000005d
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.small
Ec2Kernel: aki-00000002
Ec2Ramdisk: ari-00000002
IwConfig: Error: [Errno 2] No such file or directory
Lsusb:
 Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd
 Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: OpenStack Foundation OpenStack Nova
Package: neutron
PciMultimedia:

ProcFB:

ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.16.0-11-generic root=LABEL=cloudimg-rootfs ro console=tty1 console=ttyS0
ProcVersionSignature: Ubuntu 3.16.0-11.16-generic 3.16.1
RelatedPackageVersions:
 linux-restricted-modules-3.16.0-11-generic N/A
 linux-backports-modules-3.16.0-11-generic N/A
 linux-firmware N/A
RfKill: Error: [Errno 2] No such file or directory
Tags: utopic ec2-images
Uname: Linux 3.16.0-11-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

_MarkForUpload: True
dmi.bios.date: 01/01/2011
dmi.bios.vendor: Bochs
dmi.bios.version: Bochs
dmi.chassis.type: 1
dmi.chassis.vendor: Bochs
dmi.modalias: dmi:bvnBochs:bvrBochs:bd01/01/2011:svnOpenStackFoundation:pnOpenStackNova:pvr2014.1.2:cvnBochs:ct1:cvr:
dmi.product.name: OpenStack Nova
dmi.product.version: 2014.1.2
dmi.sys.vendor: OpenStack Foundation

Revision history for this message
James Page (james-page) wrote :
Revision history for this message
James Page (james-page) wrote :

Looking at the running kernel, I can see that the veth module is not loaded - maybe this has been switch out from core kernel functionality for this release?

Revision history for this message
James Page (james-page) wrote :

It would appear that on trusty, the veth module gets loaded by something - looking as to what does this

Revision history for this message
James Page (james-page) wrote :

On a trusty install, I can't see anything in openstack loading the module; maybe the first call to use veth is doing something magic:

Sep 3 09:07:37 juju-jamespage-machine-11 sudo: neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf ip link add tap00742668-03 type veth peer name qg-00742668-03 netns qrouter-348230c6-0f09-43b2-9aed-641a95d4ed87

Revision history for this message
Brad Figg (brad-figg) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1364828

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Revision history for this message
James Page (james-page) wrote : Re: neutron unable? to create veth devices on utopic

The same command on utopic:

$ ip link add tap14e33126-b6 type veth peer name qg-14e33126-b6 netns qrouter-0fb14595-c1ea-4c14-84d7-a05c88afcb08
Error: argument "tap14e33126-b6" is wrong: Unknown device

summary: - neutron unable? to create veth devices on utopic
+ neutron unable to create veth devices on utopic
Revision history for this message
James Page (james-page) wrote :

Same commands manually executed on 14.04:

ubuntu@juju-jamespage-machine-0:~$ sudo ip netns add qrouter-0fb14595-c1ea-4c14-84d7-a05c88afcb08
ubuntu@juju-jamespage-machine-0:~$ sudo ip link add tap14e33126-b6 type veth peer name qg-14e33126-b6 netns qrouter-0fb14595-c1ea-4c14-84d7-a05c88afcb08

All OK

Changed in linux (Ubuntu):
status: Incomplete → Confirmed
tags: added: apport-collected
description: updated
Revision history for this message
James Page (james-page) wrote : AudioDevicesInUse.txt

apport information

Revision history for this message
James Page (james-page) wrote : BootDmesg.txt

apport information

Revision history for this message
James Page (james-page) wrote : CurrentDmesg.txt

apport information

Revision history for this message
James Page (james-page) wrote : Lspci.txt

apport information

Revision history for this message
James Page (james-page) wrote : ProcCpuinfo.txt

apport information

Revision history for this message
James Page (james-page) wrote : ProcEnviron.txt

apport information

Revision history for this message
James Page (james-page) wrote : ProcInterrupts.txt

apport information

Revision history for this message
James Page (james-page) wrote : ProcModules.txt

apport information

Revision history for this message
James Page (james-page) wrote : UdevDb.txt

apport information

Revision history for this message
James Page (james-page) wrote : UdevLog.txt

apport information

Revision history for this message
James Page (james-page) wrote : WifiSyslog.txt

apport information

Revision history for this message
James Page (james-page) wrote :

Using the supposedly optional name keywork appears to resolve this problem:

$ sudo ip link add name tap14e33126-b6 type veth peer name qg-14e33126-b6 netns qrouter-0fb14595-c1ea-4c14-84d7-a05c88afcb08

Changed in linux (Ubuntu):
status: Confirmed → Invalid
Revision history for this message
James Page (james-page) wrote :

$ ip link help

Usage: ip link add [link DEV] [ name ] NAME
                   [ txqueuelen PACKETS ]
                   [ address LLADDR ]
                   [ broadcast LLADDR ]
                   [ mtu MTU ] [index IDX ]
                   [ numtxqueues QUEUE_COUNT ]
                   [ numrxqueues QUEUE_COUNT ]
                   type TYPE [ ARGS ]

Revision history for this message
James Page (james-page) wrote :
no longer affects: linux (Ubuntu)
James Page (james-page)
Changed in iproute2 (Ubuntu):
importance: Undecided → High
status: New → Triaged
Changed in neutron (Ubuntu):
status: New → Triaged
importance: Undecided → High
James Page (james-page)
Changed in iproute2 (Ubuntu Utopic):
assignee: nobody → James Page (james-page)
status: Triaged → In Progress
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package iproute2 - 3.16.0-1ubuntu1

---------------
iproute2 (3.16.0-1ubuntu1) utopic; urgency=medium

  * d/p/0003-Don-t-enforce-that-dev-already-exists.patch: Cherry pick from
    Debian bug tracker to fix break in parsing of arguments for ip link add
    commands (LP: #1364828).
 -- James Page <email address hidden> Wed, 03 Sep 2014 11:29:22 +0100

Changed in iproute2 (Ubuntu Utopic):
status: In Progress → Fix Released
Changed in iproute2 (Debian):
status: Unknown → New
Revision history for this message
James Page (james-page) wrote :

Confirmed neutron functional again with updated iproute2 package.

Changed in neutron (Ubuntu Utopic):
status: Triaged → Fix Released
Changed in iproute2 (Debian):
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.