DHCP broken for Openstack Nova instances since kernel v3.3

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1035172

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

I can confirm that the same bug happens when you run Openstack configured in Vlan mode so when the vm interface (eth1) are splitted in vlans.

@Adam,

The v3.6-rc1 kernel is now out. Would it be possible for you to test that kernel?

Also, I can assist you with the bisect and building test kernels if needed.

Thanks in advance!

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.6-rc1-quantal/

Joseph- I'm unable to get my host in working order for testing using that kernel as the initrd does not contain the required firmware for my NIC (bnx2) Is that expected to exist like it has for the other upstream kernels I've tested, or is there something I must do after installing the debs to repack the initrd with the required firmwares? Thanks

Did you happen to install both .debs, linux-image and linux-image-extra? That should be all that is needed.

Joseph- Yes, I've installed both. I was warned of possible missing firmware at package installation and sure enough at boot:

bnx2: Can't load firmware file "bnx2/bnx2-mips-09-6.2.1b.fw"

I've run into similar issues with quantal kernels in the past, see bug #1008749

Same here, i've installed on our servers and i get:

W: Possible missing firmware /lib/firmware/rtl_nic/rtl8106e-1.fw for module r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8168g-1.fw for module r8169

but it seems that eth it's still working. So the bug it's still there, I've attached a tcpdump output of the dhcp discover, offer which the vm never replies.

This is the virtual machine boot log

ApportVersion: 2.0.1-0ubuntu12
Architecture: amd64
DistroRelease: Ubuntu 12.04
InstallationMedia:

Package: linux (not installed)
ProcEnviron:
 LANGUAGE=en_US:en
 TERM=screen
 LANG=en_US.UTF-8
 SHELL=/bin/bash
Tags: precise
Uname: Linux 3.6.0-030600rc1-generic x86_64
UnreportableReason: The running kernel is not an Ubuntu kernel
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

As the comment above, i've added apport and set bug to confirmed. I've time until monday to do testing for this. Feel free to ask for more info to solve this.

@Adam,

Do you see the missing firmware error after installing the latest Quantal kernel, or only with the v3.6-rc1 mainline kernel?

@Joseph

I'm not sure this bug depends on that firmware error

@Adam,

Would it be possible for you to assist with a kernel bisect? I can build test kernels for you to test. If you can assist, we would first need to identify the kernel version that introduced this bug. We can then bisect down to the offending commit.

Would it be possible for you to test the following kernels:

v3.3 final: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.3-precise/
v3.4 final: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.4-quantal/

It appears the firmware may be missing from the mainline kernels. To get past this issue you could copy the various bnx2x firmware files into /lib/firmware. Note that you'll have to have the right versions for a specific kernel since the file names change for every kernel version.

Joseph, thanks and sorry I've been slow to respond. I'd be happy to assist with a bisect, in fact I already attempted to do this myself but I seemed to have hit some problems along the way when selecting proper starting and endpoints for the bisect, or using the wrong tree. I've even got a decent testing script that I could 'git bisect run' with, given there aren't unexpected firmware issues.

I will see about using those two kernels when I get the free cycles. But AFAICS from the testing I've already done:

- Using Ubuntu kernels that have been published in the archive already, the problem was introduced with the v3.4 packages/series. Any kernel prior (that is 3.2.0-24.37 and prior) worked as expected, 3.4.0-1.1 failed. I grabbed these kernel debs from publishing histroy @ https://launchpad.net/ubuntu/quantal/+source/linux.

- Using upstream kernels from the kernel-ppa, I was able to confirm that the regression was introduced with v3.3.

I will hopefully have a chance to test the two specific kernels you mention. In the meantime, if you might be able to provide me with a git repo + branch and the proper starting and endpoints to bisect between 3.2.0-24.37 and 3.3, I'd be happy to start doing the bisecting on my own.

Thanks

Thanks for the update, Adam.

Since you believe the regression was introduced in v3.3, can you test the following release candidates? That will allow us to identify a begin and end version for the bisect:

v3.3-rc1 http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.3-rc1-precise/
v3.3-rc2 http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.3-rc2-precise/
v3.3-rc3 http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.3-rc1-precise/
...

Basically we want to identify the first release candidate that introduced this bug. Then that will be our 'BAD' for the bisect with the prior release candidate being the 'GOOD' SHA1.

Sorry, the URL for v3.3-rc3 was posted wrong. It should be:

v3.3-rc3 http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.3-rc3-precise/

You can see the only part of the URL that changes is the -rcN piece.

Hi Joseph-

The same problem exists in v3.3-rc1. Using the mainline 3.2.28 (the last 3.2.x kernel listed) from the following URL showed no sign of the bug;

http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.2.28-precise/

Hi Adam,

I started a bisect between v3.2 final and v3.3-rc1. I'll post a test kernel shortly.

The first test kernel is built up to commit:
2ac9d7aaccbd598b5bd19ac40761b723bb675442

This kernel is available from:
http://people.canonical.com/~jsalisbury/lp1035172

Can you test that kernel and report back if it has the bug or not?

That kernel is good and not affected by the bug.

I built the next test kernel, which is built up to commit:
a429638cac1e5c656818a45aaff78df7b743004e

This kernel is available from:
http://people.canonical.com/~jsalisbury/lp1035172

Can you test that kernel and report back if it has the bug or not?

This one is good, too. Thanks Joseph.

I built the next test kernel, which is built up to commit:
16008d641670571ff4cd750b416c7caf2d89f467

This kernel is available from:
http://people.canonical.com/~jsalisbury/lp1035172

Can you test that kernel and report back if it has the bug or not?

Hi Adam,

Here's some info to perform the kernel bisect.

I was using the upstream linux-stable tree to perform the bisect. To clone that tree, you can run:
git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git linux-stable

Then cd into linux-stable

I used stable v3.2 as the first good and stable v3.3-rc1 as the first bad. Here are the actual commands:

git bisect start
git bisect good v3.2
git bisect bad v3.3-rc1

Or you could use the SHA1's for the kernel versions:
git bisect start
git bisect good 805a6af8dba5dfdd35ec35dc52ec0122400b2610
git bisect bad dcd6c92267155e70a94b3927bce681ce74b80d1f

git will then report the SHA1 for the first kernel build.

Finished bisecting over the weekend. It concluded that 7c7c7f01cc5e3e423120a4848a73dd5e4586f2f9 is the first bad commit. The following is a summary of the bisect against linux-stable:

git bisect start
git bisect good v3.2
git bisect bad v3.3-rc1

2ac9d7aaccbd598b5bd19ac40761b723bb675442 - good
a429638cac1e5c656818a45aaff78df7b743004e - good
16008d641670571ff4cd750b416c7caf2d89f467 - good
2bd43341217b6e8b75e382243328f458ac67fcbe - good
6a488979f574cb4287880db2dbc8b13cee30c5be - good
92b5abbb44e05cdbc4483219f30a435dd871a8ea - bad
fd63e9ceeeae58cfe877c2d49d41c1bf7532303c - good
4144cb2ade46d97b9c41682fd2e9064a59f23a98 - bad
3371bb3f7ed11b7b7473a202e2713bde50dc01c0 - bad
319d3b9c97b5e3191e419bb95496bf08ec50f096 - bad
088067f4f14d6ee5c6a196b015a560cbe7744224 - bad
7c7c7f01cc5e3e423120a4848a73dd5e4586f2f9 - bad
252c3d84ed398b090ac2dace46fc6faa6cfaea99 - good

7c7c7f01cc5e3e423120a4848a73dd5e4586f2f9 is the first bad commit
commit 7c7c7f01cc5e3e423120a4848a73dd5e4586f2f9
Author: stephen hemminger <email address hidden>
Date: Wed Jan 11 19:30:38 2012 +0000

    vhost-net: add module alias (v2.1)

    By adding some module aliases, programs (or users) won't have to explicitly
    call modprobe. Vhost-net will always be available if built into the kernel.
    It does require assigning a permanent minor number for depmod to work.

    Also:
      - use C99 style initialization.
      - add missing entry in documentation for loop-control

    Signed-off-by: Stephen Hemminger <email address hidden>
    Acked-By: Kay Sievers <email address hidden>
    Signed-off-by: David S. Miller <email address hidden>

:040000 040000 875fb9ee7d5a255a47161b0175ded141c5d52b0a c5f6bd70dacb0eff99381ff4fc3920f3759edc64 M Documentation
:040000 040000 4abaf797af5e7e66ec3746c45cee92decdebb175 ebb50f3aa2b1cfc8b3ac79612824d88498aba597 M drivers
:040000 040000 f6e513ed1626364e9946d8f03cc922698fc31fbb 0a530410047f948c266c4466d35ecdabb722bb4d M include

It looks like this is related to the discussion of vhost_net, packet checksumming and DHCP @ http://www.spinics.net/lists/kvm/msg37660.html and recent changes to libvirt that enable 'vhost=on' by default for KVM processes (vhost=on does not get set by default on precise)

Using iptables on the host with 'vhost=on' to workaround fixes the issue:

iptables -A POSTROUTING -t mangle -p udp --dport 68 -j CHECKSUM --checksum-fill

Also, disabled vhost explicitly in instance's libvirt.xml does as well, by setting

<driver name='qemu'/> in the <interface> section of the domain's configuration.