tcp connections hang in forwarding machine

There are a handful of commits which obviously touch code that may relate to this. They are all pretty close to each other in the history.

I'm building test kernels from points before and after this collection of patches. If you can test these two, it should speed up bisection and identification of the problem.

As soon as they are build, I'll post them and provide a link here.

Thanks!

Test kernels here:

http://people.canonical.com/~sconklin/lp791512/

If possible please report your results with each of these.

Thanks a lot Lamont!

Steve

- linux-image-2.6.32-32-generic-pae_2.6.32-32.63~01spc823e22f_i386.deb
appears to be fail.

- linux-image-2.6.32-32-generic-pae_2.6.32-32.63~01spce92585c_i386.deb
appears to be working

Said testing being done with exactly one website. I'll leave the apparently-working kernel up for a while and therefore further testing.

Lamont: kernel for testing which patches which Tim asked for reverted are at http://people.canonical.com/~herton/lp791512/ for you to test

Lamont confirms the kernel above with the following two patches reverted work for him until now:
af_unix: Only allow recv on connected seqpacket sockets.
dccp: handle invalid feature options length

Next step will be trying to revert only the af_unix one.

Lamont: when you finish testing the kernel with the two reverts, being sure about it, please try the next kernel just with the change "af_unix: Only allow recv on connected seqpacket sockets." reverted now, available on http://people.canonical.com/~herton/lp791512/r3/

When you finish testing this and if you don't have any issues, then it would be good to do a sanity check just in case, and install the kernel 2.6.32-33.68 from -proposed, and check that it really still has the bug.

r3 seems to work fine, stock 33.68 from -proposed fails (though did actually give me the oft-tried-and-failed earlier web site)

I have r3 running live to see if that continues to work, but I believe it's solid.

<tgardner> lamont, are you still happy with the kernel herton built for you yesterday regarding bug #791512 ?
<lamont> tgardner: I have heard no complaints, and experienced no issues with it
<tgardner> lamont, great. I'll add that to the bug.

This is a testcase that shows what that af_unix patch reverted on test kernel is fixing.

Booted on a natty 8.42 kernel here, and the test case didn't throw an error, recv waits, and no oops happened.
On 10.44 with the problem fixed, the testcase exits with an error.

hey Lamont,

Next time you have access to that machine, how about doing:

lsof -i

to see whether that gives any indication of which process may be using sockets

There's likely to be a lot of noise, especially if it is a NAT box - but it might tell us something

Herton - how about instrumenting a kernel with a 'WARN_ON(sk->sk_state != TCP_ESTABLISHED)' in the right place. Wouldn't that tell us the name of the offending thread ?

Yes, WARN_ON should be enough to trace what app is at fault.

@Lamont: can you install the kernel at http://people.canonical.com/~herton/lp791512/r4/ and test? It should continue to work, but spill warnings in the kernel log, please attach the log here when it happens.

Installed, will advise.

Interestingly, the r4 kernel fails to pass some (but not all) forwarded traffic, and does not log anything. I have reverted to the r3 kernel, which continues to work.

This is strange. So the code shuffling made another bug appear somehow (timing?).

@Lamont: can you test a new kernel?, it's on http://people.canonical.com/~herton/lp791512/r5/

let us know if you get failures/warnings with it.

LaMont, may be there is a memory corruption in this case here, can you also test the kernel at http://people.canonical.com/~herton/lp791512/r6/ , and boot it with 'slub_debug' kernel boot parameter added?

This bug is missing log files that will aid in dianosing the problem. From a terminal window please run:

apport-collect 791512

and then change the status of the bug back to 'New'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]