ssh X11 forwarding does not work

I have problems with X11 forwarding via SSH both with lucid and maverick. On both of these systems it simply does not work
saying (for example):

xterm Xt error: Can't open display:
xterm: DISPLAY is not set

In /var/log/auth.log I can find this:

Oct 9 13:46:45 oxygene sshd[11122]: error: Failed to allocate internet-domain X11 display socket.

After trying to strace the sshd process, I found things like this:

11122 bind(10, {sa_family=AF_INET6, sin6_port=htons(6010), inet_pton(AF_INET6, "::1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = -1 EADDRNOTAVAIL (Cannot assign requested address)

It's strange for me, since I've disabled IPv6 everywhere, but still it want to use it for some unknown reason ...

I even have -4 option in /etc/default/ssh but it does not help.

HOWEVER, if I stop ssh daemon and launch it "by hand" /usr/sbin/sshd -4 then it works nicely! It's interesting that init script (or upstart?) script of sshd does not honor /etc/default/ssh it seems ... Or it shouldn't do that in maverick anymore?

Yes, I know that problem is caused not having IPv6 in my system by will, but I haven't known that I am forced to have configured IPv6 it was not compulsory in previous versions of Ubuntu distros. Is it possible to use Ubuntu in a normal way (without dirty hacks) if I don't want IPv6 at all? On my network there is no IPv6 capable equipment, so it's totally pointless to have it for any reason but still it does introduces possible security holes etc, I think the best practice not to have _anything_ configured which is not used/needed to avoid resource wasting and maybe even security problems. Like in OpenBSD where the default install is very minimal that's why that only 2 remote holes in the default install since X ages :) :)

lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          UP LOOPBACK RUNNING MTU:16436 Metric:1
          RX packets:1777 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1777 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:6642772 (6.6 MB) TX bytes:6642772 (6.6 MB)

ProblemType: Bug
DistroRelease: Ubuntu 10.10
Package: openssh-server 1:5.5p1-4ubuntu4
ProcVersionSignature: Ubuntu 2.6.35-22.33-generic 2.6.35.4
Uname: Linux 2.6.35-22-generic i686
NonfreeKernelModules: nvidia
Architecture: i386
Date: Sat Oct 9 13:52:29 2010
EcryptfsInUse: Yes
ProcEnviron:
 LANGUAGE=en_US:en
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: openssh