big mess in devices permissions

Tested with fresh debootstraped dapper ppc

Confirmed with an up to date i386 Dapper (MSI K7T266 Pro2A mainboard, via82cxxx dirver if that matters).

Same here.

During boot the udevplug call during "Detecting and activating hardware..." hangs for a full minute, then fails. Booting is continued but many services (like network or kdm) fail to start because of wrong device permissions, missing device nodes, or kernel modules which are not loaded (for example via-rhine for network).

Logging into the console (and ignoring the permission errors on /dev/null) and stopping and starting again /etc/init.d/udev fixes everything, i.e. dev permissions are set properly and needed kernel modules loaded. Of course I have to restart some init scripts like nvidia-kernel, kdm, network, etc to get a fully working system again.

Just running /sbin/udevplug again doesn't help, it hangs for over 1 minute and the fails. /etc/init.d/udev needs to be stopped and started again.

This is on i386. Everything was working fine one and a half week ago. Sadly I didn't updated for a couple of days and thus don't know which update broke udev.

I have an eeror message now.

I booted up into Dapper, updated to the latest init scripts and kernel image today, rebooted and still no joy. So, I tried restarting udev from the console, but it just hug there. I sent the script to the background, and stopped it by doing a:

sudo /etc/init.d/udev stop

I recieved an error message:

udevplug[4144]: make_queue: unable to create /dev/.udev/queue: No such file or directory.

I then did a

sudo /etc/init.d/udev start

waited a couple of seconds and this time *it did work as expected*. That is, /dev/null has 0666 permissions instead of 0660 (as expected from reading the rules files in /etc/udev.d) and Gnome is working perfectly instead of jumping directly into a failsafe session with no operating shell.

Do notice that I have Arch Linux installed in other partition, with the save udev 079 and 2.6.15 kernel. Thus the problem is in Dapper's initialization scripts or in the ubuntu-only patches applied to either/both udev and the kernel.

I can confirm this bug. I believe the problem is that udev attempts to start before modules are loaded. Directly after udev fails, modules load, then my system fails to mount my /dev/evms/sdXX drive/partitions and I hit the e2fsck recovery console. From there, if I do:

/etc/init.d/udev stop
/etc/init.d/udev start
/etc/init.d/lvm restart
/etc/init.d/evms restart

all boots well.

Raymond has described the workaround perfectly. I just ran the test suggested by Scott in bug 29252 (that's in a PPC Mac, so i'll keep my reports here, as I have a IA32 machine).

The test is:

0) Boot up with init=/bin/bash in the kernel arguments
1) Run "udevd --daemon"
2) Check that /dev/.udev/queue does NOT exist
3) Run "udevplug -s -v"
4) Report the last line before it hands
5) Wait for up to three minutes, to see whether it times out and drops you back to the shell.

My coments on the test:

2) There was a /dev/.udev/queue. I had to REMOVE IT BY HAND.

4) The last line reported was "/sys/devices/pci0000:00"

5) udevplug hangs there for about 4 minuts and finally returns to the shell. Surprisingly, all device files are created according to the rules in /etc/udev.d/rules.d!!!!!?. Perfect permissions, timestamps show it is a runtime created udev filesystem!!!?

A small detail that comes to my mind just know. My BIOS set up is the factory one with only a couple of adjustments to front bus clock. video bus, it defaults to PCI but I have an AGP vidcard, and bootup order. Therefore, my BIOS *is not* letting the OS assign IRQ addreses but ordering them them itself (that is, it is set as "OS is not PnP aware". It works with MS WInXP, Breezy, Arch Linux, FreeBSD 6-STABLE...

Ok, the most interesting fact there is your reply to #2 ... you should not have a /dev/.udev/queue directory at that point.

So here's some more things to try

After booting (but before running udevd --daemon) does /dev/.udev/queue exist?

Is /dev/.udev/queue empty or is there something in it?

Is /dev a mountpoint and a tmpfs? (check /proc/mounts)

Is /dev empty?

Is there a /dev/.udev/failed?

Is there anything in /dev/.udev/failed?

Thanks

(The reason that udevplug is failing is because it's waiting for that directory to go away before it starts its work, and that directory never goes away -- so after 3 minutes it times out having done nothing, so you're left with whatever the initramfs prepared, which amongst other things means no permissions)

Hi Scott,

I've just run a test with your parameters and the results are:

=> After booting (but before running udevd --daemon) does /dev/.udev/queue exist?

Yes, it exists.

=> Is /dev/.udev/queue empty or is there something in it?

It is empty at first but after a couple of minutes there are a couple of entries (and udevd is not running!):

lrwxrwxrwx 1 root root 31 Jan 27 12:02 class@<email address hidden> -> /sys/class/usb_device/usbdev1.2
lrwxrwxrwx 1 root root 53 Jan 27 12:02 devices@pci0000:00@0000:00:11.2@usb1@1-1@1-1:1.0 -> /sys/devices/pci0000:00/0000:00:11.2/usb1/1-1/1-1:1.0

=> Is /dev a mountpoint and a tmpfs? (check /proc/mounts)

Yes. It shows up in /proc/mounts and /etc/mtab.

=> Is /dev empty?

No. It has a full device file suite.

=> Is there a /dev/.udev/failed?

Yes.

=> Is there anything in /dev/.udev/failed?

lrwxrwxrwx 1 root root 36 Jan 27 12:02 devices@pci0000:00@0000:00:00.0 -> /sys/devices/pci0000:00/0000:00:00.0
lrwxrwxrwx 1 root root 36 Jan 27 12:02 devices@pci0000:00@0000:00:01.0 -> /sys/devices/pci0000:00/0000:00:01.0
lrwxrwxrwx 1 root root 36 Jan 27 12:02 devices@pci0000:00@0000:00:11.0 -> /sys/devices/pci0000:00/0000:00:11.0

I think all this points out to a problem with USB devices. I have two attached at boot time, one is a UPS, the other one is an aDSL bridge.

Here are the PCI and USB settings in my box:

palopez@the-void:~$ lspci
0000:00:00.0 Host bridge: VIA Technologies, Inc. VT8366/A/7 [Apollo KT266/A/333]0000:00:01.0 PCI bridge: VIA Technologies, Inc. VT8366/A/7 [Apollo KT266/A/333 AGP]
0000:00:11.0 ISA bridge: VIA Technologies, Inc. VT8233A ISA Bridge
0000:00:11.1 IDE interface: VIA Technologies, Inc. VT82C586A/B/VT82C686/A/B/VT823x/A/C PIPC Bus Master IDE (rev 06)
0000:00:11.2 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 23)
0000:00:11.3 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 23)
0000:00:11.5 Multimedia audio controller: VIA Technologies, Inc. VT8233/A/8235/8237 AC97 Audio Controller (rev 40)
0000:01:00.0 VGA compatible controller: ATI Technologies Inc Rage 128 Pro Ultra TF

palopez@the-void:~$ lsusb
Bus 002 Device 001: ID 0000:0000
Bus 001 Device 003: ID 0e60:0600
Bus 001 Device 002: ID 051d:0002 American Power Conversion Back-UPS Pro 500/1000/1500
Bus 001 Device 001: ID 0000:0000

That makes absolutely no sense I'm afraid.

You can't get things created in /dev/.udev/queue if udevd isn't running, because that's the only thing that creates them.

Could you double-and-triple check that udevd isn't running at that point ... even if it means checking every /proc/*/exe

=> That makes absolutely no sense I'm afraid.

Yes, of course. That was a PEBCAK on my part. The contents of /dev/.udev/queue exist when I boot into the shell. I somehow didn't notice them when I checked the first time (I blame it on forgetting the US keyboard layout, have an Spanish one these days ;-)).

I've made sure to check three times, cold booting, hard resetting and soft rebooting from breezy. Each and every time I find the same USB devices I reported above in /dev/.udev/queue/ and /dev/.udev/failed/. There is no instance of udevd running (checked every
/proc/*/exe each time).

Interesting ... I bet I know what the bug is, we must be killing udevd in the initramfs while it's still processing events

I never thought of that ... amazing that it's happening so damned consistently for you, perhaps the problem is that you have a USB device that's taking a metric week to initialise

I'll add some fu to the initramfs scripts to instruct udev to stop processing events, then wait for them to complete, before killing it -- and we'll see whether that cures it for you

(it may just move the long wait to earlier in the boot process, of course, but it'd stop the damned-fucked behaviour <g>)

=> Interesting ... I bet I know what the bug is, we must be killing udevd in the initramfs while it's still processing events

Probably, and that may be the problem with bug 29541 as well. Older BIOSs were so darned buggy...

=> I never thought of that ... amazing that it's happening so damned consistently for you, perhaps the problem is that you have a USB device that's taking a metric week to initialise

Yes! And the winner is the UPS: its USB interface is slower than a sloth in painkillers. I have played with plugging and unplugging my USB toyz and now I am typing this from Dapper (today's vintage).

Yet, I don't want to lose my ability to shutdown automagically this box if power cuts off. I wonder if the detection of usbhid devices can be sent to the background somehow...

Ok, well at least it's been tracked down.

The USB detection can't be moved into the background because we need to do it before we mount the root filesystem for two reasons:

  1) the root filesystem might be on USB
  2) if it fails, you might need your keyboard

And even it was "backgrounded" (ie. we didn't wait for udevd's queue to become empty) it would be instantly foregrounded when we wanted for the filesystem hardware events.

But no matter, it actually looks to me like the device isn't even appearing in the udevd queue until after the wait is over (otherwise you'd be stuck in initramfs for a while) and is just there being processed when udevd gets killed.

Just uploaded 079-0ubuntu4 which will deliberately wipe the /dev/.udev/queue directory after killing udevd so udevplug in real userspace gets a clean start.

Hmmm.... For some reason the userspace udev process doesn't seem to start up correctly. If I plug in the UPS thingamagick, although the boot process doesn't hang, udev doesn0t kick in either. I'm left with the initramfs devices (e.g., /dev/null permissions are 0660) and no input devices. :-(

OK. I've booted into Dapper a couple of more times and now it will hang up just as it did previously. No joy :-)

Can you confirm the version of udev you have installed

I updated to 079-0ubuntu4 from Breezy before booting into Dapper.

At least my problem is gone with udev 079-0ubuntu4.

Alejandro ... could you try running "update-initramfs -u" before rebooting

=> Alejandro ... could you try running "update-initramfs -u" before rebooting

/me blushes in shame.

I forgot to do it!!! After updating the initrd from a Breezy's chroot, here I am typing this from Dapper, and I even managed to install apcupsd, the controler for my UPS, and everything is hunky-dory, yay!

Thank you very much Scott.

I'm adding the comment here rather than 29541 (which is more similar to my case) because the discussion here is more detailed. I have a hang in "detecting hardware", after which
(sometimes) the system locks solid
(othertimes) the system continues to boot after a while, but the touchpad doesn't work.

breezy worked fine on this system. I saw this around Jan 25, but it continues to be true after a dist-upgrade today (kernel 2.6.15-15-686, udev 079-0ubuntu8).

If I boot to single-user and then multi-user, it works fine.

I tried the suggestions above, and the answers are:
on booting with init=/bin/bash,
contents of /dev/.udev : db failed (no queue)

/dev is tmpfs, fully populated (afaict)

last line of udevplug -s -v:
/sys/devices/pci0000:00

contents of "failed":
lrwxrwxrwx 1 root root 36 Feb 8 13:38 devices@pci0000:00@0000:00:00.0 -> /sys/devices/pci0000:00/0000:00:00.0
lrwxrwxrwx 1 root root 36 Feb 8 13:38 devices@pci0000:00@0000:00:1e.0 -> /sys/devices/pci0000:00/0000:00:1e.0
lrwxrwxrwx 1 root root 49 Feb 8 13:38 devices@pci0000:00@0000:00:1e.0@0000:01:09.0 -> /sys/devices/pci0000:00/0000:00:1e.0/0000:01:09.0
lrwxrwxrwx 1 root root 49 Feb 8 13:38 devices@pci0000:00@0000:00:1e.0@0000:01:09.2 -> /sys/devices/pci0000:00/0000:00:1e.0/0000:01:09.2
lrwxrwxrwx 1 root root 49 Feb 8 13:38 devices@pci0000:00@0000:00:1e.0@0000:01:09.3 -> /sys/devices/pci0000:00/0000:00:1e.0/0000:01:09.3
lrwxrwxrwx 1 root root 36 Feb 8 13:38 devices@pci0000:00@0000:00:1f.0 -> /sys/devices/pci0000:00/0000:00:1f.0
lrwxrwxrwx 1 root root 36 Feb 8 13:38 devices@pci0000:00@0000:00:1f.3 -> /sys/devices/pci0000:00/0000:00:1f.3

These seem to be the PCI bridge and ISA bridge:
0000:00:1e.0 PCI bridge: Intel Corporation 82801 Mobile PCI Bridge (rev d3) (prog-if 01 [Subtractive decode])
        Flags: bus master, fast devsel, latency 0
        Bus: primary=00, secondary=01, subordinate=05, sec-latency=216
        I/O behind bridge: 00003000-00003fff
        Memory behind bridge: b0100000-b01fffff
        Prefetchable memory behind bridge: 0000000020000000-0000000021f00000
        Capabilities: <available only to root>

0000:00:1f.0 ISA bridge: Intel Corporation 82801FBM (ICH6M) LPC Interface Bridge (rev 03)
        Subsystem: Hewlett-Packard Company: Unknown device 3080
        Flags: bus master, medium devsel, latency 0

The above is booting with "pci=assign-busses" (otherwise my PCMCIA doesn't work). But the udevplug hang happens even without this option.

Any ideas? Should I post a new bug report?

Another remark: when I boot in single-user mode, I notice the message when init starts:

init: 110: cannot create /dev/.initramfs/progress_state: Directory nonexistent

Can this be related?

I did the following which seems to fix things (so far)

(1) comment out the udevplug lines from /etc/init.d/udev
(2) Create /etc/init.d/udevplug with *just* the udevplug lines in "start)" and nothing in the other sections
(3) Symlink this as /etc/rcS.d/S99udevplug (so it's the last thing to start in runlevel 1)

That seems to fix it for me, so far anyway... but it's ugly and will need attention every time I dist-upgrade.