sshfs mounts occasionally lock up

Thanks for reporting this bug and any supporting documentation. Since this bug has enough information provided for a developer to begin work, I'm going to mark it as confirmed and let them handle it from here. Thanks for taking the time to make Ubuntu better!

I have sent this upstream by message to 'fuse-sshfs <fuse-
<email address hidden>>' and we will now await a response.

Thanks for taking the time to make Ubuntu better!

The files I just attached are GDB backtraces of the SSH and SSHfs processes after this bug manifests itself.

According to “ps aux”, they were run as:

ssh -x -a -oClearAllForwardings=yes -2 tanelorn -s sftp
sshfs -s -o reconnect,transform_symlinks,allow_other,nonempty,large_read,hard_remove,intr,fsname=tanelorn:/mnt/corum tanelorn:/mnt/corum /media/corum

Everything that touched the mountpount froze, including Nautilus, and, strangely enough, the panel. Some applets on the panel *do* work, but some (and the panel itself) don’t. By the way, I have the system monitor in the panel, and it displays something like a quarter of its width filled with the IOWait color. I assume this means one of the four cores is blocked waiting for something.

By the way, the back-traces were recorded using the instructions for already-running programs at https://wiki.ubuntu.com/Backtrace

However, I skipped step 4; if I ran “continue”, gdb would hang too, and Ctrl-C would only display a “^C”. I hope this is useful.

I’ve done another few tests, this time with « sshfs »’s debug enabled. This didn’t work for the similar bug I experienced last year.

I mounted the fs with:
bogdanb@mabelode:~$ sshfs -d tanelorn:/mnt/corum /media/corum 2>&1 | tee sshfs-log.txt

Then I did things on the FS until I encountered the bug. Note that, as I said, this happens randomly, and usually when there’s a lot of activity. This time it took even longer than usual, I suspect that debugging conflicts with whatever is causing the lock-up.

(In this case, I was tagging music files with Picard; Picard reads the files to generate an acoustic fingerprint, and later writes the file with modified tags somewhere else. At the same time I was looking at images on the same mount-point, to help trigger the bug.)

As a result, the log is 16 MB long uncompressed. I also made backtraces for the SSHFS and SSH processes while locked up. I did several to see if it was completely frozen or just very slow; one of them seems to have changed between runs.

I noticed at one point a “Corrupted MAC on input” message. I had noticed the same message before, in a test run without “-d”. The earlier test run locked up very fast after that message, but this one kept going for quite a while, so I think it may be just a coincidence.

Some comment on the net suggested I look at dmesg, and I noticed the messages below there. (There were several identical copies, with ~120 delay in their timestamps.) It might help to know what Picard was doing when SSHFS locked up.

[ 1440.800052] INFO: task picard:2290 blocked for more than 120 seconds.
[ 1440.800056] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1440.800059] picard D 00000000ffffffff 0 2290 1 0x00000000
[ 1440.800065] ffff8800cfa41c78 0000000000000086 0000000000015b80 0000000000015b80
[ 1440.800071] ffff88010fe6df80 ffff8800cfa41fd8 0000000000015b80 ffff88010fe6dbc0
[ 1440.800076] 0000000000015b80 ffff8800cfa41fd8 0000000000015b80 ffff88010fe6df80
[ 1440.800081] Call Trace:
[ 1440.800091] [<ffffffff810f38d0>] ? sync_page+0x0/0x50
[ 1440.800096] [<ffffffff8153e697>] io_schedule+0x47/0x70
[ 1440.800100] [<ffffffff810f390d>] sync_page+0x3d/0x50
[ 1440.800104] [<ffffffff8153ed9a>] __wait_on_bit_lock+0x5a/0xc0
[ 1440.800108] [<ffffffff810f38a7>] __lock_page+0x67/0x70
[ 1440.800114] [<ffffffff81085340>] ? wake_bit_function+0x0/0x40
[ 1440.800118] [<ffffffff810f36de>] ? find_get_page+0x1e/0xa0
[ 1440.800121] [<ffffffff810f4f33>] find_lock_page+0x63/0x80
[ 1440.800125] [<ffffffff810f5192>] filemap_fault+0x182/0x460
[ 1440.800130] [<ffffffff811118b4>] __do_fault+0x54/0x500
[ 1440.800135] [<ffffffff81114df8>] handle_mm_fault+0x1a8/0x3c0
[ 1440.800140] [<ffffffff810397a9>] ? default_spin_lock_flags+0x9/0x10
[ 1440.800145] [<ffffffff8154334a>] do_page_fault+0x12a/0x3b0
[ 1440.800149] [<ffffffff81540ca5>] page_fault+0x25/0x30

I don't know if this is the problem here, but I often get zombie processes when I forget to unmount sshfs shares before suspending my laptop (thus breaking the network connection). I can 'fusermount -u -z' the share, but I need to kill the sshfs process before the zombies can go away. I don't think I've had to reboot.

Couldn't the sshfs process be terminated when the network connection breaks, or at least on a lazy (-z) unmount?

I'm not sure that this issue is exactly the problem I'm having but it seems that even the sshd locks up after sshfs has crashed. Since this happens on my headless server it's really tiresome. It started to happen after I upgraded to Lucid.

Would really appreciate if you looked into this asap.

Thanks!

I think it should generally not be possible for FUSE to lock processes like this.

One thing I did to improve the situation was to reduce the ssh timeout:
echo ServerAliveInterval 30 >> ~/.ssh/config

But I think in my case usually the ssh connection is really hung (though the network connection stands without a problem). Maybe this can help for you, too.

Can you please try the fix here:
http://rudd-o.com/linux-and-free-software/how-to-solve-network-data-corruption-with-realtek-cards
It worked for me on Gentoo using an MSI 890FXA-GD70 with Realtek Ethernet-chips.

Background:
https://www.zdnet.com/blog/ou/realtek-network-driver-silently-corrupts-data/639
https://www.zdnet.com/blog/ou/realtek-silent-data-corruption-caused-by-firmware/663

I have that fix activated since a long time ago. I still get issues; they seem somewhat rarer, but it’s hard to tell with things this intermittent.