sshfs locks up randomly

Bug #137514 reported by Bogdan Butnaru
62
This bug affects 9 people
Affects Status Importance Assigned to Milestone
sshfs-fuse (Ubuntu)
Expired
Medium
Unassigned

Bug Description

Binary package hint: sshfs

Hello! I'm running Gutsy on a laptop, and I use sshfs to mount directories from various computers around the house. (AFAIK, it's the "safest" solution, but I'm open to suggestions.)

It usually runs pretty well, although a bit slow (though I think it's because of the wireless more than the overhead).

However, it randomly locks up. This means that any application that tries to access a mount point (say, nautilus or ls) just freezes. I can start a terminal and kill the sshfs process, unmount the "share", and the applications (usually) come back. Either that or I kill them and restart them from the terminal. I can remount immediately, and it works for a while until it happens again.

I can't see any constant pattern with the freezes. I seems to happen more often when there's a big load on the share (eg, copy files and play media at the same time), but not always. Sometimes it seems to happen repeatedly when I try to access a certain directory on a share, but after a while it works. It doesn't seem to be linked to a certain application, either. Sometimes it happens a dozen times a day, sometimes I go for days without encountering this.

I've checked for anything that seemed obvious: the network isn't to blame (everything else keeps working), it happens both on wireless and Ethernet, and I'm connecting to a Linux (Ubuntu Feisty) and a MacOS X machine and it happens the same way.

I don't know what to search for next.

Bogdan Butnaru (bogdanb)
description: updated
Revision history for this message
Jean-Francois Saucier (jfsaucier) wrote :

I can confirm here.

I use sshfs to mount my remote folder and if I listen to a small movie file, the movie lock sometime. The network continue to work (firefox, pidgin, etc) and I can do a ssh to my server but the sshfs mount is lock. After a short while, the lock dissapear and the movie restart to play...

Revision history for this message
Jean-Francois Saucier (jfsaucier) wrote :

I know it's late but I resolve the problem for me. I have switch to bcm43xx driver for my wireless adapter (I was using ndiswrapper before) and I have not a single freeze since then.

Maybe we can close this bug?

Revision history for this message
Bogdan Butnaru (bogdanb) wrote :

I'm not using ndiswrapper, and the network connection is always working when SSHFS locks up. I've never noticed anything similar happening with other programs, not even with SCP. So this is probably a different issue in the SSH filesystem driver, not in SSH or the network drivers.

Revision history for this message
Jean-Francois Saucier (jfsaucier) wrote :

Can it be that bug : https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/111502

Because I'm also affected by this and disabling NetworkManager did solve many problem with my wireless card.

Also, can you mention the wireless card that you use?

Revision history for this message
nirk (ola-brunborg) wrote :

I've got the exact same problem as the one originally described,

and I'm convinced it's got nothing to do with the connection stability (i.e. wireless).
The cable's been in all time, and now every shell or program that tries to access the mount freezes and doesn't come back.
Killing the sshfs-process with the signal SIGKILL (9) evidently "fixes" the problem. You can't umount the mount unless you do this first.

And I've got no clue on how to approach this problem, even though I'm rather experienced with computers.

Revision history for this message
nirk (ola-brunborg) wrote :

What I forgot to say is:
I'm running the same distro and on a laptop, as the initial poster.
I think this problem is of rather high importance, because this functionality is assumed stable by most people and it can thus probably causes a lot of problems and maybe undiscovered damage to files etc.

Revision history for this message
joehill (joseph-hill) wrote :

I can confirm this. I've been using sshfs since around Breezy and am how using Gutsy and Hardy, and in every version I've had this problem, although I have the impression it's become more consistent since Gutsy. Here are some observations I've made:

1. sshfs locks up consistently (no exceptions) whenever left idle for over maybe 20 or 30 minutes, regardless of what host I'm connected to and what kind of network connection I use (wireless, t1, cable).
2. Any program (terminal, nautilus, media player, etc.) that tries to access an sshfs file/directory locks up (sometimes uninterruptibly).
3. The only way to unlock these programs is to kill the ssh process (ssh -x -a -oClearAllForwardings=yes -2 remotehost.org -s sftp). I used to lazy unmount ("fusermount -uz mountpoint"), which frees up the mount point but leaves the ssh and other processes hanging.

I'll have to try to see if it still happens without network-manager--it may be that I started having this problem when I started using NM.

I think this is a major problem that makes sshfs practically unusable for anyone who doesn't like to constantly kill and restart processes through the command line.

Revision history for this message
Bogdan Butnaru (bogdanb) wrote : Re: [Bug 137514] Re: sshfs locks up randomly
  • unnamed Edit (2.8 KiB, text/html; charset=WINDOWS-1252)

As I've mentioned before, I noticed crashes often (though inconsistently)
when the mounts are used intensely, like copying movie files or lots of
directories, and when telling Amarok to scan the (100GB+) music collection.
There are crashes in other cases, but I didn't notice a pattern for those.

However, the crashes in the reproducible case go away if I mount manually
with "sshfs -f -s -o sshfs_sync -o no_readahead -o cache=no". It might be
useful as a workaround (I think most of those options can be given in
fstab). Someone with some time for debugging might try combinations of some
options to find out which one actually causes the crashes to go away, that
should point out the source.

On Tue, Apr 1, 2008 at 8:18 PM, joehill <email address hidden> wrote:

> I can confirm this. I've been using sshfs since around Breezy and am
> how using Gutsy and Hardy, and in every version I've had this problem,
> although I have the impression it's become more consistent since Gutsy.
> Here are some observations I've made:
>
> 1. sshfs locks up consistently (no exceptions) whenever left idle for over
> maybe 20 or 30 minutes, regardless of what host I'm connected to and what
> kind of network connection I use (wireless, t1, cable).
> 2. Any program (terminal, nautilus, media player, etc.) that tries to
> access an sshfs file/directory locks up (sometimes uninterruptibly).
> 3. The only way to unlock these programs is to kill the ssh process (ssh
> -x -a -oClearAllForwardings=yes -2 remotehost.org -s sftp). I used to
> lazy unmount ("fusermount -uz mountpoint"), which frees up the mount point
> but leaves the ssh and other processes hanging.
>
> I'll have to try to see if it still happens without network-manager--it
> may be that I started having this problem when I started using NM.
>
> I think this is a major problem that makes sshfs practically unusable
> for anyone who doesn't like to constantly kill and restart processes
> through the command line.
>
> --
> sshfs locks up randomly
> https://bugs.launchpad.net/bugs/137514
> You received this bug notification because you are a direct subscriber
> of the bug.
>

--
Bogdan Butnaru — <email address hidden>
"I think I am a fallen star, I should wish on myself." – O.

Revision history for this message
Bogdan Butnaru (bogdanb) wrote :

OK, I've done a bit of testing now. Mounting with "sshfs -s" does seem to work, I have yet to see any crashes like that. The other don't seem to help, though; I've tried with all of them together, and I still got crashes (and it works veeeery slow), so I'm pretty sure multi-threading is to blame.

I can't seem to find a way to put that option in fstab, I've filed bug #213644 about that.

Revision history for this message
Daniel T Chen (crimsun) wrote :

Is this symptom still reproducible in 8.10 beta or later?

Changed in sshfs-fuse:
status: New → Incomplete
importance: Undecided → Medium
Revision history for this message
Bogdan Butnaru (bogdanb) wrote :

No, I haven't noticed any instability with Intrepid. Perhaps the current packages should be back-ported to Hardy, given it's LTS.

Bogdan Butnaru (bogdanb)
Changed in sshfs-fuse:
status: Incomplete → Fix Released
Revision history for this message
stardust85 (m-samia) wrote :

yes, I have this problem in 8.10. Somebody should debug it to find the exact reason of this issue.

Revision history for this message
stardust85 (m-samia) wrote :

I am sorry for the previous comment, it was probably because I didn't have ServerAliveInterval 15 in my .ssh/config (or use -o ServerAliveInterval=15 on the sshfs command line) This will force the ssh connection to stay alive even if you have no activity. This is written in the sshfs FAQ (zcat /usr/share/doc/sshfs/FAQ.txt.gz)

Revision history for this message
Geoffrey (geoffrey-dommett) wrote :

Probably the same bug, using ubuntu 9.04, and had the same problem is earlier versions
I have an application that appends a large number of lines to a text file, one at a time, and it always causes sshfs to crash

the only workaround I have found is to launch sshfs with sshfs -d for debug output, then it runs without crashing, but the file transfer is VERY slow

if launched with sshfs -d remote mount_point > /dev/null 2>&1 to send the debug info to /dev/null file access is much faster (though still slower than without the -d switch) and I have never seen a crash running it this way.

I don't know if this helps anyone, nut is certainly the most ridiculous bug.

Revision history for this message
Bogdan Butnaru (bogdanb) wrote :

I should note that a while ago I started using « sshfs -o 'reconnect' » for my mounts. I'm not sure if this hides the bug, or if it's gone.

Using this I don't see the same pattern as before. I don't get freezes or obvious error messages anymore. However, accesses to the SSHFS-mounted partitions do seem to break periodically. For instance, copying files to-and-from sometimes stops working mid-operation, but work upon re-trying. (I acquired the reflex to copy/delete instead of moving things this way.)

I usually have music playing in Amarok all the time, sourced on the sshfs-mounted folders. Amarok used to freeze or do other bad things when the sshfs mounts went bad. Now it only stops playing a song mid-track, and skips to the next one. If I try to replay the skipped song, it always works correctly.

I guess whatever the bug is, the reconnect option hides most of its effects.

In response to Geoffrey's comment, I also tried to do some debugging a while ago. Various options (not only debug, but also things like readahead, sync, cache, direct_io, and something related to threading that I can't find right now) seemed to have effects on the breakage, but due to the rare and intermittent nature of the failures I wasn't able to isolate their effects.

Perhaps with a sure-fire example case like Geoffrey's program somebody can figure out what's wrong.

Revision history for this message
Geoffrey (geoffrey-dommett) wrote :

I wish the -o reconnect would work at all for me. unfortunately the file handles for open files become invalid which causes all kinds of problems for me, including the program in question to crash, or in your case Amarok to skip to the next song. not really any kind of a workaround at all.

I think I tried all the sync_cache and read ahead options that I could think of that made any sense. though I will confirm suggestions.
Ironically, turning on the debug output with the -d switch is the only thing that makes the problem go away, though it cripples the execution speed, so debugging this is rather difficult.

I am 99% sure that it must be some kind of stack or buffer overflow as:

 trickle -s -u 150 -d 150 sshfs -f Geoffrey@xxxxxxxxxxx: xxxxxxxxxxxx/

executes successfully, albeit with the program executing at a quarter the speed it should.

(trickle limits the network bandwidth available to the process)

This would indicate a problem with outstanding requests, but before anyone asks, setting -o max_read=1 and / or -o max_write=1 does not solve this, or make any discernible difference whatsoever.

Revision history for this message
themuddler (mike-udall) wrote :

I have this bug with Ubuntu 10.04 exactly as described by the OP (during heavy transfers). Killing the SSHFS process, unmounting and remounting solves the problem but seems to crash parts of gnome (not sure which but my theme disappears). I'm happy to help diagnose if I can and I'll try the various suggestions mentioned in the comments above.

I have changed the status back to confirmed so that this bug isn't overlooked because of its present status despite comments to suggest that it's still a problem. Please accept my apologies if I've not followed correct protocol with this change but the system appeared to allow me to make it so I was proactive.

Changed in sshfs-fuse (Ubuntu):
status: Fix Released → Confirmed
Revision history for this message
twak (twakered) wrote :

I can confirm this in 10.04. So I use sshfs to connect by ubuntu box to a server at work. When I'm on the lan it works well, and doesn't hang. At home (over cable), it hangs after half an hour or so (possibly of inactive use). Nautilus freezes without any feedback as to what has died.

Revision history for this message
Tyson Whitehead (twhitehead) wrote :

Would the you guys happen to be mounting your home directories via sshfs?

There is an issue where the reconnect option causes a loop with home mounts because sshfs restarts ssh and it then tries and access ~/.ssh. I've posted a patch for this upstream but it hasn't been accepted yet.

http://permalink.gmane.org/gmane.comp.file-systems.fuse.sshfs/1160

Cheers! -Tyson

Revision history for this message
Bogdan Butnaru (bogdanb) wrote :

@Tyson: I don’t know about the others, but I at least don’t mount the home directory. All SSHFS mounts I use are in /mnt or /media.

Revision history for this message
kripton (kripton-r) wrote :
Revision history for this message
Frostybeard (william-smith-frostybeard) wrote :

In Ubuntu 11.10 sshfs locks up, for example when the remote end drops off. "fusermount -u /path/to/mount" has no effect

When this happens, I cannot browse the directory where the mount folder is - it just hangs. If I am in a terminal, I have to forcefully close the terminal.

searching with "ps -e | grep sshfs" shows an orphaned process. If I kill it, I can use sshfs again and browse folders again.

@kripton @kripton-r
I have no realtek hardware on my computer. It happens over USB networking.

Revision history for this message
Ken Sharp (kennybobs) wrote :

Are you still seeing this?

Changed in sshfs-fuse (Ubuntu):
status: Confirmed → Incomplete
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for sshfs-fuse (Ubuntu) because there has been no activity for 60 days.]

Changed in sshfs-fuse (Ubuntu):
status: Incomplete → Expired
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.