[Edgy Data Loss] umount progress dialog missing

Conformed by Tonio_ on #kubuntu-devel.

changing to high importance, as user's data is getting lost here.

This is a serious bug in Kubuntu (Edgy). I know already of several people who lost data due to moving data from the desktop to an USB stick and pulling out the USB stick too early.

Asking non-techniccal people to open a terminal and typing sync before pulling out USB sticks is _not_ a solution.

I confirmed this bug is present on Fedora Core 6's KDE also, so this is most likely an upstream KDE problem, too.

But Fedora mounts removable media sync, which greatly alleviates this situation.

also dapper was missing progress bar however it will keep the icon on the desktop until all data was copied... this can be done also for edgy or not?

No, the dialog was not missing in stock Dapper but it was missing in kubuntu.org's Dapper KDE update packages.

In stock Dapper, it shows an Unmounting dialog with a progress of 0% until the process was completely finished.

Today I experienced also this:
- forgot di remove usb key
- shut down normally edgy
- at the next boot -> data loss

nice!

i spoke to the maintainer earlier

this is a side effect of moving to dbus, and will be fixed in kde4.

I think leave Edgy and then Feisty in this status is the same to release an unstable OS that loss data!

> i spoke to the maintainer earlier
> this is a side effect of moving to dbus, and will be fixed in kde4.

We need a solution now and not in one year. People are loosing data for real! This is a severe bug and should get fixed.

On Fri, 2006-12-22 at 07:11 +0000, Sarah Hobbs wrote:
> i spoke to the maintainer earlier
>
> this is a side effect of moving to dbus, and will be fixed in kde4.
>
Ok, then in the meantime we will need some sort of less elegant way of
notifying the user not to remove the USB key until it is fully
unmounted, OR kubuntu-default-settings should set all removable devices
as sync.

Having it fixed in KDE4 is not an appropriate way of addressing a
serious data loss bug.

I never said it was a solution for now.

I just stated the cause of the bug.

A solution will have to be found - who's up for coding one?

I think this bug is underevaluted.

these are good steps to follow:
1) set to CRITICAL not HIGH
2) put SYNC for all devices in kubuntu-default-settings NOW (was to be done 3 months ago)
3) code a better solution
4) test it
5) then upload to edgy-updates the solution and delete SYNC again.

3 months for a data loss bug? No guys this is for an unstable product, not for a "stable" desktop or server one.
I can say with better words and trying to be more diplomatic, but the truth is this.

b.r.

On Sat, 2006-12-23 at 03:27 +0000, Sarah Hobbs wrote:
> I never said it was a solution for now.
>
> I just stated the cause of the bug.
Ok, cool, that's a relief to hear. Sorry if I got really edgy (no pun
intended) about it... When this affected GNOME, "users should be less
stupid" was the initially accepted solution, so yeah I tend to get a bit
feisty when it seems like a data loss bug is being brushed off.

>
> A solution will have to be found - who's up for coding one?
>

I think the best initial solution will be to have
kubuntu-default-settings add a hal config file that makes removable
media mount sync for Kubuntu.

The better psuedo-longterm solution is to modify KDE's unmounting logic
to first remount-ro and then unmount, so the icon does not disappear
immediately.

FYI:

* I always "sync" after using the USB storage.
* I do not use "cut/paste" in "Konqueror" anymore, because I guess that that is the main suspect of the corruption.
* The second suspect is the Trash can in the USB directory. Apparently, there is a problem with garbage collection?
* I have already fixed the USB stick twice by using (**forgive me**) M$ disk tools.

I'm not surer that sync is the right solution, read this, sync destroy removable keys!
http://readlist.com/lists/vger.kernel.org/linux-kernel/22/111748.html

anyway a solution is needed! And before kde 4.0 is out!

Thanx

I highly disagree with the original claims in that article. As someone who does a lot of work with embedded devices and CompactFlash, as well as an avid user of flash media, even the cheapest devices I can locally purchase can put up with an astounding number of read-write cycles, much more than the typical use-span of a thumbdrive can put on them.

Comparing a slightly premature failure in 15 years rather than 20 versus users ripping out the thumbdrive when KDE <b>assured</b> the user that his data was written, which sounds worse?

Perhaps remounting read-only, then unmounting is a better solution though I have no idea if dbus/hal has a backend for doing that.

In reality Linux needs behavior similar to Windows XP's handling of removable media, that is, turning off writeback caching. Right now "sync" is as close as we can get to that, but obviously it has some not-so-intuitive side effects.

Concerning mounting flash drives with "sync" as default, in my view this should only be done *until* the underlying problem is corrected. Using "sync" is not really solving the problem, it's just camouflaging it.

Bear also in mind that we are talking about a regression here: this worked perfectly in Dapper! Moreover, since it's all too easy to lose data over this, I agree with others who have said that this bug should be marked "Critical".

One question: does the problem affect only kubuntu or is it also present upstream?

Right, sync or ro+umount are just workarounds until a solid solution for tracking the unmounting state at the HAL/D-Bus level is reached.

This is indeed a KDE-wide problem that started in either 3.5.3 or 3.5.4 and affects all other KDE distros that I've tested, including Gentoo, Fedora, OpenSUSE. The only thing is they either mount sync or have a mounting daemon like submount that masks the problem.

I continue to NOT understand why this bug is ignored and also why a simple workaround is not taken.

We are not talking about a simple crash but DATA LOSS.
This is even worse than security issue in certain cases, and the bug is 4 months old!

I think Kubuntu can be called unstable or unsafe at this point, better switch to other solutions.

I can confirm this happening in x86_64 as well.
Please fix ASAP.
I cannot keep recommending Kubuntu Edgy with this problem,
I will make a fool of myself.

Also see bug #1.

I totaly agree with kubuntu_user and all you guys out there.

This bug *must* be fixed ASAP. I cannot understand why this bug is around since monthes and nothing visible happened.

Note: Ali Milis mentioned trash: My egdy doen't delete Trash in my Usb-stick. Trash remains in the stick as ".Trash" and can only be deleted with "sudo -R .Trash". Don't get me wrong, but this drives me mad. I don't want to use "sudo" just because I want to delete things that I have allready deleted....

These problems apart: Edgy rocks the house :-D

yup same here, already confirmed in another thread. dangerous bug...

On 1/17/07, "Kosmonaut" Bernd Müller <email address hidden> wrote:
> I totaly agree with kubuntu_user and all you guys out there.
>
> This bug *must* be fixed ASAP. I cannot understand why this bug is
> around since monthes and nothing visible happened.
>
> Note: Ali Milis mentioned trash: My egdy doen't delete Trash in my Usb-
> stick. Trash remains in the stick as ".Trash" and can only be deleted
> with "sudo -R .Trash". Don't get me wrong, but this drives me mad. I
> don't want to use "sudo" just because I want to delete things that I
> have allready deleted....
>
> These problems apart: Edgy rocks the house :-D
>
> --
> [Edgy Data Loss] umount progress dialog missing
> https://launchpad.net/bugs/61946
>

This is a definitely confirmed bug and we all understand the serious nature of the problem at heart, but it is also a bug that does not have any easy, unintrusive fixes. As a result, it's gonna take some time to find the optimal solution and implement it. Confirming the bug and/or lecturing on the seriousness of this bug any more will just generate more launchpad mail and not add any productiveness.

The problem is, the unmount dialog was lost on migration from KDE handling mounts to KDE asking dbus/HAL to handle mounts. There is not enough information being reported back via this new route to determine when an unmount operation is completely done... just whether or not it was successfully initiated.

So, possible workarounds and their drawbacks would be:

(1) Make global HAL policy as sync for removable media. This drags all flavors of Ubuntu into this ugly mess and reduces performance all-around, not to mention adding various degrees of additional stress on media due to VFAT's incorrect handling of the sync option. It is unfair to have this affect other flavors of Ubuntu as they are not affected by this KDE-specific bug.

(2) Make HAL policy as sync for kubuntu via kubuntu-default-settings. This in my mind is the most straightforward and also easiest to implement workaround until a true fix can be reached, but having a package modify configuration files in /etc is currently taboo (a violation of packaging policy)

(3) Extend the dbus/HAL mechanism for unmounting to differentiate between unmountING and unmountED. There may or may not be enough information exported via dbus to deduce that already; I'm not expert in dbus. But I would expect the KDE guys not to have introduced this bug if it was indeed this trivial to fix.

(4) Have KDE watch /proc/mounts or a similar mechanism for hackjobbed unmount progress detection, similar to what Ubuntu's GNOME does with its unmount dialog. Time consuming to implement?

(5) Have KDE remount read-only then unmount. This ensures that by the time the unmount is executed the medium will be in a written state safe for removal. This is also time-consuming to implement, and also has further complications given that there is currently no mechanism for ordinary users to remount read-only (That takes root), and also adds another failure mode in the read-only stage. If a application opens a file on the read-only disk, it will no longer unmount cleanly, leaving the need to roll back to read-write mount (which again may or may not succeed). This may leave users stuck with a half-crippled half-unmounted state, which is less than optimal to say the least :)

First of all, let me tank you for taking the time to explain this. Based on what you said, I am willing to manually fix this problem in all the installations I have access to, until a ditribution fix is ready.

Are there instructions to implement "Make HAL policy as sync" for just a particular device or USB bus ?

Sure, look in /etc/hal's configuration files, there already is example configuration (commented out) for implementing sync policy on removable devices.

Personally, I think it's a better workaround for the time being to educate users to issue a "sync" command at the terminal after unmounting, and waiting for that command to return before yanking out any media.

This allows users to keep enjoying the apparently faster write speed and only inconveniences the user at unmount time.

The users do not have to put any command in Windows, to report of
which it turns out to be inconvinient to do it whenever you have to
disconnect a device.
If we follow this theory, this does not advance and we will continue
having that to do everything across a terminal.

Excuse my English.

2007/1/20, John Dong <email address hidden>:
> Sure, look in /etc/hal's configuration files, there already is example
> configuration (commented out) for implementing sync policy on removable
> devices.
>
> Personally, I think it's a better workaround for the time being to
> educate users to issue a "sync" command at the terminal after
> unmounting, and waiting for that command to return before yanking out
> any media.
>
> This allows users to keep enjoying the apparently faster write speed and
> only inconveniences the user at unmount time.
>
> --
> [Edgy Data Loss] umount progress dialog missing
> https://launchpad.net/bugs/61946
>

--
El veloz murcciélago hindú comia feliz cardillo y kiwi. La cigüeña
tocaba el saxofón detrás del palenque de paja. 1234567890.

I understand that continue to talk about this bug is only spam, but hey, Feisty is near, and seems that there is NO activity for this bug.

I don't think it's a good idea to release another version affected by this bug.
Hope to understand that true is different.

Please remember BUG #1!!!

M$ OSes where not as stable as linux but they (almost) never lost data: an OS can be unstable, ugly, unfriendly, (whatever you want), but CANNOT loose data!
Nobody would use an OS that compromise data and USB keys (with big size) are very common today!
Do something... disable the incriminated GUI components... but do something!

Bye
Piero

So, after the very detailed and informativ comments of John Dong, there seems not to be a solution until Feisty+X. (Give X any number)

"As a result, it's gonna take some time to find the optimal solution and implement it. Confirming the bug and/or lecturing on the seriousness of this bug any more will just generate more launchpad mail and not add any productiveness."

Well, I really do understand that position. I guess that it is anoying to re-re-read all those allready confirmed bugreports. But I don't understand, what you have written here:

"There is not enough information being reported back via this new route to determine when an unmount operation is completely done... just whether or not it was successfully initiated."

So there are not enough informations, right? Question: Is Kubuntu the only KDE-based-distribution, that has this USB-"data loss" problem? How do other KDE-distros handle that problem. Or is this a Kubuntu special*feature* ;-).

I hope, this posting will not be misunsterstood as a *flame-posting*,or so. If I'd be able to hack this bug, I'd have done it long time ago (but I ehmm am to stupid for that:-D).

I have a temporary fix for this that works for me.

http://shinyblue.net/safelyRemove/

It uses kommander to provide a GUI to replace the KDE Safely Remove action. It calls sync, monitors /proc/meminfo, and then when sync is done, it calls eject.

great shinyblue, if it works then must be included in feisty and also backported to edgy!

thanx!

shiny fix the link ;)

On Wednesday 28 March 2007 01:22:38 Cimmo wrote:
> shiny fix the link ;)

Hey cimmo

erm, the link is correct

http://shinyblue.net/safelyRemove

site seems to be up for me, I've tried it from two different connections.

let me know if you still have probs and I'll look into it tomorrow.

smiles,
rich

shinyblue: the download link doesn't work.

Looks interesting though. We can't use it by default since we don't have Kommander installed by default but a version in python would be a great idea for feisty+1.

I've experimented a little with this.
The resulting debdiff is attached.

This progressdialog does not show any real progress, it just sits there at 1/3 till it's all finished.
I don't expect saving unwritten data to take much time.
It's a quick hack, but i belive it's better than nothing.

Please test/consider/use/enjoy.

- Frode

Got some feedback from Hobbsee, now i've made it show some (pseudo)progress.
New debdiff attached.

More feedback from Hobbsee, the progress dialog now stays open for atleast 1 second.
That way it doesn't just "flash" at the user when it finishes quickly.
Now most users manage to read what the dialog says before it finishes.

New debdiff attached

All of them work fine for me. The last one works best.

Resulting binary file is at http://ubuntu.lnix.net/misc/kio_media_mounthelper - chmod +x it, and put it into /usr/bin

Two people have tried this - no ill effects.

We really want to get this fixed for feisty.

Works for me.

This fix needs testing.
I don't expect testers to compile kdebase so this is a shortcut:
1. Fetch http://ubuntu.lnix.net/misc/kio_media_mounthelper
2. Check that md5sum is: 781e26354e3b3a1f4011d47b04107d31
3. chmod +x kio_media_mounthelper
4. Backup your existing /usr/bin/kio_media_mounthelper (cp /usr/bin/kio_media_mounthelper /tmp/)
5. sudo cp kio_media_mounthelper/usr/bin/
6. Safely Remove something.

Thanks for testing.
Please submit your testing results as comments to this bug.

Ok I've tried a bit this new way:

tried to copy a large file about 50 MB then copied another one (about 15 MB), I know that my usb key is not fast as it shown, so I safely removed, but dialog isn't appeared immediately but after some seconds (5-6) and then progress dialog start from 0% and go to 100% very quickly.

In other words seems that sync is called before showing progress dialog and then can confuse people, window dialog shouldn't be opened only at the end of sync.
Hope to be clear

Cimmo:
Thanks for your testing and reply.
Can you please test this: http://ubuntu.lnix.net/misc/kio_media_mounthelper_r2
MD5SUM:
f6f0160efa1b7cbb20d81f5e3056f8d9 kio_media_mounthelper_r2

Does this behave better?

- Frode

Frode same identical thing, dialog appears only at the end of sync and of course is very quick to disappears.

Can be that I'm with Kubuntu Edgy and not Feisty? Anyway I have kde 3.5.6 installed.
Let me know

Tested _r2 on edgy+kde 3.5.6;
-same results as Cimmo, dialog seems to be called after sync
-problem; if the copied file is too large, then sync takes long, resulting in a kde -error message (below) concerning a time-out. Tried the same thing in a console (sudo umount /dev/sdb1) and although it takes long, it unmounts cleanly.

error message
/*Unfortunately, the device system:/media/sdb1 (/dev/sdb1) named '131M Removable Media' and currently mounted at /media/usbdisk could not be unmounted. Unmounting failed due to the following error:
Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
Moreover, programs still using the device have been detected. */

I am aware of the delay before the dialog is shown, I have not found a simple way to make it appear earlier in the process.
The error message comes from the mediamanagers HALbackend. I won't touch that code this close to release.
The funny thing is, It might as well unmount successfully even though it gives that error.

I will try to figure out a way to do this better for feisty+1.
As of now I belive the question is: Is this good enought for feisty or is it better to leave it out?

Thanks for testing and feedback.
- Frode

Well I have done some more testing and tried to copy a 784.5 MB file, after it "finished" I safely removed and I got the error mentioned by Martijn (see attachment).
Tried the same identical file with shinyblue script and it worked a lot better:

window dialog appears immediately and also I didn't receive any errors.
In both case seems that file is copied correctly, but of course an error message or a late window dialog can confuse people as I said.

Frode anyway I don't want to say you did a bad job, THANK YOU SO MUCH to take care about that problem, I have only made some comparison :)
In my opinion your solution is better than nothing, shinyblue seems more reliable but uses kommander that is not installed by default.

thanx to both, hope to have a solution
Cimmo

I have to contradict myself, with Frode method when I receive the error shown in my attachment file is still syncing, if I remove the key just after received that error message then file is corrupted, tried with md5sum and they didn't match.

So I'm so sorry but shinyblue solution remains the best and most reliable one until those bugs are fixed in Frode one ;)

OK.

New proposal:

Get http://ubuntu.lnix.net/misc/kio_umountwrapper
and http://ubuntu.lnix.net/misc/media_safelyremove.desktop

chmod +x kio_umountwrapper and copy to /usr/bin/
media_safelyremove.desktop replaces /usr/share/apps/konqueror/servicemenus/media_safelyremove.desktop
(remember to backup).

kio_umountwrapper is a simple kdialog shellscript that executed kio_media_mounthelper -s.
This only adds the progress dialog, it does not remove any errors from the default behavior.
With this the dialogbox appears instantly.

- Frode

Tried also this Frode, in fact dialog appears immediately however with big files there is always the error mentioned before and sync is not finished when appears.
Anyway this is a bug present also in Edgy Default... it's a pity because without this bug can be a realiable solution

Cimmo:

In /etc/dbus-1/session.conf
Can you try to put this line:
  <limit name="activation_timeout">60000</limit>
after this:
  <limit name="service_start_timeout">60000</limit>

So the section of the file looks like this:
 <!-- raise the service start timeout to 40 seconds as it can timeout
       on the live cd on slow machines -->
  <limit name="service_start_timeout">60000</limit>
  <limit name="activation_timeout">60000</limit>

Then restart your machine or dbus and try again.
The error message you see is a D-Bus one.

Nevermind the latter comment. D-bus has changed. activation_timeout is now known as service_start_timeout.
I have no more ideas right now.

Frode in fact I had problem log into my account with that modification, the old text console let me back to previous file ;)

A question:
a script based on shinyblue solution but without kommander dependency?

Seems to be fixed by:

Changes:
 kdebase (4:3.5.6-0ubuntu20) feisty; urgency=low
 .
   * Add kubuntu_95_safely-remove_umount_dialog.diff
     On 'Safely Remove' this will make a kprogressdialog-popup telling
     the user it's unmounting.
     The popup will not close until the umount is complete. (Closes: LP #61946)

Uh? Frode is one of your solutions?

Erm, if this is properly fixed, great!

Otherwise, I give you my 2nd offering: same workings, but using Python (QT) and Perl.

details again at: http://shinyblue.net/safelyRemove/

smiles,

Rich.

thanx shiny I have tested a little your second solution here my personal suggestions (let's see you if you want to add them):

- if an error occurs can be shown why? For example if you still have an open konqueror window
- center window
- window should not be resizable
- close window after synced
- put an application icon, default X is not professional :)
- at the end says: ./media/usbdisk should be "/media/usbdisk" ?

Hey Cimmo, thanks for comments!

New tarball, just for you! Same place.

Done:
- if an error occurs can be shown why? For example if you still have an open konqueror window
- at the end says: ./media/usbdisk should be "/media/usbdisk" ?
Errors are reported, full stop removed. Also, sync works towards 99% now, not 100%, which
is more satisfying to me -- I hate seeing something say it's 100% done and then having to wait ages!
It goes to 100% when it starts calling eject.

Don't know how to do:
- center window
- window should not be resizable
- put an application icon, default X is not professional :)

Don't want to do:
- close window after synced
I like the confirmation that it has been successful. It's feasable that some error would happen and the application dissappear without it having sucessfully unmounted. So I prefer the confirmation.

smiles,
rich

> I like the confirmation that it has been successful. It's feasable that some error would happen and the application dissappear without it having sucessfully unmounted. So I prefer the confirmation.

you can always close window if succesfull or no if some errors are present

shiny tried and got 3 times errors :)
Ok 3 is better than nothing... see attachment.

tried new solution built-in in Feisty, it seems to work very well, however there is still the problem that with big files dialog appears only at the end of sync, so for a lot of seconds after safely remove is launched nothing occurs @ the video.

Stephan Kulow of KDE has just provided a patch, see http://bugs.kde.org/show_bug.cgi?id=143353.
Can anyone confirm this fixes the issue?

I have a new proposal for gutsy.
Remove current patch to kdebase. (revert to ubuntu19)

I have written a small KDE application that works as a wrapper around kio_media_mounthelper.
It provides a progressbar, and shows up instantly.

This is a feisty package for the thing at:
 http://frode.kde.no/misc/kio_umountwrapper/

To remove the old progressbar you need /usr/bin/kio_media_mounthelper from
https://launchpad.net/ubuntu/feisty/i386/kdebase-kio-plugins/4:3.5.6-0ubuntu19

You can either install the package directly and live with the conflicting versions or unpack the .deb with the following commands:
$ mkdir debtmp
$ ar xf kdebase-kio-plugins_3.5.6-0ubuntu19_i386.deb
$ tar -zxf data.tar.gz

NOTE: The next command will overwrite your current /usr/bin/kio_media_mounthelper,
to restore it use the command: 'sudo aptitude reinstall kdebase-kio-plugins'

$ sudo cp usr/bin/kio_media_mounthelper /usr/bin/

You can now test safely removing.

Feedback please!