Installer stuck at "Removing conflicting operating system files..."

Attaching screenshot with my partition scheme.

Status changed to 'Confirmed' because the bug affects multiple users.

I have the same problem. I did not install any additional drivers; Ubiquity and the release are the same version (12.04 beta 1) but I also have a non-formatted old filesystem as /home. I have / and /boot formatted as new btrfs partitions and /home as the old not-to-be-formatted reiserfs partition. /home is mounted and decrypted at /mnt/tmpmount at the time the installer hangs with the message 'Removing conflicting operating system files' and the button to skip this is greyed out.

Please attach /var/log/syslog and /var/log/partman from an installation attempt after the point where it hangs.

Can you please run the installer until it reaches this point, then run `sudo apport-collect 946663`? This will collect files needed to be able to further diagnose this problem.

Attaching /var/log/syslog

Attaching /var/log/partman

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

I ran the apport command but it crashed with the following message after I clicked "Yes" on the dialog that asked me if I wanted to submit the debug log:

The authorization page:
 (https://launchpad.net/+authorize-token?oauth_token=Fs6t9Rv5VKjjppV539Ht&allow_permission=DESKTOP_INTEGRATION)
should be opening in your browser. Use your browser to authorize
this program to access Launchpad on your behalf.
Press any key to continue or wait (5) seconds...
Waiting to hear from Launchpad about your decision...

Traceback (most recent call last):
  File "/usr/share/apport/apport-gtk", line 468, in <lambda>
    GLib.idle_add(lambda: self.collect_info(on_finished=self.ui_update_view))
  File "/usr/lib/python2.7/dist-packages/apport/ui.py", line 830, in collect_info
    if (self.report['ProblemType'] == 'Crash' and 'Stacktrace' in self.report) or \
  File "/usr/lib/python2.7/UserDict.py", line 23, in __getitem__
    raise KeyError(key)
KeyError: 'ProblemType'

Oh, I see the files got attached twice, sorry about that.

I should probably mention also that it's *not* that the installation crashes, it just stays in the "Removing conflicting operating system files" forever. You can continue entering the install information. You can also expand the console on the installation window.

Is there a workaround for this? e.g. some subprocess we can kill?

I have run a number of partition scenarios based on the inforomation from Marcelo and Anthony, but so far have been unable to reproduce this issue using todays ubuntu_precise-desktop-i386.iso image.

... but looking at syslog, we have:

  umount: /mnt/tmpmount: device is busy.

This does indeed appear to be caused by finish.d/clear_partitions failing for some reason.

If you can still reproduce this issue, please can you:

- wait for the hang to occur
- click the top-left "home" button
- type "gnome-terminal" to open a terminal
- run the following commands and attach the output to this bug:
    - process listing ("ps -efwww")
    - mounts ("mount")
    - partitions ("sudo fdisk -l")
    - open files ("sudo lsof")

What would also be very useful is a tar file of /var/lib/partman/:

tar cvfz /tmp/partman.tgz /var/lib/partman/

I will try this tonight using the latest iso. BTW, I'm not using the i386 release, I'm using the amd64 one.

@Marcelo: thank you. The architecture shouldn't make a difference in this case.

reproducible for me on amd64.

 - install with partitions for /, /boot, swap, home (in this order)
 - encrypt home directory

 - restart from the image, choose "try Ubuntu"
 - install with partitions for /, /boot, swap, home (don't format this time), (in this order)
 - encrypt home directory

after that the installation hangs.

a side note: on the second install the swap partition was not recognized, and I had to re-enable it explicity

Still happening on today's (march 20th) amd64 build.

I'm attaching a tgz file with the outputs of the commands requested by James Hunt and the contents of /var/lib/partman. I am *not* enabling the home directory encryption option (my /home directory already exists from a previous install and it is on its own partition, as you can see).

Hope this helps.

I guess this is a duplicate of #924660.

I'm affected with the same typical setup (Ubuntu, amd64, installed with existing partitions for / (with formatting) and /home (without formatting). For me, the workaround described in #924660 worked (install without /home and manually add it to the fstab after installation).

Followed the steps as per #26 using the i386 desktop image from today, and I still do not see the hang: ubiquity correctly checks the "encrypt my home directory for the 2nd install but greys it out (since I'd asked not to reformat the /home FS).

I'm still using a 32-bit image, so maybe this is amd64 related after all. Will give that a try...

I rescind that comment - the installer is actually hanging for me on i386 with the message "configuring target system..."

I'm seeing the installer hang at "Removing conflicting operating system files..." during an attempt to install a precise i386 USB image on a PC with 2 hard disks. First disk has "/" and "swap" partitions, second disk has "/home" partition (not encrypted). I choosed to format the "/" partition and not format the "/home" partition.
Attached are the files requested in comment 22 (bug-946663-i386.tgz)

I have what sounds like essentially the same problem. "Removing conflicting operating system files..." and it never leaves that state. I connect to the wireless but do not install Broadcom drivers. I choose sda8 which has Lubuntu installed. I have tried to install with and without formatting the partition, this time without. I choose / as mount point and ext 4. This is a Samsung netbook with Win 7 and ubuntu 11.10, both of which I want to keep at this point.

Same problem on an Intel MacBook...

Same on an Atom netbook.

Same on my hp pavillion dv6000.

/windows ntfs
/ ext4 (Format)
/home ext4
/media/bigpart ext4

I appear to alo have this issue. Which is annoying, I was looking forward to trying 12.04 :(

Can't remember my exact partiton, but I know I am formatting the / partition and not the /home partition.

Same problem with beta2 formatting / as extra and mounting existing extra /home and 2 ntfs volumes

I'm trying to install without volumes except / and it seems installing. Maybe I can configure fstab manually after installation process completed

Same problem confirmed on an Intel MacBook 8,2.
Using Ubuntu-12.04LTS-beta2-desktop-i386 as installation source.
I have attempted to install the os on 8 different plans on my 2 separate partitions ( / and /home ), on the basis of, for each partition, whether it 1) contains old data or not, whether it 2) is formatted already before running the installer or not and whether it 3) is marked in ubiquity "to be formatted" during the installation or not.
The result is that the problem occurs in all scenarios and has no relevance to the three controls I have set.

I have the same issue. Stuck at "Removing conflicting operating system files..."

My setup is:

NTFS - Recovery
NTFS - Diagnostics
Ext4 - /home (not to be formatted)
Extended Partition
   Ext4 - / (to be formatted)
   Swap (to be formatted)

Also encountering this issue.

Formatting / as ext4 and mounting /home on existing ext4

Installing as brtfs on / works fine however, but that leads to the annoying apt-get taking forever issue which renders apt-get more or less unusable.

Yep, same issues here. Lenovo Z575... Was looking forward to having a nice shiny new system :( I'm using extended partitioning, perhaps an issue there?

I confirm this with ubuntu-12.04-beta2-desktop-i386.iso while not formatting /home partition.
A workaround from #28 does work (install without separate /home)

Update: still met this issue on Precise beta2.

Got this as well.
Configured the installer to reformat both my /boot and root partitions, but leave my /home untouched. Home is ext4, not encrypted.

Running 12.04 beta2.

Just ran the command as per #22. Running sudo lsof produced the following warnings:

lsof: WARNING: can't stat() tmpfs file system /cow
      Output information may be incomplete.
lsof: WARNING: can't stat() fuse.gvfs-fuse-daemon file system /home/ubuntu/.gvfs
      Output information may be incomplete.

Output has been attached.

I also format /boot and / and keep /home unchanged, but /home in ReiserFS

In console I see:

ubuntu ubiquity: umount /mnt/tmpmount: device is busy
ubuntu ubiquity: rmdir: failed to remove '/mnt/tmpmount': Device or resource busy.

I was having the same problem with Precise Beta2 AMD 64 I just downloaded. I have 2 hdds, one with windows partitions (secondary) and other for linux only (primary). Ubiquity kept hanging until I assigned every single partition. Once I assigned both windows' and linux's partitions, ubiquity finished the installation as usual.

This workaround solved it for me.

https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/924660/comments/3

I had tried to format / as ext4 und mount /home as ext4 before the beta 2 (download on 04-10) got stuck. Both partitions are primary. The mentioned workaround can't be a solution, because handling the /home data is pretty dangerous in my mind. To me, this really is a severe problem.

Looking at this, it seems that mostly everybody with this problem has an additional NTFS partition. So basically to reproduce it you need this partition schema:

* One partition mapped to /, format
* One partition mapped to /home, no format
* One NTFS partition, no format

Makes sense?

In my case (comment #31) I didn't have any NTFS partition. It seems that having a /home partition, and not formatting it, is a sufficient condition to trigger the bug.

 #53

Looking at this, it seems that mostly everybody with this problem has an additional NTFS partition. So basically to reproduce it you need this partition schema:

* One partition mapped to /, format
* One partition mapped to /home, no format
* One NTFS partition, no format

Makes sense?

____

No. Because I'm experiencing this bug and have no ntfs partition in my laptop. Only /, swap and /home partitions.

The workaround above works.

I don't have any NTFS partitions either. And for me the workaround as mentioned in #51 worked, although I skipped the whole /home2 part. I just:
- kept my existing /home unmounted until the installation had finished
- made a copy of everything in the created /home just to be sure (never needed it afterwards, though)
- then edited the /etc/fstab file on the target disk to mount my existing /home
- rebooted, done.

But I agree with #52, this is NOT a solution.

I am curious what Canonical thinks is the best way (recommended flow) for existing Ubuntu users who prefer NOT to use the upgrade mechanism, but install a fresh setup instead and then re-use their existing /home.
IMHO, this flow is very complicated (to non-techies) even if this bug were not present. Perhaps I should file a different bug for this?

I have additional NTFS partition.

I don't think this bug has anything to do with ntfs. This bug is very easily reproducible with ONLY a separate ext4 home partition, and no ntfs partitions at all. Quite a serious bug as keeping your home partition unformatted so you don't need to wipe all your data (for example I have 100+ gigs of music, thats a huge pia to have to copy over every 6 months). Upgrading using distro upgrade can often be problematic, so this is the best way to "clean install" and keep your data, this is something that really needs to work on final release (not to mention this bug also borks your previous install)

I'm seeing this bug, and I have neither NTFS partitions, nor a separate /home. I've told the installer to install everything in a single partition at /. I'm installing to a btrfs partition that I created outside the installer (because I wanted to enable the compress option and the installer won't let me do that) -- so the target root filesystem already exists. If I tell it to format the root partition, it works.

So this is a bug that only exists when the user choses NOT to format an install partition. Seems it could be / or /home (or probably anything else)

agree on #54, #58, #59, etc. above that it's nothing to do with if the partition is any of NTFS, btrfs, reiserfs.

Maybe this is trigged by some archives that the installator tries to erase without sucess because a lack of permissions?

This is caused by the script /lib/partman/finish.d/10clear_partitions when umount fails after deleting critical directories from the (existing) target filesystem. There appears to be a race condition between subshell commands which access the partition mounted on /mnt/tmpmount, and the umount. e.g. commands run in a sub-shell, such as the several (rm -rf "$x" ...) have not fully exited before the umount was run.

I was able to work around this by inserting 'sleep 5' before the umount $tmp on line 82. Not an elegant solution, but it worked for me. There are three umounts in this file, on line 82 (for /) on line 93 (for /home) and line 106 (for /usr/local), the other two probably have the same problem, which is why people above were seeing this when trying to preserve /home too.

Attached is a patch which inserts a 1s delay before each of the three umounts. If that doesn't work for you, try increasing the delay.

A better solution would be to not use subshell commands like () in this script.

The attachment "Add 1s delay before umount so subshell processes on mounted partition can exit" of this bug report has been identified as being a patch. The ubuntu-reviewers team has been subscribed to the bug report so that they can review the patch. In the event that this is in fact not a patch you can resolve this situation by removing the tag 'patch' from the bug report and editing the attachment so that it is not flagged as a patch. Additionally, if you are member of the ubuntu-reviewers team please also unsubscribe the team from this bug report.

[This is an automated message performed by a Launchpad user owned by Brian Murray. Please contact him regarding any issues with the action taken in this bug report.]

The patch of comment #62 solved the bug for me. Thank you mcelrath.
However hunk 1 of the original patch was rejected due to a missing newline.
I attach an updated patch, with 2 sec. sleep delays, which worked vey well for me.

Something doesn't make sense here. Aside from the fact that 'sleep' is generally a poor solution (how do you know how long to wait?), subshells on their own don't add any asynchrony: the only way that commands in a subshell could fail to exit before subsequent commands are run would be if the subshell in question were run in the background. That isn't the case here.

I think more investigation is needed. I'm confident that there's a better answer than inserting sleeps, but we won't know what until we know the true cause of this bug.

One possibility is that something entirely outside of the installer, such as (but not necessarily!) a udev rule, is racing with the installer. We'd need to know what.

This bug makes it impossible for many (most?) testers to assess the beta at all.

Another beta should be released after this bug is fixed, in order for proper testing to be done.

That's unfortunate I know, but it's the right thing.

Re: #65, #66, I agree it's odd. FYI I also put lsof and fuser into that script to see what was holding the mount point, and got empty results. So the time it takes to run fuser/lsof is enough...and I have no evidence of a udev rule or anything else causing it.

Without lsof/fuser telling us what is holding that mount point, I don't see how to find out what it is...unless someone wants to go through the udev rules by hand...

Steve: At this point we're just doing daily builds and full testing rounds on those. There won't be another beta. (And of course there is the question of whether we can track this down at all in time ...)

Stéphane Graber made a suggestion to use 'umount -l', which should work around this for now.

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/946663

This bug was fixed in the package partman-target - 77ubuntu2

---------------
partman-target (77ubuntu2) precise; urgency=low

  * Exit 0 at the end of finish.d/clear_partitions, so that partman-commit
    will carry on even if the final rmdir fails (which is a bug in itself,
    but shouldn't blow away the whole install; LP: #985526).
  * Use 'umount -l' in finish.d/clear_partitions, to work around an as yet
    unidentified race that keeps the filesystem busy (LP: #946663).
 -- Colin Watson <email address hidden> Thu, 19 Apr 2012 11:54:45 +0100

Still not fixed

Fixed for me
---
Ubuntu Bug Squad volunteer triager
http://wiki.ubuntu.com/BugSquad

Unfortunately, we were not able to resolve this issue in time for the release. Even when simulating some of the disk layouts attached, I don't see the problem.

However, to help us track down the cause it would be extremely useful if someone would list the *exact* steps they take to see this problem: every button click and action performed.

If you are seeing this issue when installing via a VM, a screencast would be the easiest way to generate this information (xvidcap is great for this and simple to use - just drag the red rectangle over your VM window and hit 'record': https://wiki.ubuntu.com/ScreenCasts/XVidCap).

This issue occurred for me as well. I'm installing Ubuntu 12.04 on a Samsung 7 Series laptop. I added a 100GB partition for / with a 500GB partition for /home. But I also tried to add a third FAT32 partition with the remaining 150GB. There was an error creating the FAT32 partition and then the "Removing conflicting..." message appeared and the installer moved at a crawl while showing messages about "nouveau".

I worked around it by killing the installer process (ubuiquity) and then starting over and omitting the FAT32 partition. After that, the installer finished properly.

This problem continues amazingly in Lbuntu 16.10, 16.04.2 and Ubuntu 16.10
I say "amazingly" advisedly, as this bug has been around since at least Ubuntu 12.04 in 2012/04/11.
Consider renaming it Ubuntu based installers are cought in a loop at "Removing conflicting operating system files..."

The /home partition is the primary culprit, especially if doing an install where you want to keep the data in it, mainly to save the time needed for a restore, or possibly files missed in the backup.

The details pane only shows about six lines of messages at a time and I was unable to copy and paste it to a file during the "Looping" install. I have about 54 photographs of selected parts of the data scrolling through the details pane. I can e-mail cropped copies or attempt to OCR them and provide an rtf file if anyone is working on this.

There are two workarounds posted.

http://askubuntu.com/questions/613660/stuck-on-removing-conflicting-operating-system-files-while-upgrading-from-14-0

and

http://alexsleat.co.uk/2012/04/11/ubuntu-12-04-removing-conflicting-operating-system-files/comment-page-1/

both referring to this "common bug"

by in the installer version of Gparted setting /home to "Do not use this partition" My Lubuntu install finished