when booting cloned live drive 3rd partition is created without file system

Sorry for the garbled list in 'details.txt', the non-standard characters are not rendered nicely by Firefox. Here is a better list from this computer:

lubuntu@lubuntu:~$ lsblk -lo name,fstype,label,size
NAME FSTYPE LABEL SIZE
loop0 squashfs 1.6G
sda iso9660 Lubuntu 20.10 amd64 55.9G
sda1 iso9660 Lubuntu 20.10 amd64 1.7G
sda2 vfat ESP 4.9M
sda3 54.2G <--- this one is created without file system
sdb 15G
sdb1 vfat KEEP_ME 14.9G
sr0 11M
zram0 962.2M
zram1 962.2M
zram2 962.2M
zram3 962.2M
nvme0n1 238.5G
nvme0n1p1 vfat SYSTEM_DRV 260M
nvme0n1p2 16M
nvme0n1p3 ntfs Windows-SSD 134.8G
nvme0n1p4 ntfs WINRE_DRV 1000M
nvme0n1p5 ext4 102.4G
lubuntu@lubuntu:~$

Not exactly reproducible here. If I download the lubuntu desktop ISO, extend it with 'dd if=/dev/zero of=lubuntu-groovy-amd64.iso count=0 bs=1 seek=$((4*1024*1024*1024))', attach it to a VM as a virtio disk, and boot into it, I get a fully configured 'writable' partition, which is mounted at /var/crash and /var/log - NOT an unconfigured partition. So it seems the casper configuration was only partially completed in your case.

Can you reproduce this with a fresh flash of the ISO to a disk? If so, can you attach /var/log/casper.log to this report?

I have seen several cases, where bugs appear in situations typical for end users, who want to install Ubuntu and Ubuntu family flavours in their computers.

If I understand correctly, you have not seen these bugs in your internal tests. Instead of testing only in virtual machines, I think at least some of you developers should have separate test computers, where you can test things directly, for example booting from USB drives. Otherwise the debug process will be very slow because of the delay in the communication between you (developers) and us (testers).

-o-

I did some tests with yesterday's Lubuntu Groovy iso file, the same as in the previous test.

USB drives

- a 60 GB SSD, OCZ-AGILITY ITY3 in a USB to SATA adapter
- a 32 GB pendrive, Sandisk Extreme
- a 16 GB pendrive, Sandisk Extreme

Process: Cloning

Test computer: http://www.toshiba.se/laptops/satellite-pro/c850/satellite-pro-c850-19w/

(This time *not* the Lenovo V130)

Tested with the USB drives

- in used state (with various data, typically leftovers after previous tests with live or persistent live systems).

- when completely overwritten with zeros (matching your test case)

-o-

I can draw the following conclusions:

- I reproduce the bug, that a third partition is created, but without a file system, when the USB drives are 'in used state'

- I reproduce your result, that a third partition is created with an ext4 file system and the label 'writable', when the drives were completely overwritten with zeros.

Please notice, that I have not had this issue before you started to modify the boot system. I have created many live drives by cloning from Ubuntu, Lubuntu ... 20.04.x and Groovy until the beginning of July, and it was *never* necessary to overwrite the whole drive with zeros (in order to get a third partition is created with an ext4 file system and the label 'writable').

I could reproduce the bug with today's daily Xubuntu Groovy iso file (with the leftovers after testing yesterday's daily Lubuntu Groovy iso fil in the 60 GB SSD, OCZ-AGILITY ITY3 drive).

This bug is still alive when tested with the current daily Ubuntu Groovy iso file cloned to a USB drive and tested in a Dell Precision M4800.

This iso file is improved in many other ways, but this bug remains to be fixed. Maybe there is some kind of race condition, that affects drives, that were not wiped before cloning.

It could be a race, this is the code that creates the partition and filesystem:

    echo "start=$start" | sfdisk --no-reread -q $DEVICE -a || return
    for d in ${DEVICE}$newpartno ${DEVICE}p$newpartno ${DEVICE}-part$newpartno; do
        if [ -e $d ]; then
            mkfs.ext4 -q -L "$(root_persistence_label)" -F $d
            break
        fi
    done
    udevadm trigger
    udevadm settle

But my understanding is that sfdisk (with the options we use here) doesn't return until it has told the kernel about the new partitions.

I also doubt that the changes to image mastering made a difference here and suspect it was some other change that coincidentally happened around the same time but who knows. (If the partition wasn't being created, I could quite easily believe the mastering changes would make a difference). I'll see if I can reproduce on my laptop tomorrow.

Something has happened, that I discovered today: Lubuntu works.

I tested the current Groovy iso files of Ubuntu Desktop, Lubuntu and Xubuntu.

- with the USB drives in used state (with various data, typically leftovers after previous tests with live or persistent live systems)

- both in UEFI mode and BIOS mode

- Computer: Dell Precision M4800

- USB drive: Kingston DT Workspace, 32 GB

The results were the same with Ubuntu and Xubuntu: no file system was created in the third partition.

Results with Lubuntu:

- The current version was still dated Sept 20 (while the other two iso files was dated today, Sept 21).

- I repeated the order of cloning four times to get a different 'history', and could repeat success: an ext4 file system with the label 'writable' was created in the third partition.

There is a significant difference between Lubuntu and the other flavours of Ubuntu: Lubuntu uses the installer Calamares, and there is no boot option 'maybe-ubiquity'. I am not saying that it makes the difference concerning this bug. Probably there are other differences too.

Now I tested the current Lubuntu Groovy iso file dated Sept 21. I could repeat the results from yesterday: The Lubuntu iso file makes USB boot drives, that can create a file system in the third partition, while the Ubuntu Desktop and Xubuntu iso files (of the same date, Sept 21, fail to do so.

Using sudo mkfs.ntfs -f -L data /dev/sdx3 and sudo mkfs.vfat -n data /dev/sdx3
I can format the third partition.
It is then accessible in Disks but it is not accessible in Windows.
With 20.04 the third partition was accessible in Windows after such formatting.
I have not tested if It can be formatted ext4 an labeled "writable yet.

Used "sudo mkfs.ext4 -L writable /dev/sdx3"

And partition sdx3 became persistent after using F6 and persistent during boot.

Being able to make the spare space useful between Ubuntu and Windows for data is still a big deal for me.

Oops, I dont think F6 and persistent during boot works with 20.10. I think I meant pressing "e" and typing persistence, sorry.

The bug is still there in the current daily Lubuntu Groovy iso file (beta candidate). See the attached screenshot.

Please notice that the error is not temporary. It persists after reboot.

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1895329

This bug is still alive in the current Ubuntu Desktop Groovy daily iso file dated October 2.

The first test results from the current daily Ubuntu Groovy iso file

$ ls -l groovy-desktop-amd64.iso
-rw------- 1 sudodus sudodus 2936969216 okt 9 21:15 groovy-desktop-amd64.iso

$ md5sum groovy-desktop-amd64.iso
00cbcce8855c00dd2231077e3a01aebc groovy-desktop-amd64.iso

indicate that that squashing bug #1886148 did also squash bug #1895329, but now there is another 3rd partition, and we should look for a 4th partition.

sudodus@bionic64 /media/multimed-2/test/ubuntu $ sudo kpartx -av groovy-desktop-amd64.iso
add map loop0p1 (253:0): 0 5725588 linear 7:0 64
add map loop0p2 (253:1): 0 9952 linear 7:0 5725652
add map loop0p3 (253:2): 0 600 linear 7:0 5735604

sudodus@bionic64 /media/multimed-2/test/ubuntu $ lsblk -f /dev/loop0
NAME FSTYPE LABEL UUID MOUNTPOINT
loop0 iso9660 Ubuntu 20.10 amd64 2020-10-09-21-15-31-00
├─loop0p1 iso9660 Ubuntu 20.10 amd64 2020-10-09-21-15-31-00
├─loop0p2 vfat ESP 6508-0C9E
└─loop0p3

sudodus@bionic64 /media/multimed-2/test/ubuntu $ sudo kpartx -d groovy-desktop-amd64.iso
loop deleted : /dev/loop0

After booting, there is a 'writable' partition.

sudodus@bionic64 /media/multimed-2/test/ubuntu $ lsblk -f /dev/sdc
NAME FSTYPE LABEL UUID MOUNTPOINT
sdc iso9660 Ubuntu 20.10 amd64 2020-10-09-21-15-31-00
├─sdc1 iso9660 Ubuntu 20.10 amd64 2020-10-09-21-15-31-00
├─sdc2 vfat ESP 6508-0C9E
├─sdc3 iso9660 Ubuntu 20.10 amd64 2020-10-09-21-15-31-00
└─sdc4 ext4 writable 8b73f753-eb94-40ec-8995-f57c64933d3f

I zsynced the current daily Xubuntu Groovy iso date (dated 2020-10-10), and when cloned to a USB pendrive and booted, a 'writable' partition is created as it should.

This system was cloned directly after testing a larger Ubuntu Groovy system, and with earlier iso files, it was likely to fail to create an ext file system in this partition (unless the drive was overwritten with zeros). So this is a second (and stronger) indication, that squashing bug #1886148 did also squash bug #1895329.

I tested also with the current daily Lubuntu Groovy iso file (dated 2020-10-10) and when cloned to a USB pendrive and booted, a 'writable' partition is created as it should. This is a third indication that this bug is squashed.