Ubuntu
click-reviewers-tools package

snaps have different checksums on repeated mksquashfs runs

Bug #1576763 reported by Jamie Strandboge on 2016-04-29

This bug affects 1 person

	Status	Importance	Assigned to
Snapcraft	Fix Released	High	Tyler Hicks
snapd	Fix Released	Medium	Tyler Hicks
click-reviewers-tools (Ubuntu)	Fix Released	High	Jamie Strandboge

Bug Description

Reported by josepht:

root@localhost:~# for i in 1 2 3 4 5; do unsquashfs -f -d nmap_unpack_$i nmap_7.12SVN-0.4_armhf.snap; mksquashfs nmap_unpack_$i nmap_$i.snap -fstime 1461597924 -noappend -comp xz -all-root -no-xattrs; done
Parallel unsquashfs: Using 1 processor
725 inodes (845 blocks) to write

[===========================================================================================================================================================================================================================================================/] 845/845 100%

created 725 files
created 41 directories
created 0 symlinks
created 0 devices
created 0 fifos
Parallel mksquashfs: Using 1 processor
Creating 4.0 filesystem on nmap_1.snap, block size 131072.
Setting fs time: Mon Apr 25 15:25:24 2016

created 725 files
created 41 directories
created 0 symlinks
created 0 devices
created 0 fifos
Parallel mksquashfs: Using 1 processor
Creating 4.0 filesystem on nmap_2.snap, block size 131072.
Setting fs time: Mon Apr 25 15:25:24 2016

created 725 files
created 41 directories
created 0 symlinks
created 0 devices
created 0 fifos
Parallel mksquashfs: Using 1 processor
Creating 4.0 filesystem on nmap_3.snap, block size 131072.
Setting fs time: Mon Apr 25 15:25:24 2016

created 725 files
created 41 directories
created 0 symlinks
created 0 devices
created 0 fifos
Parallel mksquashfs: Using 1 processor
Creating 4.0 filesystem on nmap_4.snap, block size 131072.
Setting fs time: Mon Apr 25 15:25:24 2016

created 725 files
created 41 directories
created 0 symlinks
created 0 devices
created 0 fifos
Parallel mksquashfs: Using 1 processor
Creating 4.0 filesystem on nmap_5.snap, block size 131072.
Setting fs time: Mon Apr 25 15:25:24 2016

Exportable Squashfs 4.0 filesystem, xz compressed, data block size 131072
        compressed data, compressed metadata, compressed fragments, no xattrs
        duplicates are removed
Filesystem size 5476.70 Kbytes (5.35 Mbytes)
        24.01% of uncompressed filesystem size (22806.52 Kbytes)
Inode table size 5532 bytes (5.40 Kbytes)
        22.00% of uncompressed inode table size (25147 bytes)
Directory table size 8206 bytes (8.01 Kbytes)
        42.65% of uncompressed directory table size (19241 bytes)
Number of duplicate files found 0
Number of inodes 766
Number of files 725
Number of fragments 56
Number of symbolic links 0
Number of device nodes 0
Number of fifo nodes 0
Number of socket nodes 0
Number of directories 41
Number of ids (unique uids + gids) 1
Number of uids 1
        root (0)
Number of gids 1
        root (0)
root@localhost:~# ls
nmap_1.snap nmap_2.snap nmap_3.snap nmap_4.snap nmap_5.snap nmap_7.12SVN-0.4_armhf.snap nmap_unpack_1 nmap_unpack_2 nmap_unpack_3 nmap_unpack_4 nmap_unpack_5
root@localhost:~# sha512sum *.snap
4c9eed209ced0788975dfdbe382fa517d0f9b41dd3afd092792de7f604c708e9dec2fb40215d7d1f8aecf6ef56e9ee1974480a8a39acc40fcc6a02452577fe58 nmap_1.snap
4c9eed209ced0788975dfdbe382fa517d0f9b41dd3afd092792de7f604c708e9dec2fb40215d7d1f8aecf6ef56e9ee1974480a8a39acc40fcc6a02452577fe58 nmap_2.snap
7419b843ce769fbfde31e5f3c37597a841ddb2427a89b54f3850386cef3b08e57e31f1460aef6500fb5cdfecbc676373ad4da48fb56d00906d7f909c132b5165 nmap_3.snap
7419b843ce769fbfde31e5f3c37597a841ddb2427a89b54f3850386cef3b08e57e31f1460aef6500fb5cdfecbc676373ad4da48fb56d00906d7f909c132b5165 nmap_4.snap
4c9eed209ced0788975dfdbe382fa517d0f9b41dd3afd092792de7f604c708e9dec2fb40215d7d1f8aecf6ef56e9ee1974480a8a39acc40fcc6a02452577fe58 nmap_5.snap
4c9eed209ced0788975dfdbe382fa517d0f9b41dd3afd092792de7f604c708e9dec2fb40215d7d1f8aecf6ef56e9ee1974480a8a39acc40fcc6a02452577fe58 nmap_7.12SVN-0.4_armhf.snap

See original description

Jamie Strandboge (jdstrand) on 2016-04-29

description:

updated

Revision history for this message

Joe Talbott (joetalbott) wrote on 2016-04-29:

root@localhost:~# dpkg -l | grep squashfs
ii squashfs-tools 1:4.3-3ubuntu2 armhf Tool to create and append to squashfs filesystems

Revision history for this message

Joe Talbott (joetalbott) wrote on 2016-04-29:

nmap_7.12SVN-0.4_armhf.snap Edit (5.4 MiB, application/octet-stream)

Here's the snap I used for the tests.

Jamie Strandboge (jdstrand) on 2016-05-20

Changed in click-reviewers-tools (Ubuntu):
status:	New → Confirmed

Revision history for this message

Tyler Hicks (tyhicks) wrote on 2017-11-30:

Hello - the bug title/description makes it sound like this bug is specific to armhf (armhf's squashfs-tools deb package and the armhf nmap snap) but I'd like to confirm that's what you meant to imply.

If I use amd64's squashfs-tools deb package and the armhf nmap snap, I can still reproduce this bug. Have you tested if that's the case when using the amd64 squashfs-tools and an amd64 nmap snap?

Revision history for this message

Tyler Hicks (tyhicks) wrote on 2017-11-30:

I'm having trouble with this one because I can't reproduce it in the same way that's described in the bug description. The bug description shows that two of the snap files share one sha512 and two other snap files share another sha512. What I'm seeing is that no snap files share the same sha512 (md5sum shown below for easier reading):

$ for i in 1 2 3 4 5; do sudo sh -c "unsquashfs -f -d nmap_unpack_$i nmap_7.12SVN-0.4_armhf.snap; mksquashfs nmap_unpack_$i nmap_$i.snap -fstime 1461597924 -noappend -comp xz -all-root -no-xattrs"; done
...
$ md5sum *.snap
50c3f6ca2c64a85d89be90086c6025a6 nmap_1.snap
b14a8b87581b73e6c68ae7f1e07d1282 nmap_2.snap
91da8708b2ee90e1398215671f5aa4d5 nmap_3.snap
823f9f332a1cdd898904d02886a28f19 nmap_4.snap
1c5ec183fed50dbba8c166f2467ad43a nmap_5.snap
5fccb0276e2cd329021591129855eb83 nmap_7.12SVN-0.4_armhf.snap

This test run was on a raspi2 device inside of an ubuntu:16.04 lxd container.

Revision history for this message

Tyler Hicks (tyhicks) wrote on 2017-12-06:

This bug has to do with the frag deflator thread in mksquashfs. I haven't fully determined the problem but it is likely due to parallel processing (NOTE: the '-processors 1' option does not solve the bug). There's an out-of-tree patch that disables the thread by removing its working queue:

https://github.com/intrigeri/squashfs-tools/commit/c2c4d15580adee66b68c29eb9d2ee660225449bc

I've verified that the patch results in consistent, reproducible squashfs images:

$ md5sum *.snap
0161f2c324cb2753429b8c51818132c5 nmap_1.snap
0161f2c324cb2753429b8c51818132c5 nmap_2.snap
0161f2c324cb2753429b8c51818132c5 nmap_3.snap
0161f2c324cb2753429b8c51818132c5 nmap_4.snap
0161f2c324cb2753429b8c51818132c5 nmap_5.snap
5fccb0276e2cd329021591129855eb83 nmap_7.12SVN-0.4_armhf.snap

Note that nmap_7.12SVN-0.4_armhf.snap is the snap that was originally attached to this bug which was created with a mksquashfs that had the frag deflator thread enabled, so its snap differs from the ones created with the patched mksquashfs.

Revision history for this message

Jamie Strandboge (jdstrand) wrote on 2017-12-06:

IIUC, this would mean that snaps would have to be created with a mksquashfs with a patched/removed frag deflator to make reproducible snaps. Is that correct?

Revision history for this message

Tyler Hicks (tyhicks) wrote on 2017-12-06:

@jdstrand yes, that's unfortunately correct

Revision history for this message

Tyler Hicks (tyhicks) wrote on 2017-12-06:

I've done some performance testing of the patch that removes the frag deflator. The numbers below are in "min-max(avg+/-dev)" formatting and they were gathered with 10 consecutive runs of the following command:

$ ./mksquashfs nmap_unpack nmap_$i.snap -noappend -comp xz -all-root -no-xattrs -quiet

A) 12 core Xeon E5-2609 v3 with 16GB of ram

Without the frag deflator removal patch:
real 0.910000-1.010000(0.965+/-0.032)
user 9.760000-10.060000(9.937+/-0.076)
sys 0.150000-0.370000(0.235+/-0.073)

With the frag deflator removal patch:
real 2.930000-3.000000(2.96+/-0.024)
user 9.460000-9.590000(9.53+/-0.045)
sys 0.100000-0.140000(0.116+/-0.012)

B) Raspberry Pi 2 (4 core, 1GB of ram)

Without the frag deflator removal patch:
real 14.960000-15.320000(15.116+/-0.096)
user 55.100000-55.730000(55.365+/-0.16)
sys 3.530000-4.170000(3.764+/-0.18)

With the frag deflator removal patch:
real 20.980000-21.360000(21.107+/-0.13)
user 51.510000-52.100000(51.694+/-0.19)
sys 1.410000-1.770000(1.572+/-0.086)

There's a considerable increase in mksquashfs execution time when the frag deflator threads are removed. I would assume that this nmap snap is relatively small compared to other snaps so it might be a factor to keep in mind.

Revision history for this message

Tyler Hicks (tyhicks) wrote on 2017-12-07:

Disabling fragments, by passing the -no-fragments option to mksquashfs, retains performance while providing reproducibility:

A) 12 core Xeon E5-2609 v3 with 16GB of ram

Without the frag deflator removal patch and with the -no-fragments option:
real 0.900000-0.970000(0.928+/-0.023)
user 9.470000-9.620000(9.531+/-0.046)
sys 0.090000-0.180000(0.12+/-0.022)

B) Raspberry Pi 2 (4 core, 1GB of ram)

Without the frag deflator removal patch and with the -no-fragments option:

real 14.480000-14.610000(14.547+/-0.033)
user 53.250000-53.660000(53.467+/-0.13)
sys 2.910000-3.240000(3.056+/-0.094)

We actually see a small performance improvement. The downside here is that the resulting squashfs image isn't as small. This particular nmap snap increases from 5.4M to 5.8M in size. However, I suspect that the small increase in size is negated entirely by mksquashfs producing consistent snaps that can be accurately binary diffed for delta upgrades. More investigation is needed to make that claim but that's my current hunch.

Tyler Hicks (tyhicks) on 2017-12-13

summary:	- pi2 images generate different checksums on repeated runs + snaps have different checksums on repeated mksquashfs runs
Changed in snapcraft:
status:	New → In Progress
importance:	Undecided → High
Changed in click-reviewers-tools (Ubuntu):
importance:	Undecided → High
Changed in snapcraft:
assignee:	nobody → Tyler Hicks (tyhicks)

Tyler Hicks (tyhicks) on 2017-12-13

Changed in snapd:
status:	New → In Progress
importance:	Undecided → High
assignee:	nobody → Tyler Hicks (tyhicks)
importance:	High → Medium

Revision history for this message

Tyler Hicks (tyhicks) wrote on 2017-12-13:

#10

The snapcraft fix is committed: https://github.com/snapcore/snapcraft/commit/9a131cb5b649f0450f6c67cde575fd2286bd2b31

Changed in snapcraft:
status:	In Progress → Fix Committed

Revision history for this message

Michael Vogt (mvo) wrote on 2018-01-02:

#11

https://github.com/snapcore/snapd/pull/4396 is merged

Changed in snapd:
status:	In Progress → Fix Committed

Sergio Schvezov (sergiusens) on 2018-01-02

Changed in snapcraft:
status:	Fix Committed → Fix Released

Revision history for this message

Jamie Strandboge (jdstrand) wrote on 2018-02-06:

#12

The review-tools now use -no-fragments, so marking this bug as Fix Released for them.

Changed in snapd:
status:	Fix Committed → Fix Released
Changed in click-reviewers-tools (Ubuntu):
status:	Confirmed → Fix Released
assignee:	nobody → Jamie Strandboge (jdstrand)