autopkgtest fail due to broken PNG images in openscad-testing-data

openscad/2021.01-1 never worked.
Bad on First Februaray and up to now.
    1072 - pdfexporttest_centered (Failed)
All good tests are wit the old version 2019.05-5

This package is entangled with libzip which blocks quite a bunch of others.
So unblocking this would help proposed more than just for this package.

Works fine in Debian Ci:
https://ci.debian.net/data/autopkgtest/testing/amd64/o/openscad/10926296/log.gz

Checked a local VM based repro as-is and all-proposed.
Both failed.

We have two files that are compared:
-rw-rw-r-- 1 ubuntu ubuntu 29568 Mar 10 12:20 /tmp/autopkgtest.NS3C7t/autopkgtest_tmp/testrun/build/output/pdfexporttest/centered-actual.png
-rw-rw-r-- 1 ubuntu ubuntu 29502 Jan 31 21:17 /tmp/autopkgtest.NS3C7t/build.b25/src/tests/./regression/pdfexporttest/centered-expected.png

Unfortunately the test fail is on comparison
Image comparison cmdline: /usr/bin/convert regression/pdfexporttest/centered-expected.png /tmp/autopkgtest.NS3C7t/autopkgtest_tmp/testrun/build/output/pdfexporttest/centered-actual.png -alpha On -compose difference -composite -threshold 10% -morphology Erode Square:2 -format %[fx:w*h*mean] info:
 actual image: /tmp/autopkgtest.NS3C7t/autopkgtest_tmp/testrun/build/output/pdfexporttest/centered-actual.png

 expected image: regression/pdfexporttest/centered-expected.png

Image comparison return: 0 output: convert: iCCP: profile 'default_gray.icc': 'GRAY': Gray color space not permitted on RGB PNG `regression/pdfexporttest/centered-expected.png' @ warning/png.c/MagickPNGWarningHandler/1667.
8
Traceback (most recent call last):
  File "/tmp/autopkgtest.NS3C7t/autopkgtest_tmp/testrun/build/./test_cmdline_tool.py", line 387, in <module>
    if not verification or not compare_with_expected(resultfile): exit(1)
  File "/tmp/autopkgtest.NS3C7t/autopkgtest_tmp/testrun/build/./test_cmdline_tool.py", line 209, in compare_with_expected
    if "compare_" + options.suffix in globals(): return globals()["compare_" + options.suffix](resultfilename)
  File "/tmp/autopkgtest.NS3C7t/autopkgtest_tmp/testrun/build/./test_cmdline_tool.py", line 194, in compare_png
    pixelerr = int(float(output.strip()))
ValueError: could not convert string to float: "convert: iCCP: profile 'default_gray.icc': 'GRAY': Gray color space not permitted on RGB PNG `regression/pdfexporttest/centered-expected.png' @ warning/png.c/MagickPNGWarningHandler/1667.\n8"
<end of output>
Test time = 5.00 sec
----------------------------------------------------------
Test Failed.

But executing said comparison myself in the very same test env works fine:

ubuntu@autopkgtest:/tmp/autopkgtest.NS3C7t/build.b25/src/tests$ /usr/bin/convert regression/pdfexporttest/centered-expected.png /tmp/autopkgtest.NS3C7t/autopkgtest_tmp/testrun/build/output/pdfexporttest/centered-actual.png -alpha On -compose difference -composite -threshold 10% -morphology Erode Square:2 -format %[fx:w*h*mean] info
ubuntu@autopkgtest:/tmp/autopkgtest.NS3C7t/build.b25/src/tests$ echo $?
0

Involved versions of imagemagick
Debian: 8:6.9.11.60+dfsg-1
Ubuntu: 8:6.9.10.23+dfsg-2.1ubuntu16

Running the full testsuite recreates the issue.
The isolated way to trigger this is via:
# enter the test dir that was created
cd /tmp/autopkgtest.NS3C7t/autopkgtest_tmp/testrun-manual/build
# run the full set
"/usr/bin/python3" "./test_cmdline_tool.py" "--comparator=" "-c" "/usr/bin/convert" "-s" "png" "-e" "pdfexporttest" "-k" "Square:2" "-t" "pdfexporttest" "-f" "centered" "/usr/bin/python3" "./export_pngtest.py" "./../testdata/scad/pdf/centered.scad" "--openscad=./../openscad" "--format=PDF"

Detail per stage:
Works: /usr/bin/python3 ./export_pngtest.py ./../testdata/scad/pdf/centered.scad --openscad=./../openscad --format=PDF /tmp/autopkgtest.NS3C7t/autopkgtest_tmp/testrun-manual/build/output/pdfexporttest/centered-actual.png
Fails: /usr/bin/convert regression/pdfexporttest/centered-expected.png /tmp/autopkgtest.NS3C7t/autopkgtest_tmp/testrun-manual/build/output/pdfexporttest/centered-actual.png -alpha On -compose difference -composite -threshold 10% -morphology Erode Square:2 -format %[fx:w*h*mean] info

We can copy out the files and run it on those dirctly:
ubuntu@autopkgtest:~$ /usr/bin/convert centered-expected.png centered-actual.png -alpha On -compose difference -composite -threshold 10% -morphology Erode Square:2 -format %[fx:w*h*mean] infoconvert: iCCP: profile 'default_gray.icc': 'GRAY': Gray color space not permitted on RGB PNG `centered-expected.png' @ warning/png.c/MagickPNGWarningHandler/1667.

Also complains in Focal for: 8:6.9.10.23+dfsg-2.1ubuntu11.2
Copying these files into a Debian with 8:6.9.11.58+dfsg-1 is broken as well
Upgrading imagemagick-6.q16 in Debian to 8:6.9.11.60+dfsg-1 breaks as well.

Hmm, so it isn't just the imagemagick version.
Maybe the result of the PDF conversion really is different.

Getting those files from Debian ... and comparing them.

The test run there works fine.
The created files are different, but not only "actual" also the expected file is.

I was reading detailed attributes from these png's via
 identify -verbose ubuntu.centered-actual.png > ubuntu.centered-actual.png.info
 identify -verbose debian.centered-actual.png > debian.centered-actual.png.info
 identify -verbose ubuntu.centered-expected.png > ubuntu.centered-expected.png.info
 identify -verbose debian.centered-expected.png > debian.centered-expected.png.info
And it turns out what is broken is the ubuntu.centered-expected.png

Also it has a vastly different file size:
$ ll ubuntu.centered-actual.png ubuntu.centered-expected.png debian.centered-actual.png debian.centered-expected.png
-rw-r--r-- 1 paelzer paelzer 29568 Mär 10 14:41 debian.centered-actual.png
-rw-r--r-- 1 paelzer paelzer 29502 Mär 10 14:41 debian.centered-expected.png
-rw-rw-r-- 1 paelzer paelzer 29568 Mär 10 14:26 ubuntu.centered-actual.png
-rw-r--r-- 1 paelzer paelzer 18446 Mär 10 14:26 ubuntu.centered-expected.png

Even the check of debian.expected vs ubuntu-actial is valid
$ /usr/bin/convert debian.centered-expected.png ubuntu.centered-actual.png -alpha On -compose difference -composite -threshold 10% -morphology Erode Square:2 -format %[fx:w*h*mean] info

So why is "out" expected file broken and how is it created.

The expected file in Ubuntu has "Colorspace: sRGB" while all others
have "Colorspace: Gray". Also the properties are very different:
- icc:copyright: Copyright Artifex Software 2018
- icc:description: Artifex Software sGray ICC Profile
- png:iCCP: chunk was found
- png:IHDR.bit-depth-orig: 8
- png:IHDR.bit_depth: 8
- png:IHDR.color-type-orig: 0
- png:IHDR.color_type: 0 (Grayscale)
+ date:create: 2021-03-10T14:31:22+01:00
+ date:modify: 2021-03-10T14:26:32+01:00
+ png:IHDR.bit-depth-orig: 2
+ png:IHDR.bit_depth: 2
+ png:IHDR.color-type-orig: 3
+ png:IHDR.color_type: 3 (Indexed)

The file in the package source itself is equal.
f43804573fceeb8486c1b5c2c65f7205 /root/openscad-2021.01/tests/regression/pdfexporttest/centered-expected.png
f43804573fceeb8486c1b5c2c65f7205 /tmp/openscad-2021.01/tests/regression/pdfexporttest/centered-expected.png

But for the test oen is in
(That still is the same)
f43804573fceeb8486c1b5c2c65f7205 /tmp/autopkgtest.NS3C7t/build.b25/src/tests/regression/pdfexporttest/centered-expected.png
The bad one is
dc6148ec8d80e477e368ab82b01ed873 /usr/share/openscad/regression/pdfexporttest/centered-expected.png

That was created at build time of 2021.01-1build1
The older version of openscad-testing-data didn't contain this file.

Both 2021.01-1 builds have that bad file
https://launchpadlibrarian.net/520637662/buildlog_ubuntu-hirsute-amd64.openscad_2021.01-1_BUILDING.txt.gz
https://launchpadlibrarian.net/527107257/buildlog_ubuntu-hirsute-amd64.openscad_2021.01-1build1_BUILDING.txt.gz

This is installed by:
openscad-2021.01/debian/openscad-testing-data.install
  testdata /usr/share/openscad/
  tests/regression /usr/share/openscad/

But then in the source tests/regression/pdfexporttest/centered-expected.png was ok.
When did it get corrupted?

A local sbuild rebuild of openscad recreated the issue.
But interestingly only in the file it installed.
See the size difference:

find -name centered-expected.png -exec ls -laF {} \;
-rw-r--r-- 1 paelzer paelzer 18446 Jan 31 20:17 ./openscad-2021.01/debian/openscad-testing-data/usr/share/openscad/regression/pdfexporttest/centered-expected.png
-rw-r--r-- 1 paelzer paelzer 29502 Jan 31 20:17 ./openscad-2021.01/tests/regression/pdfexporttest/centered-expected.png

Hmm:
pkgstripfiles: Running PNG optimization (using 3 cpus) for package openscad-testing-data ...

Maybe that killed my file content?

$ rm -rf debian/openscad-mytest
$ mkdir -p debian/openscad-mytest/tests/regression/pdfexporttest/
$ f="debian/openscad-mytest/tests/regression/pdfexporttest/centered-expected.png"
$ cp -a ./tests/regression/pdfexporttest/centered-expected.png $f
$ cp -a ./tests/regression/pdfexporttest/centered-expected.png $f.orig
$ optipng -o4 -preserve "$f"
** Processing: debian/openscad-mytest/tests/regression/pdfexporttest/centered-expected.png
2479x3508 pixels, 8 bits/pixel, grayscale
Reducing image to 2 bits/pixel, 4 colors in palette
Input IDAT size = 27039 bytes
Input file size = 29502 bytes

Trying:
  zc = 9 zm = 8 zs = 0 f = 0 IDAT size = 16026
  zc = 9 zm = 8 zs = 1 f = 0 IDAT size = 15995

Selecting parameters:
  zc = 9 zm = 8 zs = 1 f = 0 IDAT size = 15995

Output IDAT size = 15995 bytes (11044 bytes decrease)
Output file size = 18446 bytes (11056 bytes = 37.48% decrease)
$ advpng -q -z4 "$f"

I double checked, optipng would break it in Debian as well.
Why it didn't in the past is a riddle to me, are our policies differently?

Turns out this isn't as much of a riddle as I thought:
[15:46] <cpaelzer> so I wonder why this runs for us but not in the Debian build, but since this is an area I never touched I wanted to ask for some experience or guidance where to look at
[15:48] <cjwatson> cpaelzer: pkgstripfiles existing at all is different between Debian/Ubuntu

Now that this is clear the fix is easy via $NO_PNG_PKG_MANGLE