PNG output is sometimes blank

[Description]

When I use the OpenSCAD CLI to create a PNG view of a design, sometimes the PNG ends up blank. This appears to be non-deterministic. Sometimes the output PNG appears correct; sometimes it appears to be a valid PNG but is blank.

I'm running Ubuntu Artful with openscad 2015.03-2+dfsg-2ubuntu1 and all my packages are up to date (2017-12-24).

I'm mostly using defaults, so GNOME on Wayland. This may matter since openscad connects to X11 even when in batch mode the CLI. I haven't yet been able to get Xvfb working for my problem case (Test Case A below), but it shouldn't matter for this bug. In any case, it matters that this won't work on a default Ubuntu system.

[Impact]

OpenSCAD can't reliably be used in scripting or other automation.

[Test Case A]

apt install openscad
echo "cube();" > test.scad
for i in `seq 1 1000`; do echo $i; openscad -o test.png test.scad; size=`stat -c%s test.png`; [ $size -lt 3635 ] && break; done

Expected results: runs to completion (1000 attempts).

Actual results: terminates early; test.png is smaller (1947 bytes) and appears to be a valid PNG file but with only background pixels.

Note: on my Artful system, the correct output PNG for the above test is 3635 bytes, and I rely on that in my reproduction steps above. It may be the case that others' correct output PNG sizes are different depending on unrelated environmental factors. In this case, please tweak the figure above to reproduce the problem. Fundamentally the issue is that the PNG sometimes turns out "blank" in a not obviously deterministic way. Repeating the openscad call above multiple times should demonstrate the problem if the output PNGs aren't always identical.

[Test Case B]

apt install openscad-testing
openscad-testrun --virtual

Expected results: all pass

Actual results: "Errors while running CTest"

Note: even with this bug fixed, not all tests in this test suite pass. Even more tests fail in sid currently. I have filed https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=885589 to track this separately.

Without this bug fixed in Artful, a large number of tests fail, but this appears to be non-deterministic. For example, on one attempt 344 failed, and on another run 321 failed.

With this bug fixed in Artful, only 8 tests fail:

The following tests FAILED:
  17 - echotest_rands (Failed)
 297 - cgalpngtest_tessellation-text-test (Failed)
 470 - opencsgtest_issue1165 (Failed)
 473 - opencsgtest_issue1215 (Failed)
 605 - csgpngtest_tessellation-text-test (Failed)
 777 - throwntogethertest_issue1215 (Failed)
 915 - openscad-camtrans-viewall_camera-tests (Failed)
 917 - openscad-camortho-viewall_camera-tests (Failed)

The results are the same regardless of whether the test runs against my GNOME Wayland session or if I use --virtual so it uses Xvfb.

I have incorporated this Test Case B into a dep8 test.

[Stable Fix]

A no change rebuild fixes the problem. Test Case B becomes deterministic and the number of failed tests reduces to 8.

I've also cherry-picked the dep8 test added in the development release to help inform SRU verification. This is expected to still fail when this bug is fixed, but with only 8 test failures instead of 300+.

[Development Fix]

The package has already been rebuilt in Bionic. The bug (Test Case A) does not reproduce on Bionic, so no fix is needed. The test suite (Test Case B) fails, but this also happens on a rebuild and appears to be unrelated. The number of individual test failures in Test Case B is larger than what I see with the bug fixed in Artful, but appears to be deterministic and thus I believe unrelated:

The following tests FAILED:
  17 - echotest_rands (Failed)
 216 - cgalpngtest_text-font-alignment-tests (Failed)
 217 - cgalpngtest_text-font-composition (Failed)
 218 - cgalpngtest_text-font-direction-tests (Failed)
 219 - cgalpngtest_text-font-simple-tests (Failed)
 284 - cgalpngtest_polyhedron-nonplanar-tests (Failed)
 297 - cgalpngtest_tessellation-text-test (Failed)
 470 - opencsgtest_issue1165 (Failed)
 473 - opencsgtest_issue1215 (Failed)
 524 - csgpngtest_text-font-alignment-tests (Failed)
 525 - csgpngtest_text-font-composition (Failed)
 526 - csgpngtest_text-font-direction-tests (Failed)
 527 - csgpngtest_text-font-simple-tests (Failed)
 528 - csgpngtest_text-font-symbol (Failed)
 529 - csgpngtest_text-font-tests (Failed)
 592 - csgpngtest_polyhedron-nonplanar-tests (Failed)
 605 - csgpngtest_tessellation-text-test (Failed)
 777 - throwntogethertest_issue1215 (Failed)
 811 - monotonepngtest_polyhedron-nonplanar-tests (Failed)
 834 - cgalstlcgalpngtest_polyhedron-nonplanar-tests (Failed)
 870 - dxfpngtest_text-font-alignment-tests (Failed)
 871 - dxfpngtest_text-font-composition (Failed)
 872 - dxfpngtest_text-font-direction-tests (Failed)
 873 - dxfpngtest_text-font-simple-tests (Failed)
 915 - openscad-camtrans-viewall_camera-tests (Failed)
 917 - openscad-camortho-viewall_camera-tests (Failed)

I've submitted a dep8 test to run Test Case B to Debian in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=885429 and cherry picked this into Ubuntu. The test is still expected to fail due to https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=885589 but since the results are still helpful to other bugs, having the tests run with history available should be useful. It will need to be force-badtested into the release pocket.

[Regression Potential]

Unfortunately there isn't any specific area on which we can focus testing. It isn't clear what caused the regression, since tests did pass at the time the package was built.

In mitigation, there is a comprehensive test suite whose results are objectively better in the rebuild. So it appears to be reliable in helping us ensure that the rebuild doesn't create a new regression for those areas tested.

For SRU verification, we should at a minimum run both reproduction steps above, which should ensure that the SRU build is demonstrably better than the current situation, and check that Test Case B is deterministic again.

I think a smoke test by running the GUI by hand would also be wise.