i386 autopkgtests are unstable

Olivier, would be good if you could look at this if you have some time ...

For some of the failures (not all of them), the log ends with:

autopkgtest [22:27:34]: ERROR: erroneous package: Test dependencies are unsatisfiable. A common reason is that your testbed is out of date with respect to the archive, and you need to use a current testbed or run apt-get update or use -U.
blame: libreoffice
badpkg: Test dependencies are unsatisfiable. A common reason is that your testbed is out of date with respect to the archive, and you need to use a current testbed or run apt-get update or use -U.

Other runs complete but there are actual failures in junit-subsequentcheck, e.g.:

There was 1 failure:
1) complex.dbaccess.RowSet
java.lang.AssertionError: expected:<0> but was:<139>
 at org.junit.Assert.fail(Assert.java:88)
 at org.junit.Assert.failNotEquals(Assert.java:834)
 at org.junit.Assert.assertEquals(Assert.java:645)
 at org.junit.Assert.assertEquals(Assert.java:631)
 at org.openoffice.test.OfficeConnection.tearDown(OfficeConnection.java:150)
 at complex.dbaccess.TestCase.tearDownConnection(TestCase.java:227)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
 at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
 at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:52)
 at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
 at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
 at org.junit.runners.Suite.runChild(Suite.java:128)
 at org.junit.runners.Suite.runChild(Suite.java:27)
 at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
 at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
 at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
 at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
 at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
 at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
 at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
 at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
 at org.junit.runner.JUnitCore.runMain(JUnitCore.java:77)
 at org.junit.runner.JUnitCore.main(JUnitCore.java:36)

Other common failures:

There were 2 failures:
1) test(org.openoffice.test.UnoApiTest)
com.sun.star.lang.DisposedException: java_remote_bridge com.sun.star.lib.uno.bridges.java_remote.java_remote_bridge@c8a2c0 is disposed
 at com.sun.star.lib.uno.bridges.java_remote.java_remote_bridge.checkDisposed(java_remote_bridge.java:688)
 at com.sun.star.lib.uno.bridges.java_remote.java_remote_bridge.sendRequest(java_remote_bridge.java:602)
 at com.sun.star.lib.uno.bridges.java_remote.ProxyFactory$Handler.request(ProxyFactory.java:145)
 at com.sun.star.lib.uno.bridges.java_remote.ProxyFactory$Handler.invoke(ProxyFactory.java:129)
 at com.sun.proxy.$Proxy10.loadComponentFromURL(Unknown Source)
 at util.SOfficeFactory.openDoc(SOfficeFactory.java:435)
 at util.SOfficeFactory.createCalcDoc(SOfficeFactory.java:108)
 at mod._sc.ScHeaderFooterContentObj.initialize(ScHeaderFooterContentObj.java:62)
 at lib.TestCase.initializeTestCase(TestCase.java:67)
 at base.java_fat.executeTest(java_fat.java:137)
 at org.openoffice.Runner.run(Runner.java:175)
 at org.openoffice.test.UnoApiTest.test(UnoApiTest.java:41)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
 at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
 at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:52)
 at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
 at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
 at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
 at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
 at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
 at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
 at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
 at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
 at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
 at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
 at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
 at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
 at org.junit.runners.Suite.runChild(Suite.java:128)
 at org.junit.runners.Suite.runChild(Suite.java:27)
 at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
 at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
 at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
 at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
 at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
 at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
 at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
 at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
 at org.junit.runner.JUnitCore.runMain(JU...

The complex.dbaccess.RowSet failure happens because the soffice executable crashed (exit code 139 == SIGSEGV). Indeed:

It looks like /usr/lib/libreoffice/program/soffice.bin generated a core file at /tmp/autopkgtest.ONgbYC/build.EzT/libreoffice-5.3.3/workdir/JunitTest/dbaccess_complex/user/core
Backtraces:
"/tmp/autopkgtest.ONgbYC/build.EzT/libreoffice-5.3.3/workdir/JunitTest/dbaccess_complex/user/core" is not a core dump: File format not recognized
/tmp/tmp.70307LjmlG:1: Error in sourced command file:
The program has no registers now.

Unfortunately the core file is not attached to the autopkgtest artifacts, and so far I have been unable to reproduce the test failures locally in an x86 VM.

This was reported for artful which is EOL nowadays.
But the very same RowSet error is still an issue and it seems that it blocks all i386-xenial tests since mid 2017 [1] - that is the last 41/41 tests.
I think we can agree that that seems pretty consistently broken for now.

All of the last errors were blocked on the very same JunitTest_dbaccess_complex test
$ check-autopkgtest-stats.sh -c 40 -p libreoffice -r xenial -a i386 -P 'gb_JunitTest_DEBUGRUN'
Check last 40 test results for src:libreoffice on releases 'xenial' on architectures 'i386'
Fetch Data to /tmp/tmp.zmMkCSCraz
Of the 40 last tests, we had these subtest failing per release/arch:

xenial
  i386
     40 junit-subsequentcheck (100.00%) FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
Pattern found, sum/order found patterns
     40 make gb_JunitTest_DEBUGRUN=T JunitTest_dbaccess_comple

Other than stated before this is reproducible with i386 in VM:
$ sudo autopkgtest-buildvm-ubuntu-cloud -a i386 -r xenial -s 15G
# Note: I use autopkgtest from git
$ sudo ~/work/autopkgtest/autopkgtest/runner/autopkgtest --test-name=junit-subsequentcheck --no-built-binaries --apt-upgrade --shell-fail libreoffice_5.1.6~rc2-0ubuntu1~xenial6.dsc -- qemu --qemu-options='-cpu host' --ram-size=2048 --cpus 4 ~/work/autopkgtest-xenial-i386.img

When you log in you can isolate the failing test via:

$ cd /tmp/autopkgtest.*/build.*/src
$ make -rk OOO_TEST_SOFFICE=path:/usr/lib/libreoffice/program/soffice bridges_SELECTED_BRIDGE=foo JunitTest_dbaccess_complex

I'll attach some debug data I got that way for you to reconsider.

[1]: http://autopkgtest.ubuntu.com/packages/libr/libreoffice/xenial/i386

Isolated run into C/Java stack trace

$ make -rk OOO_TEST_SOFFICE=path:/usr/lib/libreoffice/program/soffice bridges_SELECTED_BRIDGE=foo JunitTest_dbaccess_complex
Automatic fetching of external tarballs is disabled.
make -j 4 -rs -f /tmp/autopkgtest.2t6Gxk/build.jSG/src/Makefile.gbuild JunitTest_dbaccess_complex
[JUT] dbaccess_complex
JUnit version 4.12
..testing the basics
.checking proper dynamic of the set
.checking PropertySetAccess via sequences
.....testing RowSet Events
testing events for afterLast
testing events for next
testing events for next
testing events for next
testing events for last
testing events for next
testing events for first
testing events for previous
testing events for next
testing events for next
testing events for next
testing events for moveToInsertRow
testing events for insertRow
testing events for updateRow
testing events for deleteRow
testing events for refreshRow
testing events for cancelRowUpdates
testing events for moveToBookmark
testing events for moveRelativeToBookmark
testing events for deleteRows
.testing testRowSet
testing Thread
starting thread 1 of 10
starting thread 2 of 10
starting thread 3 of 10
starting thread 4 of 10
starting thread 5 of 10
starting thread 6 of 10
starting thread 7 of 10
starting thread 8 of 10
starting thread 9 of 10
starting thread 10 of 10

It looks like /usr/lib/libreoffice/program/soffice.bin generated a core file at /tmp/autopkgtest.2t6Gxk/build.jSG/src/workdir/JunitTest/dbaccess_complex/user/core
Backtraces:
[New LWP 4059]
[New LWP 4070]
[New LWP 4101]
[New LWP 4061]
[New LWP 4063]
[New LWP 4071]
[New LWP 4106]
[New LWP 4105]
[New LWP 4104]
[New LWP 4072]
[New LWP 4103]
[New LWP 4097]
[New LWP 4095]
[New LWP 4073]
[New LWP 4102]
[New LWP 4075]
[New LWP 4094]
[New LWP 4093]
[New LWP 4096]
[New LWP 4098]
[New LWP 4099]
[New LWP 4100]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/lib/libreoffice/program/soffice.bin -env:UserInstallation=file:///tmp/auto'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0xa6ae3845 in ?? () from /usr/lib/jvm/java-8-openjdk-i386/jre/lib/i386/server/libjvm.so
[Current thread is 1 (Thread 0xafedc900 (LWP 4059))]

Thread 22 (Thread 0x759ffb40 (LWP 4100)):
#0 0xb76ddc31 in __kernel_vsyscall ()
No symbol table info available.
#1 0xb3ed6026 in do_futex_wait.constprop () from /lib/i386-linux-gnu/libpthread.so.0
No symbol table info available.
#2 0xb3ed6117 in __new_sem_wait_slow.constprop.1 () from /lib/i386-linux-gnu/libpthread.so.0
No symbol table info available.
#3 0xa6ae4867 in ?? () from /usr/lib/jvm/java-8-openjdk-i386/jre/lib/i386/server/libjvm.so
No symbol table info available.
#4 0xa6adc86d in ?? () from /usr/lib/jvm/java-8-openjdk-i386/jre/lib/i386/server/libjvm.so
No symbol table info available.
#5 0xa6c29feb in ?? () from /usr/lib/jvm/java-8-openjdk-i386/jre/lib/i386/server/libjvm.so
No symbol table info available.
#6 0xa6c2a4be in ?? () from /usr/lib/jvm/java-8-openjdk-i386/jre/lib/i386/server/libjvm.so
No symbol table info available.
#7 0xa6ae60e5 in ?? () from /usr/lib/jvm/java-8-openjdk-i386/jre/lib/i386/server/lib...

Here the core dump that the test generates.
Here [1] is a installed package list, but it is just Xenial as installed by the test as of today - nothing special and since the issue persists since almost two years I guess that isn't too important.

[1]: http://paste.ubuntu.com/p/SczQ3cfczv/

ApportVersion: 2.20.1-0ubuntu2.18
Architecture: i386
DistroRelease: Ubuntu 16.04
Package: libreoffice
PackageArchitecture: i386
ProcVersionSignature: User Name 4.4.0-143.169-generic 4.4.170
Tags: xenial
Uname: Linux 4.4.0-143-generic i686
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm audio cdrom dialout dip floppy lxd netdev plugdev sudo video
_MarkForUpload: True

apport information

apport information

apport information

Hmm, I meant to attach apport-collect data just to have all in one place.
But it didn't add a lot :-)

BTW the dbg symbols have broken dependencies, I forked bug 1822538 for that.
Due to that debug symbols where a bit harder to install than usual.

I got a slightly more detailed stack trace for you now, but further down I'm lost in Openoffice Details I don't know - I'll leave it to you then.

--- stack trace ---
#0 0xa62e2845 in ?? () from /usr/lib/jvm/java-8-openjdk-i386/jre/lib/i386/server/libjvm.so
No symbol table info available.
#1 0xa62e5464 in ?? () from /usr/lib/jvm/java-8-openjdk-i386/jre/lib/i386/server/libjvm.so
No symbol table info available.
#2 0xa62f0290 in ?? () from /usr/lib/jvm/java-8-openjdk-i386/jre/lib/i386/server/libjvm.so
No symbol table info available.
#3 0xa60d2cef in ?? () from /usr/lib/jvm/java-8-openjdk-i386/jre/lib/i386/server/libjvm.so
No symbol table info available.
#4 0xa60d312c in ?? () from /usr/lib/jvm/java-8-openjdk-i386/jre/lib/i386/server/libjvm.so
No symbol table info available.
#5 0xb2a32ce7 in jvmaccess::VirtualMachine::attachThread(bool*) const () from /usr/lib/libreoffice/program/libjvmaccesslo.so
No symbol table info available.
#6 0xb2a32d36 in jvmaccess::VirtualMachine::AttachGuard::AttachGuard(rtl::Reference<jvmaccess::VirtualMachine> const&) () from /usr/lib/libreoffice/program/libjvmaccesslo.so
No symbol table info available.
#7 0xb2a32890 in jvmaccess::UnoVirtualMachine::~UnoVirtualMachine() () from /usr/lib/libreoffice/program/libjvmaccesslo.so
No symbol table info available.
#8 0xb2a3295a in jvmaccess::UnoVirtualMachine::~UnoVirtualMachine() () from /usr/lib/libreoffice/program/libjvmaccesslo.so
No symbol table info available.
#9 0xad711d50 in ?? () from /usr/lib/libreoffice/program/libjavavmlo.so
No symbol table info available.
#10 0xad711f0a in ?? () from /usr/lib/libreoffice/program/libjavavmlo.so
No symbol table info available.
#11 0xb2b001b1 in cppu::OWeakObject::release() () from /usr/lib/libreoffice/program/libuno_cppuhelpergcc3.so.3
No symbol table info available.
#12 0xb2a91ccb in cppu::WeakComponentImplHelperBase::release() () from /usr/lib/libreoffice/program/libuno_cppuhelpergcc3.so.3
No symbol table info available.
#13 0xad719e18 in ?? () from /usr/lib/libreoffice/program/libjavavmlo.so
No symbol table info available.
#14 0xb2acdded in ?? () from /usr/lib/libreoffice/program/libuno_cppuhelpergcc3.so.3
No symbol table info available.
#15 0xb2accac9 in ?? () from /usr/lib/libreoffice/program/libuno_cppuhelpergcc3.so.3
No symbol table info available.
#16 0xb2b001b1 in cppu::OWeakObject::release() () from /usr/lib/libreoffice/program/libuno_cppuhelpergcc3.so.3
No symbol table info available.
#17 0xb2acd508 in ?? () from /usr/lib/libreoffice/program/libuno_cppuhelpergcc3.so.3
No symbol table info available.
#18 0xb2a6959d in ?? () from /usr/lib/libreoffice/program/libuno_cppuhelpergcc3.so.3
No symbol table info available.
#19 0xb2b4f005 in uno_any_destruct () from /usr/lib/libreoffice/program/libuno_cppu.so.3
No symbol table info available.
#20 0xb2a6e4e4 in ?? () from /usr/lib/libreoffice/program/libuno_cppuhelpergcc3.so.3
No symbol table info available.
#21 0xb2a94226 in cppu::WeakComponentImplHelperBase::dispose() () from /usr/lib/libreoffice/program/libuno_cppuhelpergcc3.s...

I knew all of this seemed somewhat known to me and look at that [1].
We merged such a change to ignore the error (I was about to ask about that) in the past.
It should actually already be an ignored error, but [2] currently complains.

Well, I hope I still helped you by providing some debug info - and I'll ask the SRU Team to check why this force-badtest isn't working and go on with it.

[1]: https://bazaar.launchpad.net/~ubuntu-sru/britney/hints-ubuntu-xenial/revision/1700
[2]: http://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html

Oh it is rather trivial why the force-badtest no more works, there were uploads that got released but did not bump the test hinting - I'll provide an MP

MP up at https://code.launchpad.net/~paelzer/britney/hints-ubuntu-xenial-i386-libreoffice/+merge/365333

Thanks for the detailed investigation Christian.

This is indeed a known issue with libreoffice on i386, and it's not just autopkgtests, libreoffice applications crash at startup in java code if a JVM is installed. It affects xenial and bionic, and is about to be resolved on bionic with the backport of openjdk 11 (see bug #1814133).

Until now we have gotten away with a hint to ignore the test failures, given that i386 is a small percentage of the user base, and a JVM isn't installed by default.

Note that the inital report was about flaky tests. This JVM crash causing failure is consistently happening at every test run, so technically that's a different problem, but it's probably okay to recycle this bug.