OpenSSL CPU detection for AMD Ryzen CPUs

Here's some context after a conversation about this bug on channel : #ubuntu-release

...
[10:01:50] <slashd> hi SRU, I'm currently working on a case (no LP bug yet).... about an OpenSSL bug on new AMD CPU (Ryzen) released last Feb ... where the SHA Extension routine is not called on AMD Ryzen cores. My question is since this look like H/W enablement ... do you think this could be eligible for SRU in stable release such like Xenial ? or this will only be accepted for devel release ? This is a new CPU but Xenial is there for a couple of years still so maybe future Xenial user running Ryzen CPU may benefit on this eventually...
[10:03:20] <apw> slashd, for me some new functionality like that is ok as long as it is very self-contained so easy to review and confirm is only used on the new h/w
[10:03:52] <apw>one of our main goals is to avoid regressions
[10:04:41] <slashd>apw, make sense, thanks for your input
[10:12:24] <rbasak> The SRU policy does explicitly permit hardware enablement in an LTS IIRC, though I'd expect ~ubuntu-sru to be involved in mitigating risk and making the final risk decision, FWIW.
[10:16:11] <apw> rbasak, right, it would have to be carefully considered once we can see what the diff actually is
[10:16:34] <slashd> rbasak, apw ack, will communite the info with the proper group
[10:16:50] <apw> with a much greater level of testing and scrutiny than a regular fix only sru
[10:17:12] <slashd> apw, rbasak, FYI I have requested the new CPU from our partner to test in deep
...

Debian bug :
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=861145

So far my test reveal the following :

# Note that the below test has been made on a Ryzen system #

[Without patch]
 * Generated a checksum of a big file (e.g. 5GB file) with openssl
 $ time /usr/bin/openssl dgst -sha256 /var/tmp/5Gfile
SHA256(/var/tmp/5Gfile)= 8d448d81521cbc1bfdc04dd199d448bd3c49374221007bd0846d8d39a70dd4f8

real 0m12.835s
user 0m12.344s
sys 0m0.484s

[With patch]
 * Generated a checksum of a big file (e.g. 5GB file) with openssl
 $ time /usr/bin/openssl dgst -sha256 /var/tmp/5Gfile
SHA256(/var/tmp/5Gfile)= 8d448d81521cbc1bfdc04dd199d448bd3c49374221007bd0846d8d39a70dd4f8

real 0m3.471s
user 0m2.956s
sys 0m0.516s

Another test "Openssl speed"

[Without patch]
 $ openssl speed sha1
Doing sha1 for 3s on 16 size blocks: 9969152 sha1's in 3.00s
Doing sha1 for 3s on 64 size blocks: 8019164 sha1's in 3.00s
Doing sha1 for 3s on 256 size blocks: 5254219 sha1's in 2.99s
Doing sha1 for 3s on 1024 size blocks: 2217067 sha1's in 3.00s
Doing sha1 for 3s on 8192 size blocks: 347842 sha1's in 3.00s
OpenSSL 1.0.2g 1 Mar 2016
built on: reproducible build, date unspecified
options:bn(64,64) rc4(8x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: gcc -I. -I.. -I../include -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -Wa,--noexecstack -m64 -DL_ENDIAN -O3 -Wall -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM -DECP_NISTZ256_ASM
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
sha1 53168.81k 171075.50k 449859.55k 756758.87k 949840.55

[With patch]
 $ openssl speed sha1
Doing sha1 for 3s on 16 size blocks: 12081890 sha1's in 3.00s
Doing sha1 for 3s on 64 size blocks: 11563950 sha1's in 3.00s
Doing sha1 for 3s on 256 size blocks: 8375101 sha1's in 3.00s
Doing sha1 for 3s on 1024 size blocks: 3987643 sha1's in 3.00s
Doing sha1 for 3s on 8192 size blocks: 678036 sha1's in 3.00s
OpenSSL 1.0.2g 1 Mar 2016
built on: reproducible build, date unspecified
options:bn(64,64) rc4(8x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: gcc -I. -I.. -I../include -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -Wa,--noexecstack -m64 -DL_ENDIAN -O3 -Wall -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM -DECP_NISTZ256_ASM
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
sha1 64436.75k 246697.60k 714675.29k 1361115.48k 1851490.30k

[For SRU Verification team]

Context :
Previous IRC discussion with apw/rbasak about this case :
https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1674399/comments/2

AMD released a new CPU called "Ryzen" that now support "Intel SHA extensions" technology.

In current Ubuntu openssl package sha extension are masked on Ryzen CPU. Ryzen is available in 64-bit only CPU (Confimed with AMD representative).

There are upstream patches that solve this situation and my test revealed that openssl has significant performance increase on a AMD CPU with sha extension capability.

The upstream patches "f8418d8" & "1aed5e1" fix the situation by moving the extended feature detection from Label (.Lintel) to Label (.Lgeneric) in both 64-bit(crypto/x86_64cpuid.pl) & 32-bit (crypto/x86cpuid.pl) code where it should belongs now that non-intel CPU can also have the capability. I cannot strictly isolate the fix for Ryzen CPU only, meaning that this feature will enable sha extension for all CPUa that support the functionality, where prior this patch it was strictly reserved for Intel CPU only.

Since I don't forsee a lot of users using the 32-bit package of opensssl on a 64-bit CPU with sha extension enabled and putting myself in a SRU mindset, I proposed the following :

- SRU the 64-bit and 32-bit patch in development release (Artful/17.10)
- SRU the 64 bit only in Stable release (Xenial/Zesty)
  - 32-bit code : Remain the same, thus no behavioural change
  - 64-bit code : Enable sha extension for 64-bit CPU that has the capabilities.

The test that I proposed before "verification-done" (while package is in -proposed)

-> Do a performance test using openssl speed[1] & generate a checksum on a big file[2] (e.g. 5GB) and capture the metric to compare before and after the patch on a 64-bit AMD system WITH sha extension capability (Ryzen).
-> Do a performance test using openssl speed[1] & generate a checksum on a big file[2] (e.g. 5GB) and capture the metric to compare before and after the patch on a 64-bit AMD system WITHOUT sha extension capability.
-> Do a performance test using openssl speed[1] & generate a checksum on a big file[2] (e.g. 5GB) and capture the metric to compare before and after the patch on a 64-bit Intel system WITH sha extension capability.
-> Do a performance test using openssl speed[1] & generate a checksum on a big file[2] (e.g. 5GB) and capture the metric to compare before and after the patch on a 64-bit Intel system WITHOUT sha extension capability.
-> In all the above tests, using perf record/annotate could also be done to validate if the sha instructions has been used or not depending on the above scenarios to conclude everything is working as expected.

No extra testing will be needed for 32-bit package since no 32-bit code will be modified.
On the other hand, if a user for some reason want to use openssl in 32-bit on a 64-bit with sha extension CPU capabilities, then this user won't have all the performance benefit of the 64-bit code fix.

Considering that this request is a "HW enablement", thus not a bugfix and the above notes... Is this would be eligible for SRU in Stable Release ?

[1] - Openssl speed measurement using sha1
$...

Attaching Artful debdiff

Here's the highligh of the discussion I had in #ubuntu-release with infinity about my proposal in comment #6.

<slashd>For SRU, I had a talk with apw and rbasak about this bug a couples weeks ago LP: #1674399, could you please look at this bug and based on the Descriptions and comment #6 if this looks eligible for SRU in Stable release ? (note that this is a HW enablement, not a bug, this is why I'm requesting you to have a look at it) thanks in advance.

<infinity> slashd: I disagree with your reasoning for not fixing both 64 and 32.

<infinity> slashd: Lots of people run 64/32 multiarch and would benefit from fixing both.

<slashd> infinity, I'm fine with fixing 32bit, I proposed that approach cause apw wanted to self-contained the fix as much as possible

<infinity> That doesn't really contain it much. ;)

<slashd> infinity, so what if I do the same proposition but including 32-bit in stable release, would that work for you ?

<infinity> slashd: Conceptually, I have no issues with the plan (other than the "please do 32-bit too" comment).

<slashd> infinity, sure, I'm actually glad you are keen to see the 32-bit portion included

<infinity> slashd: Upload away, IMO.

<slashd> infinity, I'll then start the upload for Artful, note that starting next week I'll be gone for 2 weeks for sprints, and won't be able to do much testing, so do you think it's preferable we only start the SRU when I get back or we upload this week and worst case it will languish in -proposed for ~2weeks which will allow ppls to test with no stress (if any volunteer)

<infinity> slashd: I think letting it fester in proposed for two weeks to see if we get random negative feedback is entirely fine. Obviously, I'll delete/revert it if it breaks anything, but I don't need you around for that.

<infinity> slashd: 2 weeks of random user installations plus you executing a more precision test plan should give us solid confidence.

zesty_openssl_lp1674399.debdiff

xenial_openssl_lp1674399.debdiff

This bug was fixed in the package openssl - 1.0.2g-1ubuntu12

---------------
openssl (1.0.2g-1ubuntu12) artful; urgency=medium

  * crypto/x86*cpuid.pl: move extended feature detection. (LP: #1674399)
    This fix moves extended feature detection past basic feature
    detection where it belongs. 32-bit counterpart is harmonized too.

 -- Eric Desrochers <email address hidden> Tue, 25 Apr 2017 18:16:18 -0400

zesty_openssl_lp1674399.debdiff

yakkety_openssl_lp1674399.debdiff

Hello Eric, or anyone else affected,

Accepted openssl into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openssl/1.0.2g-1ubuntu9.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Verificaton YAKKETY]

# i386
- Significant performance increase using the yakkety-proposed/i386 package inside a 32-bit LXD container build using a Ryzen CPU with Intel SHA Extension capability.
- Same performance (as expected) using the yakkety-proposed/i386 package on a non SHA Extension Intel CPU (i7-6770HQ) with yakkety-proposed package.

# amd64
- Significant performance increase using the yakkety-proposed/amd64 package on Ryzen CPU with Intel SHA Extension capability.
- Same performance (as expected) using the yakkety-proposed/amd64 package on a non SHA Extension Intel CPU (i7-6770HQ) with yakkety-proposed package.

Note : I unfortunately don't (nor colleagues) have access to a Intel CPU with SHA Extension capability at our disposal. Ideally, if someone has access to one to test it would be good.
Otherwise, I think it is safe to rely on upstream author of the patch who confirmed it was working as expected using a Intel CPU with SHA extension capability.

Reference : https://github.com/openssl/openssl/issues/2848
"...Myself I tested on Intel processors, yes, with/without...."

==
* Test yakkety-proposed/i386 on a 32-bit LXD container using a non SHA Extension Intel CPU (Version before -proposed pkg):
--
ii libssl1.0.0:i386 1.0.2g-1ubuntu9.1 i386 Secure Sockets Layer toolkit - shared libraries
ii openssl 1.0.2g-1ubuntu9.1 i386 Secure Sockets Layer toolkit - cryptographic utility

# openssl speed sha1
Doing sha1 for 3s on 16 size blocks: 12441833 sha1's in 3.00s
Doing sha1 for 3s on 64 size blocks: 8997589 sha1's in 3.00s
Doing sha1 for 3s on 256 size blocks: 5074636 sha1's in 3.00s
Doing sha1 for 3s on 1024 size blocks: 1904828 sha1's in 3.00s
Doing sha1 for 3s on 8192 size blocks: 304739 sha1's in 3.00s
OpenSSL 1.0.2g 1 Mar 2016
built on: reproducible build, date unspecified
options:bn(64,32) rc4(8x,mmx) des(ptr,risc1,16,long) aes(partial) blowfish(idx)
compiler: cc -I. -I.. -I../include -fPIC -DOPENSSL_PIC -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DL_ENDIAN -g -O2 -fdebug-prefix-map=/build/openssl-OIx07U/openssl-1.0.2g=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOUR
CE=2 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wa,--noexecstack -Wall -DOPENSSL_BN_ASM_PART_WORDS -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DRMD160_ASM -DAES_ASM -DVPAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
sha1 66356.44k 191948.57k 433035.61k 650181.29k 832140.63k

# time openssl dgst -sha256 /var/tmp/5Gfile
SHA256(/var/tmp/5Gfile)= 7f06c62352aebd8125b2a1841e2b9e1ffcbed602f381c3dcb3200200e383d1d5

real 0m15.429s
user 0m14.372s
sys 0m1.052s
==
* Test yakkety-proposed/i386 on a 32-bit LXD container using a non SHA Extension Intel CPU (With -proposed pkg):
--
ii libssl1.0.0:i386 1.0.2g-1ubuntu9.2 i386 Secure Sockets Layer toolkit - shared libraries
ii openssl ...

Hello Eric, or anyone else affected,

Accepted openssl into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openssl/1.0.2g-1ubuntu11.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Eric, or anyone else affected,

Accepted openssl into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openssl/1.0.2g-1ubuntu4.7 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

We have tested these packages in zesty-proposed (openssl-1.0.2g-1ubuntu11.1) and can confirm that the SHA extension codepath is executed correctly and we see the accompanying expected performance improvements.

Thanks!

[Verificaton XENIAL]

# i386
- Significant performance increase using the xenial-proposed/i386 package inside a 32-bit LXD container build using a Ryzen CPU with Intel SHA Extension capability.
- Same performance (as expected) using the xenial-proposed/i386 package on a non SHA Extension Intel CPU (i7-6770HQ) with xenial-proposed package.

# amd64
- Significant performance increase using the xenial-proposed/amd64 package on Ryzen CPU with Intel SHA Extension capability.
- Same performance (as expected) using the xenial-proposed/amd64 package on a non SHA Extension Intel CPU (i7-6770HQ) with xenial-proposed package.

Note : I unfortunately don't (nor colleagues) have access to a Intel CPU with SHA Extension capability at our disposal. Ideally, if someone has access to one to test it would be good.
Otherwise, I think it is safe to rely on upstream author of the patch who confirmed it was working as expected using a Intel CPU with SHA extension capability.

Reference : https://github.com/openssl/openssl/issues/2848
"...Myself I tested on Intel processors, yes, with/without...."

==
* Test xenial/i386 on a 32-bit LXD container using a non SHA Extension Intel CPU:
--
ii libssl1.0.0:i386 1.0.2g-1ubuntu4.6 i386 Secure Sockets Layer toolkit - shared libraries
ii openssl 1.0.2g-1ubuntu4.6 i386 Secure Sockets Layer toolkit - cryptographic utility

# openssl speed sha1
Doing sha1 for 3s on 16 size blocks: 12391058 sha1's in 3.00s
Doing sha1 for 3s on 64 size blocks: 8934411 sha1's in 3.00s
Doing sha1 for 3s on 256 size blocks: 5048901 sha1's in 3.00s
Doing sha1 for 3s on 1024 size blocks: 1893157 sha1's in 3.00s
Doing sha1 for 3s on 8192 size blocks: 301374 sha1's in 3.00s
OpenSSL 1.0.2g 1 Mar 2016
built on: reproducible build, date unspecified
options:bn(64,32) rc4(8x,mmx) des(ptr,risc1,16,long) aes(partial) blowfish(idx)
compiler: cc -I. -I.. -I../include -fPIC -DOPENSSL_PIC -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DL_ENDIAN -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wa,--noexecstack -Wall -DOPENSSL_BN_ASM_PART_WORDS -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DRMD160_ASM -DAES_ASM -DVPAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
sha1 66085.64k 190600.77k 430839.55k 646197.59k 822951.94k

# time openssl dgst -sha256 /var/tmp/5Gfile
SHA256(/var/tmp/5Gfile)= 7f06c62352aebd8125b2a1841e2b9e1ffcbed602f381c3dcb3200200e383d1d5

real 0m15.518s
user 0m14.428s
sys 0m1.084s
==
* Test xenial-proposed/i386 on a 32-bit LXD container using a non SHA Extension Intel CPU:
--
ii libssl1.0.0:i386 1.0.2g-1ubuntu4.7 i386 Secure Sockets Layer toolkit - shared libraries
ii openssl 1.0.2g-1ubuntu4.7 i386 Secure Sockets Layer toolkit - cryptographic utility

# ope...

The same precision verification testing has been tested for zesty-proposed with the same result as X and Y :

[Verificaton zesty]

# i386
- Significant performance increase using the zesty-proposed/i386 package inside a 32-bit LXD container build using a Ryzen CPU with Intel SHA Extension capability.
- Same performance (as expected) using the zesty-proposed/i386 package on a non SHA Extension Intel CPU (i7-6770HQ) with zesty-proposed package.

# amd64
- Significant performance increase using the zesty-proposed/amd64 package on Ryzen CPU with Intel SHA Extension capability.
- Same performance (as expected) using the zesty-proposed/amd64 package on a non SHA Extension Intel CPU (i7-6770HQ) with zesty-proposed package.

Note : I unfortunately don't (nor colleagues) have access to a Intel CPU with SHA Extension capability at our disposal. Ideally, if someone has access to one to test it would be good.
Otherwise, I think it is safe to rely on upstream author of the patch who confirmed it was working as expected using a Intel CPU with SHA extension capability.

Reference : https://github.com/openssl/openssl/issues/2848
"...Myself I tested on Intel processors, yes, with/without...."

Additionally, we also had some feedbacks from
Justin Erenkrantz, a affected users using a Ryzen/Naple CPU.

Please look comment #18 to see Justin feedback:
https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1674399/comments/18

libssl1.0.0 1.0.2g-1ubuntu9.2 breaks OpenVPN (2.4.0-5ubuntu1 or 2.3.11-1ubuntu2) connections to Canonical's VPN on my Ryzen 7 1700X desktop running Linux 4.10.0-21-generic. In UDP mode the server stops responding during TLS negotiation, and in TCP mode the server closes the connection at the same stage. Downgrading to ubuntu9.1 fixes it. artful's 1.0.2g-1ubuntu12 is broken in the same way. The HMAC in use by the VPN is SHA-1.

From the server log:

ovpn-tcp[30227]: TCP connection established with [AF_INET]<REDACTED>:44544
ovpn-tcp[30227]: <REDACTED>:44544 TCP connection established with [AF_INET]<REDACTED>:47753
ovpn-tcp[30227]: <REDACTED>:44544 TLS_ERROR: BIO read tls_read_plaintext error: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac
ovpn-tcp[30227]: <REDACTED>:44544 TLS Error: TLS object -> incoming plaintext read error
ovpn-tcp[30227]: <REDACTED>:44544 TLS Error: TLS handshake failed
ovpn-tcp[30227]: <REDACTED>:44544 Fatal TLS error (check_tls_errors_co), restarting

The start of a client log is at http://paste.ubuntu.com/24603459/. Until the connection is closed by the server, it differs from a successful connection only in its keys and session IDs.

Fortunately the OpenSSL test suite also fails when run during the build on Ryzen. It turns out that the AES-NI+SHA-NI AES-CBC+SHA{1,256} implementations are both broken, so https://github.com/openssl/openssl/commit/08d09628d2c9f3ef599399d8cad021a07ab98347 needs to be backported too. I guess nobody's seriously used Ubuntu on Goldmont.

I've uploaded fixed SRU test builds to https://launchpad.net/~wgrant/+archive/ubuntu/experimental/+packages?field.name_filter=openssl, and they all build and test successfully on i386 and amd64 on Ryzen. At least 9.3~ppa1 even lets OpenVPN connect, with accelerated hashing. I don't think we really need to dig up a Goldmont device from somewhere, but if someone has one handy...

Thanks William, I'll set the proposed pkg as verification-failed, and will work on backporting the patch[1] you are suggesting and that has been proven to fix the issue.

[1] - https://github.com/openssl/openssl/commit/08d09628d2c9f3ef599399d8cad021a07ab98347

Eric

William, as per our IRC conversation, we have decided that you will do the upload for this specific fix for the 4 releases.

Thanks for your collaboration.

- Eric

Hello Eric, or anyone else affected,

Accepted openssl into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openssl/1.0.2g-1ubuntu11.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Eric, or anyone else affected,

Accepted openssl into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openssl/1.0.2g-1ubuntu9.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Eric, or anyone else affected,

Accepted openssl into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openssl/1.0.2g-1ubuntu4.8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

As a part of the Stable Release Updates quality process a search for Launchpad bug reports using the version of openssl from zesty-proposed was performed and bug 1692981 was found. Please investigate this bug report to ensure that a regression will not be created by this SRU. In the event that this is not a regression remove the "verification-failed" tag from this bug report and add the tag "bot-stop-nagging" to bug 1692981 (not this bug). Thanks!

Removing the "regression-proposed" & "verification-failed" tags now that the regression fix ins now available to testing in -proposed.

As a part of the Stable Release Updates quality process a search for Launchpad bug reports using the version of openssl from zesty-proposed was performed and bug 1692981 was found. Please investigate this bug report to ensure that a regression will not be created by this SRU. In the event that this is not a regression remove the "verification-failed" tag from this bug report and add the tag "bot-stop-nagging" to bug 1692981 (not this bug). Thanks!

William - could you add some information about the verification you did since you tagged this verification-done?

Eric - Have you tested the new version of openssl which William uploaded?

I've been running the fixed OpenSSL on my artful Ryzen desktop, zesty Haswell-U laptop and xenial Sandy Bridge server and its VMs for two weeks without incident. These hosts include a variety of VPN clients, HTTPS clients, HTTPS servers, SSH clients and servers, etc. The SRUs for all series build successfully on i386 and amd64 on Ryzen and Apollo Lake (I managed to acquire some hardware), the two shipping platforms that were exposed to the previous regression, and which FTBFS with the original SRU due to the test suite picking up the bug.

@Brian Murray (brian-murray)

> Eric - Have you tested the new version of openssl which William uploaded?
I did and my test cases results are still the same as the one pre-wgrant SRU.

All good on my side.

IMHO, the only things left to check are the 2-3 regressions in X and Y for postgresql (X,Y) and mongodb (Y) on armhf architecture.

So far at first glance I can't see how this is failure are related to our openssl changes, but I'll look at it more in deep.

The rest of the regressions found are explained in the description above in the "Regression section" where it says : * autopktest failure.

- Eric

@Brian Murray,
@sil2100

To summarize :
- wgrant did his verifications which are all successful
- I did mine that are all still successful
- I also reviewed all the regression failures reported on the pending sru page and have provided an explanation for all of them above in the SRU justification template.

IMHO, Now I don't see any blockers for the SRU to complete, but I'll let you guys (SRU) take the final decision.

Let us know if we can be of any help.

- Eric

This bug was fixed in the package openssl - 1.0.2g-1ubuntu4.8

---------------
openssl (1.0.2g-1ubuntu4.8) xenial; urgency=medium

  * aes/asm/aesni-sha*-x86_64.pl: fix IV handling in SHAEXT paths.
    (LP: #1674399)

 -- William Grant <email address hidden> Fri, 19 May 2017 18:27:58 +1000

The verification of the Stable Release Update for openssl has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package openssl - 1.0.2g-1ubuntu9.3

---------------
openssl (1.0.2g-1ubuntu9.3) yakkety; urgency=medium

  * aes/asm/aesni-sha*-x86_64.pl: fix IV handling in SHAEXT paths.
    (LP: #1674399)

 -- William Grant <email address hidden> Fri, 19 May 2017 18:25:11 +1000

This bug was fixed in the package openssl - 1.0.2g-1ubuntu11.2

---------------
openssl (1.0.2g-1ubuntu11.2) zesty; urgency=medium

  * aes/asm/aesni-sha*-x86_64.pl: fix IV handling in SHAEXT paths.
    (LP: #1674399)

 -- William Grant <email address hidden> Fri, 19 May 2017 18:29:44 +1000

William, Eric, I'm curious if this OpenSSL update may have introduced issues:

https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1697099
https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1696930
https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1696863
https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1696799
https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1692981

My own upgrades on the command line went without troubles.

Thanks

Some of the bug reports in my previous comment are actually filed against previous versions of openssl (despite all being filed recently). Eric noted that only 1692981 (zesty) and 1696799 (xenial) are for this SRU, and 1696863 1697099 1696930 are for previous updates.

I don't think there's an issue with this SRU; this just appears to be usual upgrade failure noise with unfortunate timing making me think there may be a pattern here.

Thanks, and thanks to Eric for the extra research late on a Friday.

Seth,

I don't think it is related ... 3 of them are related to package(s) prior this SRU on version "1.0.2g-1ubuntu4.6" release on "Mon, 30 Jan 2017" and only 1 on "1.0.2g-1ubuntu4.8" for Xenial & another on on "libssl1.0.0" for Zesty.

Title: package openssl 1.0.2g-1ubuntu4.6 failed to install/upgrade
1697099

Title: package libssl-dev:amd64 1.0.2g-1ubuntu11.2 failed to install/upgrade
1692981

Title: package libssl1.0.0:i386 1.0.2g-1ubuntu4.8 failed to install/upgrade:
1696799

Title: package libssl1.0.0:i386 1.0.2g-1ubuntu4.6 failed to install/upgrade:
1696863
1696930

I also did a couple of upgrade from "ubuntu4.6" to "ubuntu4.8" inside amd64 and i386 lxc container myself during the SRU and again today, and I haven't been able to reproduce the situation.

As per our irc conversation, it is probably just being some random pile of noise, but let's keep an eye on it in the next few days.

- Eric