[UBUNTU 20.04] SE-tooling: New IBM host-key subject locality (s390-tools)

------- Comment From <email address hidden> 2024-03-27 16:09 EDT-------
Full list of patches:

a54daf459e7504c0f42d3eb028100b7ab07894ff ("pvattest: Fix root-ca parsing")
5e1cb58a21ae0707d1993de3c8fc078c5cffed88 ("libpv: Support `Armonk` in IBM signing key subject")
d14e7593cc6380911ca42b09e11c53477ae13d5c ("genprotimg: support `Armonk` in IBM signing key subject")
1a3d0b74f7819f5e087e6ecbf3ec879a05a88bbc ("rust/pv: Support `Armonk` in IBM signing key subject")

Well, I already had a hard time to get the requested commits applied to noble (which is on 2.31.0).

I figured out that:
1) commit f6c6f0cc712433221fb0588c754e0d09884453dd ("rust/pv/test: Code + Certificate refactoring") is needed on top as pre-requisite, otherwise the other patches do not apply.
2) the commit id for ("libpv: Support `Armonk` in IBM signing key subject") is d7c95265cdb6217b0203efa5893c3a27838af63c (and not 5e1cb58a21ae0707d1993de3c8fc078c5cffed88 - this commit id does not exist in upstream master)
3) the commit id for ("pvattest: Fix root-ca parsing") is 2b5e7b049123aff094c7de79ba57a5df09471b2e (and not a54daf459e7504c0f42d3eb028100b7ab07894ff - again this commit id does not exist in upstream master).

I'm really wondering if it wouldn't be best to have a new minor version tagged upstream (like a 2.31.1) that includes everything needed, since I can't patch binary files with quilt (rust/pv/tests/assets/cert/der.crl and rust/pv/tests/assets/cert/der.crt), hence had to skip these hunks.

------- Comment From <email address hidden> 2024-04-02 09:17 EDT-------
(In reply to comment #11)
> Well, I already had a hard time to get the requested commits applied to
> noble (which is on 2.31.0).
>
> I figured out that:
> 1) commit f6c6f0cc712433221fb0588c754e0d09884453dd ("rust/pv/test: Code +
> Certificate refactoring") is needed on top as pre-requisite, otherwise the
> other patches do not apply.
> 2) the commit id for ("libpv: Support `Armonk` in IBM signing key subject")
> is d7c95265cdb6217b0203efa5893c3a27838af63c (and not
> 5e1cb58a21ae0707d1993de3c8fc078c5cffed88 - this commit id does not exist in
> upstream master)
> 3) the commit id for ("pvattest: Fix root-ca parsing") is
> 2b5e7b049123aff094c7de79ba57a5df09471b2e (and not
> a54daf459e7504c0f42d3eb028100b7ab07894ff - again this commit id does not
> exist in upstream master).
>
> I'm really wondering if it wouldn't be best to have a new minor version
> tagged upstream (like a 2.31.1) that includes everything needed, since I
> can't patch binary files with quilt (rust/pv/tests/assets/cert/der.crl and
> rust/pv/tests/assets/cert/der.crt), hence had to skip these hunks.

Your proposal makes sense. Let me see what we can do.

Steffen

Meanwhile I've navigated through the challenges in noble and have a successful PPA build here:
launchpad.net/~fheimes/+archive/ubuntu/lp2059303
and the s390-tools and the s390-tools-signed debdiffs attached.

Builds finally completed:
noble: https://launchpad.net/~fheimes/+archive/ubuntu/lp2059303
mantic, jammy, focal: https://launchpad.net/~fheimes/+archive/ubuntu/test

------- Comment From <email address hidden> 2024-04-05 04:40 EDT-------
(In reply to comment #15)
> Builds finally completed:
> noble: https://launchpad.net/~fheimes/+archive/ubuntu/lp2059303
> mantic, jammy, focal: https://launchpad.net/~fheimes/+archive/ubuntu/test

Hi Frank

Thank you for the supply of packages.

I currently have no noble (24.04) system by hand. I will test noble as soon as I have access.

For mantic, jammy, focal, it seems that a new s390-tools-signed package is missing in the PPA (it is there in the noble PPA). Tested only on focal.

Example for focal:

after adding the ppa + update.
> apt upgrade s390-tools
[...]
The following packages have unmet dependencies:
s390-tools : Depends: s390-tools-signed (= 2.12.0-0ubuntu3.8) but 2.12.0-0ubuntu3.7 is to be installed

Thank you in advance for a fix/solution.

And also, sorry for the Rust-backport inconvenience.
BTW, I released a new s390-tools version on Wednesday (https://github.com/ibm-s390-linux/s390-tools/releases/tag/v2.32.0) with the armonk problem resolved.

Steffen

Hi Steffen,
thanks for checking the updated packages.
Indeed, I had to restart the build of the s390-tools-signed packages for mantic/23.10, /jammy/22.04 and focal/20.04.
These build will be complete in an hour or so (so well before Monday).

Since the updates need to be done starting with the newest to oldest, a test on noble/24.04 would be most important.

Do you have a mantic/23.10 or jammy/22.04 test system over?
In this case you should be able to upgrade to noble/24.04 in just a few minutes.

First of all ensure that your current system (mantic or jammy) is up-to-date with:

sudo apt update && && sudo apt full-upgrade
(with a potential reboot)

You should then be able to upgrade with:
sudo do-release-upgrade -d

(or in case you are brave enough - but on a test/dev system ;-) :
sudo do-release-upgrade --quiet --devel-release --frontend=DistUpgradeViewNonInteractive && sudo reboot )

This bug was fixed in the package s390-tools - 2.31.0-0ubuntu5

---------------
s390-tools (2.31.0-0ubuntu5) noble; urgency=medium

  * Add the following commits as patches:
    - d/p/lp-2059303-rust-pv-test-Code-Certificate-refactoring.patch
      (excluded patching binary files rust/pv/tests/assets/cert/der.crl and
       rust/pv/tests/assets/cert/der.crt, which is not supported by quilt
       - these files are needed for testing only)
    - d/p/lp-2059303-rust-pv-Support-Armonk-in-IBM-signing-key-subject.patch
    - d/p/lp-2059303-genprotimg-support-Armonk-in-IBM-signing-key-subject.patch
    - d/p/lp-2059303-libpv-Support-Armonk-in-IBM-signing-key-subject.patch
    - d/p/lp-2059303-pvattest-Fix-root-ca-parsing.patch
    to fix Secure Execution tooling and accept new IBM host-key
    subject locality. LP: #2059303
  * Add d/p/lp-2058944-dbginfo.sh-dash-compatible-copy-sequence.patch
    to fix dash incompatibility in dbginfo.sh. LP: #2058944

 -- Frank Heimes <email address hidden> Tue, 02 Apr 2024 12:45:30 +0200

This bug was fixed in the package s390-tools-signed - 2.31.0-0ubuntu5

---------------
s390-tools-signed (2.31.0-0ubuntu5) noble; urgency=medium

  * Rebuild against 2.31.0-0ubuntu5 (LP: #2058944, LP: #2059303)

 -- Frank Heimes <email address hidden> Tue, 02 Apr 2024 12:55:52 +0200

------- Comment From <email address hidden> 2024-04-12 10:22 EDT-------
Hi Frank,

An update from my side:

!SHORT:

GOOD:
noble(with a nit), mantic, jammy
BAD/not verifyable:
focal

!LONG:

Verified for:

* 24.04 (noble):

with today's (2024-apr-12) s390tools version from noble repository (after a release-upgrade from jammy):
- genprotimg
- pvattest
- pvsecret
work.
did **not** use your PPA

But there is a mismatch from the package version to the version that the tools report:

> apt-cache showpkg s390-tools
Package: s390-tools
Versions:
2.31.0-0ubuntu5 (/var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_noble_main_binary-s390x_Packages)

> pvattest --version
pvattest version 2.31.0-build-20240409

You very likely forgot to change the DISTRELEASE variable in common.mak

* 23.10 (mantic):
All work with your PPA
- genprotimg
- pvattest
- pvsecret

* 22.04 (jammy):
All work with your PPA
- genprotimg
- pvattest

* 20:04 (focal):

the updated s390-tools package from your PPA has no genprotimg in it:

> dpkg --contents s390-tools_2.12.0-0ubuntu3.8_s390x.deb | grep genprot
-rw-r--r-- root/root 1775 2024-04-03 14:10 ./usr/share/man/man8/genprotimg.8.gz
drwxr-xr-x root/root 0 2024-04-03 14:10 ./usr/share/s390-tools/genprotimg/
-rw-r--r-- root/root 9656 2024-04-03 14:10 ./usr/share/s390-tools/genprotimg/stage3a.bin
-rw-r--r-- root/root 5498 2024-04-03 14:10 ./usr/share/s390-tools/genprotimg/stage3b_reloc.bin

(before the update(s390-tools2.12.0-ubuntu3.7))
> apt-file list s390-tools | grep genprotimg
s390-tools: /usr/bin/genprotimg
[....]

Verify process:
1) Obtain a (z15) Host-key document e.g. via the official channel
see: https://www.ibm.com/docs/en/linux-on-systems?topic=execution-obtain-host-key-document

2) Get a signing key (z15) + intermediate certificate
see: https://www.ibm.com/docs/en/linux-on-systems?topic=execution-verify-host-key-document

3) (optional) verify that the signing key is a new one (checl for: Locality Armonk)
> openssl x509 -text -in international_business_machines_corporation.crt | grep Subject
Subject: C = US, ST = New York, L = Armonk, O = International Business Machines Corporation, OU = IBM Z Host Key Signing Service, CN = International Business Machines Corporation

L **must** be Armonk, and not Poughkeepsie

4) run the tools if available:
The fixed tools will accept the cert chain and exit with exit code 0 and the output generated.
The non-fixed will print n error message, abort, and report exit != 0

> genprotimg: genprotimg -o tmp -i /boot/vmlinuz-$(uname -r) -k ~/hostkey.crt --cert ~/international_business_machines_corporation.crt --cert ~/DigiCertCA.crt

#BEFORE_FIX:
Failed to verify host-key document: please specify at least one IBM Z signing key
# AFTER_FIX:
# exit code 0

> pvattest create -VVV -o tmp --arpk arpk -k ~/hostkey.crt --cert ~/international_business_machines_corporation.crt --cert ~/DigiCertCA.crt

#BEFORE_FIX:
ERROR: Creating the attestation request failed:
Specify at least one IBM Z signing key
# AFTER_FIX:
# exit code 0

> pvsecret create --hdr ~/secure_guest.hdr -o tmp -k ~/hostkey.crt --cert ~/international_business_machines_corporation.crt --cert ~/armonk/DigiCertC...

Hi Steffen,
many thanks for the tests, that's highly appreciated!

I'm glad to see that we are fine with noble, mantic and jammy.
It was btw. good to pick the version from the noble archive, since it's in beta).
(And btw. we usually do not alter DISTRELEASE in common.mak.)

I'm uploading first of all the debdiffs for mantic and jammy (to not loose much time),
and will investigate focal in a bit...

s390-tools and s390-tools-signed debdiffs for 23.10/mantic

s390-tools and s390-tools-signed debdiffs for 22.04/jammy

For focal the commit d14e7593cc6
(https://github.com/ibm-s390-linux/s390-tools/commit/d14e7593cc6380911ca42b09e11c53477ae13d5c)
does not properly build and the logs show a few errors:
https://launchpadlibrarian.net/723098720/buildlog_ubuntu-focal-s390x.s390-tools_2.12.0-0ubuntu3.8_BUILDING.txt.gz
(search for "error:")
"
utils/crypto.c: In function ‘x509_armonk_locality_fixup’:
utils/crypto.c:770:22: error: passing argument 1 of ‘X509_NAME_dup’ discards ‘const’ qualifier from pointer target type [-Werror=discarded-qualifiers]
  770 | ret = X509_NAME_dup(name);
      | ^~~~
In file included from /usr/include/openssl/pem.h:17,
                 from utils/crypto.c:17:
/usr/include/openssl/x509.h:482:12: note: expected ‘X509_NAME *’ {aka ‘struct X509_name_st *’} but argument is of type ‘const X509_NAME *’ {aka ‘const struct X509_name_st *’}
  482 | X509_NAME *X509_NAME_dup(X509_NAME *xn);
      | ^~~~~~~~~~~~~
utils/crypto.c: In function ‘quirk_X509_STORE_ctx_get1_crls’:
utils/crypto.c:888:8: error: implicit declaration of function ‘Pv_X509_STORE_CTX_get1_crls’; did you mean ‘X509_STORE_CTX_get1_crls’? [-Werror=implicit-function-declaration]
  888 | ret = Pv_X509_STORE_CTX_get1_crls(ctx, subject);
      | ^~~~~~~~~~~~~~~~~~~~~~~~~~~
      | X509_STORE_CTX_get1_crls
utils/crypto.c:888:8: error: nested extern declaration of ‘Pv_X509_STORE_CTX_get1_crls’ [-Werror=nested-externs]
utils/crypto.c:888:6: error: assignment to ‘STACK_OF_X509_CRL_autoptr’ {aka ‘struct stack_st_X509_CRL *’} from ‘int’ makes pointer from integer without a cast [-Werror=int-conversion]
  888 | ret = Pv_X509_STORE_CTX_get1_crls(ctx, subject);
      | ^
utils/crypto.c:902:7: error: assignment to ‘STACK_OF_X509_CRL_autoptr’ {aka ‘struct stack_st_X509_CRL *’} from ‘int’ makes pointer from integer without a cast [-Werror=int-conversion]
  902 | ret = Pv_X509_STORE_CTX_get1_crls(ctx, fixed_subject);
      | ^
utils/crypto.c:913:7: error: assignment to ‘STACK_OF_X509_CRL_autoptr’ {aka ‘struct stack_st_X509_CRL *’} from ‘int’ makes pointer from integer without a cast [-Werror=int-conversion]
  913 | ret = Pv_X509_STORE_CTX_get1_crls(ctx, fixed_subject);
      | ^
utils/crypto.c:925:6: error: assignment to ‘STACK_OF_X509_CRL_autoptr’ {aka ‘struct stack_st_X509_CRL *’} from ‘int’ makes pointer from integer without a cast [-Werror=int-conversion]
  925 | ret = Pv_X509_STORE_CTX_get1_crls(ctx, fixed_subject);
      | ^
"
and
"
utils/crypto.c: In function ‘x509_armonk_locality_fixup’:
utils/crypto.c:770:22: error: passing argument 1 of ‘X509_NAME_dup’ discards ‘const’ qualifier from pointer target type [-Werror=discarded-qualifiers]
  770 | ret = X509_NAME_dup(name);
      | ^~~~
In file included from /usr/include/openssl/pem.h:17,
                 from utils/crypto.c:17:
/usr/include/openssl/x509.h:482:12: note: expected ‘X509_NAME *’ {aka ‘struct X509_name_st *’} but argument is of type ‘const X509_NAME *’ {aka ‘const struct X509_name_st *’}
  482 | X509_NAME *X509_NAME_dup(X509_NAME *xn);
      | ^~~~~~~~~~~~~
utils/crypto.c: In function ‘quirk_X509_STORE_ctx_get1_crls’:
utils/cr...

------- Comment From <email address hidden> 2024-04-16 05:22 EDT-------
Can you please pick this commit:

https://github.com/ibm-s390-linux/s390-tools/commit/f5744b95db93fa9d5cfd6fb206767ad2dcc3c804

?

Hi Marc,
I've added f5744b95db9, but I unfortunately still get the build errors:
https://launchpadlibrarian.net/724893134/buildlog_ubuntu-focal-s390x.s390-tools_2.12.0-0ubuntu3.9_BUILDING.txt.gz
(search for "error:")

Marc and me, we had a side discussion and finally found the missing commits and have a proper build for focal.
The build is available here:
https://launchpad.net/~fheimes/+archive/ubuntu/lp205930
and the debdiff(s) attached.

------- Comment From <email address hidden> 2024-04-17 04:28 EDT-------
Thanks Frank and Marc.

Verified for focal with the focal-package in PPA at http://ppa.launchpad.net/fheimes/lp2059303/ubuntu (2.12.0-0ubuntu3.8)

-> verified for all releases in service.
focal, jammy, mantic, and noble.

Steffen

Cool, thanks a lot Steffen!

I've been asked to try to help these updates along; I'm not on the SRU team so I can't give concrete directions, only suggestions.

My assumption is that these package updates should be published first to -updates for autopkgtest testing, and once they have passed testing and phased to users, then we should republish these updates to -security so that they are available to all users. Does this sound correct?

This is much easier to execute if the updates have been built in a PPA with only -security enabled, and not -updates. (The -security pocket is built with only packages from -release and -security, not -updates.) Do packages built in such a PPA exist?

The SRU workflow asks for packages to be either uploaded with dput to the queue or debdiffs provided. I see some debdiffs here, but some additional work was performed after most of the debdiffs were uploaded. Are the posted debdiffs something that the SRU team should work with? The Ubuntu Sponsors team was added around three weeks ago, before much of the work was done, it's entirely possible that this has fallen off their radar as a result. (And, the general hustle of responding to the xz-utils issue, release time goals, etc.)

So, with the reminder that I'm not on the SRU team, I think the next steps should be:

- prepare a PPA with only -security enabled
- build packages
- ask SRU team to move the packages to -proposed and see how autopkgtests go
- phase the update
- ask the security team to binary copy the packages to -security once it's proven in the field

What do you think?

Thanks

Hi @seth-arnold,
I am not in-depth familiar with security related updates (since they happen rarely for me, and if private-security are handled by the security team anyway).

"
My assumption is that these package updates should be published first to -updates for autopkgtest testing, and once they have passed testing and phased to users, then we should republish these updates to -security so that they are available to all users. Does this sound correct?
"
There are no autopkgtests (for historical reasons and since lot's of functions in that package req. the hw to be configured in a certain way, which cannot guaranteed by the build systems), but the packages get (and already got) already manually tested upfront, with focus on the changes (according to test plan in SRU justification).

"
This is much easier to execute if the updates have been built in a PPA with only -security enabled, and not -updates. (The -security pocket is built with only packages from -release and -security, not -updates.) Do packages built in such a PPA exist?
"
I've now kicked off build in a -security only PPA here:
https://launchpad.net/~fheimes/+archive/ubuntu/lp2059303-sec
(so yes, they build there, but it'll take a while until published)

"
The SRU workflow asks for packages to be either uploaded with dput to the queue or debdiffs provided. I see some debdiffs here, but some additional work was performed after most of the debdiffs were uploaded.
"
The changes are some broken URL references in some quilt patch headers, I've fixed those.

"
Are the posted debdiffs something that the SRU team should work with? The Ubuntu Sponsors team was added around three weeks ago, before much of the work was done, it's entirely possible that this has fallen off their radar as a result. (And, the general hustle of responding to the xz-utils issue, release time goals, etc.)
"
I think the debdiffs should be taken (as usual).
It's difficult to get SRUs processed around release times.

One concern I have is that copying the packages might not work, since there is a bootloader component that is signed, and the signing key is based on the location where the package is build.
Hence a package build in PPA will be signed with the PPA key and not with the official ('production') key
and so copying it over from PPA to archive will probably mess up things.

So I believe the debdiffs need to be the base for an upload (by a sponsor), then build for the archives (that will ensure signing with the proper key), then published on -proposed, verified there and then eventually released.

(I'm attaching the debdiffs again, with fixed urls)