[SRU] Include powerpc port upstream fixes to librpmem 1.10 on pmdk package

I think this affects pmdk (rather the Linux kernel).

------- Comment From <email address hidden> 2021-06-07 10:25 EDT-------
> I think this affects pmdk (rather the Linux kernel).

Correct.

Since the requested packages hould be incl. in pmdk 1.10
and since we have the following versions in the ubuntu releases:
 pmdk | 1.4.1-0ubuntu1~18.04.1 | bionic-updates/universe | source
 pmdk | 1.8-1ubuntu1 | focal | source
 pmdk | 1.9-2 | groovy | source
 pmdk | 1.10-1ubuntu1 | hirsute | source
 pmdk | 1.10-1ubuntu1 | impish | source
I think this affects hirsute and impish only.

------- Comment From <email address hidden> 2021-06-17 09:51 EDT-------
(In reply to comment #8)
> Since the requested packages hould be incl. in pmdk 1.10
> and since we have the following versions in the ubuntu releases:
> pmdk | 1.4.1-0ubuntu1~18.04.1 | bionic-updates/universe | source
> pmdk | 1.8-1ubuntu1 | focal | source
> pmdk | 1.9-2 | groovy | source
> pmdk | 1.10-1ubuntu1 | hirsute | source
> pmdk | 1.10-1ubuntu1 | impish | source
> I think this affects hirsute and impish only.

That's correct.

------- Comment From <email address hidden> 2021-07-22 08:52 EDT-------
Is there any updates on this issue?

According to the pmdk upstream git repository the two commits identified as fixing this bug have been released in version 1.11.0:

https://github.com/pmem/pmdk/commit/652659830
https://github.com/pmem/pmdk/commit/e672c09d9

which is now in Impish, so I'm going to mark the Impish task as Fix Released.

I am setting up a PPA for preliminary testing before the actual upload. Unfortunately the package build queue for ppc64el is very long at the moment; I'll share a link to the PPA as soon as the package is ready.

Hi, so here is the PPA with amd64 and s390x packages including the fix:

https://launchpad.net/~paride/+archive/ubuntu/pmdk-lp1931063

I'll work towards uploading it to -proposed for the formal SRU/verification process, however early feedback on the packages will help and speed things up. @lamm or others: if you can proceed with some testing of those packages I'll greatly appreciate. Thanks!

------- Comment From <email address hidden> 2021-07-27 13:11 EDT-------
(In reply to comment #13)
> Hi, so here is the PPA with amd64 and s390x packages including the fix:
>
> https://launchpad.net/~paride/+archive/ubuntu/pmdk-lp1931063
>
> I'll work towards uploading it to -proposed for the formal SRU/verification
> process, however early feedback on the packages will help and speed things
> up. @lamm or others: if you can proceed with some testing of those packages
> I'll greatly appreciate. Thanks!

Thanks @paride! Sure, I will test it here.

------- Comment From <email address hidden> 2021-07-29 10:36 EDT-------
@pride I'm getting the above issue with the librpmem tests. I've tried testing
the upstream code and the issue still stays on Ubuntu. However I don't get this on other distros, so I'm suspecting it's some libfabric issue.

rpmem_fip/TEST0: SETUP (check/none/nondebug/sockets/GPSPM)
A process has executed an operation involving a call
to the fork() system call to create a child process.

As a result, the libfabric EFA provider is operating in
a condition that could result in memory corruption or
other system errors.

For the libfabric EFA provider to work safely when fork()
is called, you will need to set the following environment
variable:
RDMAV_FORK_SAFE

However, setting this environment variable can result in
signficant performance impact to your application due to
increased cost of memory registration.

You may want to check with your application vendor to see
if an application-level alternative (of not using fork)
exists.

Your job will now abort.

Thanks @lamm for testing. Just to be sure: is log excerpt you just pasted a new issue which is not present in the version currently in Hirsute? In other words, does it look like a regression introduced by the patches which are included in the PPA package?

I tried reproducing the issue by recompiling the source package from my PPA and from the current tip of pmdk git on an up-to-date Hirsute container running on:

$ arch
ppc64le

$ uname -a
Linux paride-h 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:17:52 UTC 2021 ppc64le ppc64le ppc64le GNU/Linux

A simple "manual" build+test run done with:

 - make
 - make test

seems to fully succeed. Possible caveats of my approach:

 - It's a container
 - Tests not running as root (but doesn't seem to be needed)
 - No real persistent memory device present

Do you have any suggestion on how I can reproduce the "rpmem_fip/TEST0" issue you are hitting? Thanks!

------- Comment From <email address hidden> 2021-07-30 10:31 EDT-------
> Do you have any suggestion on how I can reproduce the "rpmem_fip/TEST0"
> issue you are hitting? Thanks!

Hi @paride. I spent some time trying to identify the issue. Indeed it occurs with
the provided package. Here is some steps to test it:

1. Install the provided librpmem and libfabric
2. Clone PMDK repository https://github.com/pmem/pmdk.git
3. Checkout to branch stable-1.10
4. Compile
5. Configure src/test/testconfig.sh. I will put an example on the end of the message
- To do that copy the testconfig.sh.example to testconfig.sh. Fill it with the configurations for your system.
- Be sure to set the variable PMDK_LIB_PATH_NONDEBUG with the path to librpmem.so.
- As you are not on a DAX device use PMEM_FS_DIR_FORCE_PMEM=1.
- As this will be a librpmem test we need to configure the node variables. Also ssh keys must be configured to access the nodes without password. For example, using the testconfig.sh bellow you will need to run "ssh-copy-id 127.0.0.1"
6. Inside the src/test/ directory run "./RUNTESTS -b nondebug rpmem_fip -s TEST0". This will run only one rpmem_fip/TEST0 for nondebug. It's enough to reproduce the
error

Here is an example of testconfig.sh:

PMEM_FS_DIR=/tmp/pmem-fs.d
NON_PMEM_FS_DIR=/tmp/non-pmem-fs.d
PMEM_FS_DIR_FORCE_PMEM=1
PMDK_LIB_PATH_NONDEBUG=/usr/lib/powerpc64le-linux-gnu/
TEST_BUILD="nondebug"
TEST_TIMEOUT=6m
TM=1
KEEP_GOING=y
CLEAN_FAILED=y
UNITTEST_LOG_LEVEL=0
UNITTEST_LOG_LEVEL=1
NODE[0]=127.0.0.1
NODE[1]=127.0.0.1
NODE[2]=127.0.0.1
NODE[3]=127.0.0.1
NODE_ADDR[0]=127.0.0.1
NODE_ADDR[1]=127.0.0.1
NODE_ADDR[2]=127.0.0.1
NODE_ADDR[3]=127.0.0.1
NODE_WORKING_DIR[0]=/tmp/node0
NODE_WORKING_DIR[1]=/tmp/node1
NODE_WORKING_DIR[2]=/tmp/node2
NODE_WORKING_DIR[3]=/tmp/node3
NODE_ENV[0]="PMEM_IS_PMEM_FORCE=1"
NODE_ENV[1]="PMEM_IS_PMEM_FORCE=1"
NODE_ENV[2]="PMEM_IS_PMEM_FORCE=1"
NODE_ENV[3]="PMEM_IS_PMEM_FORCE=1"
TEST_PROVIDERS=sockets

Thanks, I'll follow the steps you provided and see if I can reproduce the issue.

Hi @lamm, I managed to reproduce the "fork() system call" failure you described. For some reason my /tmp/node* directories do not get automatically created/populated, despite testconfig.sh.example saying that "they will be created". However by manually populating them with the required test files I got the save error message as yours.

Here is my take on it:

1. You are right saying that libfabric is involved. That error message has been introduced in this libfabric commit:

  https://github.com/ofiwg/libfabric/commit/b40ce3531dcfc79f3356e2c01701058a8e2ef4f4

AIUI that commit disabled the default usage of a fork-safety mode, as it affected performance and was imperfect in any case. The suggestion is to force a (different/better?) fork-safe mode by setting RDMAV_FORK_SAFE=1.

2. Apparently rpmem_fip/TEST0 has fork() calls, and thus triggers the warning and abort() implemented in that b40ce35 commit. Setting RDMAV_FORK_SAFE=1, i.e. setting the following in testconfig.sh:

NODE_ENV[0]="PMEM_IS_PMEM_FORCE=1 RDMAV_FORK_SAFE=1"

(and similar for the other NODEs) makes rpmem_fip/TEST0 pass for me. So this is probably not a bug in libfabric and strictly speaking not a bug in pmdk, but the pmdk tests may need to be updated to work by default with the newer versions of libfabric. It may be worth filing an upstream pmdk bug.

3. Commit b40ce35 was first released in libfabric 1.11.0, which was first released in Hirsute. This is consistent with the fact that you were not seeing that failure in the pre-Hirsute Ubuntu releases. It is worth checking which version of libfabric is in the other distros you tested. If it's < 1.11.0 then it's all consistent.

4. If we agree we don't have a regression in pmdk here, let's go back to the issue with RPMEM_RAW_BUFF_SIZE and LANE_ALIGN_SIZE. That issue should be fixed in the test packages in my PPA (https://launchpad.net/~paride/+archive/ubuntu/pmdk-lp1931063). By setting RDMAV_FORK_SAFE=1 you should be able to verify, so we can proceed with the SRU for Hirsute.

Let me know WDYT, and thanls again for the feedback and testing.

------- Comment From <email address hidden> 2021-08-30 15:31 EDT-------
Hi @paride,

@lamm is out of the office, so let me try to reply to your comment.

Nice catch!
I confirmed the tests do pass with RDMAV_FORK_SAFE=1.

I also verified the fix has been integrated and the issue has been fixed.

With that said, I'm closing the issue on my side.

Thanks!

Hello bugproxy, or anyone else affected,

Accepted pmdk into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/pmdk/1.10-1ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hi @tulioqm and @lamm,

The package in hirsute-proposed requiring verification is the same published in the test PPA above, modulo a change in the version string. Even if they should be otherwise identical please make sure to test the package from hirsute-proposed (version: 1.10-1ubuntu1.1) and not the PPA one.

I gave a shot at the verification locally and it LGTM, but given that the issue this bug is about is quite specific I'll wait for your verification results before marking the hirsute-proposed package as verification-done. Thanks!

------- Comment From <email address hidden> 2021-09-08 14:19 EDT-------
> The package in hirsute-proposed requiring verification is the same published
> in the test PPA above, modulo a change in the version string. Even if they
> should be otherwise identical please make sure to test the package from
> hirsute-proposed (version: 1.10-1ubuntu1.1) and not the PPA one.
>
> I gave a shot at the verification locally and it LGTM, but given that the
> issue this bug is about is quite specific I'll wait for your verification
> results before marking the hirsute-proposed package as verification-done.

I've also tested version 1.10-1ubuntu1.1 and it LGTM too.

Thanks!

@tulioqm thank you for the verification (I'm adjusting the tag accordingly ...)

This bug was fixed in the package pmdk - 1.10-1ubuntu1.1

---------------
pmdk (1.10-1ubuntu1.1) hirsute; urgency=medium

  * d/p/lp1931063-rpmem-Fix-RPMEM_RAW_BUFF_SIZE-and-LANE_ALIGN_SIZE.patch:
    Fix RPMEM_RAW_BUFF_SIZE and LANE_ALIGN_SIZE for ppc64el (LP: #1931063)

 -- Paride Legovini <email address hidden> Mon, 26 Jul 2021 14:17:53 +0200

The verification of the Stable Release Update for pmdk has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Upstream mentions the RPMEM_RAW_BUFF_SIZE and LANE_ALIGN_SIZE fix as part of the 1.11.1 release in the release notes [1], however the relevant commits that I mentioned in comment 6 are already in 1.11.0 (I had a look at the actual source package for 1.11.0-2), so I'm still convinced this is Fix Released in Impish.

[1] https://github.com/pmem/pmdk/releases/tag/1.11.1

------- Comment From <email address hidden> 2021-10-04 09:10 EDT-------
(In reply to comment #27)
> Upstream mentions the RPMEM_RAW_BUFF_SIZE and LANE_ALIGN_SIZE fix as part of
> the 1.11.1 release in the release notes [1], however the relevant commits
> that I mentioned in comment 6 are already in 1.11.0 (I had a look at the
> actual source package for 1.11.0-2), so I'm still convinced this is Fix
> Released in Impish.

Indeed, the commits are on upstream 1.11.0. The release notes are mistaken. I agree this is Fix Released in Impish.