package centos-stream-release-9.0-8.el9.noarch (which is newer than centos-stream-release-9.0-6.el9.noarch) is already installed broke cs9 content provider

Bug #1958202 reported by chandan kumar
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Unassigned

Bug Description

CS9 content provider job is broken currently with following error while building the base container.

```
Last metadata expiration check: 0:01:21 ago on Tue 18 Jan 2022 03:13:29 AM UTC.
centos-stream-release-9.0-6.el9.noarch.rpm 7.9 MB/s | 26 kB 00:00
warning: centos-stream-release-9.0-6.el9.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID 8483c65d: NOKEY
Verifying... ########################################
Preparing... ########################################
 package centos-stream-release-9.0-8.el9.noarch (which is newer than centos-stream-release-9.0-6.el9.noarch) is already installed
time="2022-01-18T03:14:50Z" level=debug msg="\"/var/tmp/buildah894016426/mnt/buildah-bind-target-12\" is apparently not really mounted, skipping"
time="2022-01-18T03:14:50Z" level=debug msg="\"/var/tmp/buildah894016426/mnt/buildah-bind-target-11\" is apparently not really mounted, skipping"
time="2022-01-18T03:14:50Z" level=debug msg="\"/var/tmp/buildah894016426/mnt/buildah-bind-target-7\" is apparently not really mounted, skipping"
time="2022-01-18T03:14:50Z" level=debug msg="\"/var/tmp/buildah894016426/mnt/buildah-bind-target-3\" is apparently not really mounted, skipping"
time="2022-01-18T03:14:50Z" level=debug msg="\"/var/tmp/buildah894016426/mnt/rootfs\" is apparently not really mounted, skipping"
time="2022-01-18T03:14:50Z" level=debug msg="\"/var/tmp/buildah894016426/mnt\" is apparently not really mounted, skipping"
time="2022-01-18T03:14:50Z" level=debug msg="error building at step {Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin LANG=en_US.UTF-8 container=oci] Command:run Args:[if [ 'centos' == 'centos' ];then dnf -y install dnf-plugins-core && dnf download centos*release && rpm -ivh --nodeps --replacepkgs --replacefiles *.rpm && if [ -n \"$(rpm -qa redhat-release)\" ];then rpm -e --nodeps redhat-release; fi && rm -rf *.rpm; fi] Flags:[] Attrs:map[] Message:RUN if [ 'centos' == 'centos' ];then dnf -y install dnf-plugins-core && dnf download centos*release && rpm -ivh --nodeps --replacepkgs --replacefiles *.rpm && if [ -n \"$(rpm -qa redhat-release)\" ];then rpm -e --nodeps redhat-release; fi && rm -rf *.rpm; fi Original:RUN if [ 'centos' == 'centos' ];then dnf -y install dnf-plugins-core && dnf download centos*release && rpm -ivh --nodeps --replacepkgs --replacefiles *.rpm && if [ -n \"$(rpm -qa redhat-release)\" ];then rpm -e --nodeps redhat-release; fi && rm -rf *.rpm; fi}: error while running runtime: exit status 1"
```

Logs:
https://563468f00e94e2338fc4-0e05369923f9d9dc83501cb0c8bf10ab.ssl.cf2.rackcdn.com/824705/5/gate/tripleo-ci-centos-9-content-provider/b9a1676/logs/undercloud/home/zuul/workspace/logs/container-builds/94ba19ac-ea0c-44c6-abe9-481819a3b861/base/base-build.log

and job status:
https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-9-content-provider&skip=0

As per the code
https://opendev.org/openstack/tripleo-common/src/branch/master/container-images/tcib/base/base.yaml#L50
```
- run: >-
    if [ '{{ tcib_distro }}' == 'centos' ];then dnf -y install dnf-plugins-core &&
    dnf download centos*release &&
    rpm -ivh --nodeps --replacepkgs --replacefiles *.rpm &&
    if [ -n "$(rpm -qa redhat-release)" ];then rpm -e --nodeps redhat-release; fi &&
    rm -rf *.rpm; fi
```
dnf download centos*release is pulling older version of centos-stream-release and higher version of that package is already installed which is causing the breakage.

Changed in tripleo:
status: Triaged → In Progress
Revision history for this message
chandan kumar (chkumar246) wrote :

https://logserver.rdoproject.org/48/38248/2/check/tripleo-ci-centos-8-content-provider/9e9aa3c/logs/container-builds/baa23530-1c1a-438f-a010-2bf9509a6962/base/base-build.log

```
level=debug msg="closing stdin"
Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

Last metadata expiration check: 0:02:39 ago on Tue 18 Jan 2022 07:32:38 AM UTC.
Error:
 Problem: The operation would result in removing the following protected packages: redhat-release
(try to add '--skip-broken' to skip uninstallable packages)
level=debug msg="error building at step {Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin container=oci LANG=en_US.UTF-8] Command:run Args:[if [ 'centos' == 'centos' ];then dnf -y install dnf-plugins-core centos-stream-release && if [ -n \"$(rpm -qa redhat-release)\" ];then rpm -e --nodeps redhat-release; fi && rm -rf *.rpm; fi] Flags:[] Attrs:map[] Message:RUN if [ 'centos' == 'centos' ];then dnf -y install dnf-plugins-core centos-stream-release && if [ -n \"$(rpm -qa redhat-release)\" ];then rpm -e --nodeps redhat-release; fi && rm -rf *.rpm; fi Original:RUN if [ 'centos' == 'centos' ];then dnf -y install dnf-plugins-core centos-stream-release && if [ -n \"$(rpm -qa redhat-release)\" ];then rpm -e --nodeps redhat-release; fi && rm -rf *.rpm; fi}: error while running runtime: exit status 1"
```

Revision history for this message
chandan kumar (chkumar246) wrote :

on CS8:
```
[root@45edcd50c57a /]# dnf whatprovides redhat-release
Last metadata expiration check: 0:00:29 ago on Tue Jan 18 06:21:16 2022.
centos-stream-release-8.5-2.el8.noarch : CentOS Stream release files
Repo : baseos
Matched from:
Provide : redhat-release = 8.5-2.el8

centos-stream-release-8.5-3.el8.noarch : CentOS Stream release files
Repo : baseos
Matched from:
Provide : redhat-release = 8.5-3.el8

centos-stream-release-8.6-1.el8.noarch : CentOS Stream release files
Repo : @System
Matched from:
Provide : redhat-release = 8.6-1.el8

centos-stream-release-8.6-1.el8.noarch : CentOS Stream release files
Repo : baseos
Matched from:
Provide : redhat-release = 8.6-1.el8

```

and on CS9
```
[root@0c05c37ff8f5 /]# dnf whatprovides redhat-release
CentOS Stream 9 - BaseOS 91 kB/s | 5.1 MB 00:57
CentOS Stream 9 - AppStream 124 kB/s | 10 MB 01:24
CentOS Stream 9 - Extras packages 40 B/s | 257 B 00:06
centos-stream-release-9.0-2.el9.noarch : CentOS Stream release files
Repo : baseos
Matched from:
Provide : redhat-release = 9.0-2.el9

centos-stream-release-9.0-3.el9.noarch : CentOS Stream release files
Repo : baseos
Matched from:
Provide : redhat-release = 9.0-3.el9

centos-stream-release-9.0-4.el9.noarch : CentOS Stream release files
Repo : baseos
Matched from:
Provide : redhat-release = 9.0-4.el9

centos-stream-release-9.0-5.el9.noarch : CentOS Stream release files
Repo : baseos
Matched from:
Provide : redhat-release = 9.0-5.el9

centos-stream-release-9.0-6.el9.noarch : CentOS Stream release files
Repo : baseos
Matched from:
Provide : redhat-release = 9.0-6.el9

```

Revision history for this message
chandan kumar (chkumar246) wrote :

[chandankumar@fedora tcib]$ podman run -it cc0656847854
[root@5692667c8df4 /]# cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="8.5 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.5"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.5 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.5
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.5"
[root@5692667c8df4 /]# rpm -e --nodeps redhat-release
[root@5692667c8df4 /]# cat /etc/os-release
cat: /etc/os-release: No such file or directory
[root@5692667c8df4 /]#

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)
Revision history for this message
Ronelle Landy (rlandy) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-common (master)

Reviewed: https://review.opendev.org/c/openstack/tripleo-common/+/825038
Committed: https://opendev.org/openstack/tripleo-common/commit/b09fb3fef0896e812ca9ce3e6dc74f297494203a
Submitter: "Zuul (22348)"
Branch: master

commit b09fb3fef0896e812ca9ce3e6dc74f297494203a
Author: Chandan Kumar (raukadah) <email address hidden>
Date: Tue Jan 18 11:20:07 2022 +0530

    Reinstall centos-stream-release from centos repos

    In CentOS Stream 8 and 9 container image, centos-stream-release
    package is already available in the container image. But in UBI,
    it is not available. It can be installed from the base OS repo.

    It removes the logic of downloading the centos-stream-release
    rpm and installing it. As it was added earlier when we were
    adding the initial support for EL9.

    since redhat-release is a part of centos-stream-release
    In UBI 8, redhat-release is a seperate package, so it's rpm
    needs to be removed otherwise it will end up with skip broken
    package.

    On removing redhat-release, /etc/os-release file also gets deleted.
    So we need to first remove redhat-release rpm if present and then
    install centos-stream-release package.

    It also removes dnf-core-plugins package installation as we donot
    need to download the centos-stream-release rpm.

    Closes-Bug: #1958202
    Related-Bug: #1958240

    Signed-off-by: Chandan Kumar (raukadah) <email address hidden>
    Change-Id: Ia06c6770ea567656b5ce380826bdb120ca32a1a8

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-common (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-common/+/831580

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-common (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/tripleo-common/+/831580
Committed: https://opendev.org/openstack/tripleo-common/commit/3db8093d0c54db9cd0fae6df851402f270bcf35d
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 3db8093d0c54db9cd0fae6df851402f270bcf35d
Author: Chandan Kumar (raukadah) <email address hidden>
Date: Tue Jan 18 11:20:07 2022 +0530

    Reinstall centos-stream-release from centos repos

    In CentOS Stream 8 and 9 container image, centos-stream-release
    package is already available in the container image. But in UBI,
    it is not available. It can be installed from the base OS repo.

    It removes the logic of downloading the centos-stream-release
    rpm and installing it. As it was added earlier when we were
    adding the initial support for EL9.

    since redhat-release is a part of centos-stream-release
    In UBI 8, redhat-release is a seperate package, so it's rpm
    needs to be removed otherwise it will end up with skip broken
    package.

    On removing redhat-release, /etc/os-release file also gets deleted.
    So we need to first remove redhat-release rpm if present and then
    install centos-stream-release package.

    It also removes dnf-core-plugins package installation as we donot
    need to download the centos-stream-release rpm.

    Closes-Bug: #1958202
    Related-Bug: #1958240
    Closes-Bug: #1962786

    Signed-off-by: Chandan Kumar (raukadah) <email address hidden>
    Change-Id: Ia06c6770ea567656b5ce380826bdb120ca32a1a8
    (cherry picked from commit b09fb3fef0896e812ca9ce3e6dc74f297494203a)

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-common 16.4.0

This issue was fixed in the openstack/tripleo-common 16.4.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.