tripleo

container image build consistent fails because of 'Error: invalid policy in "/etc/containers/policy.json": Unknown key "keyPaths"'

Bug #1988500 reported by Takashi Kajinami
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Unassigned
tripleo zed-1

Bug Description

Description
===========
content provider jobs and standalone jobs building container images are currently broken
and fail when building container images.

example:
https://zuul.opendev.org/t/openstack/build/45ef7109ba954c8f816d38af58db987e

```
2022-09-01 22:07:21.144195 | primary | TASK [container-build : Deploy the content provider registry from quay.io/tripleoci/registry:2] ***
2022-09-01 22:07:21.144217 | primary | Thursday 01 September 2022 22:07:21 +0000 (0:00:00.125) 0:09:50.691 ****
2022-09-01 22:07:23.190442 | primary | fatal: [undercloud]: FAILED! => {"changed": true, "cmd": "podman run -d --name docker_registry -p 5001:5001 -e REGISTRY_HTTP_ADDR=\"0.0.0.0:5001\" -e REGISTRY_LOG_LEVEL=debug quay.io/tripleoci/registry:2", "delta": "0:00:00.353188", "end": "2022-09-01 22:07:22.948351", "msg": "non-zero return code", "rc": 125, "start": "2022-09-01 22:07:22.595163", "stderr": "Error: invalid policy in \"/etc/containers/policy.json\": Unknown key \"keyPaths\"", "stderr_lines": ["Error: invalid policy in \"/etc/containers/policy.json\": Unknown key \"keyPaths\""], "stdout": "", "stdout_lines": []}
2022-09-01 22:07:23.202916 | primary |
2022-09-01 22:07:23.202958 | primary | TASK [container-build : Deploy the content provider registry from trunk.registry.rdoproject.org/ceph/registry:2] ***
2022-09-01 22:07:23.202998 | primary | Thursday 01 September 2022 22:07:23 +0000 (0:00:02.058) 0:09:52.750 ****
2022-09-01 22:07:24.956537 | primary | fatal: [undercloud]: FAILED! => {"changed": true, "cmd": "podman run -d --name docker_registry -p 5001:5001 -e REGISTRY_HTTP_ADDR=\"0.0.0.0:5001\" -e REGISTRY_LOG_LEVEL=debug trunk.registry.rdoproject.org/ceph/registry:2", "delta": "0:00:00.138296", "end": "2022-09-01 22:07:24.701859", "msg": "non-zero return code", "rc": 125, "start": "2022-09-01 22:07:24.563563", "stderr": "Error: invalid policy in \"/etc/containers/policy.json\": Unknown key \"keyPaths\"", "stderr_lines": ["Error: invalid policy in \"/etc/containers/policy.json\": Unknown key \"keyPaths\""], "stdout": "", "stdout_lines": []}
2022-09-01 22:07:24.957977 | primary |
```

Takashi Kajinami (kajinamit)
Changed in tripleo:
importance: Undecided → Critical
tags: added: ci promotion-blocker
Changed in tripleo:
milestone: none → zed-1
Revision history for this message
Takashi Kajinami (kajinamit) wrote (last edit ):

The KeyPaths key is available since containers/image v5.22.0. while we have podman v4.
We somehow got that "unsupported" key which messed up the whole CI.

https://github.com/containers/image/commit/d218ff3d4611d35295615adf0913352a76684220

In the failing build

https://87ce4650c4c88cf6e719-b9786354af2400d660edf4654debede0.ssl.cf2.rackcdn.com/854307/1/gate/tripleo-ci-centos-9-content-provider/45ef710/logs/undercloud/var/log/extra/rpm-list.txt

ansible-collection-community-general-4.0.0-1.el9s.noarch
ansible-collection-containers-podman-1.9.4-1.el9s.noarch
podman-4.1.1-6.el9.x86_64

https://87ce4650c4c88cf6e719-b9786354af2400d660edf4654debede0.ssl.cf2.rackcdn.com/854307/1/gate/tripleo-ci-centos-9-content-provider/45ef710/logs/undercloud/etc/containers/policy.json

 => this uses keyPath*s*

In the last successful build

https://723be80a4a0e72ac556e-dd6141d83323607d90dfeb8c1a134184.ssl.cf5.rackcdn.com/855511/2/check/tripleo-ci-centos-9-content-provider/c82a361/logs/undercloud/var/log/extra/rpm-list.txt

ansible-collection-community-general-4.0.0-1.el9s.noarch
ansible-collection-containers-podman-1.9.4-1.el9s.noarch
podman-4.1.1-6.el9.x86_64

https://723be80a4a0e72ac556e-dd6141d83323607d90dfeb8c1a134184.ssl.cf5.rackcdn.com/855511/2/check/tripleo-ci-centos-9-content-provider/c82a361/logs/undercloud/etc/containers/policy.json

 => this uses keyPath

Revision history for this message
Rabi Mishra (rabi) wrote :

I think containers-common has been bumped to whereas we're still using old podman (that uses old containers/image).

Passing Job:

containers-common.x86_64 2:1-40.el9 @quickstart-centos-appstreams

Failed Job:

containers-common.x86_64 2:1-44.el9 @quickstart-centos-appstreams

I think we need to pin containers-common to an old version.

Revision history for this message
chandan kumar (chkumar246) wrote :

https://gitlab.com/redhat/centos-stream/rpms/containers-common/-/commit/04645c4a84442da3324eea8f6538a5768e69919a adds the new keyPath which broke the job. Pinning it to an older version sounds good.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-quickstart (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/855587

Revision history for this message
chandan kumar (chkumar246) wrote :

with latest logs https://ad4a918c918d1cf92f3d-b971135d7916cb5f7bbc35fdb125d434.ssl.cf5.rackcdn.com/855552/1/check/tripleo-ci-centos-9-content-provider/b701413/logs/undercloud/var/log/extra/rpm-list.txt

```
podman-4.2.0-3.el9.x86_64
containers-common-1-44.el9.x86_64
```
and
https://ad4a918c918d1cf92f3d-b971135d7916cb5f7bbc35fdb125d434.ssl.cf5.rackcdn.com/855552/1/check/tripleo-ci-centos-9-content-provider/b701413/logs/undercloud/home/zuul/container_image_build.log

```
ime="2022-09-02T02:03:32-04:00" level=debug msg="Pull Policy for pull [ifnewer]"\nerror creating build container: copying system image from manifest list: Source image rejected: None of the signatures were accepted, reasons: open /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta: no such file or directory; open /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta: no such file or directory; open /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta: no such file or directory; open /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta: no such file or directory; open /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta: no such file or directory; open /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta: no such file or directory\nFailed to write to log, write /home/zuul/container-builds/f8866c54-53d9-435a-903c-36a38d3df83b/base/base-build.log: file already closed\n'
```

Revision history for this message
Takashi Kajinami (kajinamit) wrote :

So the initial problem was that the new podman package podman-4.2.0-3.el9.x86_64 was not installed, likely because of delay in repository sync.
Currently we are facing the problem mentioned in comment:5.

Revision history for this message
Rabi Mishra (rabi) wrote :

Downgrading containers-common still can be a work-around as "keyPath" in policy.json would work as before.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-quickstart (master)

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart/+/855587
Committed: https://opendev.org/openstack/tripleo-quickstart/commit/b10da3f993b1be0709cfe047b292c091fa7f3554
Submitter: "Zuul (22348)"
Branch: master

commit b10da3f993b1be0709cfe047b292c091fa7f3554
Author: Chandan Kumar (raukadah) <email address hidden>
Date: Fri Sep 2 11:02:28 2022 +0530

    Downgrade containers-common to 1-40

    containers-common-1-43 adds the new keypath[1] which will
    work with latest podman[2] which is not available in
    podman-4.1.1-6. It breaks the deployment.

    Downgrading containers-common to 1-40 fixes the issue
    till we get a new podman version.

    On release file changes, wallaby jobs are failing with
    ```
    Depsolve Error occurred: \n Problem: problem with installed package catatonit-3:0.1.7-7.el9.x86_64\n
    - package podman-2:4.2.0-3.el9.x86_64 conflicts with catatonit provided by catatonit-3:0.1.7-7.el9.x86_64
    ```
    during overcloud deployment. It blocks the above changes.

    We need to revert https://review.opendev.org/c/openstack/tripleo-quickstart/+/853142
    the change in this patch itself and get this patch in.

    Links:
    [1]. https://gitlab.com/redhat/centos-stream/rpms/containers-common/-/commit/04645c4a84442da3324eea8f6538a5768e69919a
    [2]. https://github.com/containers/image/commit/d218ff3d4611d35295615adf0913352a76684220

    Related-Bug: #1988500
    Related-Bug: #1988514
    Closes-Bug: #1985981

    Signed-off-by: Chandan Kumar (raukadah) <email address hidden>
    Change-Id: Ie0aea674228f011881f42b9515a2e0a73198abed

Revision history for this message
chandan kumar (chkumar246) wrote :

After merging https://review.opendev.org/c/openstack/tripleo-quickstart/+/855587 the jobs https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-content-provider&skip=0 are back to green, we can move it to done.

Rabi Mishra (rabi)
Changed in tripleo:
status: New → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-quickstart (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/877386

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-quickstart (master)

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart/+/877386
Committed: https://opendev.org/openstack/tripleo-quickstart/commit/745df87ac51562ef65c7022c121ed1363b6535a7
Submitter: "Zuul (22348)"
Branch: master

commit 745df87ac51562ef65c7022c121ed1363b6535a7
Author: Cédric Jeanneret <email address hidden>
Date: Tue Mar 14 16:30:42 2023 +0100

    Remove downgrade/exclude of containers-common-1.44.*

    This was due to a delay in repo sync, according to the associated LP

    Change-Id: Iaeaf29e73c1ec0648423350edd1ee3a3a538b408
    Closes-Bug: #2011598
    Related-Bug: #1988500

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.