tripleo

podman jobs failing after new podman(0.12.1) and container-selinux(2.77) are released

Bug #1809218 reported by yatin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
In Progress
Undecided
yatin

Bug Description

jobs running with podman are failing with below Error after new podman and container-selinux hit buildlogs:-

relabel failed "/run/netns": operation not supported

Example log:- http://logs.openstack.org/56/625256/8/check/tripleo-ci-centos-7-standalone/eae540d/logs/undercloud/var/log/containers/neutron/dhcp-agent.log.txt.gz#_2018-12-20_02_01_11_029

Logstash query:- http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22relabel%20failed%20%5C%22%5C%2Frun%5C%2Fnetns%5C%22:%20operation%20not%20supported%5C%22

This is hitting in check/gate and are multiple occurrence for this, hence it's critical.

See original description
Tags: alert ci
yatin (yatinkarel)
tags: added: alert ci
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.openstack.org/626511

Changed in tripleo:
assignee: nobody → yatin (yatinkarel)
status: New → In Progress
Revision history for this message
Marios Andreou (marios-b) wrote :

adding a note because this is super confusing to debug.

In the console you can see failing tempest but nothing earlier e.g. in https://review.openstack.org/#/c/625256/ (promotion blocker for bug #1808349 in both container-multinode and standalone, failed tempest [1][2] but you have to go looking in errors.txt [3] to find the bug here like

             2018-12-20 01:46:11.099 ERROR /var/log/containers/neutron/dhcp-agent.log.1: 82923 ERROR neutron.agent.dhcp.agent relabel failed "/run/netns": operation not supported

[1] http://logs.openstack.org/56/625256/8/check/tripleo-ci-centos-7-standalone/eae540d/logs/undercloud/home/zuul/tempest.log.txt.gz#_2018-12-20_01_05_45

[2] http://logs.openstack.org/56/625256/8/check/tripleo-ci-centos-7-containers-multinode/09096e0/logs/undercloud/home/zuul/tempest.log.txt.gz#_2018-12-20_02_31_50
[3] http://logs.openstack.org/56/625256/8/check/tripleo-ci-centos-7-containers-multinode/09096e0/logs/undercloud/var/log/extra/errors.txt.gz#_2018-12-20_01_46_11_099

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to puppet-tripleo (master)

Fix proposed to branch: master
Review: https://review.openstack.org/626546

Changed in tripleo:
assignee: yatin (yatinkarel) → Cédric Jeanneret (cjeanner)
assignee: Cédric Jeanneret (cjeanner) → yatin (yatinkarel)
Revision history for this message
Cédric Jeanneret (cjeanner) wrote :

I just created an upstream bug: https://github.com/containers/libpod/issues/2034

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-quickstart (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/626583

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-quickstart (master)

Change abandoned by wes hayutin (<email address hidden>) on branch: master
Review: https://review.openstack.org/626583
Reason: waiting on results from https://review.openstack.org/#/c/626546/ and https://review.openstack.org/#/c/626546/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to puppet-tripleo (master)

Reviewed: https://review.openstack.org/626546
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=7034cfdb680b1aeafeae4660054c3dcaa96fedae
Submitter: Zuul
Branch: master

commit 7034cfdb680b1aeafeae4660054c3dcaa96fedae
Author: Cédric Jeanneret <email address hidden>
Date: Thu Dec 20 12:03:44 2018 +0100

    Temporary remove selinux label mount option for neutron

    Neutron services failing with below Error when running with podman(0.12.1)
    and container-selinux(2.77):-

    relabel failed "/run/netns": operation not supported

    Until this is fixed in podman/container-selinux, temporary remove selinux
    relabel on /run/netns.

    Change-Id: I596074fcc2318ebb3d7efb0128a2b25527e19808
    Partial-Bug: #1809218

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.openstack.org/626511
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=c3bd127421eca552c2693faa29f11c3457d582df
Submitter: Zuul
Branch: master

commit c3bd127421eca552c2693faa29f11c3457d582df
Author: yatin <email address hidden>
Date: Thu Dec 20 13:34:47 2018 +0530

    Temporary remove selinux label mount option for neutron

    Neutron services failing with below Error when running
    with podman(0.12.1) and container-selinux(2.77):-

    relabel failed "/run/netns": operation not supported

    Until this is fixed in podman/container-selinux, temporary
    remove selinux relabel on /run/netns.

    Depends-On: https://review.openstack.org/#/c/626546/
    Change-Id: Iedbeac17a0c530ecdc7e8cbba5ddd4ffb22bb616
    Partial-Bug: #1809218

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-quickstart (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/626953

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-quickstart (master)

Reviewed: https://review.openstack.org/626583
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart/commit/?id=6822ba9a7e264f0fbc454ce830483fd5e29358db
Submitter: Zuul
Branch: master

commit 6822ba9a7e264f0fbc454ce830483fd5e29358db
Author: Wes Hayutin <email address hidden>
Date: Thu Dec 20 06:52:10 2018 -0700

    temporarily turn off podman

    We're hitting several issues in the upstream
    ci related to podman :(

    https://bugs.launchpad.net/tripleo/+bug/1804434
    https://github.com/containers/libpod/issues/1844

    Related-Bug: #1804434
    Related-Bug: #1809218
    Change-Id: I19aa04382ba159768a1748d44412bbc670facaf3

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-quickstart (master)

Change abandoned by wes hayutin (<email address hidden>) on branch: master
Review: https://review.openstack.org/626953

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.