/run/netns doesn't exist until we create a network namespace

Bug #1799638 reported by Cédric Jeanneret
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Cédric Jeanneret

Bug Description

Hello,

Apparently the /run/netns directory doesn't exist until a network namespace is created. This prevent podman containers to start properly after a system reboot.

For the records, podman doesn't create the host directory tree when we mount a volume in a container. Hence, as /run/netns doesn't exist, podman crashes when it tries to start containers having that location bind-mounted.

A possible way to mitigate it would be to add a "ExecPreStart=ip netns add placeholder" or something like that in the systemd unit for container having that mount.

Or create a dedicated unit for that, and make the container depend on that unit.

Revision history for this message
Cédric Jeanneret (cjeanner) wrote :
Changed in tripleo:
assignee: nobody → Cédric Jeanneret (cjeanner)
status: Triaged → In Progress
Revision history for this message
Cédric Jeanneret (cjeanner) wrote :

Fix merged.

Changed in tripleo:
status: In Progress → Fix Committed
Changed in tripleo:
status: Fix Committed → Fix Released
Revision history for this message
Rabi Mishra (rabi) wrote :

This isn't fixed yet:/

Please check the errors[1] in https://review.openstack.org/#/c/630631/. It was working earlier as we were creating containers that were using podman networking (cni0). Once I changed them to use net: none, the containers that mount /run/netns started to fail.

[1] http://logs.openstack.org/31/630631/4/check/tripleo-ci-centos-7-undercloud-containers/73d9cbf/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2019-01-14_15_05_47

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (master)

Reviewed: https://review.openstack.org/630936
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=9b284e74cbdcf43253edaf424534b71a483fdb69
Submitter: Zuul
Branch: master

commit 9b284e74cbdcf43253edaf424534b71a483fdb69
Author: Rabi Mishra <email address hidden>
Date: Tue Jan 15 16:13:33 2019 +0530

    Create /run/netns if does not exist

    Unlike docker with podman you can't mount a host directory
    that does not exist. We did some work as part of bug 1799638
    take care of this for reboots. However, we need it to be created
    for fresh installations, as nodes are not necessarily rebooted
    after installing paunch. The real issue without reboots did not
    surface before the next patch in this series, as we were using
    cni0 bridge network for some containers and that resulted in
    namespace being created and the directory existing for
    neutron_dhcp and neutron_l3 containers in step 4 to mount them.

    This patch creates/removes a temp namespace in host_prep_tasks
    to ensure that the directory exists.

    Change-Id: I19d660168c98887a5e352b3413235888c800760d
    Related-Bug: #1799638

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (stable/rocky)

Related fix proposed to branch: stable/rocky
Review: https://review.opendev.org/673797

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (stable/queens)

Related fix proposed to branch: stable/queens
Review: https://review.opendev.org/675359

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (stable/rocky)

Reviewed: https://review.opendev.org/673797
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=12ef8149d58d1fcb4426b65fdf09c283f3345e5c
Submitter: Zuul
Branch: stable/rocky

commit 12ef8149d58d1fcb4426b65fdf09c283f3345e5c
Author: Rabi Mishra <email address hidden>
Date: Tue Jan 15 16:13:33 2019 +0530

    Create /run/netns if does not exist

    Unlike docker with podman you can't mount a host directory
    that does not exist. We did some work as part of bug 1799638
    take care of this for reboots. However, we need it to be created
    for fresh installations, as nodes are not necessarily rebooted
    after installing paunch. The real issue without reboots did not
    surface before the next patch in this series, as we were using
    cni0 bridge network for some containers and that resulted in
    namespace being created and the directory existing for
    neutron_dhcp and neutron_l3 containers in step 4 to mount them.

    This patch creates/removes a temp namespace in host_prep_tasks
    to ensure that the directory exists.

    Change-Id: I19d660168c98887a5e352b3413235888c800760d
    Related-Bug: #1799638
    (cherry picked from commit 9b284e74cbdcf43253edaf424534b71a483fdb69)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (stable/queens)

Reviewed: https://review.opendev.org/675359
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=5d00c7076f6dc6b5b65b818358d27d5cce8a5764
Submitter: Zuul
Branch: stable/queens

commit 5d00c7076f6dc6b5b65b818358d27d5cce8a5764
Author: Rabi Mishra <email address hidden>
Date: Tue Jan 15 16:13:33 2019 +0530

    Create /run/netns if does not exist

    Unlike docker with podman you can't mount a host directory
    that does not exist. We did some work as part of bug 1799638
    take care of this for reboots. However, we need it to be created
    for fresh installations, as nodes are not necessarily rebooted
    after installing paunch. The real issue without reboots did not
    surface before the next patch in this series, as we were using
    cni0 bridge network for some containers and that resulted in
    namespace being created and the directory existing for
    neutron_dhcp and neutron_l3 containers in step 4 to mount them.

    This patch creates/removes a temp namespace in host_prep_tasks
    to ensure that the directory exists.

    Change-Id: I19d660168c98887a5e352b3413235888c800760d
    Related-Bug: #1799638
    (cherry picked from commit 9b284e74cbdcf43253edaf424534b71a483fdb69)

tags: added: in-stable-queens
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.