error overlay network swarm could not open moddep file

Bug #1618283 reported by Benoit
24
This bug affects 5 people
Affects Status Importance Assigned to Milestone
docker.io (Ubuntu)
Fix Released
High
Unassigned
Xenial
Won't Fix
Undecided
Unassigned

Bug Description

Distributor ID: Ubuntu
Description: Ubuntu 16.04.1 LTS
Release: 16.04
Codename: xenial

I am using LXC/LXD
The output of "lxc info" or if that fails:

Kernel version: 4.4.0-28-generic
LXC version:
lxc --version
2.0.3

LXD version:
lxd --version
2.0.2

Storage backend in use:
ZFS

Docker.io Version : 1.11 and 1.12 from ubuntu (same issue)

# Issue description
Docker try to load extra module from kernel, already loaded and can't use the one already loaded

# Error from docker.io

....WARN[0001] Running modprobe bridge br_netfilter failed with message: modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.4.0-28-generic/modules.dep.bin'
modprobe: WARNING: Module bridge not found in directory /lib/modules/4.4.0-28-generic
modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.4.0-28-generic/modules.dep.bin'
modprobe: WARNING: Module br_netfilter not found in directory /lib/modules/4.4.0-28-generic
, error: exit status 1

The problem is only with a swarm cluster using overlay network :

docker: Error response from daemon: Error response from daemon: error creating external connectivity network: cannot restrict inter-container communication: please ensure that br_netfilter kernel module is loaded.

Regular docker image run without problem .

root@dock-m1:~# docker -H tcp://$MANAGER_HOST_IP:22222 network create --driver overlay swarm-network

root@dock-m1:~# docker -H tcp://192.168.0.1:22222 network ls
NETWORK ID NAME DRIVER
f7c17bea2fce dock-m1/bridge bridge
3013b5577335 dock-m1/host host
a16b770083ba dock-m1/none null
6eeef94aeb4d dock-w1/bridge bridge
b2dd89a7f77f dock-w1/host host
3713d6178422 dock-w1/none null
ab0e87a92785 swarm-network overlay

docker -H tcp://$MANAGER_HOST_IP:22222 run -itd --net swarm-network -e constraint:node==dock-m1 --name u1 ubuntu
afad3f266619e44ce9ae4b6318836e6a28a7c90ef8445c0e79523ad0f371113d
docker: Error response from daemon: Error response from daemon: error creating external connectivity network: cannot restrict inter-container communication: please ensure that br_netfilter kernel module is loaded.

`root@dock-m1:~# lsmod |grep nf_nat
nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE
nf_nat_ipv4 16384 1 iptable_nat
nf_nat 24576 3 nf_nat_ipv4,xt_nat,nf_nat_masquerade_ipv4
nf_conntrack 106496 6 openvswitch,nf_nat,nf_nat_ipv4,xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4

root@dock-m1:~# lsmod |grep br_net
br_netfilter 24576 0
bridge 126976 1 br_netfilter

Joshua Powers (powersj)
Changed in docker.io (Ubuntu):
status: New → Triaged
importance: Undecided → High
Revision history for this message
Stéphane Graber (stgraber) wrote :

This was discussed with the LXD team here: https://github.com/lxc/lxd/issues/2321

This feels like a Docker bug to us, where it's attempting to load kernel modules regardless of them already being loaded, then failing because modprobe in a container can't see the .ko

Revision history for this message
RNZ (renoizer) wrote :

Same issue on debian 8.6 (jessie) with kernel proxmox 4.4.24-1-pve under lxc (2).
In this kernel, modules "br_netfilter" and "bridge" compiled with kernel.

# docker network create -d overlay --subnet=192.168.0.0/16 --gateway=192.168.0.1 --ip-range=192.168.1.0/24 multihost-network1
# docker service create --name testnginx --replicas 3 --network multihost-network1 nginx

docker Failed joining ingress sandbox to ingress endpoint: error creating external connectivity network: cannot restrict inter-container communication: please ensure that br_netfilter kernel module is loaded

Revision history for this message
Athos Ribeiro (athos-ribeiro) wrote :

It seems this bug has been fixed in https://github.com/moby/libnetwork/pull/1502, which is available since Docker 1.13. Therefore, this has been fixed since bionic.

I tried to reproduce the issue with

$ lxc launch ubuntu-daily:jammy -c security.nesting=true docker-j
$ lxc exec docker-j -- bash
# apt update && apt install -y docker.io
# docker swarm init
# docker network create -d overlay multihost-network-j
# docker service create --name testnginx --replicas 3 --network multihost-network-j nginx
# docker service ps testnginx

The results were the same for bionic as well.

Since this seems to have been fixed a while ago, and that I cannot reproduce the issue, I will mark this bug as fixed. If I have missed any steps on reproducing this issue and you believe this is still a bug to be fixed in docker.io, please, reset the bug status to "new".

Changed in docker.io (Ubuntu Xenial):
status: New → Won't Fix
Changed in docker.io (Ubuntu):
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.