Multihost Networking

As we scale out with containers, we would need containers from one host to talk to containers from other host. There are few solutions, which are tying to solve this like Flannel, Weave, Calico etc.

With container commnunication we would also need see how containers discover each other and for that we need some kind of container service discovery mmechanism, which can be achived by some form shared key-value store. For example Flannel uses etcd as kv store.

Having lots different options are good but they does not provide very good user experience. With libnetwork, Docker is aiming to provide standard interface to connect containers and satisfy composible needs. libnetwork library can be used independent of Docker.

The Container Network Model

Libnetwork implements Container Network Model (CNM) which formalizes the steps required to provide networking for containers while providing an abstraction that can be used to support multiple network drivers. The CNM is built on 3 main components.


A Sandbox contains the configuration of a container’s network stack. This includes management of the container’s interfaces, routing table and DNS settings. An implementation of a Sandbox could be a Linux Network Namespace, a FreeBSD Jail or other similar concept. A Sandbox may contain many endpoints from multiple networks.


An Endpoint joins a Sandbox to a Network. An implementation of an Endpoint could be a veth pair, an Open vSwitch internal port or similar. An Endpoint can belong to only one network but may only belong to one Sandbox.


A Network is a group of Endpoints that are able to communicate with each-other directly. An implementation of a Network could be a Linux bridge, a VLAN, etc. Networks consist of many endpoints.


Group of Docker hosts participate it multihost networking.

Libnetwork Drivers


The null driver is a noop implementation of the driver API, used only in cases where no networking is desired. This is to provide backward compatibility to the Docker’s --net=none option.


The bridge driver provides a Linux-specific bridging implementation based on the Linux Bridge.


The overlay driver implements networking that can span multiple hosts using overlay network encapsulations such as VXLAN.


The remote package does not provide a driver, but provides a means of supporting drivers over a remote transport. This can be used to write third party networking plugins for Docker.

Service Discovery

Libnetwork provides native service discovery using standard DNS protocols. It is also independent of network drivers.


For this demo we would use overlay driver and configure overlay network between two systems. We would use consul as kv pair here. Any other kv pair like etcd can be used. Once configured we would access container on one system from contaier on other system.

We would see that container can reach out to each other with IP address and service name. We would also collect the tcpdump output on one of container host machine and see how VXLAN packet look like.

[root@lab-vm-1 ~] iptables -F; systemctl stop docker
[root@lab-vm-2 ~] iptables -F; systemctl stop docker
[root@lab-vm-1 ~] ./consul agent -server -bootstrap -data-dir /tmp/consul -bind=> /dev/null 2>&1 &
[root@lab-vm-2 ~] ./consul agent -data-dir /tmp/consul -bind > /dev/null 2>&1 &
[root@lab-vm-2 ~] ./consul join
[root@lab-vm-1 ~] ./docker-latest -d --kv-store=consul:localhost:8500 > /dev/null 2>&1 &
[root@lab-vm-2 ~]./docker-latest -d --kv-store=consul:localhost:8500 > /dev/null 2>&1 &
[root@lab-vm-1 ~] ./docker-latest run -itd --name container_node1 
[root@lab-vm-2 ~]./docker-latest run -itd --name container_node2
./docker-latest network ls
./docker-latest service ls
[root@lab-vm-2 ~] ./docker-latest exec -it container_node2 bash
[root@31171f3e1da0 /]# cat /etc/hosts

ping svc1

[root@lab-vm-1 ~]# tcpdump -i eth1 

Now lets try to dig a bit and look at how things work in background.

[root@lab-vm-1 ~] ./docker-latest network ls
[root@lab-vm-1 ~]  cd /var/run/docker/netns/
[root@lab-vm-1 ~]  ls
[root@lab-vm-1 ~] nsenter --net=2-3aa4e1aa64 ip link show
[root@lab-vm-1 ~] nsenter --net=1-3aa4e1aa64 ip neigh show
[root@lab-vm-2 ~]  ./docker-latest exec -it container_node2 bash
[root@31171f3e1da0 /]# ip a 
[root@lab-vm-1 ~] nsenter --net=1-3aa4e1aa64  bridge fdb show