Docker containers suddenly using 192.168.0.0/16 instead of 172.17.0.0/16: services lost

27 November 2019 Updated 27 November 2019 by Peter
In Docker

My Docker applications suddenly could not send mail using the host Postfix MTA. I fixed this by specifying a Docker default address pool, but it is better to do this before deployment.

post main image
https://unsplash.com/@mmintel

I have an ISPConfig server with Docker applications. They use the host Postfix mail transfer agent (MTA) to deliver mail to the outside world. Before using the send mail function I have a check if Postfix can be accessed. This works fine. But suddenly mail was not sent. The log file contained error messages like:

2019-11-26 17:31:56,758 ERROR MailMessage - send_mail: self.error_message = sending message, e = {'joe@example.com': (554, b'5.7.1 <joe@example.com>: Relay access denied')}
2019-11-26 17:31:56,759 ERROR MailSend - send during send, errors = send - sending message, e = {'joe@example.com': (554, b'5.7.1 <joe@example.com>: Relay access denied')}
2019-11-26 17:31:56,759 ERROR AddonContactForm - send_contact_form: mail_sender.get_errors() = send - sending message, e = {'joe@example.com': (554, b'5.7.1 <joe@example.com>: Relay access denied')}

Relay access denied? Why? Then I remembered that Postfix has a mynetworks setting in the file:

mynetworks = 127.0.0.0/8 [::1]/128  172.16.0.0/12

This led me to the IP addresses assigned by Docker to the containers:

systemctl status docker

with result:

● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2019-10-16 21:26:07 CEST; 1 months 11 days ago
     Docs: https://docs.docker.com
 Main PID: 765 (dockerd)
    Tasks: 28
   Memory: 53.7M
      CPU: 39min 54.387s
   CGroup: /system.slice/docker.service
           ├─  765 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
           ├─ 9784 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 8001 -container-ip 192.168.32.2 -container-port 8001
           └─18316 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 8000 -container-ip 192.168.176.2 -container-port 8000

The Docker containers were suddenly using the 192 subnet instead the 172 subnet. I (still) do not understand why this happened but it had to be solved. After some searching I found the solution for the problem. From link below:

... by design we have following network range for bridge networks.
"172.17.0.0/16", "172.18.0.0/16", "172.19.0.0/16", "172.20.0.0/14", "172.24.0.0/14" "172.28.0.0/14", "192.168.0.0/16"
When we create network we check for over lapping network only within docker created networks. We have added default address pool support for bridge network. You can configure it through daemon.jason file or input param while starting Docker Daemon. In that way you can choose subnet you want dockerD to create ( default gateway or network).

I specified the default address pools for my Docker networks by creating the file (it was not there):

/etc/docker/daemon.json

with the contents:

{
	"default-address-pools":
	[
		{"base":"172.17.0.0/16","size":24}
	]
}

Then I restarted Docker:

systemctl restart docker

The containers still had 192 subnet IP addresses. Then I did for the Docker applications:

docker-compose .... down
docker-compose .... up -d

Now they had 172 subnet IP addresses again and mail was getting out again! To make sure, I also restarted the server and was happy to see everything was fine.

Some commands for testing

You can use this method, see also link below, to check if the proper network is created:

docker network create foo
docker network inspect foo | grep Subnet

Some other commands I used in the process:

route -n

docker inspect <container> 

docker network ls

docker network inspect <network>

Summary

I still do not know what caused this problem but fortunately you can specify a subnet for the default network. This really should be more emphasized when you start using Docker! You can also specify default subnets in docker-compose but I did not test this. Anyway solved.

Links / credits

Allow user to specify default address pools for docker networks #29376
https://github.com/moby/moby/pull/29376

Configuring Docker to not use the 172.17.0.0 range
https://serverfault.com/questions/916941/configuring-docker-to-not-use-the-172-17-0-0-range

Docker (compose?) suddenly creates bridge using 192.168.16.0/20 range #37823
https://github.com/moby/moby/issues/37823

How does compose chooses subnet for default network? #4336
https://github.com/docker/compose/issues/4336

networks
https://docs.docker.com/compose/compose-file/#networks