Docker containers suddenly using 192.168.0.0/16 instead of 172.17.0.0/16: services lost
My Docker applications suddenly could not send mail using the host Postfix MTA. I fixed this by specifying a Docker default address pool, but it is better to do this before deployment.
I have an ISPConfig server with Docker applications. They use the host Postfix mail transfer agent (MTA) to deliver mail to the outside world. Before using the send mail function I have a check if Postfix can be accessed. This works fine. But suddenly mail was not sent. The log file contained error messages like:
2019-11-26 17:31:56,758 ERROR MailMessage - send_mail: self.error_message = sending message, e = {'joe@example.com': (554, b'5.7.1 <joe@example.com>: Relay access denied')}
2019-11-26 17:31:56,759 ERROR MailSend - send during send, errors = send - sending message, e = {'joe@example.com': (554, b'5.7.1 <joe@example.com>: Relay access denied')}
2019-11-26 17:31:56,759 ERROR AddonContactForm - send_contact_form: mail_sender.get_errors() = send - sending message, e = {'joe@example.com': (554, b'5.7.1 <joe@example.com>: Relay access denied')}
Relay access denied? Why? Then I remembered that Postfix has a mynetworks setting in the file:
mynetworks = 127.0.0.0/8 [::1]/128 172.16.0.0/12
This led me to the IP addresses assigned by Docker to the containers:
systemctl status docker
with result:
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2019-10-16 21:26:07 CEST; 1 months 11 days ago
Docs: https://docs.docker.com
Main PID: 765 (dockerd)
Tasks: 28
Memory: 53.7M
CPU: 39min 54.387s
CGroup: /system.slice/docker.service
├─ 765 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
├─ 9784 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 8001 -container-ip 192.168.32.2 -container-port 8001
└─18316 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 8000 -container-ip 192.168.176.2 -container-port 8000
The Docker containers were suddenly using the 192 subnet instead the 172 subnet. I (still) do not understand why this happened but it had to be solved. After some searching I found the solution for the problem. From link below:
... by design we have following network range for bridge networks.
"172.17.0.0/16", "172.18.0.0/16", "172.19.0.0/16", "172.20.0.0/14", "172.24.0.0/14" "172.28.0.0/14", "192.168.0.0/16"
When we create network we check for over lapping network only within docker created networks. We have added default address pool support for bridge network. You can configure it through daemon.jason file or input param while starting Docker Daemon. In that way you can choose subnet you want dockerD to create ( default gateway or network).
I specified the default address pools for my Docker networks by creating the file (it was not there):
/etc/docker/daemon.json
with the contents:
{
"default-address-pools":
[
{"base":"172.17.0.0/16","size":24}
]
}
Then I restarted Docker:
systemctl restart docker
The containers still had 192 subnet IP addresses. Then I did for the Docker applications:
docker-compose .... down
docker-compose .... up -d
Now they had 172 subnet IP addresses again and mail was getting out again! To make sure, I also restarted the server and was happy to see everything was fine.
Some commands for testing
You can use this method, see also link below, to check if the proper network is created:
docker network create foo
docker network inspect foo | grep Subnet
Some other commands I used in the process:
route -n
docker inspect <container>
docker network ls
docker network inspect <network>
Summary
I still do not know what caused this problem but fortunately you can specify a subnet for the default network. This really should be more emphasized when you start using Docker! You can also specify default subnets in docker-compose but I did not test this. Anyway solved.
Links / credits
Allow user to specify default address pools for docker networks #29376
https://github.com/moby/moby/pull/29376
Configuring Docker to not use the 172.17.0.0 range
https://serverfault.com/questions/916941/configuring-docker-to-not-use-the-172-17-0-0-range
Docker (compose?) suddenly creates bridge using 192.168.16.0/20 range #37823
https://github.com/moby/moby/issues/37823
How does compose chooses subnet for default network? #4336
https://github.com/docker/compose/issues/4336
networks
https://docs.docker.com/compose/compose-file/#networks
Read more
Docker
Recent
- Hiding database UUID primary keys of your web application
- Don't Repeat Yourself (DRY) with Jinja2
- SQLAlchemy, PostgreSQL, maximum number of rows per user
- Show the values in SQLAlchemy dynamic filters
- Secure data transfer with Public Key encryption and pyNaCl
- rqlite: a high-availability and distributed SQLite alternative
Most viewed
- Using Python's pyOpenSSL to verify SSL certificates downloaded from a host
- Using UUIDs instead of Integer Autoincrement Primary Keys with SQLAlchemy and MariaDb
- Using PyInstaller and Cython to create a Python executable
- Connect to a service on a Docker host from a Docker container
- SQLAlchemy: Using Cascade Deletes to delete related objects
- Flask RESTful API request parameter validation with Marshmallow schemas