From Docker-Compose to Docker Swarm: Configs

You have an application that consists of a number of Docker-Compose projects, and are using Docker-Compose to build, start and deploy.

Now you want to go one step further and move some of the Docker-Compose projects to another server. The most obvious choice to do this, well to try first, is Docker Swarm. You learn a few more Docker commands and you're done. Is it really that easy?

Spoiler alert. No, it's not easy if you have containers that use volumes, for example, for configuration data, high performance data, persistent data. Many examples on the internet avoid this issue completely, or casually mention using shared storage solutions like NFS (not encrypted) or GlusterFS, or use storage solutions from cloud providers. All these storage solutions all have the same feature: They are network-based and very slow compared to native storage. Yes, you can go faster but you will pay more.

Before you even consider Docker Swarm, I recommend that you read about this issue to understand if your use case is suitable for Docker Swarm. For example here is a discussion about this, 'Data(base) persistence in docker swarm mode', see links below. And from this Stackoverflow answer 'How does Docker Swarm implement volume sharing?', see links below:

Swarm Mode itself does not do anything different with volumes, it runs any volume mount command you provide on the node where the container is running. If your volume mount is local to that node, then your data will be saved locally on that node. There is no built in functionality to move data between nodes automatically.

This means it is possible to prepare a directory structure on a worker node, and make sure you run a container only on this node using the deploy placement parameters. Options ...

In this post we use Docker Swarm Configs to pass (static) configuration data to containers on worker nodes. This is a bit of an anti-pattern, but the nice thing is that we eliminate a storage requirement.

As always I am running on Ubuntu 22.04. At the moment I am not using Docker Desktop. I used VirtualBox to create a Ubuntu server VM and use it as a worker. 

Docker Swarm does not manage volume data!

Docker Swarm is a services manager and services are containers (based on images), not volume data. This means you must manage the volume data yourself.

As an example, consider a DSN server container. By default, it connects to DNS service A on the internet, no external data here.With Docker Swarm, we can deploy this container, specify replicas, easy does it.

Now, we want the same DSN server container to connect to DNS service B on the internet. We follow the instructions of the container maintainer, create two (configuration) files, create a volume mapping, and we're done.

Part of the Docker-Compose file:

...
services:

  dns_service:
    image: ...
    volumes:
      - ./config/definitions1.conf:/opt/dns_server/config/definitions1.conf:ro
      - ./config/definitions2.conf:/opt/dns_server/config/definitions2.conf:ro
    ...

But now we have a problem with Docker Swarm because it does not manage our volume data! If we deploy the service on a worker node, the data, the configuration files, is not there!

'Baking' configuration data into the service

When using Docker Swarm, we have no data in our volumes ... if we do nothing. There is a way around this, called 'Configs', not available in Docker-Compose. We can use this to specify files that will be mounted into the container like we can do with volumes. We look first at the Docker-Compose solution and then change this to Docker Swarm.

Using Docker-Compose

When using Docker-Compose, the 'docker-compose.yml' file for our DNS server project, see above,
uses volumes and looks something like:

# docker-compose.yml, using volumes

version: '3.7' 

services:
  app:
    image: busybox_volumes
    build: .
    volumes:
      - ./config/definitions1.conf:/opt/dns_server/config/definitions1.conf:ro
      - ./config/definitions2.conf:/opt/dns_server/config/definitions2.conf:ro
    command: sh -c "ls -l /opt/dns_server/config && cat /opt/dns_server/config/definitions*.conf && tail -f /dev/null"

We add a 'Dockerfile':

# Dockerfile
FROM busybox

And build the image:

docker-compose build

And then we run it:

docker-compose up

The text printed to the screen is:

app_1  | total 8
app_1  | -rw-rw-r--    1 1000     1000            14 Aug 29 08:06 definitions1.conf
app_1  | -rw-rw-r--    1 1000     1000            14 Aug 29 08:06 definitions2.conf
app_1  | definitions=1
app_1  | definitions=2

This is as expected. But if Docker Swarm would deploy this service to another host, a worker node, our files would be missing.

Using Docker Swarm

Here we remove the 'volumes' section and add the new 'configs' section:

# docker-compose.yml, using configs

version: '3.7' 

services:
  app:
    image: busybox_configs
    build: .
    configs:
      - source: definitions1
        target: /opt/dns_server/config/definitions1.conf
      - source: definitions2
        target: /opt/dns_server/config/definitions2.conf
    command: sh -c "ls -l /opt/dns_server/config && cat /opt/dns_server/config/definitions*.conf && tail -f /dev/null"

configs:
  definitions1:
    file: ./config/definitions1.conf
  definitions2:
    file: ./config/definitions2.conf

Build the image:

docker-compose build

And then we deploy:

docker stack deploy -c docker-compose.yml configs

The result of the deploy command shows the creation of the 'configs':

Creating network configs_default
Creating config configs_definitions2
Creating config configs_definitions1
Creating service configs_app

Nothing is printed to the screen, to see what is going on we look at the log of this service:

docker service logs configs_app

The result:

configs_app.1.vqumgq21leq8@myra    | total 8
configs_app.1.vqumgq21leq8@myra    | -r--r--r--    1 root     root            14 Aug 29 09:58 definitions1.conf
configs_app.1.vqumgq21leq8@myra    | -r--r--r--    1 root     root            14 Aug 29 09:58 definitions2.conf
configs_app.1.vqumgq21leq8@myra    | definitions=1
configs_app.1.vqumgq21leq8@myra    | definitions=2

This means that the Docker Swarm manager correctly added our files.

In my case the service was deployed on the Docker Swarm manager node, not on the worker node. To run it on the worker node, we add the following to 'docker-compose.yml':

    deploy:
      placement:
        constraints:
          - node.role == worker
      replicas: 1

Tag your image and push it to your Docker registry.

Only if you are using a private registry

If you are using a private, not secure, registry, you must tell all Docker nodes it is not secure by adding this information:

{
    "insecure-registries":[
        "<your registry ip address>:<your registry port>"
    ],
}

to the file:

/etc/docker/daemon.json

Restart Docker once you have done this:

sudo systemctl restart docker

Important: You must do this on all nodes.

Now, edit your 'docker-compose.yml' file to tell Docker Swarm where it can get your image.

From:

...
services:
  app:
    image: busybox_configs:latest
    ...

To:

...
services:
  app:
    image: <registry host>:<registry port>/busybox_configs:latest
    ...

Deploy the service to the worker node

Remove the service from this node, if it is still there:

docker stack rm configs_app

And deploy again:

docker stack deploy -c docker-compose.yml configs

Now you can check that the service is running on the worker node:

docker service ps configs_app

The result contains the node, see 'NODE':

ID             NAME            IMAGE     NODE         DESIRED STATE   CURRENT STATE           ERROR     PORTS
xotczxp65lub   configs_app.1   ...       vmubs2204a   Running         Running 9 minutes ago             

You can get more information for example with command:

docker service inspect --pretty configs_app

Oh, and the most important, what did our service exactly do? Nothing is printed to the screen, we must look at the logs:

docker service logs configs_app

The result is exactly the same as before. Meaning that Docker Swarm not only deployed our our service (container), but also made our 'configs' files available to our container running on the worker node. Done.

Summary

In this post, we used Docker Configs to pass (static) configuration data to containers. In 'docker-compose.yml', we removed the 'volumes' section and replaced it with the 'configs' section. Now Docker Swarm manages this data and we do not need a shared storage volume for this purpose. The annoying thing is that 'configs' does not work in Docker-Compose, which means you are moving away from a single 'docker-compose.yml' that can be used by Docker-Compose and Docker Swarm.

Links / credits

Data(base) persistence in docker swarm mode
https://forums.docker.com/t/data-base-persistence-in-docker-swarm-mode/20665

Deploy a stack to a swarm
https://docs.docker.com/engine/swarm/stack-deploy

Docker - configs
https://docs.docker.com/compose/compose-file/compose-file-v3/#configs

Docker Swarm Tutorial | Code Along | Zero to Hero under 1 Hour
https://takacsmark.com/docker-swarm-tutorial-for-beginners

How does Docker Swarm implement volume sharing?
https://stackoverflow.com/questions/47756029/how-does-docker-swarm-implement-volume-sharing