Should I migrate my Docker Swarm to Kubernetes?

I migrated a Docker-Compose project to Kubernetes and decided to go for Kubernetes.

15 September 2023 Updated 1 October 2023
post main image

When you read posts on the internet saying that Docker Swarm is dead, you get scared. I have a Docker Swarm running and I like it, it's easy when you already use Docker.

What are the alternatives? We read the whole time that there is only one thing to do and that is to migrate to Kubernetes and forget everything else.

I was half-way migrating from Docker to Docker Swarm, and wanted to know if I should continue, or move focus to Kubernetes.

Perhaps the most logical next step would be to use Docker Desktop because it includes Kubernetes. However, I do not use Docker Desktop, I'm running Docker Swarm also on my development machine and it's perfect.

Back to Kubernetes. All my applications are running on Ubuntu and Debian and after some more reading Microk8s seemed good choice, you can use it for development and production. So let's assume we move to Microk8s, then what must be done to migrate? I used one of my current Docker-Compose projects to find out.

This post is mainly about all the things I had to do to reach the result,I do not show all details of the Docker-Compose project.

As always I am on Ubuntu 22.04.

My application

My application consists of multiple blocks. Every block consists of one or more Docker-Compose projects. At the moment only the Task Runners are managed with Docker Swarm. They connect to the Backend using an Ingress overlay network.

   +----------+   +---------+         +-------------+
 --| Frontend |---| Backend |----+----| Task Runner |   
   +----------+   +---------+    |    +-------------+
                                 |    +-------------+
                   Ingress ->    +----| Task Runner |   
                   overlay       |    +-------------+
                   network       //
                                 |    +-------------+
                                 +----| Task Runner |   

          manage with              manage with
   |<--- Docker-Compose --->|<---- Docker Swarm ---->|
                                   manage with
                            |<----- Kubernetes ---->|

The Task Runner is a Docker-Compose project with five services:


'local-web' is a slightly modified Nginx image, returning a customized web page.

                          |  local-web  |   
     +----------+       +-----------------+
     |          |       |                 |-+
     |          |------>|                 | |-+ 
 <-->| rabbitmq |       |      task       | | |
     |          |<------|                 | | |
     |          |       |                 | | |
     +----------+       +-----------------+ | |
                              |     | ------+ |
                              |     | --------+
     +--------------------+   |     |
     | unbound-cloudflare |<--+     |
     +--------------------+         |
     +--------------------+         |
     | unbound-quad9      |<--------+

The rabbitmq-service connects to the Backend. With Docker Swarm I create replica's of the task-service.

Here is a stripped version of the Docker-Compose project file:

# docker-compose.yml 


    external: true

Only rabbitmq is connected to an external network. In this post I am going to run a Task Runner on Kubernetes, with the 'task' service replicated.

Questions and answers

Before starting any work I made a list of questions and tried to get some answers. Here they are:

Is a Pod the same as a Docker Container?

No it is not. From the docs: A Pod is the smallest deployable unit of computing that you can create and manage in Kubernetes. There can be multiple containers in a Pod. A Pod provides an environment for containers to run and share resources, storage, network, and inter-process communication. They share the same network namespace and can communicate with each other using localhost.

To summarize, unless you do something special, you should use one Pod per container.

Pods communicate using Services

A container in a Pod communicates with another Pod using its IP address. But this is not the way to go. In general we create a Service for every Pod, and we access the Pod via its Service. We refer to a Service using its name, we can also map ports here.

If we have a Pod with replicated containers, the Service will distribute requests accross these containers. In this case the Service also acts as a load balancer. This is very much like Docker Swarm.

When to use Deployments?

A Deployment is used to manage a Pod. With a Deployment you can specify Pod replicas, Pod placement on nodes, how new updates are released. This means you (almost) always need a Deployment for a Pod. Because we can specify a Pod inside a Deployment, we do not need separate YAML files for the Pods!

How to do Pods logging?

The Pods do not do logging. Its the containers that generate logs. This is not different from Docker logging. On my system, the logfiles are in:


Can we assign a directory on the host to a Pod like Docker Volumes?

Yes, for this we can use the 'hostPath' directive.

Is Kubernetes only for production or can we also use it for development?

Microk8s can be used with development and production. But for production we can also use something else like a Kubernetes variant from a cloud provider. On my development machine I use Microk8s alongside Docker without problems.

In Docker Swarm We have Task.Slot, what is the equivalent in Kubernetes?

Not exactly the same but comparable is the Kubernetes 'StatefulSet'. I did not try this yet.

Can we connect a Docker Swarm network to a Kubernetes Pod (container)?

The reason why I would want this is migration. Of course we can always build our own load balancer proxy, but is there a simple / standard way of doing this? For the moment I create a 'nodePort' Service. This maps the port of a Service inside Kubernetes, to a port on the host.

Before conversion, check and correct object names

I had all these nice names in my Docker-Compose files and they were using underscores ('_'). In Kubernetes object names must comply to RFC 1123, which means:

  • Max 63 characters.
  • Must start and end with a lowercase letter or number.
  • Contain lowercase letters, numbers, and hyphens.

Before doing anything I changed this in my Docker-Compose project.

Kompose: Convert the Docker-Compose project file

Now let's try to convert the Task Runner Docker-Compose files to something Kubernetes, using Kompose. Kompose is a conversion tool for Docker Compose to Kubernetes. And we run immediately into problems. Kompose appeared not to be able handle the '.env' file which is quite essential. Also (as a consequence?) no ConfigMap was generated. There is an issue reported for this.

I will check for a new version one of the next weeks but for now we substitute the enviroment variables myself using Docker-Compose 'config':

> docker-compose -f docker_compose_shared_file.yml -f docker_compose_deployment_file.yml config > docker-compose-config.yml

And then running Kompose. We use the hostPath option here because I use volumes using local system directories:

> kompose -v convert -f docker-compose-config.yml --volumes hostPath

Kompose created the following files:


For every Docker-Compose service a '-pod.yaml' file is created, and only for one a '-service.yaml' file. I also have seen examples where Kompose generated Deployments instead of Pods. Why? For me, the Kompose results are just a starting point, I must create Deployment YAML files from the Pod YAML files and also add Service YAML files. Probably you can get better results from Kompose (by adding instructions to the 'docker-compose.yml' file) but I do not have the time for experimenting at the moment.

Installing Microk8s

Now let's get our hands dirty. I'm on Ubuntu 22.04 so this is easy, we use Microk8s To get the latest stable release:

> sudo snap install microk8s --classic

Let's see what is enabled:

> microk8s status


microk8s is running
high-availability: no
  datastore master nodes:
  datastore standby nodes: none
    dns                  # (core) CoreDNS
    ha-cluster           # (core) Configure high availability on the current node
    helm                 # (core) Helm - the package manager for Kubernetes
    helm3                # (core) Helm 3 - the package manager for Kubernetes
    cert-manager         # (core) Cloud native certificate management
    community            # (core) The community addons repository
    dashboard            # (core) The Kubernetes dashboard
    gpu                  # (core) Automatic enablement of Nvidia CUDA
    host-access          # (core) Allow Pods connecting to Host services smoothly
    hostpath-storage     # (core) Storage class; allocates storage from host directory
    ingress              # (core) Ingress controller for external access
    kube-ovn             # (core) An advanced network fabric for Kubernetes
    mayastor             # (core) OpenEBS MayaStor
    metallb              # (core) Loadbalancer for your Kubernetes cluster
    metrics-server       # (core) K8s Metrics Server for API access to service metrics
    minio                # (core) MinIO object storage
    observability        # (core) A lightweight observability stack for logs, traces and metrics
    prometheus           # (core) Prometheus operator for monitoring and logging
    rbac                 # (core) Role-Based Access Control for authorisation
    registry             # (core) Private image registry exposed on localhost:32000
    storage              # (core) Alias to hostpath-storage add-on, deprecated

Enable hostpath-storage

Some services in my docker-compose.yml file use persistent data in directories on the localhost. Turn on/enable storage services in Microk8s:

> microk8s.enable hostpath-storage

Kubernetes dashboard

Microk8s includes the Kubernetes dashboard. First enable it:

> microk8s enable dashboard

There are several options to start it, this one is easy:

> microk8s dashboard-proxy

Then, in your browser, navigate to:

Accept the self-signed certificate, copy-paste the token from the terminal and you're in.

Stop and start

To stop and start Microk8s:

> microk8s stop
> microk8s start

Using kubectl

Kubectl is a command line tool used to run commands to manage Kubernetes clusters.

To use 'kubectl' instead of 'microk8s kubectl':

> alias kubectl='microk8s kubectl'

To make this permanent, add this to your 'bashrc' file:

> echo "alias kubectl='microk8s kubectl'" >> ~/.bashrc

Make al lot of aliases, or your fingers will get tired!

Getting help for example about the ports of a container in a Pod:

> kubectl explain pod.spec.containers.ports

To create and update Kubernetes objects we can use a declarative syntax ('apply'), or imperative ('create').

Here are some commands:

> kubectl get nodes
> kubectl apply -f <manifest file>
> kubectl get pods -o wide
> kubectl get deployments
> kubectl get services
> kubectl delete pods local-web
> kubectl describe pods local-web
> kubectl logs local-web

To get the latest events:

> kubectl get events

And, to get everything:

> kubectl get all -A

Remember, a Deployment contains a ReplicaSet specification and Pod specification, meaning that if you create a Deployment, you will get one or more Pods depending on the replicas you specified.

Deploy a Pod

Can we deploy one of the Pods created by Kompose? Kompose created the following file for us:


Without editing this file we try to deploy this Pod:

> kubectl apply -f local-web-pod.yaml


pod/local-web created

Is the Pod there?

> kubectl get pods


local-web   0/1     Pending   0          119s

It is there but the status is 'Pending'. Let's check the logs:

> kubectl logs --timestamps local-web

No logs, nothing is returned. Let's check the events:

> kubectl get events


LAST SEEN   TYPE      REASON                OBJECT          MESSAGE
9m31s       Warning   FailedScheduling      pod/local-web   0/1 nodes are available: persistentvolumeclaim "local-web-claim0" not found. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod..
101s        Warning   FailedScheduling      pod/local-web   0/1 nodes are available: 1 node(s) didn't match Pod's node affinity/selector. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling..

It appeared that 'nodeAffinity' had the value of the Docker-Compose file 'deploy - node.hostname'. I had to change this to the name of the host I am using at the moment of course ... :-(

Add a private (image) registry

Delete the Pod, apply again and get status:

> kubectl delete pod local-web
> kubectl apply -f local-web-pod.yaml
> kubectl get pods


NAME        READY   STATUS             RESTARTS   AGE
local-web   0/1     ErrImagePull       0          76s

I am using a private registry for Docker already. To use this in Kubernetes, we must add authentication for this registry.

Check if a registry-credential item is present:

> kubectl get secret registry-credential --output=yaml


Error from server (NotFound): secrets "registry-credential" not found

Here I am using the information already present in Docker. Create the registry-credential:

> kubectl create secret generic registry-credential \
    --from-file=.dockerconfigjson=/home/peter/.docker/config.json \


secret/registry-credential created

To check it:

> kubectl get secret registry-credential --output=yaml

We first push the image to the registry. Next, we edit the Pod files and add the reference to the registry credentials:

    - name: registry-credential

Now we delete the Pod and apply again:

> kubectl get pods


local-web   1/1     Running   0          9s

At last, it's running!

Enter the Pod container

Enter the 'local-web' container:

> kubectl exec -it local-web -- /bin/bash

Now we can check the environment variables, mounts, etc.

You may also want to enter the container as root. My Microk8s is using 'runc' (Open Container Initiative runtime) to access containers. Get the Container ID:

> kubectl describe pod <your pod> | grep containerd

This will output something like:

Container ID: containerd://6a060ba8436f575b86b4f3fe10a373125aaf7c125af835180d792f5382836355

Then exec as root into the container:

> sudo runc --root /run/containerd/runc/ exec -t -u 0 6a060ba8436f575b86b4f3fe10a373125aaf7c125af835180d792f5382836355 sh

Accessing the Pod through a Service

After stopping and starting a Pod, a new IP address may be assigned to a Pod. That's why we use a Service to access a Pod, and use the Service name.

By creating a service for a Pod we can access the Pod using its name instead of IP address. For this we need the Kubernetes cluster DNS, it should be enabled:

> kubectl get services kube-dns --namespace=kube-system


NAME       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   <none>        53/UDP,53/TCP,9153/TCP   3d5h

Create a Service for the local-web Pod

The local-web-pod.yaml file did not have a ports entry, so I added a containerPort:

      - containerPort: 80

Because a local-web-service.yaml file was not created by Kompose, I created file myself. Important: Here I add a port that gives access to the Pod. In Docker-Compose we did not need this port because the service is on the internal network.

apiVersion: v1
kind: Service
  name: local-web
  namespace: default
    - port: 80
      targetPort: 80
      protocol: TCP
    io.kompose.service: local-web

The 'selector' must match the 'label' of the Pod.

After that we start the service:

> kubectl apply -f local-web-service.yaml

Now we should be able to access the local-web service from another Pod container.

Let's start a Busybox container. There are problems with recent Busybox images and Kubernetes, so we use version 1.28.

> kubectl run -i --tty --image busybox:128 test --restart=Never --rm /bin/sh

Then inside the container:

# wget local-web-service:80


Connecting to local-web-service:80 (
saving to 'index.html'
index.html           100% |**********************************************************************************************************************************|   400  0:00:00 ETA
'index.html' saved


Creating Deployments and Services

As already mentioned above, we do not create Pod YAML files but Deployment YAML files. And we also add a Service for every Deployment that has ports.

In Kubernetes the Service names are global, in Docker-Compose services on an internal network are local to the Docker-Compose project. This means that in Kubernetes we must namespace our Services.

We can do this in two ways:

  • Add a namespace for this Docker-Compose project
  • Add a prefix to the service name

At the moment I think adding a prefix is the best option:


This gives us the Deployment names and Service names:



And filenames:



In the Deployment I start with 1 replica. Here is the first part of a Deployment file:

# task-runner-local-web-deployment.yaml
apiVersion: apps/v1
kind: Deployment
  # metadata name must not contain dots
  name: task-runner-local-web-deployment
  namespace: default
  # number of copies of each pod we want
  replicas: 1

    type: Recreate

  # pods managed by this deployment
    # match the labels we set on the pod, see below
      task-runner.local-web.deployment: task-runner-local-web-deployment

  # template field is a regular pod configuration nested inside the deployment spec
      # set labels on the pod, used in the deployment selector, see above
        task-runner.local-web.deployment: task-runner-local-web-deployment

Make '.env' file available when the containers start

I have a lot of key-value pairs in the '.env' file used by Docker-Compose. In Kubernetes we can use a ConfigMap to import these in our Deployment YAML files.

First we create a ConfigMap. The ConfigMap is created in the 'default' namespace, you can change this by adding the namespace flag and value:

> kubectl create configmap task-runner-env-configmap --from-env-file=.env

List the ConfigMaps:

> kubectl get configmaps


NAME                      DATA   AGE
task-runner-env-configmap   51     19m

We can view the result as yaml:

> kubectl get configmaps task-runner-env-configmap -o yaml

Then in the Deployment YAML files we remove the 'env' section and add the 'envFrom' section.


      - env:
        - name: ENV_VAR1
          value: "1"
        image: ...


      - envFrom:
        - configMapRef:
            name: task-runner-env-configmap
          - name: VAR_C = $(VAR_A)/$(VAR_B)
        image: ...

Here we also create a new environment variable VAR_C from two environment variables present in the ConfigMap.

We can update the ConfigMap as follows:

> kubectl create configmap task-runner-env-configmap --from-env-file=.env -o yaml --dry-run | kubectl apply -f -

We created a ConfigMap to pass the key-value pairs in the '.env' file to the containers. But there is a big problem. We cannot use environment variables in other parts of our manifests. For example, assume you want to be able to switch the repository with an environment variable IMAGE_REGISTRY:

      - envFrom:
        - configMapRef:
            name: task-runner-env-configmap
          - name: VAR_C = $(VAR_A)/$(VAR_B)
        image: $(IMAGE_REGISTRY)...

This will not work! It is 2023 and because this (trivial feature) is not working, all over the world people are creating their own scripts to make this work. Wonderful. Anyway, what must the script do:

  1. Start a new (Bash) shell.
  2. Export the key-value pairs in the '.env' file.
  3. Use 'envsubst' to replace the variables in our manifest.

First, we create a new shell to avoid disturbing the current environment. In the new environment, we create the environment variables using the '.env' file. Finally, we replace the variables:

> envsubst < mydeploy.yaml | kubectl apply -f -

PersistentVolumes and VolumeClaims

Kompose created a VolumeMounts and a Volumes section for me in the Pods YAML files using hostPath. This means that the localhost actual directories are hard-coded in the Deployment files. Obviously not what we want.

So I created some PersistentVolume and PersistentVolumeClaim manifests and used them in the Deployment files.

Create Services for the Deployments

Here I show only the local-web service.

# task-runner-local-web-service.yaml
apiVersion: v1
kind: Service
  # metadata name must not contain dots
  name: task-runner-local-web-service
  namespace: default
    - name: http
      port: 80
      targetPort: 80
      protocol: TCP
    task-runner.local-web.deployment: task-runner-local-web-deployment

Start the service:

> kubectl apply -f task-runner-local-web-service.yaml

Application code changes

I needed only small changes to my application code. As mentioned above, I prefixed the initial Service names with 'task-runner-'. To keep compatibility with Docker-Compose I now use environment variables for the Service names and ports, specified as key-value pairs in the '.env' file.

Although ports are now specified in all Services, there is no need to change anything because the ports are bound to the Services, and every Service has its own IP address.

To check where an instance of a Pod is running I store the environment variable HOSTNAME together with the result of a task. For Docker Swarm I used .Node.Hostname. In the logs we can check this field.

Using nodePort to expose the rabbitmq service to the Back End

The Task Runner is up and running but there is no connection to the outside world. There are several ways to expose the rabbitmq Service and here I use nodePort. We simply create an extra Service with type nodePort and specify the port number:

# task-runner-rabbitmq-nodeport.yaml
apiVersion: v1
kind: Service
  # metadata name must not contain dots
  name: task-runner-rabbitmq-nodeport
  namespace: default
  type: NodePort
  # The range of valid ports is 30000-32767
    - name: amqp
      port: 10763
      targetPort: 5672
      nodePort: 31763
    - name: http
      port: 20763
      targetPort: 15672
      nodePort: 32763
    task-runner.rabbitmq.deployment: task-runner-rabbitmq-deployment

Then we can refer to this Kubernetes Service on the host like: localhost:<port>.

To connect the Backend Docker container to this port we add some lines to the 'docker-compose.yaml' file:

      - "host.docker.internal:host-gateway"

And then refer to the service as:


Adding a Worker node, DNS not working

Of course I wanted to add a new node and run some Pods there. With VitualBox on my development machine I created a VM with Ubuntu server 22.04. Then added Microk8s and joined the new node.

Problems ... Kubernetes DNS was not working on the new node. There are a lot of DNS issues mentioned on the Microk8s Github page. This probably has to do with iptables ...

I was able to have a proper working Kubernetes cluster of two VirtualBox machines. At the moment I am investigating this.

Developing and troubleshooting on Kubernetes

One of the Pods did not work. I could see in the log what was wrong but we do not want to go into a loop of:

--+-> make changes ---> build image ---> test --+--->
  ^                                             |
  |                                             v

So I mounted the external project code back into the container, which takes me back to the development environment and was able to solve the problem very fast.


I only used one extra tool during migration: yamllint to validate manifests (YAML files).

> sudo apt-get install yamllint

And then run for example:

> yamllint task-runner-local-web-deployment.yaml


To start with, I spend only two week with Kubernetes (don't shoot the piano player). I may have made some wrong choices but part of my application is now running and managed by Kubernetes.

Docker Swarm is an easy choice when it comes to orchestrating Docker-Compose projects. It just works, without serious limitations. For my application, which consists of multiple Docker-Compose projects, I don't need thousands of replicas. With Docker Swarm, you start up a new node, add some deploy directives to your docker-compose.yml files, and you're done. Docker network is one of Docker's best features. With Docker Swarm, all you have to do is turn a network into an encrypted Ingress overlay network to communicate between nodes.

Moving to Kubernetes gives more deployment flexibility, but the main reason Kubernetes has become the first choice is that it is very widespread and many tools are integrated with it. That's where Docker Swarm lags behind. At the time of writing this post, development of Docker Swarm seems more or less stalled (considered fully feature-complete?). That's a shame, because it's a great solution to a very complex problem.

In an ideal world, the development environment is identical to the production environment. If it runs in development, it also runs in production. Docker Swarm comes very close to this, without much complexity. Using Kubernetes, on the other hand, creates a gap. It's like, Ok developer, you've done your job, now leave it to the DevOps to put it into production. If the developer makes changes to Docker-Compose projects, the DevOps have to work on them as well.

If you use Kubernetes for production, I think it's inevitable that the development environment also runs Kubernetes. And in many cases, Docker Swarm as well. Does this make development and production easier or harder compared to just Docker Swarm? Microk8s is very easy to use in a development environment, but that's just the beginning. Creating Docker-Compose project files is very easy. You usually have two yaml files and an '.env' file. After the conversion, my Kubernetes project already had 14 Kubernetes yaml files and more.

It would be a very good idea if Kubernetes added some extensions for the YAML files that allow importing/ adding other files (such as Nginx) and environment data. For now, we have to write our own scripts or use something like Helm ... more things to learn.

Anyway, in this post I made an attempt to investigate what it would take to move part of my application from Docker Swarm to Kubernetes. Based on the results, I have decided to move to Kubernetes, I will not continue using Docker Swarm for production. And I will also use Kubernetes for development.

Starting with Kubernetes is not very difficult, but sometimes very confusing. After a few days you understand most of the concepts and have something working. From there you can improve and expand. There can be (very) serious and time-consuming problems, such as DNS not working. I don't want to become an iptables specialist, but there is no other way.

The number of changes I had to make to my application is very limited:

  • Changing names to RFC1123.
  • Prefixing service names, using environment variables.
  • Storing other environment variables in results.

Back to coding ...

Links / credits

Configure a Pod to Use a ConfigMap

Connecting Kubernetes and Docker

Helm - The package manager for Kubernetes

How to go from Docker to Kubernetes the right way

How To Migrate a Docker Compose Workflow to Kubernetes


Kubernetes - Documentation - Concepts

Kubernetes - Documentation - kubectl Cheat Sheet

Kustomize - Kubernetes native configuration management

MicroK8s documentation - home

Running Kubernetes locally on Linux with Microk8s

Using private docker registry inside kubernetes

When should I use envFrom for configmaps?

Leave a comment

Comment anonymously or log in to comment.


Leave a reply

Reply anonymously or log in to reply.