Should I migrate my Docker Swarm to Kubernetes?
I migrated a Docker-Compose project to Kubernetes and decided to go for Kubernetes.
When you read posts on the internet saying that Docker Swarm is dead, you get scared. I have a Docker Swarm running and I like it, it's easy when you already use Docker.
What are the alternatives? We read the whole time that there is only one thing to do and that is to migrate to Kubernetes and forget everything else.
I was half-way migrating from Docker to Docker Swarm, and wanted to know if I should continue, or move focus to Kubernetes.
Perhaps the most logical next step would be to use Docker Desktop because it includes Kubernetes. However, I do not use Docker Desktop, I'm running Docker Swarm also on my development machine and it's perfect.
Back to Kubernetes. All my applications are running on Ubuntu and Debian and after some more reading Microk8s seemed good choice, you can use it for development and production. So let's assume we move to Microk8s, then what must be done to migrate? I used one of my current Docker-Compose projects to find out.
This post is mainly about all the things I had to do to reach the result,I do not show all details of the Docker-Compose project.
As always I am on Ubuntu 22.04.
My application consists of multiple blocks. Every block consists of one or more Docker-Compose projects. At the moment only the Task Runners are managed with Docker Swarm. They connect to the Backend using an Ingress overlay network.
+----------+ +---------+ +-------------+ --| Frontend |---| Backend |----+----| Task Runner | +----------+ +---------+ | +-------------+ | | +-------------+ Ingress -> +----| Task Runner | overlay | +-------------+ network // | +-------------+ +----| Task Runner | +-------------+ manage with manage with |<--- Docker-Compose --->|<---- Docker Swarm ---->| v migrate v manage with |<----- Kubernetes ---->|
The Task Runner is a Docker-Compose project with five services:
rabbitmq task local-web unbound-cloudflare unbound-quad9
'local-web' is a slightly modified Nginx image, returning a customized web page.
+-------------+ | local-web | +-------------+ ^ | +----------+ +-----------------+ | | | |-+ | |------>| | |-+ <-->| rabbitmq | | task | | | | |<------| | | | | | | | | | +----------+ +-----------------+ | | | | ------+ | | | --------+ +--------------------+ | | | unbound-cloudflare |<--+ | +--------------------+ | | +--------------------+ | | unbound-quad9 |<--------+ +--------------------+
The rabbitmq-service connects to the Backend. With Docker Swarm I create replica's of the task-service.
Here is a stripped version of the Docker-Compose project file:
# docker-compose.yml services: rabbitmq: ... networks: task-runner-network task: ... local-web: ... unboud-cloudflare: ... networks: task-runner-network: external: true
Only rabbitmq is connected to an external network. In this post I am going to run a Task Runner on Kubernetes, with the 'task' service replicated.
Questions and answers
Before starting any work I made a list of questions and tried to get some answers. Here they are:
Is a Pod the same as a Docker Container?
No it is not. From the docs: A Pod is the smallest deployable unit of computing that you can create and manage in Kubernetes. There can be multiple containers in a Pod. A Pod provides an environment for containers to run and share resources, storage, network, and inter-process communication. They share the same network namespace and can communicate with each other using localhost.
To summarize, unless you do something special, you should use one Pod per container.
Pods communicate using Services
A container in a Pod communicates with another Pod using its IP address. But this is not the way to go. In general we create a Service for every Pod, and we access the Pod via its Service. We refer to a Service using its name, we can also map ports here.
If we have a Pod with replicated containers, the Service will distribute requests accross these containers. In this case the Service also acts as a load balancer. This is very much like Docker Swarm.
When to use Deployments?
A Deployment is used to manage a Pod. With a Deployment you can specify Pod replicas, Pod placement on nodes, how new updates are released. This means you (almost) always need a Deployment for a Pod. Because we can specify a Pod inside a Deployment, we do not need separate YAML files for the Pods!
How to do Pods logging?
The Pods do not do logging. Its the containers that generate logs. This is not different from Docker logging. On my system, the logfiles are in:
Can we assign a directory on the host to a Pod like Docker Volumes?
Yes, for this we can use the 'hostPath' directive.
Is Kubernetes only for production or can we also use it for development?
Microk8s can be used with development and production. But for production we can also use something else like a Kubernetes variant from a cloud provider. On my development machine I use Microk8s alongside Docker without problems.
In Docker Swarm We have Task.Slot, what is the equivalent in Kubernetes?
Not exactly the same but comparable is the Kubernetes 'StatefulSet'. I did not try this yet.
Can we connect a Docker Swarm network to a Kubernetes Pod (container)?
The reason why I would want this is migration. Of course we can always build our own load balancer proxy, but is there a simple / standard way of doing this? For the moment I create a 'nodePort' Service. This maps the port of a Service inside Kubernetes, to a port on the host.
Before conversion, check and correct object names
I had all these nice names in my Docker-Compose files and they were using underscores ('_'). In Kubernetes object names must comply to RFC 1123, which means:
- Max 63 characters.
- Must start and end with a lowercase letter or number.
- Contain lowercase letters, numbers, and hyphens.
Before doing anything I changed this in my Docker-Compose project.
Kompose: Convert the Docker-Compose project file
Now let's try to convert the Task Runner Docker-Compose files to something Kubernetes, using Kompose. Kompose is a conversion tool for Docker Compose to Kubernetes. And we run immediately into problems. Kompose appeared not to be able handle the '.env' file which is quite essential. Also (as a consequence?) no ConfigMap was generated. There is an issue reported for this.
I will check for a new version one of the next weeks but for now we substitute the enviroment variables myself using Docker-Compose 'config':
> docker-compose -f docker_compose_shared_file.yml -f docker_compose_deployment_file.yml config > docker-compose-config.yml
And then running Kompose. We use the hostPath option here because I use volumes using local system directories:
> kompose -v convert -f docker-compose-config.yml --volumes hostPath
Kompose created the following files:
local-web-pod.yaml rabbitmq-pod.yaml rabbitmq-service.yaml task-pod.yaml unbound-cloudflare-pod.yaml unbound-quad9-pod.yaml
For every Docker-Compose service a '-pod.yaml' file is created, and only for one a '-service.yaml' file. I also have seen examples where Kompose generated Deployments instead of Pods. Why? For me, the Kompose results are just a starting point, I must create Deployment YAML files from the Pod YAML files and also add Service YAML files. Probably you can get better results from Kompose (by adding instructions to the 'docker-compose.yml' file) but I do not have the time for experimenting at the moment.
Now let's get our hands dirty. I'm on Ubuntu 22.04 so this is easy, we use Microk8s To get the latest stable release:
> sudo snap install microk8s --classic
Let's see what is enabled:
> microk8s status
microk8s is running high-availability: no datastore master nodes: 127.0.0.1:19001 datastore standby nodes: none addons: enabled: dns # (core) CoreDNS ha-cluster # (core) Configure high availability on the current node helm # (core) Helm - the package manager for Kubernetes helm3 # (core) Helm 3 - the package manager for Kubernetes disabled: cert-manager # (core) Cloud native certificate management community # (core) The community addons repository dashboard # (core) The Kubernetes dashboard gpu # (core) Automatic enablement of Nvidia CUDA host-access # (core) Allow Pods connecting to Host services smoothly hostpath-storage # (core) Storage class; allocates storage from host directory ingress # (core) Ingress controller for external access kube-ovn # (core) An advanced network fabric for Kubernetes mayastor # (core) OpenEBS MayaStor metallb # (core) Loadbalancer for your Kubernetes cluster metrics-server # (core) K8s Metrics Server for API access to service metrics minio # (core) MinIO object storage observability # (core) A lightweight observability stack for logs, traces and metrics prometheus # (core) Prometheus operator for monitoring and logging rbac # (core) Role-Based Access Control for authorisation registry # (core) Private image registry exposed on localhost:32000 storage # (core) Alias to hostpath-storage add-on, deprecated
Some services in my docker-compose.yml file use persistent data in directories on the localhost. Turn on/enable storage services in Microk8s:
> microk8s.enable hostpath-storage
Microk8s includes the Kubernetes dashboard. First enable it:
> microk8s enable dashboard
There are several options to start it, this one is easy:
> microk8s dashboard-proxy
Then, in your browser, navigate to:
Accept the self-signed certificate, copy-paste the token from the terminal and you're in.
Stop and start
To stop and start Microk8s:
> microk8s stop > microk8s start
Kubectl is a command line tool used to run commands to manage Kubernetes clusters.
To use 'kubectl' instead of 'microk8s kubectl':
> alias kubectl='microk8s kubectl'
To make this permanent, add this to your 'bashrc' file:
> echo "alias kubectl='microk8s kubectl'" >> ~/.bashrc
Make al lot of aliases, or your fingers will get tired!
Getting help for example about the ports of a container in a Pod:
> kubectl explain pod.spec.containers.ports
To create and update Kubernetes objects we can use a declarative syntax ('apply'), or imperative ('create').
Here are some commands:
> kubectl get nodes > kubectl apply -f <manifest file> > kubectl get pods -o wide > kubectl get deployments > kubectl get services > kubectl delete pods local-web > kubectl describe pods local-web > kubectl logs local-web
To get the latest events:
> kubectl get events
And, to get everything:
> kubectl get all -A
Remember, a Deployment contains a ReplicaSet specification and Pod specification, meaning that if you create a Deployment, you will get one or more Pods depending on the replicas you specified.
Deploy a Pod
Can we deploy one of the Pods created by Kompose? Kompose created the following file for us:
Without editing this file we try to deploy this Pod:
> kubectl apply -f local-web-pod.yaml
Is the Pod there?
> kubectl get pods
NAME READY STATUS RESTARTS AGE local-web 0/1 Pending 0 119s
It is there but the status is 'Pending'. Let's check the logs:
> kubectl logs --timestamps local-web
No logs, nothing is returned. Let's check the events:
> kubectl get events
LAST SEEN TYPE REASON OBJECT MESSAGE 9m31s Warning FailedScheduling pod/local-web 0/1 nodes are available: persistentvolumeclaim "local-web-claim0" not found. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.. 101s Warning FailedScheduling pod/local-web 0/1 nodes are available: 1 node(s) didn't match Pod's node affinity/selector. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling..
It appeared that 'nodeAffinity' had the value of the Docker-Compose file 'deploy - node.hostname'. I had to change this to the name of the host I am using at the moment of course ... :-(
Add a private (image) registry
Delete the Pod, apply again and get status:
> kubectl delete pod local-web > kubectl apply -f local-web-pod.yaml > kubectl get pods
NAME READY STATUS RESTARTS AGE local-web 0/1 ErrImagePull 0 76s
I am using a private registry for Docker already. To use this in Kubernetes, we must add authentication for this registry.
Check if a registry-credential item is present:
> kubectl get secret registry-credential --output=yaml
Error from server (NotFound): secrets "registry-credential" not found
Here I am using the information already present in Docker. Create the registry-credential:
> kubectl create secret generic registry-credential \ --from-file=.dockerconfigjson=/home/peter/.docker/config.json \ --type=kubernetes.io/dockerconfigjson
To check it:
> kubectl get secret registry-credential --output=yaml
We first push the image to the registry. Next, we edit the Pod files and add the reference to the registry credentials:
imagePullSecrets: - name: registry-credential
Now we delete the Pod and apply again:
> kubectl get pods
NAME READY STATUS RESTARTS AGE local-web 1/1 Running 0 9s
At last, it's running!
Enter the Pod container
Enter the 'local-web' container:
> kubectl exec -it local-web -- /bin/bash
Now we can check the environment variables, mounts, etc.
You may also want to enter the container as root. My Microk8s is using 'runc' (Open Container Initiative runtime) to access containers. Get the Container ID:
> kubectl describe pod <your pod> | grep containerd
This will output something like:
Container ID: containerd://6a060ba8436f575b86b4f3fe10a373125aaf7c125af835180d792f5382836355
Then exec as root into the container:
> sudo runc --root /run/containerd/runc/k8s.io/ exec -t -u 0 6a060ba8436f575b86b4f3fe10a373125aaf7c125af835180d792f5382836355 sh
Accessing the Pod through a Service
After stopping and starting a Pod, a new IP address may be assigned to a Pod. That's why we use a Service to access a Pod, and use the Service name.
By creating a service for a Pod we can access the Pod using its name instead of IP address. For this we need the Kubernetes cluster DNS, it should be enabled:
> kubectl get services kube-dns --namespace=kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kube-dns ClusterIP 10.152.183.10 <none> 53/UDP,53/TCP,9153/TCP 3d5h
Create a Service for the local-web Pod
The local-web-pod.yaml file did not have a ports entry, so I added a containerPort:
ports: - containerPort: 80
Because a local-web-service.yaml file was not created by Kompose, I created file myself. Important: Here I add a port that gives access to the Pod. In Docker-Compose we did not need this port because the service is on the internal network.
apiVersion: v1 kind: Service metadata: name: local-web namespace: default spec: ports: - port: 80 targetPort: 80 protocol: TCP selector: io.kompose.service: local-web
The 'selector' must match the 'label' of the Pod.
After that we start the service:
> kubectl apply -f local-web-service.yaml
Now we should be able to access the local-web service from another Pod container.
Let's start a Busybox container. There are problems with recent Busybox images and Kubernetes, so we use version 1.28.
> kubectl run -i --tty --image busybox:128 test --restart=Never --rm /bin/sh
Then inside the container:
# wget local-web-service:80
Connecting to local-web-service:80 (10.152.183.30:80) saving to 'index.html' index.html 100% |**********************************************************************************************************************************| 400 0:00:00 ETA 'index.html' saved
Creating Deployments and Services
As already mentioned above, we do not create Pod YAML files but Deployment YAML files. And we also add a Service for every Deployment that has ports.
In Kubernetes the Service names are global, in Docker-Compose services on an internal network are local to the Docker-Compose project. This means that in Kubernetes we must namespace our Services.
We can do this in two ways:
- Add a namespace for this Docker-Compose project
- Add a prefix to the service name
At the moment I think adding a prefix is the best option:
This gives us the Deployment names and Service names:
task-runner-rabbitmq-deployment task-runner-local-web-deployment task-runner-task-deployment task-runner-unbound-cloudflare-deployment task-runner-unbound-quad9-deployment task-runner-rabbitmq-service task-runner-local-web-service task-runner-task-service task-runner-unbound-cloudflare-service task-runner-unbound-quad9-service
task-runner-rabbitmq-deployment.yaml task-runner-local-web-deployment.yaml task-runner-task-deployment.yaml task-runner-unbound-cloudflare-deployment.yaml task-runner-unbound-quad9-deployment.yaml task-runner-rabbitmq-service.yaml task-runner-local-web-service.yaml task-runner-task-service.yaml task-runner-unbound-cloudflare-service.yaml task-runner-unbound-quad9-service.yaml
In the Deployment I start with 1 replica. Here is the first part of a Deployment file:
# task-runner-local-web-deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: # metadata name must not contain dots name: task-runner-local-web-deployment namespace: default spec: # number of copies of each pod we want replicas: 1 strategy: type: Recreate # pods managed by this deployment selector: # match the labels we set on the pod, see below matchLabels: task-runner.local-web.deployment: task-runner-local-web-deployment # template field is a regular pod configuration nested inside the deployment spec template: metadata: # set labels on the pod, used in the deployment selector, see above labels: task-runner.local-web.deployment: task-runner-local-web-deployment spec: ...
Make '.env' file available when the containers start
I have a lot of key-value pairs in the '.env' file used by Docker-Compose. In Kubernetes we can use a ConfigMap to import these in our Deployment YAML files.
First we create a ConfigMap. The ConfigMap is created in the 'default' namespace, you can change this by adding the namespace flag and value:
> kubectl create configmap task-runner-env-configmap --from-env-file=.env
List the ConfigMaps:
> kubectl get configmaps
NAME DATA AGE ... task-runner-env-configmap 51 19m
We can view the result as yaml:
> kubectl get configmaps task-runner-env-configmap -o yaml
Then in the Deployment YAML files we remove the 'env' section and add the 'envFrom' section.
spec: containers: - env: - name: ENV_VAR1 value: "1" ... image: ...
spec: containers: - envFrom: - configMapRef: name: task-runner-env-configmap env: - name: VAR_C = $(VAR_A)/$(VAR_B) image: ...
Here we also create a new environment variable VAR_C from two environment variables present in the ConfigMap.
We can update the ConfigMap as follows:
> kubectl create configmap task-runner-env-configmap --from-env-file=.env -o yaml --dry-run | kubectl apply -f -
We created a ConfigMap to pass the key-value pairs in the '.env' file to the containers. But there is a big problem. We cannot use environment variables in other parts of our manifests. For example, assume you want to be able to switch the repository with an environment variable IMAGE_REGISTRY:
spec: containers: - envFrom: - configMapRef: name: task-runner-env-configmap env: - name: VAR_C = $(VAR_A)/$(VAR_B) image: $(IMAGE_REGISTRY)...
This will not work! It is 2023 and because this (trivial feature) is not working, all over the world people are creating their own scripts to make this work. Wonderful. Anyway, what must the script do:
- Start a new (Bash) shell.
- Export the key-value pairs in the '.env' file.
- Use 'envsubst' to replace the variables in our manifest.
First, we create a new shell to avoid disturbing the current environment. In the new environment, we create the environment variables using the '.env' file. Finally, we replace the variables:
> envsubst < mydeploy.yaml | kubectl apply -f -
PersistentVolumes and VolumeClaims
Kompose created a VolumeMounts and a Volumes section for me in the Pods YAML files using hostPath. This means that the localhost actual directories are hard-coded in the Deployment files. Obviously not what we want.
So I created some PersistentVolume and PersistentVolumeClaim manifests and used them in the Deployment files.
Create Services for the Deployments
Here I show only the local-web service.
# task-runner-local-web-service.yaml apiVersion: v1 kind: Service metadata: # metadata name must not contain dots name: task-runner-local-web-service namespace: default spec: ports: - name: http port: 80 targetPort: 80 protocol: TCP selector: task-runner.local-web.deployment: task-runner-local-web-deployment
Start the service:
> kubectl apply -f task-runner-local-web-service.yaml
Application code changes
I needed only small changes to my application code. As mentioned above, I prefixed the initial Service names with 'task-runner-'. To keep compatibility with Docker-Compose I now use environment variables for the Service names and ports, specified as key-value pairs in the '.env' file.
Although ports are now specified in all Services, there is no need to change anything because the ports are bound to the Services, and every Service has its own IP address.
To check where an instance of a Pod is running I store the environment variable HOSTNAME together with the result of a task. For Docker Swarm I used .Node.Hostname. In the logs we can check this field.
Using nodePort to expose the rabbitmq service to the Back End
The Task Runner is up and running but there is no connection to the outside world. There are several ways to expose the rabbitmq Service and here I use nodePort. We simply create an extra Service with type nodePort and specify the port number:
# task-runner-rabbitmq-nodeport.yaml apiVersion: v1 kind: Service metadata: # metadata name must not contain dots name: task-runner-rabbitmq-nodeport namespace: default spec: type: NodePort # The range of valid ports is 30000-32767 ports: - name: amqp port: 10763 targetPort: 5672 nodePort: 31763 - name: http port: 20763 targetPort: 15672 nodePort: 32763 selector: task-runner.rabbitmq.deployment: task-runner-rabbitmq-deployment
Then we can refer to this Kubernetes Service on the host like: localhost:<port>.
To connect the Backend Docker container to this port we add some lines to the 'docker-compose.yaml' file:
extra_hosts: - "host.docker.internal:host-gateway"
And then refer to the service as:
Adding a Worker node, DNS not working
Of course I wanted to add a new node and run some Pods there. With VitualBox on my development machine I created a VM with Ubuntu server 22.04. Then added Microk8s and joined the new node.
Problems ... Kubernetes DNS was not working on the new node. There are a lot of DNS issues mentioned on the Microk8s Github page. This probably has to do with iptables ...
I was able to have a proper working Kubernetes cluster of two VirtualBox machines. At the moment I am investigating this.
Developing and troubleshooting on Kubernetes
One of the Pods did not work. I could see in the log what was wrong but we do not want to go into a loop of:
--+-> make changes ---> build image ---> test --+---> ^ | | v +--------------------<------------------------+
So I mounted the external project code back into the container, which takes me back to the development environment and was able to solve the problem very fast.
I only used one extra tool during migration: yamllint to validate manifests (YAML files).
> sudo apt-get install yamllint
And then run for example:
> yamllint task-runner-local-web-deployment.yaml
To start with, I spend only two week with Kubernetes (don't shoot the piano player). I may have made some wrong choices but part of my application is now running and managed by Kubernetes.
Docker Swarm is an easy choice when it comes to orchestrating Docker-Compose projects. It just works, without serious limitations. For my application, which consists of multiple Docker-Compose projects, I don't need thousands of replicas. With Docker Swarm, you start up a new node, add some deploy directives to your docker-compose.yml files, and you're done. Docker network is one of Docker's best features. With Docker Swarm, all you have to do is turn a network into an encrypted Ingress overlay network to communicate between nodes.
Moving to Kubernetes gives more deployment flexibility, but the main reason Kubernetes has become the first choice is that it is very widespread and many tools are integrated with it. That's where Docker Swarm lags behind. At the time of writing this post, development of Docker Swarm seems more or less stalled (considered fully feature-complete?). That's a shame, because it's a great solution to a very complex problem.
In an ideal world, the development environment is identical to the production environment. If it runs in development, it also runs in production. Docker Swarm comes very close to this, without much complexity. Using Kubernetes, on the other hand, creates a gap. It's like, Ok developer, you've done your job, now leave it to the DevOps to put it into production. If the developer makes changes to Docker-Compose projects, the DevOps have to work on them as well.
If you use Kubernetes for production, I think it's inevitable that the development environment also runs Kubernetes. And in many cases, Docker Swarm as well. Does this make development and production easier or harder compared to just Docker Swarm? Microk8s is very easy to use in a development environment, but that's just the beginning. Creating Docker-Compose project files is very easy. You usually have two yaml files and an '.env' file. After the conversion, my Kubernetes project already had 14 Kubernetes yaml files and more.
It would be a very good idea if Kubernetes added some extensions for the YAML files that allow importing/ adding other files (such as Nginx) and environment data. For now, we have to write our own scripts or use something like Helm ... more things to learn.
Anyway, in this post I made an attempt to investigate what it would take to move part of my application from Docker Swarm to Kubernetes. Based on the results, I have decided to move to Kubernetes, I will not continue using Docker Swarm for production. And I will also use Kubernetes for development.
Starting with Kubernetes is not very difficult, but sometimes very confusing. After a few days you understand most of the concepts and have something working. From there you can improve and expand. There can be (very) serious and time-consuming problems, such as DNS not working. I don't want to become an iptables specialist, but there is no other way.
The number of changes I had to make to my application is very limited:
- Changing names to RFC1123.
- Prefixing service names, using environment variables.
- Storing other environment variables in results.
Back to coding ...
Links / credits
Configure a Pod to Use a ConfigMap
Connecting Kubernetes and Docker
Helm - The package manager for Kubernetes
How to go from Docker to Kubernetes the right way
How To Migrate a Docker Compose Workflow to Kubernetes
Kubernetes - Documentation - Concepts
Kubernetes - Documentation - kubectl Cheat Sheet
Kustomize - Kubernetes native configuration management
MicroK8s documentation - home
Running Kubernetes locally on Linux with Microk8s
Using private docker registry inside kubernetes
When should I use envFrom for configmaps?
- Using UUIDs instead of Integer Autoincrement Primary Keys with SQLAlchemy and MariaDb
- Using Python's pyOpenSSL to verify SSL certificates downloaded from a host
- Connect to a service on a Docker host from a Docker container
- Using PyInstaller and Cython to create a Python executable
- SQLAlchemy: Using Cascade Deletes to delete related objects
- Flask RESTful API request parameter validation with Marshmallow schemas