Using Ingress to access RabbitMQ on a Microk8s cluster

Some time ago I made a post about moving part of an application containing RabbitMQ running on Docker Swarm to Microk8s. We used NodePort to access the Kubernetes cluster RabbitMQ on the host.

Recently I revisted this and decided to use the Microk8s Ingress Controller to access RabbitMQ.
In this post, we assume the RabbitMQ pod is present and only focus on getting the Ingress Controller up and running.

Below, 'kubectl' stands for 'microk8s kubectl'.

As always, I do this on Ubuntu 22.04.

Warning: The default Ingress Controller configuration conflicts with applications already using ports 80 and 443

The Microk8s Ingress Controller is based on Nginx and enables ports 80 and 443 by default. If you already have another application running on your system that uses these ports, such as Nginx, then you have a problem when you enable the Ingress Controller.
In my case, my existing Nginx server no longer worked ... :-(

To avoid this, change the ports of the Ingress Controller to not much used ports BEFORE enabling it:

80  -> 9480
443 -> 9483

We do this by modifying the ports section in the file:

/var/snap/microk8s/common/addons/core/addons/ingress/ingress.yaml

Here is what it looks like after editing:

        ports:
        - name: http
          containerPort: 80
          #hostPort: 80
          hostPort: 9480
        - name: https
          containerPort: 443
          #hostPort: 443
          hostPort: 9483
        - name: health
          containerPort: 10254
          hostPort: 10254

Enabling and disabling the Microk8s Ingress Controller

Now we can safely enable the Ingress Controller:

microk8s enable ingress

Result:

Infer repository core for addon ingress
Enabling Ingress
ingressclass.networking.k8s.io/public created
ingressclass.networking.k8s.io/nginx created
namespace/ingress created
serviceaccount/nginx-ingress-microk8s-serviceaccount created
clusterrole.rbac.authorization.k8s.io/nginx-ingress-microk8s-clusterrole created
role.rbac.authorization.k8s.io/nginx-ingress-microk8s-role created
clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-microk8s created
rolebinding.rbac.authorization.k8s.io/nginx-ingress-microk8s created
configmap/nginx-load-balancer-microk8s-conf created
configmap/nginx-ingress-tcp-microk8s-conf created
configmap/nginx-ingress-udp-microk8s-conf created
daemonset.apps/nginx-ingress-microk8s-controller created
Ingress is enabled

To disable the Ingress Controller later, we use:

microk8s disable ingress

To check if the Ingress Controller pod is running:

kubectl -n ingress get pod

Result:

NAME                                      READY   STATUS    RESTARTS   AGE
nginx-ingress-microk8s-controller-fmqc7   1/1     Running   0          177m

We can check the logs:

kubectl -n ingress logs nginx-ingress-microk8s-controller-74pns

Result:

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.8.0
  Build:         35f5082ee7f211555aaff431d7c4423c17f8ce9e
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.21.6

-------------------------------------------------------------------------------
....

Check that Ingress is listening on the port we specified:

wget 127.0.0.1:9480

Result:

--2024-11-17 19:41:48--  http://127.0.0.1:9480/
Connecting to 127.0.0.1:9480... connected.
HTTP request sent, awaiting response... 404 Not Found
2024-11-17 19:41:48 ERROR 404: Not Found.

Looks good, it's responsding! As you can see, the Microk8s Ingress Contreoller is based on the Nginx web server.

Connecting Ingress to RabbitMQ in the Microk8s cluster

To achieve this, we need to do the following:

1. Edit the Ingress Controller daemonset
2. Edit the Ingress Controller TCP config map
3. Create an Ingress YAML file: 'rabbitmq-ingress.yaml'

Important:

• The name of the RabbitMQ service is 'rabbitmq-service', the port is 5672.
• RabbitMQ is in the default namespace. Change if you are using another namespace.

1. Edit the Ingress Controller daemonset

Here we add a containerPort, 5672, and hostPort, 9493. To edit the daemonset of our Ingress Controller:

kubectl -n ingress edit daemonset nginx-ingress-microk8s-controller

The 'ports' section before the edit:

        ports:
        - containerPort: 80
          hostPort: 9480
          name: http
          protocol: TCP
        - containerPort: 443
          hostPort: 9483
          name: https
          protocol: TCP
        - containerPort: 10254
          hostPort: 10254
          name: health
          protocol: TCP

Edit so that it looks like this. We'll leave the other ports as they are:

        ports:
        - containerPort: 80
          hostPort: 9480
          name: http
          protocol: TCP
        - containerPort: 443
          hostPort: 9483
          name: https
          protocol: TCP
        - containerPort: 5672
          hostPort: 9493
          name: rabbitmq5672
          protocol: TCP
        - containerPort: 10254
          hostPort: 10254
          name: health
          protocol: TCP

Save and exit.

2. Edit the Ingress Controller TCP config map

Here we route external TCP/UDP traffic from non-HTTP protocols to internal services using TCP/UDP port mappings.

To get all the config maps of the Ingress Controller:

kubectl -n ingress get configmap

Result:

NAME                                DATA   AGE
kube-root-ca.crt                    1      3h56m
nginx-ingress-tcp-microk8s-conf     0      3h56m
nginx-ingress-udp-microk8s-conf     0      3h56m
nginx-load-balancer-microk8s-conf   0      3h56m

To edit the TCP config map:

kubectl -n ingress edit configmaps nginx-ingress-tcp-microk8s-conf

Add the following lines:

data:
  5672: "default/rabbitmq-service:5672"

Check that this has been done:

kubectl -n ingress get configmap nginx-ingress-tcp-microk8s-conf -o yaml

Result:

apiVersion: v1
data:
  "5672": default/rabbitmq-service:5672
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"ConfigMap","metadata":{"annotations":{},"name":"nginx-ingress-tcp-microk8s-conf","namespace":"ingress"}}
  creationTimestamp: "2024-11-17T18:26:14Z"
  name: nginx-ingress-tcp-microk8s-conf
  namespace: ingress
  resourceVersion: "9071777"
  uid: c8495fca-7753-4b60-9542-dcb9879ab4d0

3. Create an Ingress YAML file

Finally, we create the following YAML file 'rabbitmq-ingress.yaml':

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: rabbitmq-ingress
spec:
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: rabbitmq-service
            port:
              number: 5672

Deploy the Ingress Controller

After completing the steps above, we can deploy the Ingress Controller:

kubectl apply -f rabbitmq-ingress.yaml

Result:

ingress.networking.k8s.io/rabbitmq-ingress created

Check more information:

kubectl describe ingress rabbitmq-ingress

Result:

Name:             rabbitmq-ingress
Labels:           <none>
Namespace:        default
Address:          127.0.0.1
Ingress Class:    public
Default backend:  <default>
Rules:
  Host        Path  Backends
  ----        ----  --------
  *           
              /   rabbitmq-service:5672 (10.1.105.102:5672)
Annotations:  <none>
Events:       <none>

Check if the IP address matches the RabbitMQ service!

If you later want to remove the Ingress Controller:

kubectl delete ingress rabbitmq-ingress

Test on the host with Pika

This blog is about Python, so here we use the Pika package to create a simple test program.

# rabbitmq_test.py
import logging
import sys

import pika

logging.basicConfig(
    format='%(asctime)s %(levelname)8s [%(filename)-10s%(funcName)20s():%(lineno)04s] %(message)s',
    #level=logging.DEBUG,
    level=logging.INFO,
)
logger = logging.getLogger(__name__)

RABBITMQ_HOST = '127.0.0.1'
RABBITMQ_PORT = 9493
RABBITMQ_USER = '<your-user>'
RABBITMQ_PASSWORD = '<your-password>'

def main():
    
    url_items = [
        'amqp://',
        f'{RABBITMQ_USER}:{RABBITMQ_PASSWORD}',
        '@',
        f'{RABBITMQ_HOST}:{str(RABBITMQ_PORT)}',
        '/%2F',
    ]
    url = ''.join(url_items)
    logger.info(f'url = {url}')

    parameters = pika.URLParameters(url)
    logger.info(f'url = {url}')

    logger.info(f'parameters = {parameters}')
    try:
        connection = pika.BlockingConnection(parameters)
        logger.info(f'connection = {connection}')
    except Exception as e:
        logger.exception(f'connection error')
        raise
    logger.info(f'connected')

    channel = connection.channel()
    queue = 'test_queue'
    channel.queue_declare(queue=queue)

    logger.info(f'publishing ...')
    try:
        channel.basic_publish(
			exchange='test_exchange',
			routing_key=queue,
			body='test_message',
			properties=pika.BasicProperties(
                content_type='text/plain',
                delivery_mode=pika.DeliveryMode.Transient
            ),
        )
    except Exception as e:
        logger.exception(f'publish error')
        raise
    logger.info(f'publish ready')

if __name__ == '__main__':
    main()

Multiple Ingress Controllers

You may want to add another route between the host and the cluster. It is also possible to use multiple Ingress Controllers.

Summary

I got this working without much difficulty. If you check the IP address above, you will note that this is the IP address of the RabbitMQ pod. It looks like the Ingress Controller is connecting to the RabbitMQ pod, not the RabbitMQ service!
This appears to be correct (?). Hmmm ... will look into this later ...

Links / credits

Does an ingress talk to pods directly or its through a service?
https://stackoverflow.com/questions/71955811/does-an-ingress-talk-to-pods-directly-or-its-through-a-service

Exposing Redis with Ingress Nginx Controller
https://stackoverflow.com/questions/62939846/exposing-redis-with-ingress-nginx-controller

How to expose Kubernetes-hosted RabbitMQ to the outside?
https://stackoverflow.com/questions/63334992/how-to-expose-kubernetes-hosted-rabbitmq-to-the-outside

How to Set up Kubernetes Ingress with MicroK8s
https://phoenixnap.com/kb/microk8s-ingress

Issue with exposing rabbitmq port 5671
https://discuss.kubernetes.io/t/issue-with-exposing-rabbitmq-port-5671/15883

Kubernetes - Ingress-Nginx Controller
https://kubernetes.github.io/ingress-nginx

Kubernetes: microk8s with multiple metalLB endpoints and nginx ingress controllers
https://fabianlee.org/2021/07/29/kubernetes-microk8s-with-multiple-metallb-endpoints-and-nginx-ingress-controllers

Microk8s - Addon: Ingress
https://microk8s.io/docs/ingress

Minikube - Ingress nginx for TCP and UDP services
https://minikube.sigs.k8s.io/docs/tutorials/nginx_tcp_udp_ingress