MQTT K8s Setup on Ubuntu 20.04

Install the k8s cluster

1
2
microk8s enable dns storage helm3
microk8s status

Pull the Mqtt helm chart

1
2
microk8s helm repo add truecharts https://charts.truecharts.org/
microk8s helm pull truecharts/mosquitto --version 8.0.11

Modify the values.yaml file to set the configuration

We’re going to set the mosquitto configuration within the container.

1
2
3
tar -xvf ./mosquitto-8.0.11.tgz
cd mosquitto
cat values.yamls

Bottom of values.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
persistence:
  data:
    enabled: true
    mountPath: "/mosquitto/data"
  configinc:
    enabled: true
    mountPath: "/mosquitto/configinc"
  mosquitto-config:
    enabled: "true"
    mountPath: "/mosquitto/config/mosquitto.conf"
    subPath: "mosquitto.conf"
    type: "custom"
    volumeSpec:
      configMap:
        name: '{{ template "tc.common.names.fullname" . }}-config'

mosquitto-config is a config map.
configinc and data are both directories on the host machine.

Make the data directory

1
2
3
4
5
mkdir /mnt/Poseidon/k8s/mosquitto
mkdir /mnt/Poseidon/k8s/mosquitto/data
mkdir /mnt/Poseidon/k8s/mosquitto/configinc

chown -R 777 /mnt/Poseidon/k8s/mosquitto

Create Persistent Volumes and Persistent Volume Claims

Persistent Volumes

configinc

HOSTNAME: example_host
FILE_PATH: /path/to/files/

mosquitto-configinc-pv.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
apiVersion: v1
kind: PersistentVolume
metadata:
  name: mosquitto-configinc-pv
spec:
  capacity:
    storage: 3Ti
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /path/to/files/mosquitto/configinc # This must exist on the host
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - example_host

microk8s kubectl apply -f mosquitto-configinc-pv.yaml

1
persistentvolume/mosquitto-configinc-pv created

Data

HOSTNAME: example_host

mosquitto-data-pv.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
apiVersion: v1
kind: PersistentVolume
metadata:
  name: mosquitto-data-pv
spec:
  capacity:
    storage: 3Ti
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /path/to/files/mosquitto/data # This must exist on the host
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - example_host

microk8s kubectl apply -f mosquitto-data-pv.yaml

1
persistentvolume/mosquitto-data-pv created

Verify PV

microk8s kubectl get pv 

1
2
3
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                                 STORAGECLASS        REASON   AGE
mosquitto-configinc-pv                     3Ti        RWO            Retain           Available                                         local-storage                97s
mosquitto-data-pv                          3Ti        RWO            Retain           Available                                         local-storage                35s

Persistent Volume Claims

configinc

mosquitto-configinc-pvc.yaml

mosquitto-configinc-pvc.yaml
Bonds to mosquitto-configinc-pv

1
2
3
4
5
6
7
8
9
10
11
12
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mosquitto-configinc-pvc
spec:
  storageClassName: local-storage # Empty string must be explicitly set otherwise default StorageClass will be set
  accessModes:
    - ReadWriteOnce
  volumeName: mosquitto-configinc-pv
  resources:
    requests:
      storage: 3Ti

microk8s kubectl apply -f mosquitto-configinc-pvc.yaml

Data

mosquitto-data-pvc.yaml
Bonds to mosquitto-data-pv

1
2
3
4
5
6
7
8
9
10
11
12
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mosquitto-data-pvc
spec:
  storageClassName: local-storage # Empty string must be explicitly set otherwise default StorageClass will be set
  accessModes:
    - ReadWriteOnce
  volumeName: mosquitto-data-pv
  resources:
    requests:
      storage: 3Ti

microk8s kubectl apply -f mosquitto-data-pvc.yaml

Verify PV

microk8s kubectl get pvc 

1
2
mosquitto-configinc-pvc       Bound    mosquitto-configinc-pv                     3Ti        RWO            local-storage       4m44s
mosquitto-data-pvc            Bound    mosquitto-data-pv                          3Ti        RWO            local-storage       2m7s

Update the values file

We know need to tell the helm chart to mount our directories 

1
nano values.yamls

Update the persistence section as follows

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
image:
  repository: tccr.io/truecharts/eclipse-mosquitto
  tag: 2.0.15@sha256:9e1fbb32ae27aaaf18432ff4e7e046c54fda8a630851dadf0c332a94e81fbf67
  pullPolicy: IfNotPresent

service:
  main:
    ports:
      main:
        port: 1883
        targetPort: 1883
  websockets:
    enabled: true
    ports:
      websockets:
        enabled: true
        port: 9001
        targetPort: 9001

ingress:
  websockets:
    autoLink: true

auth:
  # -- By enabling this, `allow_anonymous` gets set to `false` in the mosquitto config.
  enabled: false

websockets:
  # -- By enabling this, an additional listener with protocol websockets is added in the mosquitto config.
  enabled: false

configmap:
  config:
    enabled: true
    data:
      mosquitto.conf: |
        listener {{ .Values.service.main.ports.main.targetPort }}
        {{- if .Values.websockets.enabled }}
        listener {{ .Values.service.websockets.ports.websockets.targetPort }}
        protocol websockets
        {{- end }}
        {{- if .Values.auth.enabled }}
        allow_anonymous false
        {{- else }}
        allow_anonymous true
        {{- end }}
        {{- if .Values.persistence.data.enabled }}
        persistence true
        persistence_location {{ .Values.persistence.data.mountPath }}
        autosave_interval 1800
        {{- end }}
        {{- if .Values.persistence.configinc.enabled }}
        include_dir {{ .Values.persistence.configinc.mountPath }}
        {{- end }}

persistence:
  data:
    enabled: true
    mountPath: "/mosquitto/data"
    type: pvc
    existingClaim: mosquitto-data-pvc
  configinc:
    enabled: true
    mountPath: "/mosquitto/configinc"
    type: pvc
    existingClaim: mosquitto-configinc-pvc
  mosquitto-config:
    enabled: "true"
    mountPath: "/mosquitto/config/mosquitto.conf"
    subPath: "mosquitto.conf"
    type: "custom"
    volumeSpec:
      configMap:
        name: '{{ template "tc.common.names.fullname" . }}-config'

portal:
  enabled: false

Install the helm chart

1
microk8s helm install mosquitto truecharts/mosquitto --version 8.0.11 --values ./values.yaml

Check the status of the installed application.

1
2
3
4
5
microk8s status
microk8s kubectl show pods
microk8s kubectl
microk8s kubectl logs
microk8s kubectl describe pods

Common Errors

Error from server (BadRequest): container in pod is waiting to start: ContainerCreating
You probably need to change the permissions on the PV directory. This path is what is written in the PersistentVolume in the path varaibale. A quick chmod -R 777 to this path will most likely fix the issue. The conatiner should update the permissions once it runs.

Service yaml file

We need to expose the service to the outside world.
Thankfully microk8s has a built in loadbalancer called metallb

Replace Y with the MQTT port number. Default 1883
Replace Z with the MQTT API number Default 9001

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
apiVersion: v1
kind: Service
metadata:
  name: mqtt-service
spec:
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: mosquitto
  ports:
    - name: http
      protocol: TCP
      port: Y
      targetPort: Y
    - name: https
      protocol: TCP
      port: Z
      targetPort: Z
  externalIPs:
    - X.X.X.X

More information here
https://medium.com/swlh/kubernetes-external-ip-service-type-5e5e9ad62fcd

Apply the service

1
microk8s kubectl apply -f ./mqtt-service.yaml

Confirm the service is active

1
microk8s kubectl describe services mqtt-service

Local test client

You can test the connection locally on the server with this simple CLI mqtt client
Test client
https://mqttx.app/cli

Firewall Rules

This assumes you are using ufw.
ufw is bascally a wrapper for IPTABLES. If you have ever used IPTABLES before you understand why ufw exists.

1
2
3
sudo ufw default allow routed 
sudo ufw allow from X.X.X.0/X to any port Y proto tcp
sudo ufw status

Test external Connections

Once again try to connect to port Y with https://mqttx.app/cli

References

https://kubernetes.io/docs/concepts/services-networking/service/