Kubernetes#

Mainflux can be easily deployed on Kubernetes platform by using Helm Chart from official Mainflux DevOps GitHub repository.

Prerequisites#

  • Kubernetes
  • kubectl
  • Helm v3
  • Stable Helm repository
  • Nginx Ingress Controller

Kubernetes#

Kubernetes is an open source container orchestration engine for automating deployment, scaling, and management of containerised applications. Install it locally or have access to a cluster. Follow these instructions if you need more information.

Kubectl#

Kubectl is official Kubernetes command line client. Follow these instructions to install it.

Regarding the cluster control with kubectl, default config .yaml file should be ~/.kube/config.

Helm v3#

Helm is the package manager for Kubernetes. Follow these instructions to install it.

Stable Helm Repository#

Add a stable chart repository:

helm repo add stable https://charts.helm.sh/stable

Add a bitnami chart repository:

helm repo add bitnami https://charts.bitnami.com/bitnami

Nginx Ingress Controller#

Follow these instructions to install it or:

helm install ingress-nginx ingress-nginx/ingress-nginx --version 3.26.0 --create-namespace -n ingress-nginx

Deploying Mainflux#

Get Helm charts from Mainflux DevOps GitHub repository:

git clone https://github.com/MainfluxLabs/devops.git
cd devops/charts/mainflux

Update the on-disk dependencies to mirror Chart.yaml:

helm dependency update

If you didn't already have namespace created you should do it with:

kubectl create namespace mf

Deploying release named mainflux in namespace named mf is done with just:

helm install mainflux . -n mf

Mainflux is now deployed on your Kubernetes.

Customizing Installation#

You can override default values while installing with --set option. For example, if you want to specify ingress hostname and pull latest tag of users image:

helm install mainflux -n mf --set ingress.hostname='example.com' --set users.image.tag='latest'

Or if release is already installed, you can update it:

helm upgrade mainflux -n mf --set ingress.hostname='example.com' --set users.image.tag='latest'

The following table lists the configurable parameters and their default values.

Parameter Description Default
defaults.logLevel Log level debug
defaults.image.pullPolicy Docker Image Pull Policy IfNotPresent
defaults.image.repository Docker Image Repository mainflux
defaults.image.tag Docker Image Tag 0.11.0
defaults.replicaCount Replicas of MQTT adapter, Things, Envoy and Authn 3
defaults.natsPort NATS port 4222
defaults.jaegerPort Jaeger port 6831
nginxInternal.mtls.tls TLS secret which contains the server cert/key
nginxInternal.mtls.intermediateCrt Generic secret which contains the intermediate cert used to verify clients
ingress.enabled Should the Nginx Ingress be created true
ingress.hostname Hostname for the Nginx Ingress
ingress.tls.hostname Hostname of the Nginx Ingress certificate
ingress.tls.secret TLS secret for the Nginx Ingress
nats.maxPayload Maximum payload size in bytes that the NATS server will accept 268435456
nats.replicaCount NATS replicas 3
authn.dbPort AuthN service DB port 5432
authn.grpcPort AuthN service gRPC port 8181
authn.httpPort AuthN service HTTP port 8189
users.dbPort Users service DB port 5432
users.httpPort Users service HTTP port 8180
things.dbPort Things service DB port 5432
things.httpPort Things service HTTP port 8182
things.authGrpcPort Things service Auth gRPC port 8183
things.authHttpPort Things service Auth HTTP port 8989
things.redisESPort Things service Redis Event Store port 6379
things.redisCachePort Things service Redis Auth Cache port 6379
adapter_http.httpPort HTTP adapter port 8185
mqtt.proxy.mqttPort MQTT adapter proxy port 1884
mqtt.proxy.wsPort MQTT adapter proxy WS port 8081
mqtt.broker.mqttPort MQTT adapter broker port 1883
mqtt.broker.wsPort MQTT adapter broker WS port 8080
mqtt.broker.persistentVolume.size MQTT adapter broker data Persistent Volume size 5Gi
mqtt.redisESPort MQTT adapter Event Store port 6379
mqtt.redisCachePort MQTT adapter Redis Auth Cache port 6379
adapter_coap.udpPort CoAP adapter UDP port 5683
ui.port UI port 3000
influxdb.enabled Enable InfluxDB reader & writer false
influxdb.dbPort InfluxDB port 8086
influxdb.writer.httpPort InfluxDB writer HTTP port 8900
influxdb.reader.httpPort InfluxDB reader HTTP port 8905
adapter_lora.enabled Enable LoRa adapter false
adapter_lora.httpPort LoRa adapter HTTP port 8187
adapter_lora.redisRouteMapPort LoRa adapter Redis Auth Cache port 6379

All Mainflux services (both core and add-ons) can have their logLevel, image.pullPolicy, image.repository and image.tag overridden.

Mainflux Core is a minimalistic set of required Mainflux services. They are all installed by default:

  • auth
  • users
  • things
  • adapter_http
  • adapter_mqtt
  • adapter_coap
  • adapter_ws
  • ui

Mainflux Add-ons are optional services that are disabled by default. Find in Configuration table parameters for enabling them, i.e. to enable influxdb reader & writer you should run helm install with --set influxdb=true. List of add-ons services in charts:

  • influxdb.writer
  • influxdb.reader
  • adapter_lora

By default scale of MQTT adapter, Things, Envoy, Authn and NATS will be set to 3. It's recommended that you set this values to number of your nodes in Kubernetes cluster, i.e. --set defaults.replicaCount=3 --set nats.replicaCount=3

Additional Steps to Configure Ingress Controller#

To send MQTT messages to your host on ports 1883 and 8883 some additional steps are required in configuring NGINX Ingress Controller.

NGINX Ingress Controller uses ConfigMap to expose TCP services. That ConfigMap is included in helm chart in ingress.yaml file assuming that location of ConfigMap should be ingress-nginx/tcp-services. If Ingress Controller expects it in some other namespace or with other name you should edit metadata in ingress.yaml. This location was set with --tcp-services-configmap flag and you can check it in deployment of Ingress Controller or add it there in args section for nginx-ingress-controller if it's not already specified. This is explained in NGINX Ingress documentation

Also, those two ports need to be exposed in the Service defined for the Ingress. You can do that with command that edit your service:

kubectl edit svc -n ingress-nginx nginx-ingress-ingress-nginx-controller

and add in spec->ports:

  - name: mqtt
    port: 1883
    protocol: TCP
    targetPort: 1883
  - name: mqtts
    port: 8883
    protocol: TCP
    targetPort: 8883

TLS & mTLS#

For testing purposes you can generate certificates as explained in detail in authentication chapter of this document. So, you can use this script and after replacing all localhost with your hostname, run:

make ca
make server_cert
make thing_cert KEY=<thing_key>

you should get in certs folder these certificates that we will use for setting up TLS and mTLS:

ca.crt
ca.key
ca.srl
mainflux-server.crt
mainflux-server.key
thing.crt
thing.key

Create kubernetes secrets using those certificates with running commands from (secrets script)[https://github.com/MainfluxLabs/devops/blob/master/charts/mainflux/secrets/secrets.sh]. In this example secrets are created in mf namespace:

kubectl -n mf create secret tls mainflux-server \
    --key mainflux-server.key \
    --cert mainflux-server.crt

kubectl -n mf create secret generic ca \
    --from-file=ca.crt

You can check if they are succesfully created: kubectl get secrets -n mf

And now set ingress.hostname, ingress.tls.hostname to your hostname and ingress.tls.secret to mainflux-server and after helm update you have secured ingress with TLS certificate.

For mTLS you need to set nginx_internal.mtls.tls="mainflux-server" and nginx_internal.mtls.intermediate_crt="ca".

Now you can test sending mqtt message with this parameters:

mosquitto_pub -d -L mqtts://<thing_id>:<thing_key>@example.com:8883/channels/<channel_id>/messages  --cert  thing.crt --key thing.key --cafile ca.crt  -m "test-message"