Mainflux IIoT Distributed MQTT Load Testing.

The repository contains mzbench production-ready scenarios for Mainflux IIoT MQTT load test and detailed reports from running those tests on Kubernetes cluster, hosted on different cloud providers like AWS, GKE, Digitalocean, etc...

The main purpose is to measure Mainflux MQTT messaging performance and scaling in different production environments with different compute capacity and scenarios. It should also help you with a rough estimation of cluster capacity and compute power required for a certain number of concurrent connections or messages per second and monthly costs for such clusters.

Introduction

Mainflux is a scalable, secure, open-source, and patent-free IIoT cloud platform.

MQTT is a messaging backbone protocol in Mainflux and in general most commonly used protocol for IoT. Mainflux uses mproxy which follow sidecar pattern to proxy MQTT traffic betwen client and broker. For more info checkout architecture diagram.

Motivation comes due to MQTT complexity in real-life and production environments, reliable, secure and scalable connectivity and messaging is hard. We decided to remove the limitation and introduced mproxy with "Bring your own broker" philosophy. Thanks to this idea, Mainflux is capable to work with any MQTT Broker from the market. AuthN and AuthZ (mTLS, RBAC policies) if fully offloaded to mproxy and we left MQTT part to MQTT broker.

Default MQTT broker which comes with the Mainflux IIoT platform, out of the box is VerneMQ.

It's our preferred and recommended broker, highly scalable and clustered on Kubernetes with Mainflux Helm charts but NOT required, you are free to use ANY MQTT broker, thanks to mproxy you are welcome to "Bring your own broker" which suits best for your use case.

As I said, reliable, secure and scalable communication is hard, the holy grail of IoT. Different environments and use cases require different test cases and testing is hard. This repository and test case scenarios will help you to test Mainflux platform MQTT messaging with the most commonly used connectivity patterns like fan-in, fan-out, etc... with periodic high load on production-ready Kubernetes cluster.

How this is supposed to be a production environment load testing, you will not see single node's here, testing on localhost, "tested on my Mac", etc... no fairy tales!

Testing toolbelt

Running Mainflux platform on Kubernetes.

Thanks to Helm, you can sping Mainflux on running Kubernetes cluster in 2min using Mainflux helm charts.
mzbench Thanks to @satori-com

Follow official installtion guide For UI follow How to use Dashboard
vmq_mzbenc worker plugin for MQTT testing. Thanks to @vernemq

You don't need to install it, it will be pulled from Github with running scenario. For more info check out great VerneMQ blog
mzb_api_ec2_plugin built-in AWS cloud plugin for workers distribution.

Prerequisites

Two provisioned Mainflux things connected to the same channel. We will call it thing_1 and thing_2 in our test scenarios. We need two things because we need to pub and sub at the same time.

For more details on how to do provisioning, checkout Mainflux provisioning documentation

How it works

You can't get the right number of concurred connections from one host/client, due to OS limit, client library, socket limit, etc... Testing from a single client is not realistic and far away from production-ready environments, but it's better than no testing at all. Besides connections and load, it's crucial to test real-life scenarios, together with AuthN and AuthZ, TLS determination, etc... everything that one production environment requires. All those layers and network hops add extra delays and milliseconds are important, especially with a high number of connections or messages.

Scenarios

Here is a list of scenarios that we will test.

Results

Kubernetes is a defacto standard for production-grade container orchestration, that's why we use Kubernetes in all our test environments with all scenarios. Testing is done on different infrastructures, regarding cluster resources, a number of nodes, compute power, etc... with different cloud providers. Each report contains details about cluster resources, testing scenario, test report and rough monthly cost estimate (its impossible to do precise costs estimation due to variable costs and use case details).

Digitalocean Kubernetes. Detailed report available here
Google GKE
AWS EKS

AWS Distributed testing

There is a mzbench community optimized AMI available on AWS (search for mzbench AMI) which can be used with AWS built in cloud plugin for distributed testing.

License

mteodor/benchmark

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Packages