Implementation of a Large Scale Monitoring System

Imagine you are setting up a backend system that is providing services to over one million autonomous devices that are in constant, real-time communication with the backend – Internet of Things at scale.

Data from the devices is passed through a chain of systems and even routed to 3rd party systems for further computation. Results are then passed back to the devices in near real-time so that they can make autonomous decisions.

The challenge is to keep an overview of what is going in in this huge, distributed system. There is no lack of data about the backend system and the devices connected to it: on the contrary, you need to analyze huge amounts of it, present aggregated facts to decision makers and even make autonomous control decisions, all in near real-time.

We designed and implemented a monitoring system based on the Elastic stack: using Logstash to filter and extract the relevant data, Elasticsearch to store it and provide fast search capabilities and Kibana to display it.

The system is running worldwide and consists of dozens of Elasticsearch clusters, the largest of which has over 100 nodes. Over 8 terabytes of data are ingested daily.