MonitoFi: Solution for Monitoring Health & Performance of Apache NiFi Cluster

[Tushar Dhadiwal <tu...@gmail.com>]

Hello Everyone,

Few Months ago, I was looking for a comprehensive solution that can help me
monitor our Apache NiFi clusters.

Some of the expectations I had in Mind from this solution were:

   - It should be customisable, So I can choose which processors , queues,
   etc. to monitor
   - I should be able to set thresholds on certain parameters and get
   email/slack/teams alerts whenever these thresholds are crossed.
   - If a node is down. I should be notified.
   - I should be able to see historical data about the state of my cluster,
   for example how is the disk space filling up, are certain connections under
   backpressure during specific times etc.
   - It should be reliable i.e. even if the Cluster goes down, I should get
   an alert. So something that runs externally and not within NiFi.
   - It Should support on-prem or air gapped NiFi Clusters.


I didn't find anything open source that would help on this and was easy to
use.

Thanks to the community, I came across Apache NiFi Rest API
https://nifi.apache.org/docs/nifi-docs/rest-api/index.html
<https://nifi.apache.org/docs/nifi-docs/rest-api/index.html>.

Using Data from this Rest API, Over the past few months I have developed &
Tested *MonitoFi*.
It is supposed to work as a health and performance monitor for Apache NiFi.

How does it work?

   1. It runs externally in a docker container and collects data
   periodically from any of the GET endpoints (You can choose which ones) in
   Apache NiFi Rest API.
   2. This Data is Stored in InfluxDB that is running in another container
   along with Grafana.
   https://github.com/tushardhadiwal/docker-influxdb-grafana
   <https://github.com/tushardhadiwal/docker-influxdb-grafana>
   3. Grafana is used to showcase Graphs and Manage Alerts etc. It is also
   possible to store data in Azure if needed for backups etc.


If someone wants to try it, I have included a deployment script
<http://install.monitofi.com/> that handles end2end deployment of
containers, Grafana and InfluxDB Configuration and Imports Grafana
Dashboards which include about 10 sample queries tracking various
parameters for a 3 node NiFi Cluster. Feel free to open up an issue on
github if you have any trouble getting this to run.

I am happy to share it with the community and gather some feedback on
whether this is something that people find to be useful. I have few more
improvements in mind for the near future.

*More info is available at http://www.monitofi.com/
<http://www.monitofi.com/>*

Let me know what you think?!

Cheers,
Tushar Dhadiwal <https://www.linkedin.com/in/tushar-dhadiwal/>

[image: MonitoFiDashboard.png]