[jira] [Created] (METRON-1213) Flow diagram for Metron Kafka topics

["Jasper Knulst (JIRA)" <ji...@apache.org>]

Jasper Knulst created METRON-1213:
-------------------------------------

             Summary: Flow diagram for Metron Kafka topics
                 Key: METRON-1213
                 URL: https://issues.apache.org/jira/browse/METRON-1213
             Project: Metron
          Issue Type: New Feature
    Affects Versions: 0.4.1
         Environment: Management UI
            Reporter: Jasper Knulst
             Fix For: Next + 1


The insight that is really missing in Metron is an overview of Kafka topic sizes and flow metrics (topic influx and outflux)

It is hard to answer questions like:
-Is the enrichments topic growing over time?
-What parser topics feed into enrichments, at what capacity?
-Is the enrichment topology keeping up with the projected influx from parsers?
-Same questions, but then for indexing topic

When the capacity of the chained topologies is not aligned bad thing can happen, like a serious threat not reaching ES fast enough to react upon. Metron can only be a realtime alerter if there is no congestion along the way.

This would take merging metrics from various Metron moving parts like Storm API, kafka-offset-checker (state/snapshots and flow metrics units/sec)  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)