[jira] [Updated] (METRON-1213) Flow diagram for Metron Kafka topics

["Justin Leet (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/METRON-1213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Justin Leet updated METRON-1213:
--------------------------------
    Fix Version/s:     (was: Next + 1)

> Flow diagram for Metron Kafka topics
> ------------------------------------
>
>                 Key: METRON-1213
>                 URL: https://issues.apache.org/jira/browse/METRON-1213
>             Project: Metron
>          Issue Type: New Feature
>    Affects Versions: 0.4.1
>         Environment: Management UI
>            Reporter: Jasper Knulst
>            Priority: Major
>
> The insight that is really missing in Metron is an overview of Kafka topic sizes and flow metrics (topic influx and outflux)
> It is hard to answer questions like:
> -Is the enrichments topic (lag) growing over time?
> -What parser topics feed into enrichments, at what rate?
> -Is the enrichment topology keeping up with the projected influx from all parsers?
> -Same questions, but then for indexing topic
> When the capacity of the chained topologies is not aligned bad things can happen, like a serious threat not reaching ES/Kibana fast enough to react upon. Metron can only be a realtime alerter if there is no congestion along the way.
> This would take merging metrics from various Metron moving parts like Storm API, kafka-offset-checker (state/snapshots and flow metrics units/sec)  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)