[jira] [Created] (METRON-1897) Warn About Max Spout Pending Exceeded in Profiler Topology

["Nick Allen (JIRA)" <ji...@apache.org>]

Nick Allen created METRON-1897:
----------------------------------

             Summary: Warn About Max Spout Pending Exceeded in Profiler Topology
                 Key: METRON-1897
                 URL: https://issues.apache.org/jira/browse/METRON-1897
             Project: Metron
          Issue Type: Improvement
            Reporter: Nick Allen


If the user sets the max spout pending value too low for the Profiler topology, it can cause the topology to stall and stop consuming messages.

This is an annoying issue to debug because I am unable to find anything logged by Storm indicating that the max spout pending limit is breached. When this limit is exceeded it just appears that the KafkaSpout stops consuming messages from Kafka and everything else would indicate the topology is healthy.

The Profiler topology is the only Metron topology that uses Storm's windowing support. Storm will queue up messages for the window (by default 30 seconds) and process each of the tuples in the window as one unit. This causes Storm to need to "hang on" to tuples longer than the other topologies and this requires a greater max spout pending value.

Ideally, we need to make it more obvious that exceeding the max spout pending value in the Profiler topology is causing the KafkaSpout to stop consuming messages.  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)