[jira] [Created] (STORM-1471) Make FetchRequestBuilder.minBytes a configurable parameter in SpoutConfig

["Garrick Dasbach (JIRA)" <ji...@apache.org>]

Garrick Dasbach created STORM-1471:
--------------------------------------

             Summary: Make FetchRequestBuilder.minBytes a configurable parameter in SpoutConfig
                 Key: STORM-1471
                 URL: https://issues.apache.org/jira/browse/STORM-1471
             Project: Apache Storm
          Issue Type: Bug
          Components: storm-kafka
            Reporter: Garrick Dasbach


We currently have an issue in our storm cluster where our Kafka brokers are under heavy load due to too many fetch requests from storm.  We've narrowed the problem to the way Fetch Requests are build in KafkaUtils.  When using the FetchRequestBuilder, storm provides overrides for all the properties except minBytes.  The default for that field is 0 (even though the Kafka default for the high-level consumer is 1).  When paired with a maxWait > 0, this creates a situation where the broker can immediately return a response without waiting (due to minBytes 0).  This puts a heavy load on the brokers and defeats the purpose of any long polling.

By making this a SpoutConfig option, it will allow the user to set that as appropriate for their situation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)