[jira] [Closed] (STORM-1471) Make FetchRequestBuilder.minBytes a configurable parameter in SpoutConfig

["Jungtaek Lim (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/STORM-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jungtaek Lim closed STORM-1471.
-------------------------------
    Resolution: Not A Problem

> Make FetchRequestBuilder.minBytes a configurable parameter in SpoutConfig
> -------------------------------------------------------------------------
>
>                 Key: STORM-1471
>                 URL: https://issues.apache.org/jira/browse/STORM-1471
>             Project: Apache Storm
>          Issue Type: Bug
>          Components: storm-kafka
>            Reporter: Garrick Dasbach
>
> We currently have an issue in our storm cluster where our Kafka brokers are under heavy load due to too many fetch requests from storm.  We've narrowed the problem to the way Fetch Requests are build in KafkaUtils.  When using the FetchRequestBuilder, storm provides overrides for all the properties except minBytes.  The default for that field is 0 (even though the Kafka default for the high-level consumer is 1).  When paired with a maxWait > 0, this creates a situation where the broker can immediately return a response without waiting (due to minBytes 0).  This puts a heavy load on the brokers and defeats the purpose of any long polling.
> By making this a SpoutConfig option, it will allow the user to set that as appropriate for their situation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)