[jira] [Commented] (STORM-1471) Make FetchRequestBuilder.minBytes a configurable parameter in SpoutConfig

["Jungtaek Lim (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/STORM-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15097329#comment-15097329 ] 

Jungtaek Lim commented on STORM-1471:
-------------------------------------

AFAIK, It is intended behavior since nextTuple() in Spout shouldn't be blocked.
You can refer http://storm.apache.org/documentation/Concepts.html - Spouts section.

JStorm provides multi-thread mode of Spout, and it could be applied to Storm when merging Storm and JStorm.
https://issues.apache.org/jira/browse/STORM-1358

> Make FetchRequestBuilder.minBytes a configurable parameter in SpoutConfig
> -------------------------------------------------------------------------
>
>                 Key: STORM-1471
>                 URL: https://issues.apache.org/jira/browse/STORM-1471
>             Project: Apache Storm
>          Issue Type: Bug
>          Components: storm-kafka
>            Reporter: Garrick Dasbach
>
> We currently have an issue in our storm cluster where our Kafka brokers are under heavy load due to too many fetch requests from storm.  We've narrowed the problem to the way Fetch Requests are build in KafkaUtils.  When using the FetchRequestBuilder, storm provides overrides for all the properties except minBytes.  The default for that field is 0 (even though the Kafka default for the high-level consumer is 1).  When paired with a maxWait > 0, this creates a situation where the broker can immediately return a response without waiting (due to minBytes 0).  This puts a heavy load on the brokers and defeats the purpose of any long polling.
> By making this a SpoutConfig option, it will allow the user to set that as appropriate for their situation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)