You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2016/04/04 21:31:25 UTC
[jira] [Commented] (STORM-1680) Provide configuration to set min
fetch size in KafkaSpout
[ https://issues.apache.org/jira/browse/STORM-1680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15224877#comment-15224877 ]
ASF GitHub Bot commented on STORM-1680:
---------------------------------------
GitHub user supermonk opened a pull request:
https://github.com/apache/storm/pull/1309
STORM-1680, Added Kafka Spout Config FetchByte to Storm-Kafka
Added Kafka Spout Config FetchByte to Storm-Kafka https://issues.apache.org/jira/browse/STORM-1680
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Symantec/storm 1.x-branch
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/1309.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1309
----
commit 89f4d44dee45567d6d855842856d5439f9501987
Author: narendra_bidari <na...@symantec.com>
Date: 2016-04-04T19:28:09Z
STORM-1680, Added Kafka Spout Config FetchByte to Storm-Kafka
----
> Provide configuration to set min fetch size in KafkaSpout
> ---------------------------------------------------------
>
> Key: STORM-1680
> URL: https://issues.apache.org/jira/browse/STORM-1680
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-kafka, trident
> Reporter: Sachin Pasalkar
> Assignee: Narendra Bidari
>
> Kafka consumer has provided the configuration to set minimum fetch size. However, storms kafka spout is not exposing these functionality. This is helpful in some case where someone writing data to hdfs & want file size of X.
> Below are changes needs to be done
> 1. In KafkaUtils class update fetchMessages API with below change
> FetchRequest fetchRequest = builder.addFetch(topic, partitionId, offset, config.fetchSizeBytes).clientId(config.clientId).maxWait(config.fetchMaxWait).minBytes(config.minFetchByte).build();
> 2. Update KafkaConfig class with instance variable as minFetchByte (Default value is 0 as mentioned in FetchRequestBuilder class)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)