You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Shushant Arora <sh...@gmail.com> on 2015/08/26 17:39:33 UTC
spark streaming 1.3 kafka buffer size
whats the default buffer in spark streaming 1.3 for kafka messages.
Say In this run it has to fetch messages from offset 1 to 10000. will it
fetch all in one go or internally it fetches messages in few messages
batch.
Is there any setting to configure this no of offsets fetched in one batch?
Re: spark streaming 1.3 kafka buffer size
Posted by Cody Koeninger <co...@koeninger.org>.
As it stands currently, no.
If you're already overriding the dstream, it would be pretty
straightforward to change the kafka parameters used when creating the rdd
for the next batch though
On Wed, Aug 26, 2015 at 11:41 PM, Shushant Arora <sh...@gmail.com>
wrote:
> Can I change this param fetch.message.max.bytes or spark.streaming.kafka.maxRatePerPartition
> at run time across batches.
> Say I detected some fail condition in my system and I decided to sonsume i
> next batch interval only 10 messages per partition and if that succeed I
> reset the max limit to unlimited again .
>
> On Wed, Aug 26, 2015 at 9:32 PM, Cody Koeninger <co...@koeninger.org>
> wrote:
>
>> see http://kafka.apache.org/documentation.html#consumerconfigs
>>
>> fetch.message.max.bytes
>>
>> in the kafka params passed to the constructor
>>
>>
>> On Wed, Aug 26, 2015 at 10:39 AM, Shushant Arora <
>> shushantarora09@gmail.com> wrote:
>>
>>> whats the default buffer in spark streaming 1.3 for kafka messages.
>>>
>>> Say In this run it has to fetch messages from offset 1 to 10000. will it
>>> fetch all in one go or internally it fetches messages in few messages
>>> batch.
>>>
>>> Is there any setting to configure this no of offsets fetched in one
>>> batch?
>>>
>>
>>
>
Re: spark streaming 1.3 kafka buffer size
Posted by Shushant Arora <sh...@gmail.com>.
Can I change this param fetch.message.max.bytes or
spark.streaming.kafka.maxRatePerPartition
at run time across batches.
Say I detected some fail condition in my system and I decided to sonsume i
next batch interval only 10 messages per partition and if that succeed I
reset the max limit to unlimited again .
On Wed, Aug 26, 2015 at 9:32 PM, Cody Koeninger <co...@koeninger.org> wrote:
> see http://kafka.apache.org/documentation.html#consumerconfigs
>
> fetch.message.max.bytes
>
> in the kafka params passed to the constructor
>
>
> On Wed, Aug 26, 2015 at 10:39 AM, Shushant Arora <
> shushantarora09@gmail.com> wrote:
>
>> whats the default buffer in spark streaming 1.3 for kafka messages.
>>
>> Say In this run it has to fetch messages from offset 1 to 10000. will it
>> fetch all in one go or internally it fetches messages in few messages
>> batch.
>>
>> Is there any setting to configure this no of offsets fetched in one batch?
>>
>
>
Re: spark streaming 1.3 kafka buffer size
Posted by Cody Koeninger <co...@koeninger.org>.
see http://kafka.apache.org/documentation.html#consumerconfigs
fetch.message.max.bytes
in the kafka params passed to the constructor
On Wed, Aug 26, 2015 at 10:39 AM, Shushant Arora <sh...@gmail.com>
wrote:
> whats the default buffer in spark streaming 1.3 for kafka messages.
>
> Say In this run it has to fetch messages from offset 1 to 10000. will it
> fetch all in one go or internally it fetches messages in few messages
> batch.
>
> Is there any setting to configure this no of offsets fetched in one batch?
>