spark streaming max receiver rate doubts

[Shushant Arora <sh...@gmail.com>]

1.In spark 1.3(Non receiver)  - If my batch interval is 1 sec and I don't
set spark.streaming.kafka.maxRatePerPartition - so default behavious is to
bring all messages from kafka from last offset to current offset ?

Say no of messages were large and it took 5 sec to process those so will
all jobs for interval 2-5 sec be queued and created afterwards or should
not be created since all messages are already processed for those interval
also?

2.In spark streaming 1.2(Receiver based) if I don't set
spark.streaming.receiver.maxRate
- will it consume all messages from last offset or it will just consume
messages whatever it can consume in this batch interval of 1 sec.

Re: spark streaming max receiver rate doubts

[Cody Koeninger <co...@koeninger.org>]

Those jobs will still be created for each valid time, they just may not
have many messages in them

On Mon, Aug 3, 2015 at 11:11 PM, Shushant Arora <sh...@gmail.com>
wrote:

> 1.In spark 1.3(Non receiver)  - If my batch interval is 1 sec and I don't
> set spark.streaming.kafka.maxRatePerPartition - so default behavious is to
> bring all messages from kafka from last offset to current offset ?
>
> Say no of messages were large and it took 5 sec to process those so will
> all jobs for interval 2-5 sec be queued and created afterwards or should
> not be created since all messages are already processed for those interval
> also?
>
> 2.In spark streaming 1.2(Receiver based) if I don't set spark.streaming.receiver.maxRate
> - will it consume all messages from last offset or it will just consume
> messages whatever it can consume in this batch interval of 1 sec.
>