Re: Kafka directsream receiving rate

[Cody Koeninger <co...@koeninger.org>]

If you're using the direct stream, you have 0 receivers.  Do you mean you
have 1 executor?

Can you post the relevant call to createDirectStream from your code, as
well as any relevant spark configuration?

On Thu, Feb 4, 2016 at 8:13 PM, Diwakar Dhanuskodi <
diwakar.dhanuskodi@gmail.com> wrote:

> Adding more info
>
> Batch  interval  is  2000ms.
> I expect all 100 messages  go thru one  dstream from  directsream but it
> receives at rate of 10 messages at time. Am  I missing  some
>  configurations here. Any help appreciated.
>
> Regards
> Diwakar.
>
>
> Sent from Samsung Mobile.
>
>
> -------- Original message --------
> From: Diwakar Dhanuskodi <di...@gmail.com>
> Date:05/02/2016 07:33 (GMT+05:30)
> To: user@spark.apache.org
> Cc:
> Subject: Kafka directsream receiving rate
>
> Hi,
> Using spark 1.5.1.
> I have a topic with 20 partitions.  When I publish 100 messages. Spark
> direct stream is receiving 10 messages per  dstream. I have  only  one
>  receiver . When I used createStream the  receiver  received  entire 100
> messages  at once.
>
> Appreciate  any  help .
>
> Regards
> Diwakar
>
>
> Sent from Samsung Mobile.
>