spark kafka partitioning

[Gaurav Agarwal <ga...@gmail.com>]

Hello

Regarding Spark Streaming and Kafka Partitioning

When i send message on kafka topic with 3 partitions and listens on
kafkareceiver with local value[4] . how will i come to know in Spark
Streaming that different Dstreams are created according to partitions of
kafka messages .

Thanks

Re: spark kafka partitioning

[Gaurav Agarwal <ga...@gmail.com>]

when i send the message from kafka topic having three partitions.

Spark will listen the message when i say kafkautils.createStream or
createDirectstSream have local[4]
Now i want to see if spark will create partitions when it receive
message from kafka using dstream, how and where ,prwhich method of
spark api i have to see to find out

On 8/21/15, Gaurav Agarwal <ga...@gmail.com> wrote:
> Hello
>
> Regarding Spark Streaming and Kafka Partitioning
>
> When i send message on kafka topic with 3 partitions and listens on
> kafkareceiver with local value[4] . how will i come to know in Spark
> Streaming that different Dstreams are created according to partitions of
> kafka messages .
>
> Thanks
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: spark kafka partitioning

[Cody Koeninger <co...@koeninger.org>]

I'm not clear on your question, can you rephrase it?  Also, are you talking
about createStream or createDirectStream?

On Thu, Aug 20, 2015 at 9:48 PM, Gaurav Agarwal <ga...@gmail.com>
wrote:

> Hello
>
> Regarding Spark Streaming and Kafka Partitioning
>
> When i send message on kafka topic with 3 partitions and listens on
> kafkareceiver with local value[4] . how will i come to know in Spark
> Streaming that different Dstreams are created according to partitions of
> kafka messages .
>
> Thanks
>
>

Re: spark kafka partitioning

[ayan guha <gu...@gmail.com>]

If you have 1 topic, that means you have 1 DStream, which will have a
series of RDDs for each batch interval. In receiver-based integration,
there is no direct relationship b/w Kafka paritions with spark partitions.
in Direct approach, 1 partition will be created for each kafka partition.

On Fri, Aug 21, 2015 at 12:48 PM, Gaurav Agarwal <ga...@gmail.com>
wrote:

> Hello
>
> Regarding Spark Streaming and Kafka Partitioning
>
> When i send message on kafka topic with 3 partitions and listens on
> kafkareceiver with local value[4] . how will i come to know in Spark
> Streaming that different Dstreams are created according to partitions of
> kafka messages .
>
> Thanks
>
>


-- 
Best Regards,
Ayan Guha