sparkStreaming how to work with partitions,how tp create partition

[Gaurav Agarwal <ga...@gmail.com>]

1. how to work with partition in spark streaming from kafka
2. how to create partition in spark streaming from kafka



when i send the message from kafka topic having three partitions.

Spark will listen the message when i say kafkautils.createStream or
createDirectstSream have local[4]
Now i want to see if spark will create partitions when it receive
message from kafka using dstream, how and where ,prwhich method of
spark api i have to see to find out

Re: sparkStreaming how to work with partitions,how tp create partition

[Akhil Das <ak...@sigmoidanalytics.com>]

You can use the driver ui and click on the Jobs -> Stages to see the number
of partitions being created for that job. If you want to increase the
partitions, then you could do a .repartition too. With directStream api i
guess the # partitions in spark will be equal to the number of partitions
in the kafka topic.

Thanks
Best Regards

On Sat, Aug 22, 2015 at 4:09 PM, Gaurav Agarwal <ga...@gmail.com>
wrote:

> 1. how to work with partition in spark streaming from kafka
> 2. how to create partition in spark streaming from kafka
>
>
>
> when i send the message from kafka topic having three partitions.
>
> Spark will listen the message when i say kafkautils.createStream or
> createDirectstSream have local[4]
> Now i want to see if spark will create partitions when it receive
> message from kafka using dstream, how and where ,prwhich method of
> spark api i have to see to find out
>