You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Kashyap Mhaisekar <ka...@gmail.com> on 2014/07/23 23:11:30 UTC
Parallelism for KafkaSpout
Hi,
Is the no. of executors for KafkaSpout dependent on the partitions for the
topic?
For E.g.,
Say kafka TopicA has 5 partitions.
If I have a KafkaSpout that has the parallelism hint set to 10, will all
the executors get the data? Or am i constrained by the no. of partitions
declared for the topic?
Regards,
Kashyap
Re: Parallelism for KafkaSpout
Posted by Raphael Hsieh <ra...@gmail.com>.
how do you tell how many partitions are used in a specific bolt on the
topology ?
On Wed, Jul 23, 2014 at 2:15 PM, Nathan Leung <nc...@gmail.com> wrote:
> In your example, five spouts would get data and the other five would not.
> On Jul 23, 2014 5:11 PM, "Kashyap Mhaisekar" <ka...@gmail.com> wrote:
>
>> Hi,
>> Is the no. of executors for KafkaSpout dependent on the partitions for
>> the topic?
>> For E.g.,
>> Say kafka TopicA has 5 partitions.
>> If I have a KafkaSpout that has the parallelism hint set to 10, will all
>> the executors get the data? Or am i constrained by the no. of partitions
>> declared for the topic?
>>
>> Regards,
>> Kashyap
>>
>
--
Raphael Hsieh
Re: Parallelism for KafkaSpout
Posted by Kashyap Mhaisekar <ka...@gmail.com>.
Thanks all. Thats the same thing I saw. Wanted to confirm on the behavior!
Regards,
Kashyap
On Jul 23, 2014 4:15 PM, "Nathan Leung" <nc...@gmail.com> wrote:
> In your example, five spouts would get data and the other five would not.
> On Jul 23, 2014 5:11 PM, "Kashyap Mhaisekar" <ka...@gmail.com> wrote:
>
>> Hi,
>> Is the no. of executors for KafkaSpout dependent on the partitions for
>> the topic?
>> For E.g.,
>> Say kafka TopicA has 5 partitions.
>> If I have a KafkaSpout that has the parallelism hint set to 10, will all
>> the executors get the data? Or am i constrained by the no. of partitions
>> declared for the topic?
>>
>> Regards,
>> Kashyap
>>
>
Re: Parallelism for KafkaSpout
Posted by Nathan Leung <nc...@gmail.com>.
In your example, five spouts would get data and the other five would not.
On Jul 23, 2014 5:11 PM, "Kashyap Mhaisekar" <ka...@gmail.com> wrote:
> Hi,
> Is the no. of executors for KafkaSpout dependent on the partitions for the
> topic?
> For E.g.,
> Say kafka TopicA has 5 partitions.
> If I have a KafkaSpout that has the parallelism hint set to 10, will all
> the executors get the data? Or am i constrained by the no. of partitions
> declared for the topic?
>
> Regards,
> Kashyap
>
Re: Parallelism for KafkaSpout
Posted by Robert Lee <le...@gmail.com>.
>From my understanding, the executors without a partition to read remain
idle. Its recommended for efficiency to set the number of executors to the
number of partitions.
On Wed, Jul 23, 2014 at 5:11 PM, Kashyap Mhaisekar <ka...@gmail.com>
wrote:
> Hi,
> Is the no. of executors for KafkaSpout dependent on the partitions for the
> topic?
> For E.g.,
> Say kafka TopicA has 5 partitions.
> If I have a KafkaSpout that has the parallelism hint set to 10, will all
> the executors get the data? Or am i constrained by the no. of partitions
> declared for the topic?
>
> Regards,
> Kashyap
>