You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Sreeram <sr...@gmail.com> on 2017/11/04 03:33:13 UTC
KafkaSpout not reading from some kafka partitions !
Hi,
I am seeing that storm spout in production is not reading from some
kafka partitions.
Any clues what can be going wrong?
Storm version: 1.1.0
Kafka version: 0.9.0
Number of partitions in kafka topic: 12
spout parallelism: 6
This is quite urgent, will appreciate any help.
Thank you.
Regards,
Sreeram
Re: KafkaSpout not reading from some kafka partitions !
Posted by Stig Rohde Døssing <sr...@apache.org>.
Are you using storm-kafka or storm-kafka-client?
2017-11-04 9:58 GMT+01:00 Arnaud BOS <ar...@gmail.com>:
> Shouldn't the KafkaSpout's tasks read from multiple partitions if the
> parallelism is inferior to the number of partitions ?
>
> Did you set the groupId in your KafkaSpoutConfig so that your tasks behave
> as a consumer group ?
>
> I'm new here so I'm just thinking out loud.
>
> On Sat, Nov 4, 2017, 7:21 AM Susheel Kumar Gadalay <sk...@gmail.com>
> wrote:
>
>> Change spout parallelism equal to Kafka partitions.
>>
>> On Saturday, November 4, 2017, Sreeram <sr...@gmail.com> wrote:
>> > Hi,
>> >
>> > I am seeing that storm spout in production is not reading from some
>> > kafka partitions.
>> >
>> > Any clues what can be going wrong?
>> >
>> > Storm version: 1.1.0
>> > Kafka version: 0.9.0
>> > Number of partitions in kafka topic: 12
>> > spout parallelism: 6
>> >
>> > This is quite urgent, will appreciate any help.
>> >
>> > Thank you.
>> >
>> > Regards,
>> > Sreeram
>> >
>
>
Re: KafkaSpout not reading from some kafka partitions !
Posted by Arnaud BOS <ar...@gmail.com>.
Shouldn't the KafkaSpout's tasks read from multiple partitions if the
parallelism is inferior to the number of partitions ?
Did you set the groupId in your KafkaSpoutConfig so that your tasks behave
as a consumer group ?
I'm new here so I'm just thinking out loud.
On Sat, Nov 4, 2017, 7:21 AM Susheel Kumar Gadalay <sk...@gmail.com>
wrote:
> Change spout parallelism equal to Kafka partitions.
>
> On Saturday, November 4, 2017, Sreeram <sr...@gmail.com> wrote:
> > Hi,
> >
> > I am seeing that storm spout in production is not reading from some
> > kafka partitions.
> >
> > Any clues what can be going wrong?
> >
> > Storm version: 1.1.0
> > Kafka version: 0.9.0
> > Number of partitions in kafka topic: 12
> > spout parallelism: 6
> >
> > This is quite urgent, will appreciate any help.
> >
> > Thank you.
> >
> > Regards,
> > Sreeram
> >
Re: KafkaSpout not reading from some kafka partitions !
Posted by Susheel Kumar Gadalay <sk...@gmail.com>.
Change spout parallelism equal to Kafka partitions.
On Saturday, November 4, 2017, Sreeram <sr...@gmail.com> wrote:
> Hi,
>
> I am seeing that storm spout in production is not reading from some
> kafka partitions.
>
> Any clues what can be going wrong?
>
> Storm version: 1.1.0
> Kafka version: 0.9.0
> Number of partitions in kafka topic: 12
> spout parallelism: 6
>
> This is quite urgent, will appreciate any help.
>
> Thank you.
>
> Regards,
> Sreeram
>