storm-kafka-client 1.1.0 tag - Multiple Spouts Eventually Freeze

[Zach Schoenberger <zs...@gmail.com>]

Hi all,

I am working with the storm-kafka-client and am seeing some odd behavior.
Whenever I setup a topology with the storm-kafka-client with 2 tasks, 2
workers, and parallelism hint of 2, I see that the spout will eventually
only poll 0 events from kafka, even when I have verified that there are new
events flowing in. The number of partitions is 180 for the topic. The kafka
version I am using is 0.10.1.

Any help/insight would be greatly appreciated.

Thanks,
Zach

Re: storm-kafka-client 1.1.0 tag - Multiple Spouts Eventually Freeze

[James Tyack <jt...@zignallabs.com>]

Zach,
We have been seeing a similar issue with storm kafka since upgrade and
posted about it earlier this week. We produced tests and opened pull
request for feedback on what we saw.
https://github.com/apache/storm/pull/1679
Regards
James
On Sep 12, 2016 12:03 PM, "Zach Schoenberger" <zs...@gmail.com> wrote:

> Hi all,
>
> I am working with the storm-kafka-client and am seeing some odd behavior.
> Whenever I setup a topology with the storm-kafka-client with 2 tasks, 2
> workers, and parallelism hint of 2, I see that the spout will eventually
> only poll 0 events from kafka, even when I have verified that there are new
> events flowing in. The number of partitions is 180 for the topic. The kafka
> version I am using is 0.10.1.
>
> Any help/insight would be greatly appreciated.
>
> Thanks,
> Zach
>

Re: storm-kafka-client 1.1.0 tag - Multiple Spouts Eventually Freeze

[Ambud Sharma <as...@gmail.com>]

I have seen that behavior only when running in local mode of storm and
there is no data flowing in.

This sounds like it might have something to do with zookeeper as in your
offsets in zookeeper are either not updated or the watches are not being
triggered for the spout to consume.

Try using the zkcli and manually ls the consumer group znodes and cat them
to see what data is there.

Thanks,
Ambud

On Sep 12, 2016 12:03 PM, "Zach Schoenberger" <zs...@gmail.com> wrote:

> Hi all,
>
> I am working with the storm-kafka-client and am seeing some odd behavior.
> Whenever I setup a topology with the storm-kafka-client with 2 tasks, 2
> workers, and parallelism hint of 2, I see that the spout will eventually
> only poll 0 events from kafka, even when I have verified that there are new
> events flowing in. The number of partitions is 180 for the topic. The kafka
> version I am using is 0.10.1.
>
> Any help/insight would be greatly appreciated.
>
> Thanks,
> Zach
>