You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by "Kang, Kamaljeet" <kk...@verisign.com> on 2016/12/06 16:31:57 UTC
Integration testing using LocalCluster in Storm 1.0.1
Hi all,
We recently upgrade our cluster from Strom 0.11 to 1.0.1.2.5.0.0-1245 (HortonWorks), Kafka from 0.8.2 to 0.10.0.1. We had integration tests using Storm LocalCluster, KafkaEmbedded and EmbeddedZookeeper that started failing sometimes (under load) after the upgrade. Just wanted to see if anybody had similar experience.
Another question, we were thinking of upgrading KafkaSpouts to new Consumer API but looks like Kafka Client in Maven central is not compatible with Kafka 0.10.0.1. If we build Kafka Client jar from Git, is it production ready?
Thanks
Kamal
Re: Integration testing using LocalCluster in Storm 1.0.1
Posted by Erik Weathers <ew...@groupon.com>.
I would add that it would "help me help you" [1][2] if you explained
"started failing sometimes (under load)". i.e., *what* is failing?
- Erik
[1] https://www.youtube.com/watch?v=XmlXU4uK5rA
[2] That quote is meant as a joke, I don't *literally* mean *me*, I mean
"us", but that's not the movie quote...
On Tue, Dec 6, 2016 at 6:25 PM, Jungtaek Lim <ka...@gmail.com> wrote:
> Hi Kang,
>
> I didn't see that issue and don't know about first question.
>
> I'll answer the second question. storm-kafka-client 1.1.0 SNAPSHOT is
> compatible with Kafka 0.10, whereas storm-kafka-client 1.0.x is compatible
> with Kafka 0.9.
>
> Btw, HDP 1.0.1.2.5.0.0-1245 is based on snapshot of Storm 1.1.0 SNAPSHOT
> so you can pull it with adding Hortonworks remote repository to Maven.
>
> Hope this helps.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 2016년 12월 7일 (수) 오전 1:32, Kang, Kamaljeet <kk...@verisign.com>님이 작성:
>
>> Hi all,
>>
>>
>>
>> We recently upgrade our cluster from Strom 0.11 to 1.0.1.2.5.0.0-1245
>> (HortonWorks), Kafka from 0.8.2 to 0.10.0.1. We had integration tests using
>> Storm LocalCluster, KafkaEmbedded and EmbeddedZookeeper that started
>> failing sometimes (under load) after the upgrade. Just wanted to see if
>> anybody had similar experience.
>>
>>
>>
>> Another question, we were thinking of upgrading KafkaSpouts to new
>> Consumer API but looks like Kafka Client in Maven central is not compatible
>> with Kafka 0.10.0.1. If we build Kafka Client jar from Git, is it
>> production ready?
>>
>>
>>
>>
>>
>> Thanks
>>
>>
>>
>> Kamal
>>
>>
>>
>
Re: Integration testing using LocalCluster in Storm 1.0.1
Posted by "Kang, Kamaljeet" <kk...@verisign.com>.
Hi Lim,
Thanks for your response and sorry for being so vague regarding my first question. Actually I am unable to pinpoint the problem. We have several storm topology integration tests that we run in parallel and had been working very well before the upgrade. The tests started failing after the upgrade, looks like messages are getting dropped somehow. Just wanted to see if anybody is seeing any issues with 1.0.1 LocalCluster.
Thanks
Kamal
From: Jungtaek Lim <ka...@gmail.com>
Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
Date: Tuesday, December 6, 2016 at 9:25 PM
To: "user@storm.apache.org" <us...@storm.apache.org>
Subject: [EXTERNAL] Re: Integration testing using LocalCluster in Storm 1.0.1
Hi Kang,
I didn't see that issue and don't know about first question.
I'll answer the second question. storm-kafka-client 1.1.0 SNAPSHOT is compatible with Kafka 0.10, whereas storm-kafka-client 1.0.x is compatible with Kafka 0.9.
Btw, HDP 1.0.1.2.5.0.0-1245 is based on snapshot of Storm 1.1.0 SNAPSHOT so you can pull it with adding Hortonworks remote repository to Maven.
Hope this helps.
Thanks,
Jungtaek Lim (HeartSaVioR)
2016년 12월 7일 (수) 오전 1:32, Kang, Kamaljeet <kk...@verisign.com>>님이 작성:
Hi all,
We recently upgrade our cluster from Strom 0.11 to 1.0.1.2.5.0.0-1245 (HortonWorks), Kafka from 0.8.2 to 0.10.0.1. We had integration tests using Storm LocalCluster, KafkaEmbedded and EmbeddedZookeeper that started failing sometimes (under load) after the upgrade. Just wanted to see if anybody had similar experience.
Another question, we were thinking of upgrading KafkaSpouts to new Consumer API but looks like Kafka Client in Maven central is not compatible with Kafka 0.10.0.1. If we build Kafka Client jar from Git, is it production ready?
Thanks
Kamal
Re: Integration testing using LocalCluster in Storm 1.0.1
Posted by Jungtaek Lim <ka...@gmail.com>.
Hi Kang,
I didn't see that issue and don't know about first question.
I'll answer the second question. storm-kafka-client 1.1.0 SNAPSHOT is
compatible with Kafka 0.10, whereas storm-kafka-client 1.0.x is compatible
with Kafka 0.9.
Btw, HDP 1.0.1.2.5.0.0-1245 is based on snapshot of Storm 1.1.0 SNAPSHOT so
you can pull it with adding Hortonworks remote repository to Maven.
Hope this helps.
Thanks,
Jungtaek Lim (HeartSaVioR)
2016년 12월 7일 (수) 오전 1:32, Kang, Kamaljeet <kk...@verisign.com>님이 작성:
> Hi all,
>
>
>
> We recently upgrade our cluster from Strom 0.11 to 1.0.1.2.5.0.0-1245
> (HortonWorks), Kafka from 0.8.2 to 0.10.0.1. We had integration tests using
> Storm LocalCluster, KafkaEmbedded and EmbeddedZookeeper that started
> failing sometimes (under load) after the upgrade. Just wanted to see if
> anybody had similar experience.
>
>
>
> Another question, we were thinking of upgrading KafkaSpouts to new
> Consumer API but looks like Kafka Client in Maven central is not compatible
> with Kafka 0.10.0.1. If we build Kafka Client jar from Git, is it
> production ready?
>
>
>
>
>
> Thanks
>
>
>
> Kamal
>
>
>