You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Alexandre Vermeerbergen <av...@gmail.com> on 2017/01/07 17:03:45 UTC
Is it possible to choose Kafa version with StormKafka from Storm
1.0.2 ?
Hello,
We have been happyly running our real-time supervision homebrewed
application with Apache Storm 1.0.1 and its out-of-the-box StormKafka spout
& bold with our Kafka Broker 0.9.0.1 cluster for more than 6 months.
We're been recently interested in upgrading our setup to Apache Storm
1.0.2, mainly because we sometime hit
https://issues.apache.org/jira/browse/STORM-1879
We also plan to soon upgrade our Kafka Broker cluster to 0.10.x
However, we noticed that Storm 1.0.2's KafkaSpout has been downgraded to
Kafka 0.8.2.2 : STORM-1755: Revert the kafka client version upgrade in
storm-kafka module
Kafka 0.10 brokers have a new message format which can be consumed and
produced to using older Kafka API, but at the price of performance
Is it posible to build a KafkaStorm support based on Kafka 0.10.x API ?
Best regards,
Alexandre
Re: Is it possible to choose Kafa version with StormKafka from Storm
1.0.2 ?
Posted by Jungtaek Lim <ka...@gmail.com>.
Alexandre,
I guess there's no issue on deploying storm-kafka-client 1.1.0-SNAPSHOT to
Storm 1.0.2 cluster since there was no huge change on the API, though I
haven't tried.
- Jungtaek Lim (HeartSaVioR)
2017년 1월 12일 (목) 오후 6:22, Alexandre Vermeerbergen <av...@gmail.com>님이
작성:
> Hello Aaron,
>
> Thanks for your reply.
> Was your test based on the existing storm-kafka external "lib" or on the
> newer "storm-kafka-client" one ?
> Do you remember which code changes you had to perform? was that on the
> "lib" or on your topology ?
>
> best regards,
> Alexandre
>
>
> 2017-01-12 4:30 GMT+01:00 Aaron Niskodé-Dossett <do...@gmail.com>:
>
> Hi Alexandre,
>
> My recollection from trying this when 0.10 was released is that it is
> possible, but some small code changes are necessary. If you're comfortable
> building Storm from source, updating the kafka version in the appropriate
> poms and compiling would show you what needs to be updated.
>
> On Wed, Jan 11, 2017 at 2:20 PM Alexandre Vermeerbergen <
> avermeerbergen@gmail.com> wrote:
>
> Hello,
>
> Any clue on whether or not it's possible to build a KafkaSpout / KafkaBolt
> based on Apache Kafka 0.10.x ?
>
> Best regards,
> Alexandre
>
>
> 2017-01-07 18:03 GMT+01:00 Alexandre Vermeerbergen <
> avermeerbergen@gmail.com>:
>
> Hello,
>
> We have been happyly running our real-time supervision homebrewed
> application with Apache Storm 1.0.1 and its out-of-the-box StormKafka spout
> & bold with our Kafka Broker 0.9.0.1 cluster for more than 6 months.
>
> We're been recently interested in upgrading our setup to Apache Storm
> 1.0.2, mainly because we sometime hit
> https://issues.apache.org/jira/browse/STORM-1879
>
> We also plan to soon upgrade our Kafka Broker cluster to 0.10.x
>
> However, we noticed that Storm 1.0.2's KafkaSpout has been downgraded to
> Kafka 0.8.2.2 : STORM-1755: Revert the kafka client version upgrade in
> storm-kafka module
>
> Kafka 0.10 brokers have a new message format which can be consumed and
> produced to using older Kafka API, but at the price of performance
>
> Is it posible to build a KafkaStorm support based on Kafka 0.10.x API ?
>
> Best regards,
> Alexandre
>
>
>
>
Re: Is it possible to choose Kafa version with StormKafka from Storm
1.0.2 ?
Posted by Alexandre Vermeerbergen <av...@gmail.com>.
Hello Aaron,
Thanks for your reply.
Was your test based on the existing storm-kafka external "lib" or on the
newer "storm-kafka-client" one ?
Do you remember which code changes you had to perform? was that on the
"lib" or on your topology ?
best regards,
Alexandre
2017-01-12 4:30 GMT+01:00 Aaron Niskodé-Dossett <do...@gmail.com>:
> Hi Alexandre,
>
> My recollection from trying this when 0.10 was released is that it is
> possible, but some small code changes are necessary. If you're comfortable
> building Storm from source, updating the kafka version in the appropriate
> poms and compiling would show you what needs to be updated.
>
> On Wed, Jan 11, 2017 at 2:20 PM Alexandre Vermeerbergen <
> avermeerbergen@gmail.com> wrote:
>
>> Hello,
>>
>> Any clue on whether or not it's possible to build a KafkaSpout /
>> KafkaBolt based on Apache Kafka 0.10.x ?
>>
>> Best regards,
>> Alexandre
>>
>>
>> 2017-01-07 18:03 GMT+01:00 Alexandre Vermeerbergen <
>> avermeerbergen@gmail.com>:
>>
>> Hello,
>>
>> We have been happyly running our real-time supervision homebrewed
>> application with Apache Storm 1.0.1 and its out-of-the-box StormKafka spout
>> & bold with our Kafka Broker 0.9.0.1 cluster for more than 6 months.
>>
>> We're been recently interested in upgrading our setup to Apache Storm
>> 1.0.2, mainly because we sometime hit https://issues.apache.org/
>> jira/browse/STORM-1879
>>
>> We also plan to soon upgrade our Kafka Broker cluster to 0.10.x
>>
>> However, we noticed that Storm 1.0.2's KafkaSpout has been downgraded to
>> Kafka 0.8.2.2 : STORM-1755: Revert the kafka client version upgrade in
>> storm-kafka module
>>
>> Kafka 0.10 brokers have a new message format which can be consumed and
>> produced to using older Kafka API, but at the price of performance
>>
>> Is it posible to build a KafkaStorm support based on Kafka 0.10.x API ?
>>
>> Best regards,
>> Alexandre
>>
>>
>>
Re: Is it possible to choose Kafa version with StormKafka from Storm
1.0.2 ?
Posted by Alexandre Vermeerbergen <av...@gmail.com>.
Hello Jungtae,
Thank you very much for your reply.
Is it possible for us to used storm-kafka-client 1.1.0-SNAPSHOT in our fat
jar, but deploy this fat jar with our existing Storm 1.0.1 cluster ; or do
we need to also upgrade our cluster to 1.1.0-SNAPSHOT as well (I hope not) ?
Best regards,
Alexandre
2017-01-12 9:48 GMT+01:00 Jungtaek Lim <ka...@gmail.com>:
> Hi Alexandre,
>
> README of storm-kafka describes how to change Kafka version.
> https://github.com/apache/storm/tree/master/external/
> storm-kafka#using-storm-kafka-with-different-versions-of-kafka
>
> As README stated, if you use Kafka 0.10.x, please try out new module
> storm-kafka-client which is using new client API.
>
> storm-kafka-client 1.0.2 is compatible with Kafka 0.9, and
> storm-kafka-client 1.1.0 SNAPSHOT is compatible with Kafka 0.10.x.
>
> I think we're close to start release process of Storm 1.1.0. Before that
> you may need to build Storm 1.x branch via mvn install, and include
> storm-kafka-client 1.1.0-SNAPSHOT as your fat jar.
>
> Thanks,
> Jungtaek Lim
>
> 2017년 1월 12일 (목) 오후 12:30, Aaron Niskodé-Dossett <do...@gmail.com>님이 작성:
>
> Hi Alexandre,
>
> My recollection from trying this when 0.10 was released is that it is
> possible, but some small code changes are necessary. If you're comfortable
> building Storm from source, updating the kafka version in the appropriate
> poms and compiling would show you what needs to be updated.
>
> On Wed, Jan 11, 2017 at 2:20 PM Alexandre Vermeerbergen <
> avermeerbergen@gmail.com> wrote:
>
> Hello,
>
> Any clue on whether or not it's possible to build a KafkaSpout / KafkaBolt
> based on Apache Kafka 0.10.x ?
>
> Best regards,
> Alexandre
>
>
> 2017-01-07 18:03 GMT+01:00 Alexandre Vermeerbergen <
> avermeerbergen@gmail.com>:
>
> Hello,
>
> We have been happyly running our real-time supervision homebrewed
> application with Apache Storm 1.0.1 and its out-of-the-box StormKafka spout
> & bold with our Kafka Broker 0.9.0.1 cluster for more than 6 months.
>
> We're been recently interested in upgrading our setup to Apache Storm
> 1.0.2, mainly because we sometime hit https://issues.apache.org/
> jira/browse/STORM-1879
>
> We also plan to soon upgrade our Kafka Broker cluster to 0.10.x
>
> However, we noticed that Storm 1.0.2's KafkaSpout has been downgraded to
> Kafka 0.8.2.2 : STORM-1755: Revert the kafka client version upgrade in
> storm-kafka module
>
> Kafka 0.10 brokers have a new message format which can be consumed and
> produced to using older Kafka API, but at the price of performance
>
> Is it posible to build a KafkaStorm support based on Kafka 0.10.x API ?
>
> Best regards,
> Alexandre
>
>
>
Re: Is it possible to choose Kafa version with StormKafka from Storm
1.0.2 ?
Posted by Jungtaek Lim <ka...@gmail.com>.
Hi Alexandre,
README of storm-kafka describes how to change Kafka version.
https://github.com/apache/storm/tree/master/external/storm-kafka#using-storm-kafka-with-different-versions-of-kafka
As README stated, if you use Kafka 0.10.x, please try out new module
storm-kafka-client which is using new client API.
storm-kafka-client 1.0.2 is compatible with Kafka 0.9, and
storm-kafka-client 1.1.0 SNAPSHOT is compatible with Kafka 0.10.x.
I think we're close to start release process of Storm 1.1.0. Before that
you may need to build Storm 1.x branch via mvn install, and include
storm-kafka-client 1.1.0-SNAPSHOT as your fat jar.
Thanks,
Jungtaek Lim
2017년 1월 12일 (목) 오후 12:30, Aaron Niskodé-Dossett <do...@gmail.com>님이 작성:
Hi Alexandre,
My recollection from trying this when 0.10 was released is that it is
possible, but some small code changes are necessary. If you're comfortable
building Storm from source, updating the kafka version in the appropriate
poms and compiling would show you what needs to be updated.
On Wed, Jan 11, 2017 at 2:20 PM Alexandre Vermeerbergen <
avermeerbergen@gmail.com> wrote:
Hello,
Any clue on whether or not it's possible to build a KafkaSpout / KafkaBolt
based on Apache Kafka 0.10.x ?
Best regards,
Alexandre
2017-01-07 18:03 GMT+01:00 Alexandre Vermeerbergen <avermeerbergen@gmail.com
>:
Hello,
We have been happyly running our real-time supervision homebrewed
application with Apache Storm 1.0.1 and its out-of-the-box StormKafka spout
& bold with our Kafka Broker 0.9.0.1 cluster for more than 6 months.
We're been recently interested in upgrading our setup to Apache Storm
1.0.2, mainly because we sometime hit
https://issues.apache.org/jira/browse/STORM-1879
We also plan to soon upgrade our Kafka Broker cluster to 0.10.x
However, we noticed that Storm 1.0.2's KafkaSpout has been downgraded to
Kafka 0.8.2.2 : STORM-1755: Revert the kafka client version upgrade in
storm-kafka module
Kafka 0.10 brokers have a new message format which can be consumed and
produced to using older Kafka API, but at the price of performance
Is it posible to build a KafkaStorm support based on Kafka 0.10.x API ?
Best regards,
Alexandre
Re: Is it possible to choose Kafa version with StormKafka from Storm
1.0.2 ?
Posted by Aaron Niskodé-Dossett <do...@gmail.com>.
Hi Alexandre,
My recollection from trying this when 0.10 was released is that it is
possible, but some small code changes are necessary. If you're comfortable
building Storm from source, updating the kafka version in the appropriate
poms and compiling would show you what needs to be updated.
On Wed, Jan 11, 2017 at 2:20 PM Alexandre Vermeerbergen <
avermeerbergen@gmail.com> wrote:
> Hello,
>
> Any clue on whether or not it's possible to build a KafkaSpout / KafkaBolt
> based on Apache Kafka 0.10.x ?
>
> Best regards,
> Alexandre
>
>
> 2017-01-07 18:03 GMT+01:00 Alexandre Vermeerbergen <
> avermeerbergen@gmail.com>:
>
> Hello,
>
> We have been happyly running our real-time supervision homebrewed
> application with Apache Storm 1.0.1 and its out-of-the-box StormKafka spout
> & bold with our Kafka Broker 0.9.0.1 cluster for more than 6 months.
>
> We're been recently interested in upgrading our setup to Apache Storm
> 1.0.2, mainly because we sometime hit
> https://issues.apache.org/jira/browse/STORM-1879
>
> We also plan to soon upgrade our Kafka Broker cluster to 0.10.x
>
> However, we noticed that Storm 1.0.2's KafkaSpout has been downgraded to
> Kafka 0.8.2.2 : STORM-1755: Revert the kafka client version upgrade in
> storm-kafka module
>
> Kafka 0.10 brokers have a new message format which can be consumed and
> produced to using older Kafka API, but at the price of performance
>
> Is it posible to build a KafkaStorm support based on Kafka 0.10.x API ?
>
> Best regards,
> Alexandre
>
>
>
Re: Is it possible to choose Kafa version with StormKafka from Storm
1.0.2 ?
Posted by Alexandre Vermeerbergen <av...@gmail.com>.
Hello,
Any clue on whether or not it's possible to build a KafkaSpout / KafkaBolt
based on Apache Kafka 0.10.x ?
Best regards,
Alexandre
2017-01-07 18:03 GMT+01:00 Alexandre Vermeerbergen <avermeerbergen@gmail.com
>:
> Hello,
>
> We have been happyly running our real-time supervision homebrewed
> application with Apache Storm 1.0.1 and its out-of-the-box StormKafka spout
> & bold with our Kafka Broker 0.9.0.1 cluster for more than 6 months.
>
> We're been recently interested in upgrading our setup to Apache Storm
> 1.0.2, mainly because we sometime hit https://issues.apache.org/
> jira/browse/STORM-1879
>
> We also plan to soon upgrade our Kafka Broker cluster to 0.10.x
>
> However, we noticed that Storm 1.0.2's KafkaSpout has been downgraded to
> Kafka 0.8.2.2 : STORM-1755: Revert the kafka client version upgrade in
> storm-kafka module
>
> Kafka 0.10 brokers have a new message format which can be consumed and
> produced to using older Kafka API, but at the price of performance
>
> Is it posible to build a KafkaStorm support based on Kafka 0.10.x API ?
>
> Best regards,
> Alexandre
>