You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by stephen <zh...@126.com> on 2018/08/20 02:32:39 UTC
upgrade kafka from 0.8.X to 1.1 using storm-core 1.0.3
Dear all,
I have a question about upgrade kafka from 0.8.X to 1.1 using storm-core 1.0.3. Are there some feasible methods for me without changing the version of storm-core?
Best regards,
Kun
Re: upgrade kafka from 0.8.X to 1.1 using storm-core 1.0.3
Posted by stephen <zh...@126.com>.
Many thanks for your generous support! Erik and all
It works for me by using the shade pom. Eventually, we plan to upgrade storm cluster to 1.0.7. Is it stable for us?
And we hope it can upgrade smoothly :)
Cheers
Kun
> On 21 Aug 2018, at 14:03, Jungtaek Lim <ka...@gmail.com> wrote:
>
> stephen,
>
> we migrated latest storm-kafka-client to Storm 1.0.6, so I guess there should be no problem on Storm 1.0.6 regarding any issues including Time.nanoTime() you've observed or so.
>
> I'd suggest upgrading Storm to 1.0.6 which upgrade would be expected to be done smoothly, but actually you should be better to upgrade to latest version since Apache Storm community no longer maintains 1.0.x version line actively.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 2018년 8월 21일 (화) 오후 2:44, Stig Rohde Døssing <stigdoessing@gmail.com <ma...@gmail.com>>님이 작성:
> Just want to add a quick note as well that you should use the latest storm-kafka-client. The intent is that you add a dependency both to storm-kafka-client, and org.apache.kafka:kafka-clients in your project. This will let you control the Kafka client version. See https://github.com/apache/storm/blob/master/examples/storm-kafka-client-examples/pom.xml#L41 <https://github.com/apache/storm/blob/master/examples/storm-kafka-client-examples/pom.xml#L41> for example.
>
> 2018-08-21 4:46 GMT+02:00 Erik Weathers <erik@apache.org <ma...@apache.org>>:
> Thanks for providing all those details.
>
> > Time.nanoTime is not a method in storm-core 1.0.3 until 1.0.6.
> > So, I'm wondering is there a low-influence idea (without changing version of storm-core 1.0.3) for storm job to consume message from kafka 1.1. Thanks a lot!
>
> I'm going to assume that you care about the version of storm-core that is in the cluster (Nimbus, Supervisor, etc.), and that you are fine with having storm-core v1.0.6 in your topology.
> If so you can try doing something I have done before. I used the maven shade plugin to:
>
> (1) build my topology's uber jar with a newer version of storm-core shaded into my uber jar.
> (2) relocated the org.apache.storm.utils class to avoid any conflicts.
> (3) did some exclusions to avoid complaints about not including storm-core in the topology uber jar.
>
> I am unsure if the latter 2 steps were fully necessary. But my general strategy was just to include whatever bits my topology needed that weren't being found in the version of the storm-core jar in the runtime environment (Nimbus, Supervisor), and to work around any errors that were thrown.
>
> Here's the related parts of the maven pom file that I had for my earlier experiment:
> https://gist.github.com/erikdw/3500c74b23ed2b5e2e808b50c9b39661 <https://gist.github.com/erikdw/3500c74b23ed2b5e2e808b50c9b39661>
> Note that I *was* able to build a storm topology with storm-core v1.1 shaded in and then run it on a v1.0.3 storm cluster using this technique, so you can try to do the similar exercise with your versions of storm-core.
> I'm guessing this is considered a bad practice though, so looking into getting the cluster upgraded to v1.0.6 should be pursued instead of leaving the topology in this weird state for a long time. Also, if you do follow this route then you should *definitely* mention this oddity if you ask future questions of these storm email lists about problems with the topology.
>
> - Erik
>
> P.S., not that it negatively impacted me here, but for future reference having technical text (that exception you included) in an image is a bad practice for 2 reasons:
> (1) It is not accessibility friendly for vision impaired folks. They may need to use screen-reading software that may lack the ability to translate the image to text.
> (2) the content is not indexed for searching the email threads.
>
> On Mon, Aug 20, 2018 at 5:22 PM stephen <zhaotkun@126.com <ma...@126.com>> wrote:
> Sorry Erik,
> I'll give the details. The versions I am currently using are as follows：
>
> storm-core: 1.0.3
> storm-kafka: 1.0.3
> kafka: 0.8.1.1
>
> For some reasons, I need to consuming from kafka 1.1. So, I tried to use the storm-kafka-client (1.0.3), but I found the version of kafka-clients is 0.9.0.1 in this storm-kafka-client. I want to use >= 0.10.0 version of kafka-client because of the cost of Message Conversion. If a consumer client is on a version before 0.10.0.0, the kafka broker will convert messages from the 0.10.0 format to an earlier format before sending the response to the consumer on an older version, that will result in additional overhead of performance and may cause an OOM in some cases.
>
> I found the version of kafka-client in storm-kafka-client 1.0.6 is 0.10.1. So, I used it running with storm-core 1.0.3, and found an Exception as follow:
>
>
> Time.nanoTime is not a method in storm-core 1.0.3 until 1.0.6.
>
> So, I'm wondering is there a low-influence idea (without changing version of storm-core 1.0.3) for storm job to consume message from kafka 1.1. Thanks a lot!
>
> Best regards,
> Kun
>
>> On 20 Aug 2018, at 11:45, Erik Weathers <erik@apache.org <ma...@apache.org>> wrote:
>>
>> hey Kun,
>>
>> I'm unclear which version of storm-kafka and storm-core you are currently using, please explain in more detail what you are trying to accomplish.
>> You'll need to change from using storm-kafka to storm-kafka-client, but again, please explain the versions of everything you are using.
>>
>> Thanks!
>>
>> - Erik
>>
>> On Sun, Aug 19, 2018 at 7:33 PM stephen <zhaotkun@126.com <ma...@126.com>> wrote:
>> Dear all,
>>
>> I have a question about upgrade kafka from 0.8.X to 1.1 using storm-core 1.0.3. Are there some feasible methods for me without changing the version of storm-core?
>>
>> Best regards,
>> Kun
>>
>
>
Re: upgrade kafka from 0.8.X to 1.1 using storm-core 1.0.3
Posted by Jungtaek Lim <ka...@gmail.com>.
stephen,
we migrated latest storm-kafka-client to Storm 1.0.6, so I guess there
should be no problem on Storm 1.0.6 regarding any issues including
Time.nanoTime() you've observed or so.
I'd suggest upgrading Storm to 1.0.6 which upgrade would be expected to be
done smoothly, but actually you should be better to upgrade to latest
version since Apache Storm community no longer maintains 1.0.x version line
actively.
Thanks,
Jungtaek Lim (HeartSaVioR)
2018년 8월 21일 (화) 오후 2:44, Stig Rohde Døssing <st...@gmail.com>님이 작성:
> Just want to add a quick note as well that you should use the latest
> storm-kafka-client. The intent is that you add a dependency both to
> storm-kafka-client, and org.apache.kafka:kafka-clients in your project.
> This will let you control the Kafka client version. See
> https://github.com/apache/storm/blob/master/examples/storm-kafka-client-examples/pom.xml#L41
> for example.
>
> 2018-08-21 4:46 GMT+02:00 Erik Weathers <er...@apache.org>:
>
>> Thanks for providing all those details.
>>
>> > Time.nanoTime is not a method in storm-core 1.0.3 until 1.0.6.
>> > So, I'm wondering is there a low-influence idea (without changing
>> version of storm-core 1.0.3) for storm job to consume message from kafka
>> 1.1. Thanks a lot!
>>
>> I'm going to assume that you care about the version of storm-core that is
>> in the cluster (Nimbus, Supervisor, etc.), and that you are fine with
>> having storm-core v1.0.6 in your topology.
>> If so you can try doing something I have done before. I used the maven
>> shade plugin to:
>>
>> (1) build my topology's uber jar with a newer version of storm-core
>> shaded into my uber jar.
>> (2) relocated the org.apache.storm.utils class to avoid any conflicts.
>> (3) did some exclusions to avoid complaints about not including
>> storm-core in the topology uber jar.
>>
>> I am unsure if the latter 2 steps were fully necessary. But my general
>> strategy was just to include whatever bits my topology needed that weren't
>> being found in the version of the storm-core jar in the runtime environment
>> (Nimbus, Supervisor), and to work around any errors that were thrown.
>>
>> Here's the related parts of the maven pom file that I had for my earlier
>> experiment:
>>
>> - https://gist.github.com/erikdw/3500c74b23ed2b5e2e808b50c9b39661
>>
>> Note that I *was* able to build a storm topology with storm-core v1.1
>> shaded in and then run it on a v1.0.3 storm cluster using this technique,
>> so you can try to do the similar exercise with your versions of storm-core.
>> I'm guessing this is considered a bad practice though, so looking into
>> getting the cluster upgraded to v1.0.6 should be pursued instead of leaving
>> the topology in this weird state for a long time. Also, if you do follow
>> this route then you should *definitely* mention this oddity if you ask
>> future questions of these storm email lists about problems with the
>> topology.
>>
>> - Erik
>>
>> P.S., not that it negatively impacted me here, but for future reference
>> having technical text (that exception you included) in an *image* is a
>> bad practice for 2 reasons:
>> (1) It is not accessibility friendly for vision impaired folks. They may
>> need to use screen-reading software that may lack the ability to translate
>> the image to text.
>> (2) the content is not indexed for searching the email threads.
>>
>> On Mon, Aug 20, 2018 at 5:22 PM stephen <zh...@126.com> wrote:
>>
>>> Sorry Erik,
>>> I'll give the details. The versions I am currently using are as follows：
>>>
>>> storm-core: 1.0.3
>>> storm-kafka: 1.0.3
>>> kafka: 0.8.1.1
>>>
>>> For some reasons, I need to consuming from kafka 1.1. So, I tried to use
>>> the storm-kafka-client (1.0.3), but I found the version of kafka-clients is
>>> 0.9.0.1 in this storm-kafka-client. I want to use >= 0.10.0 version of
>>> kafka-client because of the cost of Message Conversion. If a consumer
>>> client is on a version before 0.10.0.0, the kafka broker will convert
>>> messages from the 0.10.0 format to an earlier format before sending the
>>> response to the consumer on an older version, that will result in
>>> additional overhead of performance and may cause an OOM in some cases.
>>>
>>> I found the version of kafka-client in storm-kafka-client 1.0.6 is
>>> 0.10.1. So, I used it running with storm-core 1.0.3, and found an Exception
>>> as follow:
>>>
>>> Time.nanoTime is not a method in storm-core 1.0.3 until 1.0.6.
>>>
>>> So, I'm wondering is there a low-influence idea (without changing
>>> version of storm-core 1.0.3) for storm job to consume message from kafka
>>> 1.1. Thanks a lot!
>>>
>>> Best regards,
>>> Kun
>>>
>>> On 20 Aug 2018, at 11:45, Erik Weathers <er...@apache.org> wrote:
>>>
>>> hey Kun,
>>>
>>> I'm unclear which version of storm-kafka and storm-core you are
>>> currently using, please explain in more detail what you are trying to
>>> accomplish.
>>> You'll need to change from using storm-kafka to storm-kafka-client, but
>>> again, please explain the versions of everything you are using.
>>>
>>> Thanks!
>>>
>>> - Erik
>>>
>>> On Sun, Aug 19, 2018 at 7:33 PM stephen <zh...@126.com> wrote:
>>>
>>>> Dear all,
>>>>
>>>> I have a question about upgrade kafka from 0.8.X to 1.1 using
>>>> storm-core 1.0.3. Are there some feasible methods for me without changing
>>>> the version of storm-core?
>>>>
>>>> Best regards,
>>>> Kun
>>>>
>>>>
>>>
>
Re: upgrade kafka from 0.8.X to 1.1 using storm-core 1.0.3
Posted by Stig Rohde Døssing <st...@gmail.com>.
Just want to add a quick note as well that you should use the latest
storm-kafka-client. The intent is that you add a dependency both to
storm-kafka-client, and org.apache.kafka:kafka-clients in your project.
This will let you control the Kafka client version. See
https://github.com/apache/storm/blob/master/examples/storm-kafka-client-examples/pom.xml#L41
for example.
2018-08-21 4:46 GMT+02:00 Erik Weathers <er...@apache.org>:
> Thanks for providing all those details.
>
> > Time.nanoTime is not a method in storm-core 1.0.3 until 1.0.6.
> > So, I'm wondering is there a low-influence idea (without changing
> version of storm-core 1.0.3) for storm job to consume message from kafka
> 1.1. Thanks a lot!
>
> I'm going to assume that you care about the version of storm-core that is
> in the cluster (Nimbus, Supervisor, etc.), and that you are fine with
> having storm-core v1.0.6 in your topology.
> If so you can try doing something I have done before. I used the maven
> shade plugin to:
>
> (1) build my topology's uber jar with a newer version of storm-core shaded
> into my uber jar.
> (2) relocated the org.apache.storm.utils class to avoid any conflicts.
> (3) did some exclusions to avoid complaints about not including storm-core
> in the topology uber jar.
>
> I am unsure if the latter 2 steps were fully necessary. But my general
> strategy was just to include whatever bits my topology needed that weren't
> being found in the version of the storm-core jar in the runtime environment
> (Nimbus, Supervisor), and to work around any errors that were thrown.
>
> Here's the related parts of the maven pom file that I had for my earlier
> experiment:
>
> - https://gist.github.com/erikdw/3500c74b23ed2b5e2e808b50c9b39661
>
> Note that I *was* able to build a storm topology with storm-core v1.1
> shaded in and then run it on a v1.0.3 storm cluster using this technique,
> so you can try to do the similar exercise with your versions of storm-core.
> I'm guessing this is considered a bad practice though, so looking into
> getting the cluster upgraded to v1.0.6 should be pursued instead of leaving
> the topology in this weird state for a long time. Also, if you do follow
> this route then you should *definitely* mention this oddity if you ask
> future questions of these storm email lists about problems with the
> topology.
>
> - Erik
>
> P.S., not that it negatively impacted me here, but for future reference
> having technical text (that exception you included) in an *image* is a
> bad practice for 2 reasons:
> (1) It is not accessibility friendly for vision impaired folks. They may
> need to use screen-reading software that may lack the ability to translate
> the image to text.
> (2) the content is not indexed for searching the email threads.
>
> On Mon, Aug 20, 2018 at 5:22 PM stephen <zh...@126.com> wrote:
>
>> Sorry Erik,
>> I'll give the details. The versions I am currently using are as follows：
>>
>> storm-core: 1.0.3
>> storm-kafka: 1.0.3
>> kafka: 0.8.1.1
>>
>> For some reasons, I need to consuming from kafka 1.1. So, I tried to use
>> the storm-kafka-client (1.0.3), but I found the version of kafka-clients is
>> 0.9.0.1 in this storm-kafka-client. I want to use >= 0.10.0 version of
>> kafka-client because of the cost of Message Conversion. If a consumer
>> client is on a version before 0.10.0.0, the kafka broker will convert
>> messages from the 0.10.0 format to an earlier format before sending the
>> response to the consumer on an older version, that will result in
>> additional overhead of performance and may cause an OOM in some cases.
>>
>> I found the version of kafka-client in storm-kafka-client 1.0.6 is
>> 0.10.1. So, I used it running with storm-core 1.0.3, and found an Exception
>> as follow:
>>
>> Time.nanoTime is not a method in storm-core 1.0.3 until 1.0.6.
>>
>> So, I'm wondering is there a low-influence idea (without changing version
>> of storm-core 1.0.3) for storm job to consume message from kafka 1.1.
>> Thanks a lot!
>>
>> Best regards,
>> Kun
>>
>> On 20 Aug 2018, at 11:45, Erik Weathers <er...@apache.org> wrote:
>>
>> hey Kun,
>>
>> I'm unclear which version of storm-kafka and storm-core you are currently
>> using, please explain in more detail what you are trying to accomplish.
>> You'll need to change from using storm-kafka to storm-kafka-client, but
>> again, please explain the versions of everything you are using.
>>
>> Thanks!
>>
>> - Erik
>>
>> On Sun, Aug 19, 2018 at 7:33 PM stephen <zh...@126.com> wrote:
>>
>>> Dear all,
>>>
>>> I have a question about upgrade kafka from 0.8.X to 1.1 using storm-core
>>> 1.0.3. Are there some feasible methods for me without changing the version
>>> of storm-core?
>>>
>>> Best regards,
>>> Kun
>>>
>>>
>>
Re: upgrade kafka from 0.8.X to 1.1 using storm-core 1.0.3
Posted by Erik Weathers <er...@apache.org>.
Thanks for providing all those details.
> Time.nanoTime is not a method in storm-core 1.0.3 until 1.0.6.
> So, I'm wondering is there a low-influence idea (without changing version
of storm-core 1.0.3) for storm job to consume message from kafka 1.1.
Thanks a lot!
I'm going to assume that you care about the version of storm-core that is
in the cluster (Nimbus, Supervisor, etc.), and that you are fine with
having storm-core v1.0.6 in your topology.
If so you can try doing something I have done before. I used the maven
shade plugin to:
(1) build my topology's uber jar with a newer version of storm-core shaded
into my uber jar.
(2) relocated the org.apache.storm.utils class to avoid any conflicts.
(3) did some exclusions to avoid complaints about not including storm-core
in the topology uber jar.
I am unsure if the latter 2 steps were fully necessary. But my general
strategy was just to include whatever bits my topology needed that weren't
being found in the version of the storm-core jar in the runtime environment
(Nimbus, Supervisor), and to work around any errors that were thrown.
Here's the related parts of the maven pom file that I had for my earlier
experiment:
- https://gist.github.com/erikdw/3500c74b23ed2b5e2e808b50c9b39661
Note that I *was* able to build a storm topology with storm-core v1.1
shaded in and then run it on a v1.0.3 storm cluster using this technique,
so you can try to do the similar exercise with your versions of storm-core.
I'm guessing this is considered a bad practice though, so looking into
getting the cluster upgraded to v1.0.6 should be pursued instead of leaving
the topology in this weird state for a long time. Also, if you do follow
this route then you should *definitely* mention this oddity if you ask
future questions of these storm email lists about problems with the
topology.
- Erik
P.S., not that it negatively impacted me here, but for future reference
having technical text (that exception you included) in an *image* is a bad
practice for 2 reasons:
(1) It is not accessibility friendly for vision impaired folks. They may
need to use screen-reading software that may lack the ability to translate
the image to text.
(2) the content is not indexed for searching the email threads.
On Mon, Aug 20, 2018 at 5:22 PM stephen <zh...@126.com> wrote:
> Sorry Erik,
> I'll give the details. The versions I am currently using are as follows：
>
> storm-core: 1.0.3
> storm-kafka: 1.0.3
> kafka: 0.8.1.1
>
> For some reasons, I need to consuming from kafka 1.1. So, I tried to use
> the storm-kafka-client (1.0.3), but I found the version of kafka-clients is
> 0.9.0.1 in this storm-kafka-client. I want to use >= 0.10.0 version of
> kafka-client because of the cost of Message Conversion. If a consumer
> client is on a version before 0.10.0.0, the kafka broker will convert
> messages from the 0.10.0 format to an earlier format before sending the
> response to the consumer on an older version, that will result in
> additional overhead of performance and may cause an OOM in some cases.
>
> I found the version of kafka-client in storm-kafka-client 1.0.6 is 0.10.1.
> So, I used it running with storm-core 1.0.3, and found an Exception as
> follow:
>
> Time.nanoTime is not a method in storm-core 1.0.3 until 1.0.6.
>
> So, I'm wondering is there a low-influence idea (without changing version
> of storm-core 1.0.3) for storm job to consume message from kafka 1.1.
> Thanks a lot!
>
> Best regards,
> Kun
>
> On 20 Aug 2018, at 11:45, Erik Weathers <er...@apache.org> wrote:
>
> hey Kun,
>
> I'm unclear which version of storm-kafka and storm-core you are currently
> using, please explain in more detail what you are trying to accomplish.
> You'll need to change from using storm-kafka to storm-kafka-client, but
> again, please explain the versions of everything you are using.
>
> Thanks!
>
> - Erik
>
> On Sun, Aug 19, 2018 at 7:33 PM stephen <zh...@126.com> wrote:
>
>> Dear all,
>>
>> I have a question about upgrade kafka from 0.8.X to 1.1 using storm-core
>> 1.0.3. Are there some feasible methods for me without changing the version
>> of storm-core?
>>
>> Best regards,
>> Kun
>>
>>
>
Re: upgrade kafka from 0.8.X to 1.1 using storm-core 1.0.3
Posted by stephen <zh...@126.com>.
Sorry Erik,
I'll give the details. The versions I am currently using are as follows：
storm-core: 1.0.3
storm-kafka: 1.0.3
kafka: 0.8.1.1
For some reasons, I need to consuming from kafka 1.1. So, I tried to use the storm-kafka-client (1.0.3), but I found the version of kafka-clients is 0.9.0.1 in this storm-kafka-client. I want to use >= 0.10.0 version of kafka-client because of the cost of Message Conversion. If a consumer client is on a version before 0.10.0.0, the kafka broker will convert messages from the 0.10.0 format to an earlier format before sending the response to the consumer on an older version, that will result in additional overhead of performance and may cause an OOM in some cases.
I found the version of kafka-client in storm-kafka-client 1.0.6 is 0.10.1. So, I used it running with storm-core 1.0.3, and found an Exception as follow:
Time.nanoTime is not a method in storm-core 1.0.3 until 1.0.6.
So, I'm wondering is there a low-influence idea (without changing version of storm-core 1.0.3) for storm job to consume message from kafka 1.1. Thanks a lot!
Best regards,
Kun
> On 20 Aug 2018, at 11:45, Erik Weathers <er...@apache.org> wrote:
>
> hey Kun,
>
> I'm unclear which version of storm-kafka and storm-core you are currently using, please explain in more detail what you are trying to accomplish.
> You'll need to change from using storm-kafka to storm-kafka-client, but again, please explain the versions of everything you are using.
>
> Thanks!
>
> - Erik
>
> On Sun, Aug 19, 2018 at 7:33 PM stephen <zhaotkun@126.com <ma...@126.com>> wrote:
> Dear all,
>
> I have a question about upgrade kafka from 0.8.X to 1.1 using storm-core 1.0.3. Are there some feasible methods for me without changing the version of storm-core?
>
> Best regards,
> Kun
>
Re: upgrade kafka from 0.8.X to 1.1 using storm-core 1.0.3
Posted by Erik Weathers <er...@apache.org>.
hey Kun,
I'm unclear which version of storm-kafka and storm-core you are currently
using, please explain in more detail what you are trying to accomplish.
You'll need to change from using storm-kafka to storm-kafka-client, but
again, please explain the versions of everything you are using.
Thanks!
- Erik
On Sun, Aug 19, 2018 at 7:33 PM stephen <zh...@126.com> wrote:
> Dear all,
>
> I have a question about upgrade kafka from 0.8.X to 1.1 using storm-core
> 1.0.3. Are there some feasible methods for me without changing the version
> of storm-core?
>
> Best regards,
> Kun
>
>