You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by "P. Taylor Goetz" <pt...@gmail.com> on 2014/02/07 02:44:58 UTC
Re: http-client version conflict
I’m glad the shader plugin worked for you.
Updating dependencies can be tricky as it can easily introduce regressions.
Ultimately we need to figure out the best solution to avoiding conflicts between user code (i.e. dependencies in topology jar files) and Storm’s libraries.
The classloader approach has been attempted, but IMO Storm’s use of serialization complicates things significantly. Package relocation seems to be a relatively lightweight solution.
If that’s a direction we pursue, then it introduces the question of whether Storm should relocate its dependencies, or if that should be left up to the user (topology developer).
Elastic Search has gone down the path of relocating some of their dependencies [1] (not necessarily an endorsement, just an observation).
I’ve CC’d dev@ since this is all related to the infamous issue #115, which is now STORM-129 [2].
- Taylor
[1] https://github.com/elasticsearch/elasticsearch/blob/master/pom.xml#L474
[2] https://issues.apache.org/jira/browse/STORM-129
On Feb 6, 2014, at 7:25 PM, Vinay Pothnis <vi...@gmail.com> wrote:
> Thank you all for replies! The shader-plugin solution seems to work for us.
>
> I wonder if we can create a JIRA ticket for storm to upgrade the http-client library as part of their next release.
>
> -Vinay
>
>
> On Thu, Feb 6, 2014 at 2:38 PM, Michael Rose <mi...@fullcontact.com> wrote:
> We've done this with SLF4j and Guava as well without issues.
>
> Michael Rose (@Xorlev)
> Senior Platform Engineer, FullContact
> michael@fullcontact.com
>
>
>
> On Thu, Feb 6, 2014 at 3:03 PM, Mark Greene <ma...@evertrue.com> wrote:
> We had this problem as well. We modified our chef cookbook to just replace the older version with the newer one and storm didn't complain or have any other issues as a result.
>
>
> On Wed, Feb 5, 2014 at 10:31 AM, P. Taylor Goetz <pt...@gmail.com> wrote:
> Your best bet is probably to use the shade plugin to relocate the http-client package so it doesn’t conflict with the version storm uses.
>
> Storm does this with the libtrhift dependency in storm-core:
>
> https://github.com/apache/incubator-storm/blob/master/storm-core/pom.xml#L220
>
> (You can ignore the clojure transformer in that config, unless you have non-AOT clojure code that uses the http-client library).
>
> More information on using the shade plugin to do package relocations can be found here:
>
> http://maven.apache.org/plugins/maven-shade-plugin/examples/class-relocation.html
>
> - Taylor
>
> On Feb 4, 2014, at 4:27 PM, Vinay Pothnis <vi...@gmail.com> wrote:
>
> > Hello,
> >
> > I am using storm version 0.9.0.1.
> > My application depends on apache http-client version 4.3.2 - but storm depends on http-client version 4.1.1.
> >
> > What is the best way to override this dependency?
> >
> > Thanks
> > Vinay
>
>
>
>
Re: http-client version conflict
Posted by Vinay Pothnis <vi...@gmail.com>.
Thanks Taylor and Adam!
I agree with Adam in that it would be better for storm to relocate its
dependencies. I am sure that as storm's usage grows there will be more and
more users with such conflicts. Doing it once and testing well within storm
seems like the best approach.
-Vinay
On Thu, Feb 6, 2014 at 6:28 PM, Adam Lewis <ma...@adamlewis.com> wrote:
> My $0.02 on this subject:
>
> Without going down the path of class loader or OSGi mania and becoming a
> full container, I'd definitely be in favor of storm relocating its own
> dependencies. In this way edge cases around things like reflection can be
> handled once within storm rather than burdening every topology builder with
> those details. Part of the problem seems to be that storm makes extensive
> use (directly or transitively) of a lot of go-to utility libraries like
> guava, thrift, jodatime, json-simple, snakeyaml, commons-io, etc... I'm
> sure that leveraging these libraries allowed storm's development to proceed
> rapidly, but from a maturity perspective, it is problematic to impose these
> version choices on users. And while I might want Storm to, say, try to
> track the latest Guava version, that same policy could be very problematic
> for others.
>
> If storm can relocate even some of its own dependencies, I think that
> would be a great help to me at least. Longer term, I wonder how much of
> some of these libraries are really being used. For example, is clj-time
> (and by extension joda-time) really needed? Or just a small fraction of the
> functionality in that library? I can probably pitch in some of the effort
> required to do this, if this is the direction people want to go in.
>
>
>
>
> On Thu, Feb 6, 2014 at 8:44 PM, P. Taylor Goetz <pt...@gmail.com> wrote:
>
>> I'm glad the shader plugin worked for you.
>>
>> Updating dependencies can be tricky as it can easily introduce
>> regressions.
>>
>> Ultimately we need to figure out the best solution to avoiding conflicts
>> between user code (i.e. dependencies in topology jar files) and Storm's
>> libraries.
>>
>> The classloader approach has been attempted, but IMO Storm's use of
>> serialization complicates things significantly. Package relocation seems to
>> be a relatively lightweight solution.
>>
>> If that's a direction we pursue, then it introduces the question of
>> whether Storm should relocate its dependencies, or if that should be left
>> up to the user (topology developer).
>>
>> Elastic Search has gone down the path of relocating some of their
>> dependencies [1] (not necessarily an endorsement, just an observation).
>>
>> I've CC'd dev@ since this is all related to the infamous issue #115,
>> which is now STORM-129 [2].
>>
>> - Taylor
>>
>> [1]
>> https://github.com/elasticsearch/elasticsearch/blob/master/pom.xml#L474
>> [2] https://issues.apache.org/jira/browse/STORM-129
>>
>>
>>
>>
>>
>> On Feb 6, 2014, at 7:25 PM, Vinay Pothnis <vi...@gmail.com>
>> wrote:
>>
>> Thank you all for replies! The shader-plugin solution seems to work for
>> us.
>>
>> I wonder if we can create a JIRA ticket for storm to upgrade the
>> http-client library as part of their next release.
>>
>> -Vinay
>>
>>
>> On Thu, Feb 6, 2014 at 2:38 PM, Michael Rose <mi...@fullcontact.com>wrote:
>>
>>> We've done this with SLF4j and Guava as well without issues.
>>>
>>> Michael Rose (@Xorlev <https://twitter.com/xorlev>)
>>> Senior Platform Engineer, FullContact <http://www.fullcontact.com/>
>>> michael@fullcontact.com
>>>
>>>
>>> On Thu, Feb 6, 2014 at 3:03 PM, Mark Greene <ma...@evertrue.com> wrote:
>>>
>>>> We had this problem as well. We modified our chef cookbook to just
>>>> replace the older version with the newer one and storm didn't complain or
>>>> have any other issues as a result.
>>>>
>>>>
>>>> On Wed, Feb 5, 2014 at 10:31 AM, P. Taylor Goetz <pt...@gmail.com>wrote:
>>>>
>>>>> Your best bet is probably to use the shade plugin to relocate the
>>>>> http-client package so it doesn't conflict with the version storm uses.
>>>>>
>>>>> Storm does this with the libtrhift dependency in storm-core:
>>>>>
>>>>>
>>>>> https://github.com/apache/incubator-storm/blob/master/storm-core/pom.xml#L220
>>>>>
>>>>> (You can ignore the clojure transformer in that config, unless you
>>>>> have non-AOT clojure code that uses the http-client library).
>>>>>
>>>>> More information on using the shade plugin to do package relocations
>>>>> can be found here:
>>>>>
>>>>>
>>>>> http://maven.apache.org/plugins/maven-shade-plugin/examples/class-relocation.html
>>>>>
>>>>> - Taylor
>>>>>
>>>>> On Feb 4, 2014, at 4:27 PM, Vinay Pothnis <vi...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> > Hello,
>>>>> >
>>>>> > I am using storm version 0.9.0.1.
>>>>> > My application depends on apache http-client version 4.3.2 - but
>>>>> storm depends on http-client version 4.1.1.
>>>>> >
>>>>> > What is the best way to override this dependency?
>>>>> >
>>>>> > Thanks
>>>>> > Vinay
>>>>>
>>>>>
>>>>
>>>
>>
>>
>
Re: http-client version conflict
Posted by Vinay Pothnis <vi...@gmail.com>.
Thanks Taylor and Adam!
I agree with Adam in that it would be better for storm to relocate its
dependencies. I am sure that as storm's usage grows there will be more and
more users with such conflicts. Doing it once and testing well within storm
seems like the best approach.
-Vinay
On Thu, Feb 6, 2014 at 6:28 PM, Adam Lewis <ma...@adamlewis.com> wrote:
> My $0.02 on this subject:
>
> Without going down the path of class loader or OSGi mania and becoming a
> full container, I'd definitely be in favor of storm relocating its own
> dependencies. In this way edge cases around things like reflection can be
> handled once within storm rather than burdening every topology builder with
> those details. Part of the problem seems to be that storm makes extensive
> use (directly or transitively) of a lot of go-to utility libraries like
> guava, thrift, jodatime, json-simple, snakeyaml, commons-io, etc... I'm
> sure that leveraging these libraries allowed storm's development to proceed
> rapidly, but from a maturity perspective, it is problematic to impose these
> version choices on users. And while I might want Storm to, say, try to
> track the latest Guava version, that same policy could be very problematic
> for others.
>
> If storm can relocate even some of its own dependencies, I think that
> would be a great help to me at least. Longer term, I wonder how much of
> some of these libraries are really being used. For example, is clj-time
> (and by extension joda-time) really needed? Or just a small fraction of the
> functionality in that library? I can probably pitch in some of the effort
> required to do this, if this is the direction people want to go in.
>
>
>
>
> On Thu, Feb 6, 2014 at 8:44 PM, P. Taylor Goetz <pt...@gmail.com> wrote:
>
>> I'm glad the shader plugin worked for you.
>>
>> Updating dependencies can be tricky as it can easily introduce
>> regressions.
>>
>> Ultimately we need to figure out the best solution to avoiding conflicts
>> between user code (i.e. dependencies in topology jar files) and Storm's
>> libraries.
>>
>> The classloader approach has been attempted, but IMO Storm's use of
>> serialization complicates things significantly. Package relocation seems to
>> be a relatively lightweight solution.
>>
>> If that's a direction we pursue, then it introduces the question of
>> whether Storm should relocate its dependencies, or if that should be left
>> up to the user (topology developer).
>>
>> Elastic Search has gone down the path of relocating some of their
>> dependencies [1] (not necessarily an endorsement, just an observation).
>>
>> I've CC'd dev@ since this is all related to the infamous issue #115,
>> which is now STORM-129 [2].
>>
>> - Taylor
>>
>> [1]
>> https://github.com/elasticsearch/elasticsearch/blob/master/pom.xml#L474
>> [2] https://issues.apache.org/jira/browse/STORM-129
>>
>>
>>
>>
>>
>> On Feb 6, 2014, at 7:25 PM, Vinay Pothnis <vi...@gmail.com>
>> wrote:
>>
>> Thank you all for replies! The shader-plugin solution seems to work for
>> us.
>>
>> I wonder if we can create a JIRA ticket for storm to upgrade the
>> http-client library as part of their next release.
>>
>> -Vinay
>>
>>
>> On Thu, Feb 6, 2014 at 2:38 PM, Michael Rose <mi...@fullcontact.com>wrote:
>>
>>> We've done this with SLF4j and Guava as well without issues.
>>>
>>> Michael Rose (@Xorlev <https://twitter.com/xorlev>)
>>> Senior Platform Engineer, FullContact <http://www.fullcontact.com/>
>>> michael@fullcontact.com
>>>
>>>
>>> On Thu, Feb 6, 2014 at 3:03 PM, Mark Greene <ma...@evertrue.com> wrote:
>>>
>>>> We had this problem as well. We modified our chef cookbook to just
>>>> replace the older version with the newer one and storm didn't complain or
>>>> have any other issues as a result.
>>>>
>>>>
>>>> On Wed, Feb 5, 2014 at 10:31 AM, P. Taylor Goetz <pt...@gmail.com>wrote:
>>>>
>>>>> Your best bet is probably to use the shade plugin to relocate the
>>>>> http-client package so it doesn't conflict with the version storm uses.
>>>>>
>>>>> Storm does this with the libtrhift dependency in storm-core:
>>>>>
>>>>>
>>>>> https://github.com/apache/incubator-storm/blob/master/storm-core/pom.xml#L220
>>>>>
>>>>> (You can ignore the clojure transformer in that config, unless you
>>>>> have non-AOT clojure code that uses the http-client library).
>>>>>
>>>>> More information on using the shade plugin to do package relocations
>>>>> can be found here:
>>>>>
>>>>>
>>>>> http://maven.apache.org/plugins/maven-shade-plugin/examples/class-relocation.html
>>>>>
>>>>> - Taylor
>>>>>
>>>>> On Feb 4, 2014, at 4:27 PM, Vinay Pothnis <vi...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> > Hello,
>>>>> >
>>>>> > I am using storm version 0.9.0.1.
>>>>> > My application depends on apache http-client version 4.3.2 - but
>>>>> storm depends on http-client version 4.1.1.
>>>>> >
>>>>> > What is the best way to override this dependency?
>>>>> >
>>>>> > Thanks
>>>>> > Vinay
>>>>>
>>>>>
>>>>
>>>
>>
>>
>
Re: http-client version conflict
Posted by Matt Franklin <m....@gmail.com>.
On Fri, Feb 7, 2014 at 1:21 PM, Bobby Evans <ev...@yahoo-inc.com> wrote:
> I am +1 for relocating storm's dependencies, but I also think that we want
> to split some parts out too. There are a number of things on the worker
> classpath that the worker just does not need. Jetty for example. I think
> in these cases it seems simpler to have the worker process/client process
> and it's dependencies separated out from other storm system daemons. And
> then only look at relocating the dependencies the client/worker uses.
>
+1. This makes the most sense to me.
>
> --Bobby
>
> From: "Cody A. Ray" <co...@gmail.com>>
> Reply-To: "user@storm.incubator.apache.org<mailto:
> user@storm.incubator.apache.org>" <user@storm.incubator.apache.org<mailto:
> user@storm.incubator.apache.org>>
> Date: Thursday, February 6, 2014 at 9:09 PM
> To: "user@storm.incubator.apache.org<mailto:
> user@storm.incubator.apache.org>" <user@storm.incubator.apache.org<mailto:
> user@storm.incubator.apache.org>>
> Cc: "dev@storm.incubator.apache.org<ma...@storm.incubator.apache.org>"
> <de...@storm.incubator.apache.org>>
> Subject: Re: http-client version conflict
>
> +1 for relocating storm's dependencies. I just ran into this a couple
> weeks ago and have had to side-shelf the project until I find a
> work-around. Will check-out the shade plugin. Thanks for the rec!
>
> -Cody
>
>
> On Thu, Feb 6, 2014 at 9:07 PM, P. Taylor Goetz <ptgoetz@gmail.com<mailto:
> ptgoetz@gmail.com>> wrote:
> I tend to agree.
>
> The move to maven makes package relocation very easy, but we should be
> careful not to abuse it if that's a direction we decide to take.
>
> IMHO, users having to modify the contents of $STORM_HOME/lib is really bad
> and should only be done as a last resort. If someone asks for help with an
> issue after they've modified the contents of $STORM_HOM/lib, all bets are
> off.
>
> I also agree that we need to keep a close watch on our dependencies, and
> that some might warrant re-evaluation.
>
> I'd love to others opinions.
>
> - Taylor
>
> On Feb 6, 2014, at 9:28 PM, Adam Lewis <mail@adamlewis.com<mailto:
> mail@adamlewis.com>> wrote:
>
> My $0.02 on this subject:
>
> Without going down the path of class loader or OSGi mania and becoming a
> full container, I'd definitely be in favor of storm relocating its own
> dependencies. In this way edge cases around things like reflection can be
> handled once within storm rather than burdening every topology builder with
> those details. Part of the problem seems to be that storm makes extensive
> use (directly or transitively) of a lot of go-to utility libraries like
> guava, thrift, jodatime, json-simple, snakeyaml, commons-io, etc... I'm
> sure that leveraging these libraries allowed storm's development to proceed
> rapidly, but from a maturity perspective, it is problematic to impose these
> version choices on users. And while I might want Storm to, say, try to
> track the latest Guava version, that same policy could be very problematic
> for others.
>
> If storm can relocate even some of its own dependencies, I think that
> would be a great help to me at least. Longer term, I wonder how much of
> some of these libraries are really being used. For example, is clj-time
> (and by extension joda-time) really needed? Or just a small fraction of the
> functionality in that library? I can probably pitch in some of the effort
> required to do this, if this is the direction people want to go in.
>
>
>
>
> On Thu, Feb 6, 2014 at 8:44 PM, P. Taylor Goetz <ptgoetz@gmail.com<mailto:
> ptgoetz@gmail.com>> wrote:
> I'm glad the shader plugin worked for you.
>
> Updating dependencies can be tricky as it can easily introduce regressions.
>
> Ultimately we need to figure out the best solution to avoiding conflicts
> between user code (i.e. dependencies in topology jar files) and Storm's
> libraries.
>
> The classloader approach has been attempted, but IMO Storm's use of
> serialization complicates things significantly. Package relocation seems to
> be a relatively lightweight solution.
>
> If that's a direction we pursue, then it introduces the question of
> whether Storm should relocate its dependencies, or if that should be left
> up to the user (topology developer).
>
> Elastic Search has gone down the path of relocating some of their
> dependencies [1] (not necessarily an endorsement, just an observation).
>
> I've CC'd dev@ since this is all related to the infamous issue #115,
> which is now STORM-129 [2].
>
> - Taylor
>
> [1]
> https://github.com/elasticsearch/elasticsearch/blob/master/pom.xml#L474
> [2] https://issues.apache.org/jira/browse/STORM-129
>
>
>
>
>
> On Feb 6, 2014, at 7:25 PM, Vinay Pothnis <vinay.pothnis@gmail.com<mailto:
> vinay.pothnis@gmail.com>> wrote:
>
> Thank you all for replies! The shader-plugin solution seems to work for us.
>
> I wonder if we can create a JIRA ticket for storm to upgrade the
> http-client library as part of their next release.
>
> -Vinay
>
>
> On Thu, Feb 6, 2014 at 2:38 PM, Michael Rose <michael@fullcontact.com
> <ma...@fullcontact.com>> wrote:
> We've done this with SLF4j and Guava as well without issues.
>
>
> Michael Rose (@Xorlev<https://twitter.com/xorlev>)
> Senior Platform Engineer, FullContact<http://www.fullcontact.com/>
> michael@fullcontact.com<ma...@fullcontact.com>
>
>
> On Thu, Feb 6, 2014 at 3:03 PM, Mark Greene <mark@evertrue.com<mailto:
> mark@evertrue.com>> wrote:
> We had this problem as well. We modified our chef cookbook to just replace
> the older version with the newer one and storm didn't complain or have any
> other issues as a result.
>
>
> On Wed, Feb 5, 2014 at 10:31 AM, P. Taylor Goetz <ptgoetz@gmail.com
> <ma...@gmail.com>> wrote:
> Your best bet is probably to use the shade plugin to relocate the
> http-client package so it doesn't conflict with the version storm uses.
>
> Storm does this with the libtrhift dependency in storm-core:
>
>
> https://github.com/apache/incubator-storm/blob/master/storm-core/pom.xml#L220
>
> (You can ignore the clojure transformer in that config, unless you have
> non-AOT clojure code that uses the http-client library).
>
> More information on using the shade plugin to do package relocations can
> be found here:
>
>
> http://maven.apache.org/plugins/maven-shade-plugin/examples/class-relocation.html
>
> - Taylor
>
> On Feb 4, 2014, at 4:27 PM, Vinay Pothnis <vinay.pothnis@gmail.com<mailto:
> vinay.pothnis@gmail.com>> wrote:
>
> > Hello,
> >
> > I am using storm version 0.9.0.1.
> > My application depends on apache http-client version 4.3.2 - but storm
> depends on http-client version 4.1.1.
> >
> > What is the best way to override this dependency?
> >
> > Thanks
> > Vinay
>
>
>
>
>
>
>
>
>
>
> --
> Cody A. Ray, LEED AP
> codyaray@drexel.edu<ma...@drexel.edu>
> 215.501.7891
>
Re: http-client version conflict
Posted by Bobby Evans <ev...@yahoo-inc.com>.
I am +1 for relocating storm’s dependencies, but I also think that we want to split some parts out too. There are a number of things on the worker classpath that the worker just does not need. Jetty for example. I think in these cases it seems simpler to have the worker process/client process and it’s dependencies separated out from other storm system daemons. And then only look at relocating the dependencies the client/worker uses.
—Bobby
From: "Cody A. Ray" <co...@gmail.com>>
Reply-To: "user@storm.incubator.apache.org<ma...@storm.incubator.apache.org>" <us...@storm.incubator.apache.org>>
Date: Thursday, February 6, 2014 at 9:09 PM
To: "user@storm.incubator.apache.org<ma...@storm.incubator.apache.org>" <us...@storm.incubator.apache.org>>
Cc: "dev@storm.incubator.apache.org<ma...@storm.incubator.apache.org>" <de...@storm.incubator.apache.org>>
Subject: Re: http-client version conflict
+1 for relocating storm's dependencies. I just ran into this a couple weeks ago and have had to side-shelf the project until I find a work-around. Will check-out the shade plugin. Thanks for the rec!
-Cody
On Thu, Feb 6, 2014 at 9:07 PM, P. Taylor Goetz <pt...@gmail.com>> wrote:
I tend to agree.
The move to maven makes package relocation very easy, but we should be careful not to abuse it if that’s a direction we decide to take.
IMHO, users having to modify the contents of $STORM_HOME/lib is really bad and should only be done as a last resort. If someone asks for help with an issue after they’ve modified the contents of $STORM_HOM/lib, all bets are off.
I also agree that we need to keep a close watch on our dependencies, and that some might warrant re-evaluation.
I’d love to others opinions.
- Taylor
On Feb 6, 2014, at 9:28 PM, Adam Lewis <ma...@adamlewis.com>> wrote:
My $0.02 on this subject:
Without going down the path of class loader or OSGi mania and becoming a full container, I'd definitely be in favor of storm relocating its own dependencies. In this way edge cases around things like reflection can be handled once within storm rather than burdening every topology builder with those details. Part of the problem seems to be that storm makes extensive use (directly or transitively) of a lot of go-to utility libraries like guava, thrift, jodatime, json-simple, snakeyaml, commons-io, etc... I'm sure that leveraging these libraries allowed storm's development to proceed rapidly, but from a maturity perspective, it is problematic to impose these version choices on users. And while I might want Storm to, say, try to track the latest Guava version, that same policy could be very problematic for others.
If storm can relocate even some of its own dependencies, I think that would be a great help to me at least. Longer term, I wonder how much of some of these libraries are really being used. For example, is clj-time (and by extension joda-time) really needed? Or just a small fraction of the functionality in that library? I can probably pitch in some of the effort required to do this, if this is the direction people want to go in.
On Thu, Feb 6, 2014 at 8:44 PM, P. Taylor Goetz <pt...@gmail.com>> wrote:
I’m glad the shader plugin worked for you.
Updating dependencies can be tricky as it can easily introduce regressions.
Ultimately we need to figure out the best solution to avoiding conflicts between user code (i.e. dependencies in topology jar files) and Storm’s libraries.
The classloader approach has been attempted, but IMO Storm’s use of serialization complicates things significantly. Package relocation seems to be a relatively lightweight solution.
If that’s a direction we pursue, then it introduces the question of whether Storm should relocate its dependencies, or if that should be left up to the user (topology developer).
Elastic Search has gone down the path of relocating some of their dependencies [1] (not necessarily an endorsement, just an observation).
I’ve CC’d dev@ since this is all related to the infamous issue #115, which is now STORM-129 [2].
- Taylor
[1] https://github.com/elasticsearch/elasticsearch/blob/master/pom.xml#L474
[2] https://issues.apache.org/jira/browse/STORM-129
On Feb 6, 2014, at 7:25 PM, Vinay Pothnis <vi...@gmail.com>> wrote:
Thank you all for replies! The shader-plugin solution seems to work for us.
I wonder if we can create a JIRA ticket for storm to upgrade the http-client library as part of their next release.
-Vinay
On Thu, Feb 6, 2014 at 2:38 PM, Michael Rose <mi...@fullcontact.com>> wrote:
We've done this with SLF4j and Guava as well without issues.
Michael Rose (@Xorlev<https://twitter.com/xorlev>)
Senior Platform Engineer, FullContact<http://www.fullcontact.com/>
michael@fullcontact.com<ma...@fullcontact.com>
On Thu, Feb 6, 2014 at 3:03 PM, Mark Greene <ma...@evertrue.com>> wrote:
We had this problem as well. We modified our chef cookbook to just replace the older version with the newer one and storm didn't complain or have any other issues as a result.
On Wed, Feb 5, 2014 at 10:31 AM, P. Taylor Goetz <pt...@gmail.com>> wrote:
Your best bet is probably to use the shade plugin to relocate the http-client package so it doesn’t conflict with the version storm uses.
Storm does this with the libtrhift dependency in storm-core:
https://github.com/apache/incubator-storm/blob/master/storm-core/pom.xml#L220
(You can ignore the clojure transformer in that config, unless you have non-AOT clojure code that uses the http-client library).
More information on using the shade plugin to do package relocations can be found here:
http://maven.apache.org/plugins/maven-shade-plugin/examples/class-relocation.html
- Taylor
On Feb 4, 2014, at 4:27 PM, Vinay Pothnis <vi...@gmail.com>> wrote:
> Hello,
>
> I am using storm version 0.9.0.1.
> My application depends on apache http-client version 4.3.2 - but storm depends on http-client version 4.1.1.
>
> What is the best way to override this dependency?
>
> Thanks
> Vinay
--
Cody A. Ray, LEED AP
codyaray@drexel.edu<ma...@drexel.edu>
215.501.7891
Re: http-client version conflict
Posted by "Cody A. Ray" <co...@gmail.com>.
+1 for relocating storm's dependencies. I just ran into this a couple weeks
ago and have had to side-shelf the project until I find a work-around. Will
check-out the shade plugin. Thanks for the rec!
-Cody
On Thu, Feb 6, 2014 at 9:07 PM, P. Taylor Goetz <pt...@gmail.com> wrote:
> I tend to agree.
>
> The move to maven makes package relocation very easy, but we should be
> careful not to abuse it if that's a direction we decide to take.
>
> IMHO, users having to modify the contents of $STORM_HOME/lib is really bad
> and should only be done as a last resort. If someone asks for help with an
> issue after they've modified the contents of $STORM_HOM/lib, all bets are
> off.
>
> I also agree that we need to keep a close watch on our dependencies, and
> that some might warrant re-evaluation.
>
> I'd love to others opinions.
>
> - Taylor
>
> On Feb 6, 2014, at 9:28 PM, Adam Lewis <ma...@adamlewis.com> wrote:
>
> My $0.02 on this subject:
>
> Without going down the path of class loader or OSGi mania and becoming a
> full container, I'd definitely be in favor of storm relocating its own
> dependencies. In this way edge cases around things like reflection can be
> handled once within storm rather than burdening every topology builder with
> those details. Part of the problem seems to be that storm makes extensive
> use (directly or transitively) of a lot of go-to utility libraries like
> guava, thrift, jodatime, json-simple, snakeyaml, commons-io, etc... I'm
> sure that leveraging these libraries allowed storm's development to proceed
> rapidly, but from a maturity perspective, it is problematic to impose these
> version choices on users. And while I might want Storm to, say, try to
> track the latest Guava version, that same policy could be very problematic
> for others.
>
> If storm can relocate even some of its own dependencies, I think that
> would be a great help to me at least. Longer term, I wonder how much of
> some of these libraries are really being used. For example, is clj-time
> (and by extension joda-time) really needed? Or just a small fraction of the
> functionality in that library? I can probably pitch in some of the effort
> required to do this, if this is the direction people want to go in.
>
>
>
>
> On Thu, Feb 6, 2014 at 8:44 PM, P. Taylor Goetz <pt...@gmail.com> wrote:
>
>> I'm glad the shader plugin worked for you.
>>
>> Updating dependencies can be tricky as it can easily introduce
>> regressions.
>>
>> Ultimately we need to figure out the best solution to avoiding conflicts
>> between user code (i.e. dependencies in topology jar files) and Storm's
>> libraries.
>>
>> The classloader approach has been attempted, but IMO Storm's use of
>> serialization complicates things significantly. Package relocation seems to
>> be a relatively lightweight solution.
>>
>> If that's a direction we pursue, then it introduces the question of
>> whether Storm should relocate its dependencies, or if that should be left
>> up to the user (topology developer).
>>
>> Elastic Search has gone down the path of relocating some of their
>> dependencies [1] (not necessarily an endorsement, just an observation).
>>
>> I've CC'd dev@ since this is all related to the infamous issue #115,
>> which is now STORM-129 [2].
>>
>> - Taylor
>>
>> [1]
>> https://github.com/elasticsearch/elasticsearch/blob/master/pom.xml#L474
>> [2] https://issues.apache.org/jira/browse/STORM-129
>>
>>
>>
>>
>>
>> On Feb 6, 2014, at 7:25 PM, Vinay Pothnis <vi...@gmail.com>
>> wrote:
>>
>> Thank you all for replies! The shader-plugin solution seems to work for
>> us.
>>
>> I wonder if we can create a JIRA ticket for storm to upgrade the
>> http-client library as part of their next release.
>>
>> -Vinay
>>
>>
>> On Thu, Feb 6, 2014 at 2:38 PM, Michael Rose <mi...@fullcontact.com>wrote:
>>
>>> We've done this with SLF4j and Guava as well without issues.
>>>
>>> Michael Rose (@Xorlev <https://twitter.com/xorlev>)
>>> Senior Platform Engineer, FullContact <http://www.fullcontact.com/>
>>> michael@fullcontact.com
>>>
>>>
>>> On Thu, Feb 6, 2014 at 3:03 PM, Mark Greene <ma...@evertrue.com> wrote:
>>>
>>>> We had this problem as well. We modified our chef cookbook to just
>>>> replace the older version with the newer one and storm didn't complain or
>>>> have any other issues as a result.
>>>>
>>>>
>>>> On Wed, Feb 5, 2014 at 10:31 AM, P. Taylor Goetz <pt...@gmail.com>wrote:
>>>>
>>>>> Your best bet is probably to use the shade plugin to relocate the
>>>>> http-client package so it doesn't conflict with the version storm uses.
>>>>>
>>>>> Storm does this with the libtrhift dependency in storm-core:
>>>>>
>>>>>
>>>>> https://github.com/apache/incubator-storm/blob/master/storm-core/pom.xml#L220
>>>>>
>>>>> (You can ignore the clojure transformer in that config, unless you
>>>>> have non-AOT clojure code that uses the http-client library).
>>>>>
>>>>> More information on using the shade plugin to do package relocations
>>>>> can be found here:
>>>>>
>>>>>
>>>>> http://maven.apache.org/plugins/maven-shade-plugin/examples/class-relocation.html
>>>>>
>>>>> - Taylor
>>>>>
>>>>> On Feb 4, 2014, at 4:27 PM, Vinay Pothnis <vi...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> > Hello,
>>>>> >
>>>>> > I am using storm version 0.9.0.1.
>>>>> > My application depends on apache http-client version 4.3.2 - but
>>>>> storm depends on http-client version 4.1.1.
>>>>> >
>>>>> > What is the best way to override this dependency?
>>>>> >
>>>>> > Thanks
>>>>> > Vinay
>>>>>
>>>>>
>>>>
>>>
>>
>>
>
>
--
Cody A. Ray, LEED AP
codyaray@drexel.edu
215.501.7891
Re: http-client version conflict
Posted by "Cody A. Ray" <co...@gmail.com>.
+1 for relocating storm's dependencies. I just ran into this a couple weeks
ago and have had to side-shelf the project until I find a work-around. Will
check-out the shade plugin. Thanks for the rec!
-Cody
On Thu, Feb 6, 2014 at 9:07 PM, P. Taylor Goetz <pt...@gmail.com> wrote:
> I tend to agree.
>
> The move to maven makes package relocation very easy, but we should be
> careful not to abuse it if that's a direction we decide to take.
>
> IMHO, users having to modify the contents of $STORM_HOME/lib is really bad
> and should only be done as a last resort. If someone asks for help with an
> issue after they've modified the contents of $STORM_HOM/lib, all bets are
> off.
>
> I also agree that we need to keep a close watch on our dependencies, and
> that some might warrant re-evaluation.
>
> I'd love to others opinions.
>
> - Taylor
>
> On Feb 6, 2014, at 9:28 PM, Adam Lewis <ma...@adamlewis.com> wrote:
>
> My $0.02 on this subject:
>
> Without going down the path of class loader or OSGi mania and becoming a
> full container, I'd definitely be in favor of storm relocating its own
> dependencies. In this way edge cases around things like reflection can be
> handled once within storm rather than burdening every topology builder with
> those details. Part of the problem seems to be that storm makes extensive
> use (directly or transitively) of a lot of go-to utility libraries like
> guava, thrift, jodatime, json-simple, snakeyaml, commons-io, etc... I'm
> sure that leveraging these libraries allowed storm's development to proceed
> rapidly, but from a maturity perspective, it is problematic to impose these
> version choices on users. And while I might want Storm to, say, try to
> track the latest Guava version, that same policy could be very problematic
> for others.
>
> If storm can relocate even some of its own dependencies, I think that
> would be a great help to me at least. Longer term, I wonder how much of
> some of these libraries are really being used. For example, is clj-time
> (and by extension joda-time) really needed? Or just a small fraction of the
> functionality in that library? I can probably pitch in some of the effort
> required to do this, if this is the direction people want to go in.
>
>
>
>
> On Thu, Feb 6, 2014 at 8:44 PM, P. Taylor Goetz <pt...@gmail.com> wrote:
>
>> I'm glad the shader plugin worked for you.
>>
>> Updating dependencies can be tricky as it can easily introduce
>> regressions.
>>
>> Ultimately we need to figure out the best solution to avoiding conflicts
>> between user code (i.e. dependencies in topology jar files) and Storm's
>> libraries.
>>
>> The classloader approach has been attempted, but IMO Storm's use of
>> serialization complicates things significantly. Package relocation seems to
>> be a relatively lightweight solution.
>>
>> If that's a direction we pursue, then it introduces the question of
>> whether Storm should relocate its dependencies, or if that should be left
>> up to the user (topology developer).
>>
>> Elastic Search has gone down the path of relocating some of their
>> dependencies [1] (not necessarily an endorsement, just an observation).
>>
>> I've CC'd dev@ since this is all related to the infamous issue #115,
>> which is now STORM-129 [2].
>>
>> - Taylor
>>
>> [1]
>> https://github.com/elasticsearch/elasticsearch/blob/master/pom.xml#L474
>> [2] https://issues.apache.org/jira/browse/STORM-129
>>
>>
>>
>>
>>
>> On Feb 6, 2014, at 7:25 PM, Vinay Pothnis <vi...@gmail.com>
>> wrote:
>>
>> Thank you all for replies! The shader-plugin solution seems to work for
>> us.
>>
>> I wonder if we can create a JIRA ticket for storm to upgrade the
>> http-client library as part of their next release.
>>
>> -Vinay
>>
>>
>> On Thu, Feb 6, 2014 at 2:38 PM, Michael Rose <mi...@fullcontact.com>wrote:
>>
>>> We've done this with SLF4j and Guava as well without issues.
>>>
>>> Michael Rose (@Xorlev <https://twitter.com/xorlev>)
>>> Senior Platform Engineer, FullContact <http://www.fullcontact.com/>
>>> michael@fullcontact.com
>>>
>>>
>>> On Thu, Feb 6, 2014 at 3:03 PM, Mark Greene <ma...@evertrue.com> wrote:
>>>
>>>> We had this problem as well. We modified our chef cookbook to just
>>>> replace the older version with the newer one and storm didn't complain or
>>>> have any other issues as a result.
>>>>
>>>>
>>>> On Wed, Feb 5, 2014 at 10:31 AM, P. Taylor Goetz <pt...@gmail.com>wrote:
>>>>
>>>>> Your best bet is probably to use the shade plugin to relocate the
>>>>> http-client package so it doesn't conflict with the version storm uses.
>>>>>
>>>>> Storm does this with the libtrhift dependency in storm-core:
>>>>>
>>>>>
>>>>> https://github.com/apache/incubator-storm/blob/master/storm-core/pom.xml#L220
>>>>>
>>>>> (You can ignore the clojure transformer in that config, unless you
>>>>> have non-AOT clojure code that uses the http-client library).
>>>>>
>>>>> More information on using the shade plugin to do package relocations
>>>>> can be found here:
>>>>>
>>>>>
>>>>> http://maven.apache.org/plugins/maven-shade-plugin/examples/class-relocation.html
>>>>>
>>>>> - Taylor
>>>>>
>>>>> On Feb 4, 2014, at 4:27 PM, Vinay Pothnis <vi...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> > Hello,
>>>>> >
>>>>> > I am using storm version 0.9.0.1.
>>>>> > My application depends on apache http-client version 4.3.2 - but
>>>>> storm depends on http-client version 4.1.1.
>>>>> >
>>>>> > What is the best way to override this dependency?
>>>>> >
>>>>> > Thanks
>>>>> > Vinay
>>>>>
>>>>>
>>>>
>>>
>>
>>
>
>
--
Cody A. Ray, LEED AP
codyaray@drexel.edu
215.501.7891
Re: http-client version conflict
Posted by "P. Taylor Goetz" <pt...@gmail.com>.
I tend to agree.
The move to maven makes package relocation very easy, but we should be careful not to abuse it if that’s a direction we decide to take.
IMHO, users having to modify the contents of $STORM_HOME/lib is really bad and should only be done as a last resort. If someone asks for help with an issue after they’ve modified the contents of $STORM_HOM/lib, all bets are off.
I also agree that we need to keep a close watch on our dependencies, and that some might warrant re-evaluation.
I’d love to others opinions.
- Taylor
On Feb 6, 2014, at 9:28 PM, Adam Lewis <ma...@adamlewis.com> wrote:
> My $0.02 on this subject:
>
> Without going down the path of class loader or OSGi mania and becoming a full container, I'd definitely be in favor of storm relocating its own dependencies. In this way edge cases around things like reflection can be handled once within storm rather than burdening every topology builder with those details. Part of the problem seems to be that storm makes extensive use (directly or transitively) of a lot of go-to utility libraries like guava, thrift, jodatime, json-simple, snakeyaml, commons-io, etc... I'm sure that leveraging these libraries allowed storm's development to proceed rapidly, but from a maturity perspective, it is problematic to impose these version choices on users. And while I might want Storm to, say, try to track the latest Guava version, that same policy could be very problematic for others.
>
> If storm can relocate even some of its own dependencies, I think that would be a great help to me at least. Longer term, I wonder how much of some of these libraries are really being used. For example, is clj-time (and by extension joda-time) really needed? Or just a small fraction of the functionality in that library? I can probably pitch in some of the effort required to do this, if this is the direction people want to go in.
>
>
>
>
> On Thu, Feb 6, 2014 at 8:44 PM, P. Taylor Goetz <pt...@gmail.com> wrote:
> I’m glad the shader plugin worked for you.
>
> Updating dependencies can be tricky as it can easily introduce regressions.
>
> Ultimately we need to figure out the best solution to avoiding conflicts between user code (i.e. dependencies in topology jar files) and Storm’s libraries.
>
> The classloader approach has been attempted, but IMO Storm’s use of serialization complicates things significantly. Package relocation seems to be a relatively lightweight solution.
>
> If that’s a direction we pursue, then it introduces the question of whether Storm should relocate its dependencies, or if that should be left up to the user (topology developer).
>
> Elastic Search has gone down the path of relocating some of their dependencies [1] (not necessarily an endorsement, just an observation).
>
> I’ve CC’d dev@ since this is all related to the infamous issue #115, which is now STORM-129 [2].
>
> - Taylor
>
> [1] https://github.com/elasticsearch/elasticsearch/blob/master/pom.xml#L474
> [2] https://issues.apache.org/jira/browse/STORM-129
>
>
>
>
>
> On Feb 6, 2014, at 7:25 PM, Vinay Pothnis <vi...@gmail.com> wrote:
>
>> Thank you all for replies! The shader-plugin solution seems to work for us.
>>
>> I wonder if we can create a JIRA ticket for storm to upgrade the http-client library as part of their next release.
>>
>> -Vinay
>>
>>
>> On Thu, Feb 6, 2014 at 2:38 PM, Michael Rose <mi...@fullcontact.com> wrote:
>> We've done this with SLF4j and Guava as well without issues.
>>
>> Michael Rose (@Xorlev)
>> Senior Platform Engineer, FullContact
>> michael@fullcontact.com
>>
>>
>>
>> On Thu, Feb 6, 2014 at 3:03 PM, Mark Greene <ma...@evertrue.com> wrote:
>> We had this problem as well. We modified our chef cookbook to just replace the older version with the newer one and storm didn't complain or have any other issues as a result.
>>
>>
>> On Wed, Feb 5, 2014 at 10:31 AM, P. Taylor Goetz <pt...@gmail.com> wrote:
>> Your best bet is probably to use the shade plugin to relocate the http-client package so it doesn’t conflict with the version storm uses.
>>
>> Storm does this with the libtrhift dependency in storm-core:
>>
>> https://github.com/apache/incubator-storm/blob/master/storm-core/pom.xml#L220
>>
>> (You can ignore the clojure transformer in that config, unless you have non-AOT clojure code that uses the http-client library).
>>
>> More information on using the shade plugin to do package relocations can be found here:
>>
>> http://maven.apache.org/plugins/maven-shade-plugin/examples/class-relocation.html
>>
>> - Taylor
>>
>> On Feb 4, 2014, at 4:27 PM, Vinay Pothnis <vi...@gmail.com> wrote:
>>
>> > Hello,
>> >
>> > I am using storm version 0.9.0.1.
>> > My application depends on apache http-client version 4.3.2 - but storm depends on http-client version 4.1.1.
>> >
>> > What is the best way to override this dependency?
>> >
>> > Thanks
>> > Vinay
>>
>>
>>
>>
>
>
Re: http-client version conflict
Posted by "P. Taylor Goetz" <pt...@gmail.com>.
I tend to agree.
The move to maven makes package relocation very easy, but we should be careful not to abuse it if that’s a direction we decide to take.
IMHO, users having to modify the contents of $STORM_HOME/lib is really bad and should only be done as a last resort. If someone asks for help with an issue after they’ve modified the contents of $STORM_HOM/lib, all bets are off.
I also agree that we need to keep a close watch on our dependencies, and that some might warrant re-evaluation.
I’d love to others opinions.
- Taylor
On Feb 6, 2014, at 9:28 PM, Adam Lewis <ma...@adamlewis.com> wrote:
> My $0.02 on this subject:
>
> Without going down the path of class loader or OSGi mania and becoming a full container, I'd definitely be in favor of storm relocating its own dependencies. In this way edge cases around things like reflection can be handled once within storm rather than burdening every topology builder with those details. Part of the problem seems to be that storm makes extensive use (directly or transitively) of a lot of go-to utility libraries like guava, thrift, jodatime, json-simple, snakeyaml, commons-io, etc... I'm sure that leveraging these libraries allowed storm's development to proceed rapidly, but from a maturity perspective, it is problematic to impose these version choices on users. And while I might want Storm to, say, try to track the latest Guava version, that same policy could be very problematic for others.
>
> If storm can relocate even some of its own dependencies, I think that would be a great help to me at least. Longer term, I wonder how much of some of these libraries are really being used. For example, is clj-time (and by extension joda-time) really needed? Or just a small fraction of the functionality in that library? I can probably pitch in some of the effort required to do this, if this is the direction people want to go in.
>
>
>
>
> On Thu, Feb 6, 2014 at 8:44 PM, P. Taylor Goetz <pt...@gmail.com> wrote:
> I’m glad the shader plugin worked for you.
>
> Updating dependencies can be tricky as it can easily introduce regressions.
>
> Ultimately we need to figure out the best solution to avoiding conflicts between user code (i.e. dependencies in topology jar files) and Storm’s libraries.
>
> The classloader approach has been attempted, but IMO Storm’s use of serialization complicates things significantly. Package relocation seems to be a relatively lightweight solution.
>
> If that’s a direction we pursue, then it introduces the question of whether Storm should relocate its dependencies, or if that should be left up to the user (topology developer).
>
> Elastic Search has gone down the path of relocating some of their dependencies [1] (not necessarily an endorsement, just an observation).
>
> I’ve CC’d dev@ since this is all related to the infamous issue #115, which is now STORM-129 [2].
>
> - Taylor
>
> [1] https://github.com/elasticsearch/elasticsearch/blob/master/pom.xml#L474
> [2] https://issues.apache.org/jira/browse/STORM-129
>
>
>
>
>
> On Feb 6, 2014, at 7:25 PM, Vinay Pothnis <vi...@gmail.com> wrote:
>
>> Thank you all for replies! The shader-plugin solution seems to work for us.
>>
>> I wonder if we can create a JIRA ticket for storm to upgrade the http-client library as part of their next release.
>>
>> -Vinay
>>
>>
>> On Thu, Feb 6, 2014 at 2:38 PM, Michael Rose <mi...@fullcontact.com> wrote:
>> We've done this with SLF4j and Guava as well without issues.
>>
>> Michael Rose (@Xorlev)
>> Senior Platform Engineer, FullContact
>> michael@fullcontact.com
>>
>>
>>
>> On Thu, Feb 6, 2014 at 3:03 PM, Mark Greene <ma...@evertrue.com> wrote:
>> We had this problem as well. We modified our chef cookbook to just replace the older version with the newer one and storm didn't complain or have any other issues as a result.
>>
>>
>> On Wed, Feb 5, 2014 at 10:31 AM, P. Taylor Goetz <pt...@gmail.com> wrote:
>> Your best bet is probably to use the shade plugin to relocate the http-client package so it doesn’t conflict with the version storm uses.
>>
>> Storm does this with the libtrhift dependency in storm-core:
>>
>> https://github.com/apache/incubator-storm/blob/master/storm-core/pom.xml#L220
>>
>> (You can ignore the clojure transformer in that config, unless you have non-AOT clojure code that uses the http-client library).
>>
>> More information on using the shade plugin to do package relocations can be found here:
>>
>> http://maven.apache.org/plugins/maven-shade-plugin/examples/class-relocation.html
>>
>> - Taylor
>>
>> On Feb 4, 2014, at 4:27 PM, Vinay Pothnis <vi...@gmail.com> wrote:
>>
>> > Hello,
>> >
>> > I am using storm version 0.9.0.1.
>> > My application depends on apache http-client version 4.3.2 - but storm depends on http-client version 4.1.1.
>> >
>> > What is the best way to override this dependency?
>> >
>> > Thanks
>> > Vinay
>>
>>
>>
>>
>
>
Re: http-client version conflict
Posted by Adam Lewis <ma...@adamlewis.com>.
My $0.02 on this subject:
Without going down the path of class loader or OSGi mania and becoming a
full container, I'd definitely be in favor of storm relocating its own
dependencies. In this way edge cases around things like reflection can be
handled once within storm rather than burdening every topology builder with
those details. Part of the problem seems to be that storm makes extensive
use (directly or transitively) of a lot of go-to utility libraries like
guava, thrift, jodatime, json-simple, snakeyaml, commons-io, etc... I'm
sure that leveraging these libraries allowed storm's development to proceed
rapidly, but from a maturity perspective, it is problematic to impose these
version choices on users. And while I might want Storm to, say, try to
track the latest Guava version, that same policy could be very problematic
for others.
If storm can relocate even some of its own dependencies, I think that would
be a great help to me at least. Longer term, I wonder how much of some of
these libraries are really being used. For example, is clj-time (and by
extension joda-time) really needed? Or just a small fraction of the
functionality in that library? I can probably pitch in some of the effort
required to do this, if this is the direction people want to go in.
On Thu, Feb 6, 2014 at 8:44 PM, P. Taylor Goetz <pt...@gmail.com> wrote:
> I'm glad the shader plugin worked for you.
>
> Updating dependencies can be tricky as it can easily introduce
> regressions.
>
> Ultimately we need to figure out the best solution to avoiding conflicts
> between user code (i.e. dependencies in topology jar files) and Storm's
> libraries.
>
> The classloader approach has been attempted, but IMO Storm's use of
> serialization complicates things significantly. Package relocation seems to
> be a relatively lightweight solution.
>
> If that's a direction we pursue, then it introduces the question of
> whether Storm should relocate its dependencies, or if that should be left
> up to the user (topology developer).
>
> Elastic Search has gone down the path of relocating some of their
> dependencies [1] (not necessarily an endorsement, just an observation).
>
> I've CC'd dev@ since this is all related to the infamous issue #115,
> which is now STORM-129 [2].
>
> - Taylor
>
> [1]
> https://github.com/elasticsearch/elasticsearch/blob/master/pom.xml#L474
> [2] https://issues.apache.org/jira/browse/STORM-129
>
>
>
>
>
> On Feb 6, 2014, at 7:25 PM, Vinay Pothnis <vi...@gmail.com> wrote:
>
> Thank you all for replies! The shader-plugin solution seems to work for
> us.
>
> I wonder if we can create a JIRA ticket for storm to upgrade the
> http-client library as part of their next release.
>
> -Vinay
>
>
> On Thu, Feb 6, 2014 at 2:38 PM, Michael Rose <mi...@fullcontact.com>wrote:
>
>> We've done this with SLF4j and Guava as well without issues.
>>
>> Michael Rose (@Xorlev <https://twitter.com/xorlev>)
>> Senior Platform Engineer, FullContact <http://www.fullcontact.com/>
>> michael@fullcontact.com
>>
>>
>> On Thu, Feb 6, 2014 at 3:03 PM, Mark Greene <ma...@evertrue.com> wrote:
>>
>>> We had this problem as well. We modified our chef cookbook to just
>>> replace the older version with the newer one and storm didn't complain or
>>> have any other issues as a result.
>>>
>>>
>>> On Wed, Feb 5, 2014 at 10:31 AM, P. Taylor Goetz <pt...@gmail.com>wrote:
>>>
>>>> Your best bet is probably to use the shade plugin to relocate the
>>>> http-client package so it doesn't conflict with the version storm uses.
>>>>
>>>> Storm does this with the libtrhift dependency in storm-core:
>>>>
>>>>
>>>> https://github.com/apache/incubator-storm/blob/master/storm-core/pom.xml#L220
>>>>
>>>> (You can ignore the clojure transformer in that config, unless you have
>>>> non-AOT clojure code that uses the http-client library).
>>>>
>>>> More information on using the shade plugin to do package relocations
>>>> can be found here:
>>>>
>>>>
>>>> http://maven.apache.org/plugins/maven-shade-plugin/examples/class-relocation.html
>>>>
>>>> - Taylor
>>>>
>>>> On Feb 4, 2014, at 4:27 PM, Vinay Pothnis <vi...@gmail.com>
>>>> wrote:
>>>>
>>>> > Hello,
>>>> >
>>>> > I am using storm version 0.9.0.1.
>>>> > My application depends on apache http-client version 4.3.2 - but
>>>> storm depends on http-client version 4.1.1.
>>>> >
>>>> > What is the best way to override this dependency?
>>>> >
>>>> > Thanks
>>>> > Vinay
>>>>
>>>>
>>>
>>
>
>
Re: http-client version conflict
Posted by Adam Lewis <ma...@adamlewis.com>.
My $0.02 on this subject:
Without going down the path of class loader or OSGi mania and becoming a
full container, I'd definitely be in favor of storm relocating its own
dependencies. In this way edge cases around things like reflection can be
handled once within storm rather than burdening every topology builder with
those details. Part of the problem seems to be that storm makes extensive
use (directly or transitively) of a lot of go-to utility libraries like
guava, thrift, jodatime, json-simple, snakeyaml, commons-io, etc... I'm
sure that leveraging these libraries allowed storm's development to proceed
rapidly, but from a maturity perspective, it is problematic to impose these
version choices on users. And while I might want Storm to, say, try to
track the latest Guava version, that same policy could be very problematic
for others.
If storm can relocate even some of its own dependencies, I think that would
be a great help to me at least. Longer term, I wonder how much of some of
these libraries are really being used. For example, is clj-time (and by
extension joda-time) really needed? Or just a small fraction of the
functionality in that library? I can probably pitch in some of the effort
required to do this, if this is the direction people want to go in.
On Thu, Feb 6, 2014 at 8:44 PM, P. Taylor Goetz <pt...@gmail.com> wrote:
> I'm glad the shader plugin worked for you.
>
> Updating dependencies can be tricky as it can easily introduce
> regressions.
>
> Ultimately we need to figure out the best solution to avoiding conflicts
> between user code (i.e. dependencies in topology jar files) and Storm's
> libraries.
>
> The classloader approach has been attempted, but IMO Storm's use of
> serialization complicates things significantly. Package relocation seems to
> be a relatively lightweight solution.
>
> If that's a direction we pursue, then it introduces the question of
> whether Storm should relocate its dependencies, or if that should be left
> up to the user (topology developer).
>
> Elastic Search has gone down the path of relocating some of their
> dependencies [1] (not necessarily an endorsement, just an observation).
>
> I've CC'd dev@ since this is all related to the infamous issue #115,
> which is now STORM-129 [2].
>
> - Taylor
>
> [1]
> https://github.com/elasticsearch/elasticsearch/blob/master/pom.xml#L474
> [2] https://issues.apache.org/jira/browse/STORM-129
>
>
>
>
>
> On Feb 6, 2014, at 7:25 PM, Vinay Pothnis <vi...@gmail.com> wrote:
>
> Thank you all for replies! The shader-plugin solution seems to work for
> us.
>
> I wonder if we can create a JIRA ticket for storm to upgrade the
> http-client library as part of their next release.
>
> -Vinay
>
>
> On Thu, Feb 6, 2014 at 2:38 PM, Michael Rose <mi...@fullcontact.com>wrote:
>
>> We've done this with SLF4j and Guava as well without issues.
>>
>> Michael Rose (@Xorlev <https://twitter.com/xorlev>)
>> Senior Platform Engineer, FullContact <http://www.fullcontact.com/>
>> michael@fullcontact.com
>>
>>
>> On Thu, Feb 6, 2014 at 3:03 PM, Mark Greene <ma...@evertrue.com> wrote:
>>
>>> We had this problem as well. We modified our chef cookbook to just
>>> replace the older version with the newer one and storm didn't complain or
>>> have any other issues as a result.
>>>
>>>
>>> On Wed, Feb 5, 2014 at 10:31 AM, P. Taylor Goetz <pt...@gmail.com>wrote:
>>>
>>>> Your best bet is probably to use the shade plugin to relocate the
>>>> http-client package so it doesn't conflict with the version storm uses.
>>>>
>>>> Storm does this with the libtrhift dependency in storm-core:
>>>>
>>>>
>>>> https://github.com/apache/incubator-storm/blob/master/storm-core/pom.xml#L220
>>>>
>>>> (You can ignore the clojure transformer in that config, unless you have
>>>> non-AOT clojure code that uses the http-client library).
>>>>
>>>> More information on using the shade plugin to do package relocations
>>>> can be found here:
>>>>
>>>>
>>>> http://maven.apache.org/plugins/maven-shade-plugin/examples/class-relocation.html
>>>>
>>>> - Taylor
>>>>
>>>> On Feb 4, 2014, at 4:27 PM, Vinay Pothnis <vi...@gmail.com>
>>>> wrote:
>>>>
>>>> > Hello,
>>>> >
>>>> > I am using storm version 0.9.0.1.
>>>> > My application depends on apache http-client version 4.3.2 - but
>>>> storm depends on http-client version 4.1.1.
>>>> >
>>>> > What is the best way to override this dependency?
>>>> >
>>>> > Thanks
>>>> > Vinay
>>>>
>>>>
>>>
>>
>
>