Simple and Elegant Way to Include Libraries / Jar Files

[Jafar Hafiz <ja...@adsquare.com>]

Hi all,

I am building a topology with a load of libraries, such as spring,
geotools, etc. Right now, I extract each jar files and pack it into one
jar, (as suggested by many forums), using maven. The problem here, each jar
sometime has conflicting files, so I have to merge manually. Another thing,
though I pack my libraries in jar, storm loads its libraries first. For
example, I use guava 16, and storm uses guava 13. My program won't work
because guava 13 is loaded.

Current solution is I change the clojure script to put my jar first in the
classpath before other jars. Or, I put my jars in lib/ directory.

Is there any simple and reliable way to include jar in Storm? Sorry if this
email is too long. I want to make myself clear.

Best regards,
Jafar

Re: Simple and Elegant Way to Include Libraries / Jar Files

[Fritz Richter <fr...@adsquare.com>]

I would like to know how to deal with these libraries in general. Is it the fact that your project code must depend on the version and libraries Storm is adding to the classpath automatically? Do you always have to be aligned to the „old“ libs, storm is using by itself? There is and old discussion pager https://www.bountysource.com/issues/87805-keep-storm-dependencies-and-user-dependencies-independent . Unfortunately it ends in mid 2012 so I am wondering if there is any progress or final answer to this problem?

best
fr



Am 04.08.2014 um 12:22 schrieb Bruno D. Rodrigues <br...@litux.org>:

> 
>> No dia 04/08/2014, às 10:17, Jafar Hafiz <ja...@adsquare.com> escreveu:
>> 
>> Hi all,
>> 
>> I am building a topology with a load of libraries, such as spring, geotools, etc. Right now, I extract each jar files and pack it into one jar, (as suggested by many forums), using maven. The problem here, each jar sometime has conflicting files, so I have to merge manually. Another thing, though I pack my libraries in jar, storm loads its libraries first. For example, I use guava 16, and storm uses guava 13. My program won't work because guava 13 is loaded.
>> 
>> Current solution is I change the clojure script to put my jar first in the classpath before other jars. Or, I put my jars in lib/ directory.
>> 
>> Is there any simple and reliable way to include jar in Storm? Sorry if this email is too long. I want to make myself clear.
>> 
>> Best regards,
>> Jafar
> 
> I’d personally also like to know how to do this kind of thing. I’ve tried to join Kafka and Storm and failed miserably with the conflicting libraries. It would be nice if Storm (and Kafka) could split the libraries in such a way that a topology would depend only on the minimum amount of classes, and not the whole package, and then Storm should use the class loader to ensure the classes used internally wouldn’t conflict with the ones used by the topologies. For example, Kafka forces me to depend on ZooKeeper, weirdly only for the producer and not for the consumer, when it should depend on a stable minimal “zookeeper-client” interface, and not the whole shebang. 
> 

Re: Simple and Elegant Way to Include Libraries / Jar Files

["Bruno D. Rodrigues" <br...@litux.org>]

> No dia 04/08/2014, às 10:17, Jafar Hafiz <ja...@adsquare.com> escreveu:
> 
> Hi all,
> 
> I am building a topology with a load of libraries, such as spring, geotools, etc. Right now, I extract each jar files and pack it into one jar, (as suggested by many forums), using maven. The problem here, each jar sometime has conflicting files, so I have to merge manually. Another thing, though I pack my libraries in jar, storm loads its libraries first. For example, I use guava 16, and storm uses guava 13. My program won't work because guava 13 is loaded.
> 
> Current solution is I change the clojure script to put my jar first in the classpath before other jars. Or, I put my jars in lib/ directory.
> 
> Is there any simple and reliable way to include jar in Storm? Sorry if this email is too long. I want to make myself clear.
> 
> Best regards,
> Jafar

I’d personally also like to know how to do this kind of thing. I’ve tried to join Kafka and Storm and failed miserably with the conflicting libraries. It would be nice if Storm (and Kafka) could split the libraries in such a way that a topology would depend only on the minimum amount of classes, and not the whole package, and then Storm should use the class loader to ensure the classes used internally wouldn’t conflict with the ones used by the topologies. For example, Kafka forces me to depend on ZooKeeper, weirdly only for the producer and not for the consumer, when it should depend on a stable minimal “zookeeper-client” interface, and not the whole shebang.