You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Richards Peter <hb...@gmail.com> on 2015/07/29 14:21:37 UTC
Clarification regarding packaging of storm-core jar
Hi,
I was using storm-0.8.2 so far in my project. I am starting to evaluate
storm-0.9.*. I found a difference in the way storm-core(0.9.4)
/storm(0.8.2) jar has been packaged. I am not an expert in this packaging
concept. So I thought of taking some advice from the experts.
In storm 0.8.* jar, I find the transitive dependencies maintained
separately in $STORM_HOME/lib directory, i.e. the dependencies such as
http-core, commons-io, etc. are maintained separately in the lib directory.
Storm jar had only the classes related to storm. With such a design we were
able to upgrade these dependencies when we had to integrate another third
party component in our topology (by replacing older versions of jar in
storm lib with newer versions used by the third party component).
In storm 0.9.4, I find that some of the transitive dependencies of storm
are packaged within storm-core jar file. If we encounter a situation when
we have to upgrade an individual transitive dependency how should I go
forward?
Could you please share your thoughts about the same?
Thanks,
Richards Peter.
Re: Clarification regarding packaging of storm-core jar
Posted by "Grant Overby (groverby)" <gr...@cisco.com>.
I’d leave storm/lib vanilla and package the higher versions of those libs with your topology jar using maven shade to relocate the necessary packages to avoid conflict. Here is an example from a storm topology that relocates guava (com.google.common).
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>${version.shade}</version>
<configuration>
<relocations>
<relocation>
<pattern>com.google.common</pattern>
<shadedPattern>com.cisco.com.google.common</shadedPattern>
</relocation>
</relocations>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<artifactSet>
<excludes>
<exclude>org.datanucleus</exclude>
</excludes>
</artifactSet>
<promoteTransitiveDependencies>true</promoteTransitiveDependencies>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
</transformers>
</configuration>
</execution>
</executions>
</plugin>
[http://www.cisco.com/web/europe/images/email/signature/est2014/logo_06.png?ct=1398192119726]
Grant Overby
Software Engineer
Cisco.com<http://www.cisco.com/>
groverby@cisco.com<ma...@cisco.com>
Mobile: 865 724 4910
[http://www.cisco.com/assets/swa/img/thinkbeforeyouprint.gif] Think before you print.
This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message.
Please click here<http://www.cisco.com/web/about/doing_business/legal/cri/index.html> for Company Registration Information.
From: Richards Peter <hb...@gmail.com>>
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Date: Wednesday, July 29, 2015 at 8:21 AM
To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Subject: Clarification regarding packaging of storm-core jar
Hi,
I was using storm-0.8.2 so far in my project. I am starting to evaluate storm-0.9.*. I found a difference in the way storm-core(0.9.4) /storm(0.8.2) jar has been packaged. I am not an expert in this packaging concept. So I thought of taking some advice from the experts.
In storm 0.8.* jar, I find the transitive dependencies maintained separately in $STORM_HOME/lib directory, i.e. the dependencies such as http-core, commons-io, etc. are maintained separately in the lib directory. Storm jar had only the classes related to storm. With such a design we were able to upgrade these dependencies when we had to integrate another third party component in our topology (by replacing older versions of jar in storm lib with newer versions used by the third party component).
In storm 0.9.4, I find that some of the transitive dependencies of storm are packaged within storm-core jar file. If we encounter a situation when we have to upgrade an individual transitive dependency how should I go forward?
Could you please share your thoughts about the same?
Thanks,
Richards Peter.
Re: Clarification regarding packaging of storm-core jar
Posted by Richards Peter <hb...@gmail.com>.
Thank you Taylor and Grant for your replies.
I will try the approaches suggested by both of you.
Richards Peter.
Re: Clarification regarding packaging of storm-core jar
Posted by "P. Taylor Goetz" <pt...@gmail.com>.
Hi Richards,
The transitive dependencies that are now included in the storm-core jar file have their package names “relocated” so users can include their own versions of those dependencies without encountering dependency conflicts.
The best practice is to include your own version of those dependencies in your topology jar. Then if you need to upgrade, you simply repackage your topology jar.
An alternate option is to use the storm-bundled version of the dependencies. To do that you simply modify the import statements in your code to reference the relocated package (they will be prefixed with “org.apache.storm”). With this approach, your topology jar will be smaller, but you will be dependent on the specific versions bundled with storm, so if you upgrade storm later, you may need to recompile/repackage your topology code.
-Taylor
> On Jul 29, 2015, at 8:21 AM, Richards Peter <hb...@gmail.com> wrote:
>
> Hi,
>
> I was using storm-0.8.2 so far in my project. I am starting to evaluate storm-0.9.*. I found a difference in the way storm-core(0.9.4) /storm(0.8.2) jar has been packaged. I am not an expert in this packaging concept. So I thought of taking some advice from the experts.
>
> In storm 0.8.* jar, I find the transitive dependencies maintained separately in $STORM_HOME/lib directory, i.e. the dependencies such as http-core, commons-io, etc. are maintained separately in the lib directory. Storm jar had only the classes related to storm. With such a design we were able to upgrade these dependencies when we had to integrate another third party component in our topology (by replacing older versions of jar in storm lib with newer versions used by the third party component).
>
> In storm 0.9.4, I find that some of the transitive dependencies of storm are packaged within storm-core jar file. If we encounter a situation when we have to upgrade an individual transitive dependency how should I go forward?
>
> Could you please share your thoughts about the same?
>
> Thanks,
> Richards Peter.
>