You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by Abhishek Agarwal <ab...@gmail.com> on 2015/07/30 08:55:25 UTC
Submitting multiple jars to a topology classpath
Currently, as far as I know one has to package all the dependencies into
one jar and then submit it along with topology class. StormSubmitter
interface also allows only one jar. Is there any particular reason for this
limitation?
We have a use case where we want to upload more than one jar without
packaging them together. How could this be achieved?
--
Regards,
Abhishek Agarwal
Re: Submitting multiple jars to a topology classpath
Posted by Ravi Sharma <pi...@gmail.com>.
Hi Abhishek,
Storm is distributed computing framework, that means your code will be
running on any of the node in your cluster. Also in a cluster you can run
many Topologies.
So now when Storm move one of your topology from Node A to node B , it
needs to know what all code it need to move. One jar seems the best option
else you will have to provide a seprate list for your topology(exactly what
single jar does).
Ravi.
On Thu, Jul 30, 2015 at 2:07 PM, Matthias J. Sax <
mjsax@informatik.hu-berlin.de> wrote:
> Storm itself does not provide any support to upload multiple jars. As a
> workaround you can put required jars into $STORM/lib folder manually
> (you need to do this on every node in the cluster!)
>
> -Matthias
>
> On 07/30/2015 08:55 AM, Abhishek Agarwal wrote:
> > Currently, as far as I know one has to package all the dependencies into
> > one jar and then submit it along with topology class. StormSubmitter
> > interface also allows only one jar. Is there any particular reason for
> this
> > limitation?
> >
> > We have a use case where we want to upload more than one jar without
> > packaging them together. How could this be achieved?
> >
>
>
Re: Submitting multiple jars to a topology classpath
Posted by "Matthias J. Sax" <mj...@informatik.hu-berlin.de>.
Storm itself does not provide any support to upload multiple jars. As a
workaround you can put required jars into $STORM/lib folder manually
(you need to do this on every node in the cluster!)
-Matthias
On 07/30/2015 08:55 AM, Abhishek Agarwal wrote:
> Currently, as far as I know one has to package all the dependencies into
> one jar and then submit it along with topology class. StormSubmitter
> interface also allows only one jar. Is there any particular reason for this
> limitation?
>
> We have a use case where we want to upload more than one jar without
> packaging them together. How could this be achieved?
>
Re: Submitting multiple jars to a topology classpath
Posted by Abhishek Agarwal <ab...@gmail.com>.
Thanks Bobby. I have put a similar workaround in our code, wherein we
download the jars from HDFS and put them in the worker directory.
On Thu, Aug 6, 2015 at 2:19 AM, Bobby Evans <ev...@yahoo-inc.com.invalid>
wrote:
> We are working on a distributed cache like system for storm that should
> allow you to do this. Please watch
> https://issues.apache.org/jira/browse/STORM-876 for more information
> about when we merge the basics of it in.
> - Bobby
>
>
> On Saturday, August 1, 2015 8:00 AM, Abhishek Agarwal <
> abhishcool@gmail.com> wrote:
>
>
> For an end user submitting topologies manually, it is probably easier to
> just upload one uber jar. However, we have a system in place where a client
> can come and submit the code which will run alongside the system code. In
> such case, this system would like to spawn a storm topology and run the
> client jar and the system jar. That is not possible right now.
>
> Also there should not be any dependency conflicts across the topologies, be
> it single jar or multiple jars. If the topology is moved to another node,
> supervisor should download all the jars just like it downloads the uber
> jar.
>
> On Fri, Jul 31, 2015 at 7:52 PM, Harsha <st...@harsha.io> wrote:
>
> > Abhishek,
> > Can you explain whats your use case and the need for uploading
> > multiple jars without packaging together. As others have noted since
> storm
> > expected to have multiple topologies in ordered to prevent dependency
> > conflicts across the topologies its better to submit one jar with all of
> > its dependencies included in the jar.
> >
> > Thanks,
> > Harsha
> >
> >
> > On July 29, 2015 at 11:57:19 PM, Abhishek Agarwal (abhishcool@gmail.com)
> > wrote:
> >
> > Currently, as far as I know one has to package all the dependencies into
> > one jar and then submit it along with topology class. StormSubmitter
> > interface also allows only one jar. Is there any particular reason for
> > this
> > limitation?
> >
> > We have a use case where we want to upload more than one jar without
> > packaging them together. How could this be achieved?
> >
> > --
> > Regards,
> > Abhishek Agarwal
> >
> >
>
>
> --
> Regards,
> Abhishek Agarwal
>
>
>
>
--
Regards,
Abhishek Agarwal
Re: Submitting multiple jars to a topology classpath
Posted by Bobby Evans <ev...@yahoo-inc.com.INVALID>.
We are working on a distributed cache like system for storm that should allow you to do this. Please watch https://issues.apache.org/jira/browse/STORM-876 for more information about when we merge the basics of it in.
- Bobby
On Saturday, August 1, 2015 8:00 AM, Abhishek Agarwal <ab...@gmail.com> wrote:
For an end user submitting topologies manually, it is probably easier to
just upload one uber jar. However, we have a system in place where a client
can come and submit the code which will run alongside the system code. In
such case, this system would like to spawn a storm topology and run the
client jar and the system jar. That is not possible right now.
Also there should not be any dependency conflicts across the topologies, be
it single jar or multiple jars. If the topology is moved to another node,
supervisor should download all the jars just like it downloads the uber
jar.
On Fri, Jul 31, 2015 at 7:52 PM, Harsha <st...@harsha.io> wrote:
> Abhishek,
> Can you explain whats your use case and the need for uploading
> multiple jars without packaging together. As others have noted since storm
> expected to have multiple topologies in ordered to prevent dependency
> conflicts across the topologies its better to submit one jar with all of
> its dependencies included in the jar.
>
> Thanks,
> Harsha
>
>
> On July 29, 2015 at 11:57:19 PM, Abhishek Agarwal (abhishcool@gmail.com)
> wrote:
>
> Currently, as far as I know one has to package all the dependencies into
> one jar and then submit it along with topology class. StormSubmitter
> interface also allows only one jar. Is there any particular reason for
> this
> limitation?
>
> We have a use case where we want to upload more than one jar without
> packaging them together. How could this be achieved?
>
> --
> Regards,
> Abhishek Agarwal
>
>
--
Regards,
Abhishek Agarwal
Re: Submitting multiple jars to a topology classpath
Posted by Abhishek Agarwal <ab...@gmail.com>.
For an end user submitting topologies manually, it is probably easier to
just upload one uber jar. However, we have a system in place where a client
can come and submit the code which will run alongside the system code. In
such case, this system would like to spawn a storm topology and run the
client jar and the system jar. That is not possible right now.
Also there should not be any dependency conflicts across the topologies, be
it single jar or multiple jars. If the topology is moved to another node,
supervisor should download all the jars just like it downloads the uber
jar.
On Fri, Jul 31, 2015 at 7:52 PM, Harsha <st...@harsha.io> wrote:
> Abhishek,
> Can you explain whats your use case and the need for uploading
> multiple jars without packaging together. As others have noted since storm
> expected to have multiple topologies in ordered to prevent dependency
> conflicts across the topologies its better to submit one jar with all of
> its dependencies included in the jar.
>
> Thanks,
> Harsha
>
>
> On July 29, 2015 at 11:57:19 PM, Abhishek Agarwal (abhishcool@gmail.com)
> wrote:
>
> Currently, as far as I know one has to package all the dependencies into
> one jar and then submit it along with topology class. StormSubmitter
> interface also allows only one jar. Is there any particular reason for
> this
> limitation?
>
> We have a use case where we want to upload more than one jar without
> packaging them together. How could this be achieved?
>
> --
> Regards,
> Abhishek Agarwal
>
>
--
Regards,
Abhishek Agarwal
Re: Submitting multiple jars to a topology classpath
Posted by Harsha <st...@harsha.io>.
Abhishek,
Can you explain whats your use case and the need for uploading multiple jars without packaging together. As others have noted since storm expected to have multiple topologies in ordered to prevent dependency conflicts across the topologies its better to submit one jar with all of its dependencies included in the jar.
Thanks,
Harsha
On July 29, 2015 at 11:57:19 PM, Abhishek Agarwal (abhishcool@gmail.com) wrote:
Currently, as far as I know one has to package all the dependencies into
one jar and then submit it along with topology class. StormSubmitter
interface also allows only one jar. Is there any particular reason for this
limitation?
We have a use case where we want to upload more than one jar without
packaging them together. How could this be achieved?
--
Regards,
Abhishek Agarwal