You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by Abhishek Agarwal <ab...@gmail.com> on 2016/05/19 07:50:29 UTC
Submitting multiple jars in storm
I have come across this requirement multiple times. For an end user,
creating a uber jar and submitting it directly works fine. But we have
systems which provide processing as a service. These systems take business
logic, completely agnostic of storm, from client in a jar. Service has its
own code (Topology main method etc) in a separate jar which invokes the
business code. storm-sql is a similar type of system.
There are some solutions -
1. Preparing an uber jar on runtime - This changes the internal structure
of jar. META-INF entries have to be handled correctly.
2. Loading the client jar dynamically in a class loader - This approach has
its own set of problems related to class visibility.
I wanted to know if someone else here has run into problem and how did they
solve it. I have also submitted a PR as one possible solution. Any feedback
is most welcome -
https://github.com/apache/storm/pull/1296/files
--
Regards,
Abhishek Agarwal
Re: Submitting multiple jars in storm
Posted by Abhishek Agarwal <ab...@gmail.com>.
Thanks Bobby. I will try that out.
On Thu, May 19, 2016 at 7:09 PM, Bobby Evans <ev...@yahoo-inc.com.invalid>
wrote:
> It should be very simple to do with the distributed cache. Have them add
> the jar to the blob store and configure it to be downloaded with a link of
> their choosing. Then include that link in the topology.classpath. The
> only thing to be careful of is that if you upload a new version of the jar
> while your topology is running you may end up with some workers using one
> version of the jar and others using a different version, at least until
> they are restarted. - Bobby
>
> On Thursday, May 19, 2016 2:52 AM, Abhishek Agarwal <
> abhishcool@gmail.com> wrote:
>
>
> I have come across this requirement multiple times. For an end user,
> creating a uber jar and submitting it directly works fine. But we have
> systems which provide processing as a service. These systems take business
> logic, completely agnostic of storm, from client in a jar. Service has its
> own code (Topology main method etc) in a separate jar which invokes the
> business code. storm-sql is a similar type of system.
>
> There are some solutions -
> 1. Preparing an uber jar on runtime - This changes the internal structure
> of jar. META-INF entries have to be handled correctly.
>
> 2. Loading the client jar dynamically in a class loader - This approach has
> its own set of problems related to class visibility.
>
> I wanted to know if someone else here has run into problem and how did they
> solve it. I have also submitted a PR as one possible solution. Any feedback
> is most welcome -
> https://github.com/apache/storm/pull/1296/files
>
>
>
>
> --
> Regards,
> Abhishek Agarwal
>
>
>
>
--
Regards,
Abhishek Agarwal
Re: Submitting multiple jars in storm
Posted by Bobby Evans <ev...@yahoo-inc.com.INVALID>.
It should be very simple to do with the distributed cache. Have them add the jar to the blob store and configure it to be downloaded with a link of their choosing. Then include that link in the topology.classpath. The only thing to be careful of is that if you upload a new version of the jar while your topology is running you may end up with some workers using one version of the jar and others using a different version, at least until they are restarted. - Bobby
On Thursday, May 19, 2016 2:52 AM, Abhishek Agarwal <ab...@gmail.com> wrote:
I have come across this requirement multiple times. For an end user,
creating a uber jar and submitting it directly works fine. But we have
systems which provide processing as a service. These systems take business
logic, completely agnostic of storm, from client in a jar. Service has its
own code (Topology main method etc) in a separate jar which invokes the
business code. storm-sql is a similar type of system.
There are some solutions -
1. Preparing an uber jar on runtime - This changes the internal structure
of jar. META-INF entries have to be handled correctly.
2. Loading the client jar dynamically in a class loader - This approach has
its own set of problems related to class visibility.
I wanted to know if someone else here has run into problem and how did they
solve it. I have also submitted a PR as one possible solution. Any feedback
is most welcome -
https://github.com/apache/storm/pull/1296/files
--
Regards,
Abhishek Agarwal