You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by John Fang <xi...@alibaba-inc.com> on 2016/03/10 07:52:13 UTC
答复: Dynamic compiled object/classes and topology
You can put the needed compiled classes in the directory: your topology's src/main/resources, then you package your topology with resources. If you submit the topology, storm will extract the needed the compiled classes to a directory which is in the worker's scope. So the spout/bolt can find the classes.
Regards
John Fang
-----邮件原件-----
发件人: "Carlos Chávez" [mailto:cchavez@agssa.net]
发送时间: 2016年3月10日 11:29
收件人: user@storm.apache.org
主题: Dynamic compiled object/classes and topology
Hi All.
I have some bolt that compile some classes from a string, my classes are in Scala and I use ToolBox to compile. The compile is done in the prepare method and that work fine. Then in the execute method I'm able to call the methods from this classes.
The thing is that compile the classes take some time mostly because there are many thread compiling the same set of classes. I was trying to reduce the time compiling of classes before summit the topology, I was able to pass the compiled objects to the bolt and when the bolt deserialize the object it throw the class not found error.
I suspect that when I submit the topology the compiled classes are not in scope, not sure if the class loader when submit the topology is reset or something.
So, is there a way to add this dynamic compiled object/classes to the topology at runtime?
----
Cheers.
Carlos Chávez.
Re: 答复: Dynamic compiled object/classes and topology
Posted by Alexander T <mi...@gmail.com>.
Hi Carlos,
Glad to hear that it worked out!
Cheers
Alexander
On Mar 15, 2016 19:18, "Carlos Chávez" <cc...@agssa.net> wrote:
> Hi Alexander.
>
> As you pointed, I created a custom class loader and read the bytes from
> the .class files and load in this custom class loader, that works just
> fine.
>
> Now, the bolt is created instances of the objects that I need using this
> custom class loader.
>
> thank you.
>
> On 03/10/2016 07:29 AM, Alexander T wrote:
> > Hi Carlos,
> >
> > Just a wild guess here, but when you compile your source into a new
> > class you are defining a new class which will only be available in the
> > JVM where you compiled it, since it's not part of the built-time
> > classpath and not distributed with your app. Therefore you will need to
> > send the class bytes along with the serialized instance to the other
> > JVM. On the receiving JVM you will need to define your own classloader,
> > call defineClass with the class bytes (similar to the NetworkClassLoader
> > example
> > at https://docs.oracle.com/javase/7/docs/api/java/lang/ClassLoader.html)
> > and then de-serialize the instance. This should work.
> >
> > Best regards,
> > Alexander
> >
> > On Thu, Mar 10, 2016 at 2:15 PM, Carlos Chávez <cchavez@agssa.net
> > <ma...@agssa.net>> wrote:
> >
> > Hi John.
> >
> > But that should be done at build time, right?
> >
> > The thing is that I have already packed the fat jar and there is this
> > class that submit the topology, in this class I read from somewhere
> the
> > source code of the class and compile at runtime, but then when the
> bolt
> > try to deseralize the object it throw the class not found error.
> >
> > On 03/10/2016 12:52 AM, John Fang wrote:
> > > You can put the needed compiled classes in the directory: your
> > topology's src/main/resources, then you package your topology with
> > resources. If you submit the topology, storm will extract the needed
> > the compiled classes to a directory which is in the worker's scope.
> > So the spout/bolt can find the classes.
> > >
> > > Regards
> > > John Fang
> > >
> > >
> > > -----邮件原件-----
> > > 发件人: "Carlos Chávez" [mailto:cchavez@agssa.net
> > <ma...@agssa.net>]
> > > 发送时间: 2016年3月10日 11:29
> > > 收件人: user@storm.apache.org <ma...@storm.apache.org>
> > > 主题: Dynamic compiled object/classes and topology
> > >
> > > Hi All.
> > >
> > > I have some bolt that compile some classes from a string, my
> > classes are in Scala and I use ToolBox to compile. The compile is
> > done in the prepare method and that work fine. Then in the execute
> > method I'm able to call the methods from this classes.
> > >
> > > The thing is that compile the classes take some time mostly
> > because there are many thread compiling the same set of classes. I
> > was trying to reduce the time compiling of classes before summit the
> > topology, I was able to pass the compiled objects to the bolt and
> > when the bolt deserialize the object it throw the class not found
> error.
> > >
> > > I suspect that when I submit the topology the compiled classes are
> > not in scope, not sure if the class loader when submit the topology
> > is reset or something.
> > >
> > > So, is there a way to add this dynamic compiled object/classes to
> > the topology at runtime?
> > >
> > > ----
> > > Cheers.
> > > Carlos Chávez.
> > >
> >
> >
> > --
> > Saludos.
> > Carlos Chávez.
> > AG Software, S.A.
> >
> >
>
>
> --
> Saludos.
> Carlos Chávez.
> AG Software, S.A.
>
Re: 答复: Dynamic compiled object/classes and topology
Posted by Carlos Chávez <cc...@agssa.net>.
Hi Alexander.
As you pointed, I created a custom class loader and read the bytes from
the .class files and load in this custom class loader, that works just fine.
Now, the bolt is created instances of the objects that I need using this
custom class loader.
thank you.
On 03/10/2016 07:29 AM, Alexander T wrote:
> Hi Carlos,
>
> Just a wild guess here, but when you compile your source into a new
> class you are defining a new class which will only be available in the
> JVM where you compiled it, since it's not part of the built-time
> classpath and not distributed with your app. Therefore you will need to
> send the class bytes along with the serialized instance to the other
> JVM. On the receiving JVM you will need to define your own classloader,
> call defineClass with the class bytes (similar to the NetworkClassLoader
> example
> at https://docs.oracle.com/javase/7/docs/api/java/lang/ClassLoader.html)
> and then de-serialize the instance. This should work.
>
> Best regards,
> Alexander
>
> On Thu, Mar 10, 2016 at 2:15 PM, Carlos Chávez <cchavez@agssa.net
> <ma...@agssa.net>> wrote:
>
> Hi John.
>
> But that should be done at build time, right?
>
> The thing is that I have already packed the fat jar and there is this
> class that submit the topology, in this class I read from somewhere the
> source code of the class and compile at runtime, but then when the bolt
> try to deseralize the object it throw the class not found error.
>
> On 03/10/2016 12:52 AM, John Fang wrote:
> > You can put the needed compiled classes in the directory: your
> topology's src/main/resources, then you package your topology with
> resources. If you submit the topology, storm will extract the needed
> the compiled classes to a directory which is in the worker's scope.
> So the spout/bolt can find the classes.
> >
> > Regards
> > John Fang
> >
> >
> > -----邮件原件-----
> > 发件人: "Carlos Chávez" [mailto:cchavez@agssa.net
> <ma...@agssa.net>]
> > 发送时间: 2016年3月10日 11:29
> > 收件人: user@storm.apache.org <ma...@storm.apache.org>
> > 主题: Dynamic compiled object/classes and topology
> >
> > Hi All.
> >
> > I have some bolt that compile some classes from a string, my
> classes are in Scala and I use ToolBox to compile. The compile is
> done in the prepare method and that work fine. Then in the execute
> method I'm able to call the methods from this classes.
> >
> > The thing is that compile the classes take some time mostly
> because there are many thread compiling the same set of classes. I
> was trying to reduce the time compiling of classes before summit the
> topology, I was able to pass the compiled objects to the bolt and
> when the bolt deserialize the object it throw the class not found error.
> >
> > I suspect that when I submit the topology the compiled classes are
> not in scope, not sure if the class loader when submit the topology
> is reset or something.
> >
> > So, is there a way to add this dynamic compiled object/classes to
> the topology at runtime?
> >
> > ----
> > Cheers.
> > Carlos Chávez.
> >
>
>
> --
> Saludos.
> Carlos Chávez.
> AG Software, S.A.
>
>
--
Saludos.
Carlos Chávez.
AG Software, S.A.
Re: 答复: Dynamic compiled object/classes and topology
Posted by Alexander T <mi...@gmail.com>.
Hi Carlos,
Just a wild guess here, but when you compile your source into a new class
you are defining a new class which will only be available in the JVM where
you compiled it, since it's not part of the built-time classpath and not
distributed with your app. Therefore you will need to send the class bytes
along with the serialized instance to the other JVM. On the receiving JVM
you will need to define your own classloader, call defineClass with the
class bytes (similar to the NetworkClassLoader example at
https://docs.oracle.com/javase/7/docs/api/java/lang/ClassLoader.html) and
then de-serialize the instance. This should work.
Best regards,
Alexander
On Thu, Mar 10, 2016 at 2:15 PM, Carlos Chávez <cc...@agssa.net> wrote:
> Hi John.
>
> But that should be done at build time, right?
>
> The thing is that I have already packed the fat jar and there is this
> class that submit the topology, in this class I read from somewhere the
> source code of the class and compile at runtime, but then when the bolt
> try to deseralize the object it throw the class not found error.
>
> On 03/10/2016 12:52 AM, John Fang wrote:
> > You can put the needed compiled classes in the directory: your
> topology's src/main/resources, then you package your topology with
> resources. If you submit the topology, storm will extract the needed the
> compiled classes to a directory which is in the worker's scope. So the
> spout/bolt can find the classes.
> >
> > Regards
> > John Fang
> >
> >
> > -----邮件原件-----
> > 发件人: "Carlos Chávez" [mailto:cchavez@agssa.net]
> > 发送时间: 2016年3月10日 11:29
> > 收件人: user@storm.apache.org
> > 主题: Dynamic compiled object/classes and topology
> >
> > Hi All.
> >
> > I have some bolt that compile some classes from a string, my classes are
> in Scala and I use ToolBox to compile. The compile is done in the prepare
> method and that work fine. Then in the execute method I'm able to call the
> methods from this classes.
> >
> > The thing is that compile the classes take some time mostly because
> there are many thread compiling the same set of classes. I was trying to
> reduce the time compiling of classes before summit the topology, I was able
> to pass the compiled objects to the bolt and when the bolt deserialize the
> object it throw the class not found error.
> >
> > I suspect that when I submit the topology the compiled classes are not
> in scope, not sure if the class loader when submit the topology is reset or
> something.
> >
> > So, is there a way to add this dynamic compiled object/classes to the
> topology at runtime?
> >
> > ----
> > Cheers.
> > Carlos Chávez.
> >
>
>
> --
> Saludos.
> Carlos Chávez.
> AG Software, S.A.
>
Re: 答复: Dynamic compiled object/classes and topology
Posted by Carlos Chávez <cc...@agssa.net>.
Hi John.
But that should be done at build time, right?
The thing is that I have already packed the fat jar and there is this
class that submit the topology, in this class I read from somewhere the
source code of the class and compile at runtime, but then when the bolt
try to deseralize the object it throw the class not found error.
On 03/10/2016 12:52 AM, John Fang wrote:
> You can put the needed compiled classes in the directory: your topology's src/main/resources, then you package your topology with resources. If you submit the topology, storm will extract the needed the compiled classes to a directory which is in the worker's scope. So the spout/bolt can find the classes.
>
> Regards
> John Fang
>
>
> -----邮件原件-----
> 发件人: "Carlos Chávez" [mailto:cchavez@agssa.net]
> 发送时间: 2016年3月10日 11:29
> 收件人: user@storm.apache.org
> 主题: Dynamic compiled object/classes and topology
>
> Hi All.
>
> I have some bolt that compile some classes from a string, my classes are in Scala and I use ToolBox to compile. The compile is done in the prepare method and that work fine. Then in the execute method I'm able to call the methods from this classes.
>
> The thing is that compile the classes take some time mostly because there are many thread compiling the same set of classes. I was trying to reduce the time compiling of classes before summit the topology, I was able to pass the compiled objects to the bolt and when the bolt deserialize the object it throw the class not found error.
>
> I suspect that when I submit the topology the compiled classes are not in scope, not sure if the class loader when submit the topology is reset or something.
>
> So, is there a way to add this dynamic compiled object/classes to the topology at runtime?
>
> ----
> Cheers.
> Carlos Chávez.
>
--
Saludos.
Carlos Chávez.
AG Software, S.A.