You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Software Dev <st...@gmail.com> on 2014/03/25 17:45:03 UTC

DI with Storm

How would you go about use DI (Google Guice) with a Storm cluster? The
worker nodes themselves will not have access to the Injector instance
after job submission.

Thanks

Re: DI with Storm

Posted by Patricio Echagüe <pa...@gmail.com>.
Correct. But if that is part of the jar it should be available in the
classpath.


On Tue, Mar 25, 2014 at 8:15 PM, Adam Lewis <ma...@adamlewis.com> wrote:

> Yeah, that's basically what I do.  My point about being hackish is that if
> your module classes need configuration, you need to have that same
> configuration available from every call site of InjectorProvider#get()
>
>
> On Tue, Mar 25, 2014 at 9:55 PM, Patricio Echagüe <pa...@gmail.com>wrote:
>
>> It's fine. You can synchronize with a static monitor the creation on the
>> injector per worker. That's how I do it.
>>
>> public class InjectorProvider {
>>
>> private static Injector injector;
>>
>> public static Injector get() {
>> if (injector == null) {
>> synchronized (InjectorProvider.class) {
>>  if (injector == null) {
>> injector = Guice.createInjector(
>> new DAOModule(),
>>  new S3Module();
>> }
>> }
>> }
>>
>> return injector;
>> }
>>
>>
>> On Tue, Mar 25, 2014 at 6:24 PM, Adam Lewis <ma...@adamlewis.com> wrote:
>>
>>>
>>> Doesn't Storm 0.9 have a prepare for the worker?
>>>>
>>>
>>> No, I don't think it does, but please point this out if I'm mistaken.  I
>>> found the right JIRA issue though:
>>> https://issues.apache.org/jira/browse/STORM-126
>>>
>>> Seems like the patch was well along but hasn't seen any recent activity.
>>>
>>
>>
>

Re: DI with Storm

Posted by Adam Lewis <ma...@adamlewis.com>.
Yeah, that's basically what I do.  My point about being hackish is that if
your module classes need configuration, you need to have that same
configuration available from every call site of InjectorProvider#get()


On Tue, Mar 25, 2014 at 9:55 PM, Patricio Echagüe <pa...@gmail.com>wrote:

> It's fine. You can synchronize with a static monitor the creation on the
> injector per worker. That's how I do it.
>
> public class InjectorProvider {
>
> private static Injector injector;
>
> public static Injector get() {
> if (injector == null) {
> synchronized (InjectorProvider.class) {
>  if (injector == null) {
> injector = Guice.createInjector(
> new DAOModule(),
>  new S3Module();
> }
> }
> }
>
> return injector;
> }
>
>
> On Tue, Mar 25, 2014 at 6:24 PM, Adam Lewis <ma...@adamlewis.com> wrote:
>
>>
>> Doesn't Storm 0.9 have a prepare for the worker?
>>>
>>
>> No, I don't think it does, but please point this out if I'm mistaken.  I
>> found the right JIRA issue though:
>> https://issues.apache.org/jira/browse/STORM-126
>>
>> Seems like the patch was well along but hasn't seen any recent activity.
>>
>
>

Re: DI with Storm

Posted by Software Dev <st...@gmail.com>.
Thanks again

On Fri, Mar 28, 2014 at 6:49 AM, Adam Lewis <ma...@adamlewis.com> wrote:
> I don't think you are necessarily limited to one approach.  Personally, I do
> something more like the former, although rather than relying on putting
> custom properties in the storm conf, I just pass them as arguments to
> topology component constructors...although this may not be best practice.
>
> The key is that the injector isn't going to be serializable; and in my case
> the object graph I build with the injector is also not serializable, so that
> entire construct has to be built once the topology is thrown "over the wall"
> to the workers.  Think of all the pre-serialization parameters as
> "configuration" (either via StormConf or a more pojo based approach) and
> post-serialization objects as true dependencies that can acquire and hold
> resources from the target environment, etc.
>
> There is nothing stopping you from using DI to build the topology components
> pre-serialization if all the injected values are serializable, although
> doing both pre and post serialization DI is probably going to get hairy as
> there is no clear way to tell guice where to "stop" injecting the object
> graph.
>
> It really comes down to what you're trying to achieve.  I hope this helps.
>
>
> On Thu, Mar 27, 2014 at 7:23 PM, Software Dev <st...@gmail.com>
> wrote:
>>
>> Actually.. some more questions. We are running 0.9.1-incubating so we
>> do in fact have the prepare method.
>>
>> Confused on..
>>
>> Should I add all configuration values (from a properties file) onto
>> the StormConf and then in the prepare method of my workers just
>> instantiate an Injector?
>>
>> Or...
>>
>> Should I instantiate an injector right away that is used to
>> instantiate some factory classes that will create bolts with primitive
>> arguments?
>>
>> On Thu, Mar 27, 2014 at 10:13 AM, Software Dev
>> <st...@gmail.com> wrote:
>> > Thanks. I think that cleared up most of my misunderstanding.
>> >
>> > On Thu, Mar 27, 2014 at 6:16 AM, Adam Lewis <ma...@adamlewis.com> wrote:
>> >> Yes that is exactly right, the submission to Nimbus is in the form of a
>> >> big
>> >> thrift message describing the topology...this message includes java
>> >> serialized blobs of your topology components (spouts/bolts). They get
>> >> instantiated within the VM calling StormSubmitter.  Typically you would
>> >> pass
>> >> configuration info to the constructor, but "dependencies" (e.g. DB
>> >> connection pool, etc) are transient fields.  Then in the prepare method
>> >> (called after deserialization on the worker) you use the serialized
>> >> configuration fields to initialize the transient ones.  Of course Guice
>> >> fits
>> >> naturally into that step.
>> >>
>> >>
>> >> On Thu, Mar 27, 2014 at 12:37 AM, Software Dev
>> >> <st...@gmail.com>
>> >> wrote:
>> >>>
>> >>> Ok so you would configure the map in the main method before submitting
>> >>> the topology. Then this conf can be used to create guice injectors. Is
>> >>> that correct?
>> >>>
>> >>> In the book "Getting Started with Storm" it states:
>> >>>
>> >>> "To customize a bolt, you should set parameters in its constructor and
>> >>> save them as instance variables so they will be serialized when
>> >>> submitting the bolt to the cluster."
>> >>>
>> >>> Does this mean bolts are instantiated on the client side before being
>> >>> submitted to nimbus/cluster?
>> >>>
>> >>> On Wed, Mar 26, 2014 at 2:05 PM, Svend Vanderveken
>> >>> <sv...@gmail.com> wrote:
>> >>> >
>> >>> > The storm configuration map is part of the arguments received by
>> >>> > each
>> >>> > prepare() method, in most Storm primitives, on each worker. It's
>> >>> > serialised
>> >>> > to each worker when a topology instance is started there. The
>> >>> > initial
>> >>> > storm
>> >>> > configuration map is provided at deploy time to Nimbus, in the class
>> >>> > containing the main() method, specified in the "storm jar blabla.jar
>> >>> > some.class.here" command.
>> >>> >
>> >>> >
>> >>> >
>> >>> >
>> >>> > On Wed, Mar 26, 2014 at 4:42 PM, Software Dev
>> >>> > <st...@gmail.com>
>> >>> > wrote:
>> >>> >>
>> >>> >> How does one get the configuration map to each worker?
>> >>> >>
>> >>> >> On Wed, Mar 26, 2014 at 6:41 AM, Adam Lewis <ma...@adamlewis.com>
>> >>> >> wrote:
>> >>> >> > Or, since this is only being called from prepare at startup
>> >>> >> > anyway,
>> >>> >> > simpler:
>> >>> >> >
>> >>> >> > public class InjectorProvider {
>> >>> >> >
>> >>> >> > private static Injector injector;
>> >>> >> > public static synchronized Injector get(Map conf) {
>> >>> >> > if (injector == null) {
>> >>> >> > injector = Guice.createInjector(
>> >>> >> > new DAOModule(conf),
>> >>> >> > new S3Module(conf));
>> >>> >> > }
>> >>> >> >
>> >>> >> > return injector;
>> >>> >> > }
>> >>> >> > }
>> >>> >> >
>> >>> >> >
>> >>> >> >
>> >>> >> >
>> >>> >> >
>> >>> >> > On Wed, Mar 26, 2014 at 9:26 AM, Svend Vanderveken
>> >>> >> > <sv...@gmail.com> wrote:
>> >>> >> >>
>> >>> >> >> > private static Injector injector;
>> >>> >> >>
>> >>> >> >> or better:
>> >>> >> >>
>> >>> >> >> private static volatile Injector injector;
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> see
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> http://www.cs.umd.edu/~pugh/java/memoryModel/DoubleCheckedLocking.html
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> On Tue, Mar 25, 2014 at 9:55 PM, Patricio Echagüe
>> >>> >> >> <pa...@gmail.com>
>> >>> >> >> wrote:
>> >>> >> >>>
>> >>> >> >>> It's fine. You can synchronize with a static monitor the
>> >>> >> >>> creation
>> >>> >> >>> on
>> >>> >> >>> the
>> >>> >> >>> injector per worker. That's how I do it.
>> >>> >> >>>
>> >>> >> >>> public class InjectorProvider {
>> >>> >> >>>
>> >>> >> >>> private static Injector injector;
>> >>> >> >>>
>> >>> >> >>> public static Injector get() {
>> >>> >> >>> if (injector == null) {
>> >>> >> >>> synchronized (InjectorProvider.class) {
>> >>> >> >>> if (injector == null) {
>> >>> >> >>> injector = Guice.createInjector(
>> >>> >> >>> new DAOModule(),
>> >>> >> >>> new S3Module();
>> >>> >> >>> }
>> >>> >> >>> }
>> >>> >> >>> }
>> >>> >> >>>
>> >>> >> >>> return injector;
>> >>> >> >>> }
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> On Tue, Mar 25, 2014 at 6:24 PM, Adam Lewis
>> >>> >> >>> <ma...@adamlewis.com>
>> >>> >> >>> wrote:
>> >>> >> >>>>
>> >>> >> >>>>
>> >>> >> >>>>> Doesn't Storm 0.9 have a prepare for the worker?
>> >>> >> >>>>
>> >>> >> >>>>
>> >>> >> >>>> No, I don't think it does, but please point this out if I'm
>> >>> >> >>>> mistaken.
>> >>> >> >>>> I
>> >>> >> >>>> found the right JIRA issue though:
>> >>> >> >>>> https://issues.apache.org/jira/browse/STORM-126
>> >>> >> >>>>
>> >>> >> >>>> Seems like the patch was well along but hasn't seen any recent
>> >>> >> >>>> activity.
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>
>> >>> >> >
>> >>> >
>> >>> >
>> >>
>> >>
>
>

Re: DI with Storm

Posted by Adam Lewis <ma...@adamlewis.com>.
I don't think you are necessarily limited to one approach.  Personally, I
do something more like the former, although rather than relying on putting
custom properties in the storm conf, I just pass them as arguments to
topology component constructors...although this may not be best practice.

The key is that the injector isn't going to be serializable; and in my case
the object graph I build with the injector is also not serializable, so
that entire construct has to be built once the topology is thrown "over the
wall" to the workers.  Think of all the pre-serialization parameters as
"configuration" (either via StormConf or a more pojo based approach) and
post-serialization objects as true dependencies that can acquire and hold
resources from the target environment, etc.

There is nothing stopping you from using DI to build the topology
components pre-serialization if all the injected values are serializable,
although doing both pre and post serialization DI is probably going to get
hairy as there is no clear way to tell guice where to "stop" injecting the
object graph.

It really comes down to what you're trying to achieve.  I hope this helps.


On Thu, Mar 27, 2014 at 7:23 PM, Software Dev <st...@gmail.com>wrote:

> Actually.. some more questions. We are running 0.9.1-incubating so we
> do in fact have the prepare method.
>
> Confused on..
>
> Should I add all configuration values (from a properties file) onto
> the StormConf and then in the prepare method of my workers just
> instantiate an Injector?
>
> Or...
>
> Should I instantiate an injector right away that is used to
> instantiate some factory classes that will create bolts with primitive
> arguments?
>
> On Thu, Mar 27, 2014 at 10:13 AM, Software Dev
> <st...@gmail.com> wrote:
> > Thanks. I think that cleared up most of my misunderstanding.
> >
> > On Thu, Mar 27, 2014 at 6:16 AM, Adam Lewis <ma...@adamlewis.com> wrote:
> >> Yes that is exactly right, the submission to Nimbus is in the form of a
> big
> >> thrift message describing the topology...this message includes java
> >> serialized blobs of your topology components (spouts/bolts). They get
> >> instantiated within the VM calling StormSubmitter.  Typically you would
> pass
> >> configuration info to the constructor, but "dependencies" (e.g. DB
> >> connection pool, etc) are transient fields.  Then in the prepare method
> >> (called after deserialization on the worker) you use the serialized
> >> configuration fields to initialize the transient ones.  Of course Guice
> fits
> >> naturally into that step.
> >>
> >>
> >> On Thu, Mar 27, 2014 at 12:37 AM, Software Dev <
> static.void.dev@gmail.com>
> >> wrote:
> >>>
> >>> Ok so you would configure the map in the main method before submitting
> >>> the topology. Then this conf can be used to create guice injectors. Is
> >>> that correct?
> >>>
> >>> In the book "Getting Started with Storm" it states:
> >>>
> >>> "To customize a bolt, you should set parameters in its constructor and
> >>> save them as instance variables so they will be serialized when
> >>> submitting the bolt to the cluster."
> >>>
> >>> Does this mean bolts are instantiated on the client side before being
> >>> submitted to nimbus/cluster?
> >>>
> >>> On Wed, Mar 26, 2014 at 2:05 PM, Svend Vanderveken
> >>> <sv...@gmail.com> wrote:
> >>> >
> >>> > The storm configuration map is part of the arguments received by each
> >>> > prepare() method, in most Storm primitives, on each worker. It's
> >>> > serialised
> >>> > to each worker when a topology instance is started there. The initial
> >>> > storm
> >>> > configuration map is provided at deploy time to Nimbus, in the class
> >>> > containing the main() method, specified in the "storm jar blabla.jar
> >>> > some.class.here" command.
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > On Wed, Mar 26, 2014 at 4:42 PM, Software Dev
> >>> > <st...@gmail.com>
> >>> > wrote:
> >>> >>
> >>> >> How does one get the configuration map to each worker?
> >>> >>
> >>> >> On Wed, Mar 26, 2014 at 6:41 AM, Adam Lewis <ma...@adamlewis.com>
> wrote:
> >>> >> > Or, since this is only being called from prepare at startup
> anyway,
> >>> >> > simpler:
> >>> >> >
> >>> >> > public class InjectorProvider {
> >>> >> >
> >>> >> > private static Injector injector;
> >>> >> > public static synchronized Injector get(Map conf) {
> >>> >> > if (injector == null) {
> >>> >> > injector = Guice.createInjector(
> >>> >> > new DAOModule(conf),
> >>> >> > new S3Module(conf));
> >>> >> > }
> >>> >> >
> >>> >> > return injector;
> >>> >> > }
> >>> >> > }
> >>> >> >
> >>> >> >
> >>> >> >
> >>> >> >
> >>> >> >
> >>> >> > On Wed, Mar 26, 2014 at 9:26 AM, Svend Vanderveken
> >>> >> > <sv...@gmail.com> wrote:
> >>> >> >>
> >>> >> >> > private static Injector injector;
> >>> >> >>
> >>> >> >> or better:
> >>> >> >>
> >>> >> >> private static volatile Injector injector;
> >>> >> >>
> >>> >> >>
> >>> >> >> see
> >>> >> >>
> >>> >> >>
> http://www.cs.umd.edu/~pugh/java/memoryModel/DoubleCheckedLocking.html
> >>> >> >>
> >>> >> >>
> >>> >> >> On Tue, Mar 25, 2014 at 9:55 PM, Patricio Echagüe
> >>> >> >> <pa...@gmail.com>
> >>> >> >> wrote:
> >>> >> >>>
> >>> >> >>> It's fine. You can synchronize with a static monitor the
> creation
> >>> >> >>> on
> >>> >> >>> the
> >>> >> >>> injector per worker. That's how I do it.
> >>> >> >>>
> >>> >> >>> public class InjectorProvider {
> >>> >> >>>
> >>> >> >>> private static Injector injector;
> >>> >> >>>
> >>> >> >>> public static Injector get() {
> >>> >> >>> if (injector == null) {
> >>> >> >>> synchronized (InjectorProvider.class) {
> >>> >> >>> if (injector == null) {
> >>> >> >>> injector = Guice.createInjector(
> >>> >> >>> new DAOModule(),
> >>> >> >>> new S3Module();
> >>> >> >>> }
> >>> >> >>> }
> >>> >> >>> }
> >>> >> >>>
> >>> >> >>> return injector;
> >>> >> >>> }
> >>> >> >>>
> >>> >> >>>
> >>> >> >>> On Tue, Mar 25, 2014 at 6:24 PM, Adam Lewis <mail@adamlewis.com
> >
> >>> >> >>> wrote:
> >>> >> >>>>
> >>> >> >>>>
> >>> >> >>>>> Doesn't Storm 0.9 have a prepare for the worker?
> >>> >> >>>>
> >>> >> >>>>
> >>> >> >>>> No, I don't think it does, but please point this out if I'm
> >>> >> >>>> mistaken.
> >>> >> >>>> I
> >>> >> >>>> found the right JIRA issue though:
> >>> >> >>>> https://issues.apache.org/jira/browse/STORM-126
> >>> >> >>>>
> >>> >> >>>> Seems like the patch was well along but hasn't seen any recent
> >>> >> >>>> activity.
> >>> >> >>>
> >>> >> >>>
> >>> >> >>
> >>> >> >
> >>> >
> >>> >
> >>
> >>
>

Re: DI with Storm

Posted by Software Dev <st...@gmail.com>.
Actually.. some more questions. We are running 0.9.1-incubating so we
do in fact have the prepare method.

Confused on..

Should I add all configuration values (from a properties file) onto
the StormConf and then in the prepare method of my workers just
instantiate an Injector?

Or...

Should I instantiate an injector right away that is used to
instantiate some factory classes that will create bolts with primitive
arguments?

On Thu, Mar 27, 2014 at 10:13 AM, Software Dev
<st...@gmail.com> wrote:
> Thanks. I think that cleared up most of my misunderstanding.
>
> On Thu, Mar 27, 2014 at 6:16 AM, Adam Lewis <ma...@adamlewis.com> wrote:
>> Yes that is exactly right, the submission to Nimbus is in the form of a big
>> thrift message describing the topology...this message includes java
>> serialized blobs of your topology components (spouts/bolts). They get
>> instantiated within the VM calling StormSubmitter.  Typically you would pass
>> configuration info to the constructor, but "dependencies" (e.g. DB
>> connection pool, etc) are transient fields.  Then in the prepare method
>> (called after deserialization on the worker) you use the serialized
>> configuration fields to initialize the transient ones.  Of course Guice fits
>> naturally into that step.
>>
>>
>> On Thu, Mar 27, 2014 at 12:37 AM, Software Dev <st...@gmail.com>
>> wrote:
>>>
>>> Ok so you would configure the map in the main method before submitting
>>> the topology. Then this conf can be used to create guice injectors. Is
>>> that correct?
>>>
>>> In the book "Getting Started with Storm" it states:
>>>
>>> "To customize a bolt, you should set parameters in its constructor and
>>> save them as instance variables so they will be serialized when
>>> submitting the bolt to the cluster."
>>>
>>> Does this mean bolts are instantiated on the client side before being
>>> submitted to nimbus/cluster?
>>>
>>> On Wed, Mar 26, 2014 at 2:05 PM, Svend Vanderveken
>>> <sv...@gmail.com> wrote:
>>> >
>>> > The storm configuration map is part of the arguments received by each
>>> > prepare() method, in most Storm primitives, on each worker. It's
>>> > serialised
>>> > to each worker when a topology instance is started there. The initial
>>> > storm
>>> > configuration map is provided at deploy time to Nimbus, in the class
>>> > containing the main() method, specified in the "storm jar blabla.jar
>>> > some.class.here" command.
>>> >
>>> >
>>> >
>>> >
>>> > On Wed, Mar 26, 2014 at 4:42 PM, Software Dev
>>> > <st...@gmail.com>
>>> > wrote:
>>> >>
>>> >> How does one get the configuration map to each worker?
>>> >>
>>> >> On Wed, Mar 26, 2014 at 6:41 AM, Adam Lewis <ma...@adamlewis.com> wrote:
>>> >> > Or, since this is only being called from prepare at startup anyway,
>>> >> > simpler:
>>> >> >
>>> >> > public class InjectorProvider {
>>> >> >
>>> >> > private static Injector injector;
>>> >> > public static synchronized Injector get(Map conf) {
>>> >> > if (injector == null) {
>>> >> > injector = Guice.createInjector(
>>> >> > new DAOModule(conf),
>>> >> > new S3Module(conf));
>>> >> > }
>>> >> >
>>> >> > return injector;
>>> >> > }
>>> >> > }
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> > On Wed, Mar 26, 2014 at 9:26 AM, Svend Vanderveken
>>> >> > <sv...@gmail.com> wrote:
>>> >> >>
>>> >> >> > private static Injector injector;
>>> >> >>
>>> >> >> or better:
>>> >> >>
>>> >> >> private static volatile Injector injector;
>>> >> >>
>>> >> >>
>>> >> >> see
>>> >> >>
>>> >> >> http://www.cs.umd.edu/~pugh/java/memoryModel/DoubleCheckedLocking.html
>>> >> >>
>>> >> >>
>>> >> >> On Tue, Mar 25, 2014 at 9:55 PM, Patricio Echagüe
>>> >> >> <pa...@gmail.com>
>>> >> >> wrote:
>>> >> >>>
>>> >> >>> It's fine. You can synchronize with a static monitor the creation
>>> >> >>> on
>>> >> >>> the
>>> >> >>> injector per worker. That's how I do it.
>>> >> >>>
>>> >> >>> public class InjectorProvider {
>>> >> >>>
>>> >> >>> private static Injector injector;
>>> >> >>>
>>> >> >>> public static Injector get() {
>>> >> >>> if (injector == null) {
>>> >> >>> synchronized (InjectorProvider.class) {
>>> >> >>> if (injector == null) {
>>> >> >>> injector = Guice.createInjector(
>>> >> >>> new DAOModule(),
>>> >> >>> new S3Module();
>>> >> >>> }
>>> >> >>> }
>>> >> >>> }
>>> >> >>>
>>> >> >>> return injector;
>>> >> >>> }
>>> >> >>>
>>> >> >>>
>>> >> >>> On Tue, Mar 25, 2014 at 6:24 PM, Adam Lewis <ma...@adamlewis.com>
>>> >> >>> wrote:
>>> >> >>>>
>>> >> >>>>
>>> >> >>>>> Doesn't Storm 0.9 have a prepare for the worker?
>>> >> >>>>
>>> >> >>>>
>>> >> >>>> No, I don't think it does, but please point this out if I'm
>>> >> >>>> mistaken.
>>> >> >>>> I
>>> >> >>>> found the right JIRA issue though:
>>> >> >>>> https://issues.apache.org/jira/browse/STORM-126
>>> >> >>>>
>>> >> >>>> Seems like the patch was well along but hasn't seen any recent
>>> >> >>>> activity.
>>> >> >>>
>>> >> >>>
>>> >> >>
>>> >> >
>>> >
>>> >
>>
>>

Re: DI with Storm

Posted by Software Dev <st...@gmail.com>.
Thanks. I think that cleared up most of my misunderstanding.

On Thu, Mar 27, 2014 at 6:16 AM, Adam Lewis <ma...@adamlewis.com> wrote:
> Yes that is exactly right, the submission to Nimbus is in the form of a big
> thrift message describing the topology...this message includes java
> serialized blobs of your topology components (spouts/bolts). They get
> instantiated within the VM calling StormSubmitter.  Typically you would pass
> configuration info to the constructor, but "dependencies" (e.g. DB
> connection pool, etc) are transient fields.  Then in the prepare method
> (called after deserialization on the worker) you use the serialized
> configuration fields to initialize the transient ones.  Of course Guice fits
> naturally into that step.
>
>
> On Thu, Mar 27, 2014 at 12:37 AM, Software Dev <st...@gmail.com>
> wrote:
>>
>> Ok so you would configure the map in the main method before submitting
>> the topology. Then this conf can be used to create guice injectors. Is
>> that correct?
>>
>> In the book "Getting Started with Storm" it states:
>>
>> "To customize a bolt, you should set parameters in its constructor and
>> save them as instance variables so they will be serialized when
>> submitting the bolt to the cluster."
>>
>> Does this mean bolts are instantiated on the client side before being
>> submitted to nimbus/cluster?
>>
>> On Wed, Mar 26, 2014 at 2:05 PM, Svend Vanderveken
>> <sv...@gmail.com> wrote:
>> >
>> > The storm configuration map is part of the arguments received by each
>> > prepare() method, in most Storm primitives, on each worker. It's
>> > serialised
>> > to each worker when a topology instance is started there. The initial
>> > storm
>> > configuration map is provided at deploy time to Nimbus, in the class
>> > containing the main() method, specified in the "storm jar blabla.jar
>> > some.class.here" command.
>> >
>> >
>> >
>> >
>> > On Wed, Mar 26, 2014 at 4:42 PM, Software Dev
>> > <st...@gmail.com>
>> > wrote:
>> >>
>> >> How does one get the configuration map to each worker?
>> >>
>> >> On Wed, Mar 26, 2014 at 6:41 AM, Adam Lewis <ma...@adamlewis.com> wrote:
>> >> > Or, since this is only being called from prepare at startup anyway,
>> >> > simpler:
>> >> >
>> >> > public class InjectorProvider {
>> >> >
>> >> > private static Injector injector;
>> >> > public static synchronized Injector get(Map conf) {
>> >> > if (injector == null) {
>> >> > injector = Guice.createInjector(
>> >> > new DAOModule(conf),
>> >> > new S3Module(conf));
>> >> > }
>> >> >
>> >> > return injector;
>> >> > }
>> >> > }
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > On Wed, Mar 26, 2014 at 9:26 AM, Svend Vanderveken
>> >> > <sv...@gmail.com> wrote:
>> >> >>
>> >> >> > private static Injector injector;
>> >> >>
>> >> >> or better:
>> >> >>
>> >> >> private static volatile Injector injector;
>> >> >>
>> >> >>
>> >> >> see
>> >> >>
>> >> >> http://www.cs.umd.edu/~pugh/java/memoryModel/DoubleCheckedLocking.html
>> >> >>
>> >> >>
>> >> >> On Tue, Mar 25, 2014 at 9:55 PM, Patricio Echagüe
>> >> >> <pa...@gmail.com>
>> >> >> wrote:
>> >> >>>
>> >> >>> It's fine. You can synchronize with a static monitor the creation
>> >> >>> on
>> >> >>> the
>> >> >>> injector per worker. That's how I do it.
>> >> >>>
>> >> >>> public class InjectorProvider {
>> >> >>>
>> >> >>> private static Injector injector;
>> >> >>>
>> >> >>> public static Injector get() {
>> >> >>> if (injector == null) {
>> >> >>> synchronized (InjectorProvider.class) {
>> >> >>> if (injector == null) {
>> >> >>> injector = Guice.createInjector(
>> >> >>> new DAOModule(),
>> >> >>> new S3Module();
>> >> >>> }
>> >> >>> }
>> >> >>> }
>> >> >>>
>> >> >>> return injector;
>> >> >>> }
>> >> >>>
>> >> >>>
>> >> >>> On Tue, Mar 25, 2014 at 6:24 PM, Adam Lewis <ma...@adamlewis.com>
>> >> >>> wrote:
>> >> >>>>
>> >> >>>>
>> >> >>>>> Doesn't Storm 0.9 have a prepare for the worker?
>> >> >>>>
>> >> >>>>
>> >> >>>> No, I don't think it does, but please point this out if I'm
>> >> >>>> mistaken.
>> >> >>>> I
>> >> >>>> found the right JIRA issue though:
>> >> >>>> https://issues.apache.org/jira/browse/STORM-126
>> >> >>>>
>> >> >>>> Seems like the patch was well along but hasn't seen any recent
>> >> >>>> activity.
>> >> >>>
>> >> >>>
>> >> >>
>> >> >
>> >
>> >
>
>

Re: DI with Storm

Posted by Adam Lewis <ma...@adamlewis.com>.
Yes that is exactly right, the submission to Nimbus is in the form of a big
thrift message describing the topology...this message includes java
serialized blobs of your topology components (spouts/bolts). They get
instantiated within the VM calling StormSubmitter.  Typically you would
pass configuration info to the constructor, but "dependencies" (e.g. DB
connection pool, etc) are transient fields.  Then in the prepare method
(called after deserialization on the worker) you use the serialized
configuration fields to initialize the transient ones.  Of course Guice
fits naturally into that step.


On Thu, Mar 27, 2014 at 12:37 AM, Software Dev <st...@gmail.com>wrote:

> Ok so you would configure the map in the main method before submitting
> the topology. Then this conf can be used to create guice injectors. Is
> that correct?
>
> In the book "Getting Started with Storm" it states:
>
> "To customize a bolt, you should set parameters in its constructor and
> save them as instance variables so they will be serialized when
> submitting the bolt to the cluster."
>
> Does this mean bolts are instantiated on the client side before being
> submitted to nimbus/cluster?
>
> On Wed, Mar 26, 2014 at 2:05 PM, Svend Vanderveken
> <sv...@gmail.com> wrote:
> >
> > The storm configuration map is part of the arguments received by each
> > prepare() method, in most Storm primitives, on each worker. It's
> serialised
> > to each worker when a topology instance is started there. The initial
> storm
> > configuration map is provided at deploy time to Nimbus, in the class
> > containing the main() method, specified in the "storm jar blabla.jar
> > some.class.here" command.
> >
> >
> >
> >
> > On Wed, Mar 26, 2014 at 4:42 PM, Software Dev <static.void.dev@gmail.com
> >
> > wrote:
> >>
> >> How does one get the configuration map to each worker?
> >>
> >> On Wed, Mar 26, 2014 at 6:41 AM, Adam Lewis <ma...@adamlewis.com> wrote:
> >> > Or, since this is only being called from prepare at startup anyway,
> >> > simpler:
> >> >
> >> > public class InjectorProvider {
> >> >
> >> > private static Injector injector;
> >> > public static synchronized Injector get(Map conf) {
> >> > if (injector == null) {
> >> > injector = Guice.createInjector(
> >> > new DAOModule(conf),
> >> > new S3Module(conf));
> >> > }
> >> >
> >> > return injector;
> >> > }
> >> > }
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Wed, Mar 26, 2014 at 9:26 AM, Svend Vanderveken
> >> > <sv...@gmail.com> wrote:
> >> >>
> >> >> > private static Injector injector;
> >> >>
> >> >> or better:
> >> >>
> >> >> private static volatile Injector injector;
> >> >>
> >> >>
> >> >> see
> >> >>
> http://www.cs.umd.edu/~pugh/java/memoryModel/DoubleCheckedLocking.html
> >> >>
> >> >>
> >> >> On Tue, Mar 25, 2014 at 9:55 PM, Patricio Echagüe <
> patricioe@gmail.com>
> >> >> wrote:
> >> >>>
> >> >>> It's fine. You can synchronize with a static monitor the creation on
> >> >>> the
> >> >>> injector per worker. That's how I do it.
> >> >>>
> >> >>> public class InjectorProvider {
> >> >>>
> >> >>> private static Injector injector;
> >> >>>
> >> >>> public static Injector get() {
> >> >>> if (injector == null) {
> >> >>> synchronized (InjectorProvider.class) {
> >> >>> if (injector == null) {
> >> >>> injector = Guice.createInjector(
> >> >>> new DAOModule(),
> >> >>> new S3Module();
> >> >>> }
> >> >>> }
> >> >>> }
> >> >>>
> >> >>> return injector;
> >> >>> }
> >> >>>
> >> >>>
> >> >>> On Tue, Mar 25, 2014 at 6:24 PM, Adam Lewis <ma...@adamlewis.com>
> >> >>> wrote:
> >> >>>>
> >> >>>>
> >> >>>>> Doesn't Storm 0.9 have a prepare for the worker?
> >> >>>>
> >> >>>>
> >> >>>> No, I don't think it does, but please point this out if I'm
> mistaken.
> >> >>>> I
> >> >>>> found the right JIRA issue though:
> >> >>>> https://issues.apache.org/jira/browse/STORM-126
> >> >>>>
> >> >>>> Seems like the patch was well along but hasn't seen any recent
> >> >>>> activity.
> >> >>>
> >> >>>
> >> >>
> >> >
> >
> >
>

Re: DI with Storm

Posted by Software Dev <st...@gmail.com>.
Ok so you would configure the map in the main method before submitting
the topology. Then this conf can be used to create guice injectors. Is
that correct?

In the book "Getting Started with Storm" it states:

"To customize a bolt, you should set parameters in its constructor and
save them as instance variables so they will be serialized when
submitting the bolt to the cluster."

Does this mean bolts are instantiated on the client side before being
submitted to nimbus/cluster?

On Wed, Mar 26, 2014 at 2:05 PM, Svend Vanderveken
<sv...@gmail.com> wrote:
>
> The storm configuration map is part of the arguments received by each
> prepare() method, in most Storm primitives, on each worker. It's serialised
> to each worker when a topology instance is started there. The initial storm
> configuration map is provided at deploy time to Nimbus, in the class
> containing the main() method, specified in the "storm jar blabla.jar
> some.class.here" command.
>
>
>
>
> On Wed, Mar 26, 2014 at 4:42 PM, Software Dev <st...@gmail.com>
> wrote:
>>
>> How does one get the configuration map to each worker?
>>
>> On Wed, Mar 26, 2014 at 6:41 AM, Adam Lewis <ma...@adamlewis.com> wrote:
>> > Or, since this is only being called from prepare at startup anyway,
>> > simpler:
>> >
>> > public class InjectorProvider {
>> >
>> > private static Injector injector;
>> > public static synchronized Injector get(Map conf) {
>> > if (injector == null) {
>> > injector = Guice.createInjector(
>> > new DAOModule(conf),
>> > new S3Module(conf));
>> > }
>> >
>> > return injector;
>> > }
>> > }
>> >
>> >
>> >
>> >
>> >
>> > On Wed, Mar 26, 2014 at 9:26 AM, Svend Vanderveken
>> > <sv...@gmail.com> wrote:
>> >>
>> >> > private static Injector injector;
>> >>
>> >> or better:
>> >>
>> >> private static volatile Injector injector;
>> >>
>> >>
>> >> see
>> >> http://www.cs.umd.edu/~pugh/java/memoryModel/DoubleCheckedLocking.html
>> >>
>> >>
>> >> On Tue, Mar 25, 2014 at 9:55 PM, Patricio Echagüe <pa...@gmail.com>
>> >> wrote:
>> >>>
>> >>> It's fine. You can synchronize with a static monitor the creation on
>> >>> the
>> >>> injector per worker. That's how I do it.
>> >>>
>> >>> public class InjectorProvider {
>> >>>
>> >>> private static Injector injector;
>> >>>
>> >>> public static Injector get() {
>> >>> if (injector == null) {
>> >>> synchronized (InjectorProvider.class) {
>> >>> if (injector == null) {
>> >>> injector = Guice.createInjector(
>> >>> new DAOModule(),
>> >>> new S3Module();
>> >>> }
>> >>> }
>> >>> }
>> >>>
>> >>> return injector;
>> >>> }
>> >>>
>> >>>
>> >>> On Tue, Mar 25, 2014 at 6:24 PM, Adam Lewis <ma...@adamlewis.com>
>> >>> wrote:
>> >>>>
>> >>>>
>> >>>>> Doesn't Storm 0.9 have a prepare for the worker?
>> >>>>
>> >>>>
>> >>>> No, I don't think it does, but please point this out if I'm mistaken.
>> >>>> I
>> >>>> found the right JIRA issue though:
>> >>>> https://issues.apache.org/jira/browse/STORM-126
>> >>>>
>> >>>> Seems like the patch was well along but hasn't seen any recent
>> >>>> activity.
>> >>>
>> >>>
>> >>
>> >
>
>

Re: DI with Storm

Posted by Svend Vanderveken <sv...@gmail.com>.
The storm configuration map is part of the arguments received by each
prepare() method, in most Storm primitives, on each worker. It's serialised
to each worker when a topology instance is started there. The initial storm
configuration map is provided at deploy time to Nimbus, in the class
containing the main() method, specified in the "storm jar blabla.jar
some.class.here" command.




On Wed, Mar 26, 2014 at 4:42 PM, Software Dev <st...@gmail.com>wrote:

> How does one get the configuration map to each worker?
>
> On Wed, Mar 26, 2014 at 6:41 AM, Adam Lewis <ma...@adamlewis.com> wrote:
> > Or, since this is only being called from prepare at startup anyway,
> simpler:
> >
> > public class InjectorProvider {
> >
> > private static Injector injector;
> > public static synchronized Injector get(Map conf) {
> > if (injector == null) {
> > injector = Guice.createInjector(
> > new DAOModule(conf),
> > new S3Module(conf));
> > }
> >
> > return injector;
> > }
> > }
> >
> >
> >
> >
> >
> > On Wed, Mar 26, 2014 at 9:26 AM, Svend Vanderveken
> > <sv...@gmail.com> wrote:
> >>
> >> > private static Injector injector;
> >>
> >> or better:
> >>
> >> private static volatile Injector injector;
> >>
> >>
> >> see
> http://www.cs.umd.edu/~pugh/java/memoryModel/DoubleCheckedLocking.html
> >>
> >>
> >> On Tue, Mar 25, 2014 at 9:55 PM, Patricio Echagüe <pa...@gmail.com>
> >> wrote:
> >>>
> >>> It's fine. You can synchronize with a static monitor the creation on
> the
> >>> injector per worker. That's how I do it.
> >>>
> >>> public class InjectorProvider {
> >>>
> >>> private static Injector injector;
> >>>
> >>> public static Injector get() {
> >>> if (injector == null) {
> >>> synchronized (InjectorProvider.class) {
> >>> if (injector == null) {
> >>> injector = Guice.createInjector(
> >>> new DAOModule(),
> >>> new S3Module();
> >>> }
> >>> }
> >>> }
> >>>
> >>> return injector;
> >>> }
> >>>
> >>>
> >>> On Tue, Mar 25, 2014 at 6:24 PM, Adam Lewis <ma...@adamlewis.com>
> wrote:
> >>>>
> >>>>
> >>>>> Doesn't Storm 0.9 have a prepare for the worker?
> >>>>
> >>>>
> >>>> No, I don't think it does, but please point this out if I'm mistaken.
>  I
> >>>> found the right JIRA issue though:
> >>>> https://issues.apache.org/jira/browse/STORM-126
> >>>>
> >>>> Seems like the patch was well along but hasn't seen any recent
> activity.
> >>>
> >>>
> >>
> >
>

Re: DI with Storm

Posted by Software Dev <st...@gmail.com>.
How does one get the configuration map to each worker?

On Wed, Mar 26, 2014 at 6:41 AM, Adam Lewis <ma...@adamlewis.com> wrote:
> Or, since this is only being called from prepare at startup anyway, simpler:
>
> public class InjectorProvider {
>
> private static Injector injector;
> public static synchronized Injector get(Map conf) {
> if (injector == null) {
> injector = Guice.createInjector(
> new DAOModule(conf),
> new S3Module(conf));
> }
>
> return injector;
> }
> }
>
>
>
>
>
> On Wed, Mar 26, 2014 at 9:26 AM, Svend Vanderveken
> <sv...@gmail.com> wrote:
>>
>> > private static Injector injector;
>>
>> or better:
>>
>> private static volatile Injector injector;
>>
>>
>> see http://www.cs.umd.edu/~pugh/java/memoryModel/DoubleCheckedLocking.html
>>
>>
>> On Tue, Mar 25, 2014 at 9:55 PM, Patricio Echagüe <pa...@gmail.com>
>> wrote:
>>>
>>> It's fine. You can synchronize with a static monitor the creation on the
>>> injector per worker. That's how I do it.
>>>
>>> public class InjectorProvider {
>>>
>>> private static Injector injector;
>>>
>>> public static Injector get() {
>>> if (injector == null) {
>>> synchronized (InjectorProvider.class) {
>>> if (injector == null) {
>>> injector = Guice.createInjector(
>>> new DAOModule(),
>>> new S3Module();
>>> }
>>> }
>>> }
>>>
>>> return injector;
>>> }
>>>
>>>
>>> On Tue, Mar 25, 2014 at 6:24 PM, Adam Lewis <ma...@adamlewis.com> wrote:
>>>>
>>>>
>>>>> Doesn't Storm 0.9 have a prepare for the worker?
>>>>
>>>>
>>>> No, I don't think it does, but please point this out if I'm mistaken.  I
>>>> found the right JIRA issue though:
>>>> https://issues.apache.org/jira/browse/STORM-126
>>>>
>>>> Seems like the patch was well along but hasn't seen any recent activity.
>>>
>>>
>>
>

Re: DI with Storm

Posted by Adam Lewis <ma...@adamlewis.com>.
Or, since this is only being called from prepare at startup anyway, simpler:

public class InjectorProvider {

private static Injector injector;
public static synchronized Injector get(Map conf) {
 if (injector == null) {
injector = Guice.createInjector(
 new DAOModule(conf),
new S3Module(conf));
 }

 return injector;
}
}





On Wed, Mar 26, 2014 at 9:26 AM, Svend Vanderveken <
svend.vanderveken@gmail.com> wrote:

> > private static Injector injector;
>
> or better:
>
> private static volatile Injector injector;
>
>
> see http://www.cs.umd.edu/~pugh/java/memoryModel/DoubleCheckedLocking.html
>
>
> On Tue, Mar 25, 2014 at 9:55 PM, Patricio Echagüe <pa...@gmail.com>wrote:
>
>> It's fine. You can synchronize with a static monitor the creation on the
>> injector per worker. That's how I do it.
>>
>> public class InjectorProvider {
>>
>> private static Injector injector;
>>
>> public static Injector get() {
>> if (injector == null) {
>> synchronized (InjectorProvider.class) {
>>  if (injector == null) {
>> injector = Guice.createInjector(
>> new DAOModule(),
>>  new S3Module();
>> }
>> }
>> }
>>
>> return injector;
>> }
>>
>>
>> On Tue, Mar 25, 2014 at 6:24 PM, Adam Lewis <ma...@adamlewis.com> wrote:
>>
>>>
>>> Doesn't Storm 0.9 have a prepare for the worker?
>>>>
>>>
>>> No, I don't think it does, but please point this out if I'm mistaken.  I
>>> found the right JIRA issue though:
>>> https://issues.apache.org/jira/browse/STORM-126
>>>
>>> Seems like the patch was well along but hasn't seen any recent activity.
>>>
>>
>>
>

Re: DI with Storm

Posted by Svend Vanderveken <sv...@gmail.com>.
> private static Injector injector;

or better:

private static volatile Injector injector;


see http://www.cs.umd.edu/~pugh/java/memoryModel/DoubleCheckedLocking.html


On Tue, Mar 25, 2014 at 9:55 PM, Patricio Echagüe <pa...@gmail.com>wrote:

> It's fine. You can synchronize with a static monitor the creation on the
> injector per worker. That's how I do it.
>
> public class InjectorProvider {
>
> private static Injector injector;
>
> public static Injector get() {
> if (injector == null) {
> synchronized (InjectorProvider.class) {
>  if (injector == null) {
> injector = Guice.createInjector(
> new DAOModule(),
>  new S3Module();
> }
> }
> }
>
> return injector;
> }
>
>
> On Tue, Mar 25, 2014 at 6:24 PM, Adam Lewis <ma...@adamlewis.com> wrote:
>
>>
>> Doesn't Storm 0.9 have a prepare for the worker?
>>>
>>
>> No, I don't think it does, but please point this out if I'm mistaken.  I
>> found the right JIRA issue though:
>> https://issues.apache.org/jira/browse/STORM-126
>>
>> Seems like the patch was well along but hasn't seen any recent activity.
>>
>
>

Re: DI with Storm

Posted by Patricio Echagüe <pa...@gmail.com>.
It's fine. You can synchronize with a static monitor the creation on the
injector per worker. That's how I do it.

public class InjectorProvider {

private static Injector injector;

public static Injector get() {
if (injector == null) {
synchronized (InjectorProvider.class) {
 if (injector == null) {
injector = Guice.createInjector(
new DAOModule(),
 new S3Module();
}
}
}

return injector;
}


On Tue, Mar 25, 2014 at 6:24 PM, Adam Lewis <ma...@adamlewis.com> wrote:

>
> Doesn't Storm 0.9 have a prepare for the worker?
>>
>
> No, I don't think it does, but please point this out if I'm mistaken.  I
> found the right JIRA issue though:
> https://issues.apache.org/jira/browse/STORM-126
>
> Seems like the patch was well along but hasn't seen any recent activity.
>

Re: DI with Storm

Posted by Adam Lewis <ma...@adamlewis.com>.
> Doesn't Storm 0.9 have a prepare for the worker?
>

No, I don't think it does, but please point this out if I'm mistaken.  I
found the right JIRA issue though:
https://issues.apache.org/jira/browse/STORM-126

Seems like the patch was well along but hasn't seen any recent activity.

Re: DI with Storm

Posted by Patricio Echagüe <pa...@gmail.com>.
why do you need to do that? Just instantiate the injector on the prepare.
You can make it thread safe and visible to the whole JVM for the worker.

Doesn't Storm 0.9 have a prepare for the worker?


On Tue, Mar 25, 2014 at 10:25 AM, Software Dev <st...@gmail.com>wrote:

> I'm guessing my only other choice is to add everything I need onto the
> Conf map then right?
>
> On Tue, Mar 25, 2014 at 10:14 AM, Adam Lewis <ma...@adamlewis.com> wrote:
> > As I recall there was an open issue for adding all the needed lifecycle
> > hooks for integrating cleanly with something like Guice (worker lifecycle
> > callbacks?) but I can't find it now...maybe STORM-172
> > .
> >
> > In any case,
> > I think there is an issue with not having quite enough
> > exposed
> > from storm to properly integrate things like Guice.
> > Having said that,
> > I'm using
> > Guice now, but it is basically a hack until I can find a better way.
> >
> > The key to my hack is that injectors are not serializable, but modules
> are.
> > I use the singleton pattern with static accessor to hold the injector
> which
> > is created lazily (i.e. after topology components have been deserialized
> and
> > are running in the worker VM).  I instantiate my guice module during
> > topology build and it gets serialized as part of the topo definition.
>  In my
> > storm components, I pass the module to the singleton injector factory
> which
> > creates the injector from the passed module if one doesn't exist in the
> VM,
> > and just returns the existing injector otherwise.  Finally, with an
> injector
> > in hand for the prepare method, I can then either do a
> > injector.getInstance() to create objects (e.g. inside a trident state
> > factory) or injector.injectMembers(this) in a prepare method if my
> topology
> > component has transient @Inject fields for dependencies (since storm
> > controls the object lifecycle and I can't use constructor injection
> there)
> >
> > The biggest problem with this hack is it assumes that all module
> instances
> > are interchangeable (and identical) and the first component to init gets
> to
> > build the injector.  With the right lifecycle hooks, the injector could
> be
> > created in a cleaner way.
> >
> >
> > On Tue, Mar 25, 2014 at 12:45 PM, Software Dev <
> static.void.dev@gmail.com>
> > wrote:
> >>
> >> How would you go about use DI (Google Guice) with a Storm cluster? The
> >> worker nodes themselves will not have access to the Injector instance
> >> after job submission.
> >>
> >> Thanks
>

Re: DI with Storm

Posted by Software Dev <st...@gmail.com>.
I'm guessing my only other choice is to add everything I need onto the
Conf map then right?

On Tue, Mar 25, 2014 at 10:14 AM, Adam Lewis <ma...@adamlewis.com> wrote:
> As I recall there was an open issue for adding all the needed lifecycle
> hooks for integrating cleanly with something like Guice (worker lifecycle
> callbacks?) but I can't find it now...maybe STORM-172
> .
>
> In any case,
> I think there is an issue with not having quite enough
> exposed
> from storm to properly integrate things like Guice.
> Having said that,
> I'm using
> Guice now, but it is basically a hack until I can find a better way.
>
> The key to my hack is that injectors are not serializable, but modules are.
> I use the singleton pattern with static accessor to hold the injector which
> is created lazily (i.e. after topology components have been deserialized and
> are running in the worker VM).  I instantiate my guice module during
> topology build and it gets serialized as part of the topo definition.  In my
> storm components, I pass the module to the singleton injector factory which
> creates the injector from the passed module if one doesn't exist in the VM,
> and just returns the existing injector otherwise.  Finally, with an injector
> in hand for the prepare method, I can then either do a
> injector.getInstance() to create objects (e.g. inside a trident state
> factory) or injector.injectMembers(this) in a prepare method if my topology
> component has transient @Inject fields for dependencies (since storm
> controls the object lifecycle and I can't use constructor injection there)
>
> The biggest problem with this hack is it assumes that all module instances
> are interchangeable (and identical) and the first component to init gets to
> build the injector.  With the right lifecycle hooks, the injector could be
> created in a cleaner way.
>
>
> On Tue, Mar 25, 2014 at 12:45 PM, Software Dev <st...@gmail.com>
> wrote:
>>
>> How would you go about use DI (Google Guice) with a Storm cluster? The
>> worker nodes themselves will not have access to the Injector instance
>> after job submission.
>>
>> Thanks

Re: DI with Storm

Posted by Adam Lewis <ma...@adamlewis.com>.
As I recall there was an open issue for adding all the needed lifecycle
hooks for integrating cleanly with something like Guice (worker lifecycle
callbacks?) but I can't find it now...maybe STORM-172
.

In any case,
I think there is an issue with not having quite enough
exposed
from storm to properly integrate things like Guice.
Having said that,
I'm using
Guice now, but it is basically a hack until I can find a better way.

The key to my hack is that injectors are not serializable, but modules are.
 I use the singleton pattern with static accessor to hold the injector
which is created lazily (i.e. after topology components have been
deserialized and are running in the worker VM).  I instantiate my guice
module during topology build and it gets serialized as part of the topo
definition.  In my storm components, I pass the module to the singleton
injector factory which creates the injector from the passed module if one
doesn't exist in the VM, and just returns the existing injector otherwise.
 Finally, with an injector in hand for the prepare method, I can then
either do a injector.getInstance() to create objects (e.g. inside a trident
state factory) or injector.injectMembers(this) in a prepare method if my
topology component has transient @Inject fields for dependencies (since
storm controls the object lifecycle and I can't use constructor injection
there)

The biggest problem with this hack is it assumes that all module instances
are interchangeable (and identical) and the first component to init gets to
build the injector.  With the right lifecycle hooks, the injector could be
created in a cleaner way.


On Tue, Mar 25, 2014 at 12:45 PM, Software Dev <st...@gmail.com>
wrote:
>
> How would you go about use DI (Google Guice) with a Storm cluster? The
> worker nodes themselves will not have access to the Injector instance
> after job submission.
>
> Thanks