How to implement static initialization in storm for several topolgies

[Dzmitry Viarzhbitski <dz...@gmail.com>]

Let's I assume that I have several topologies in the same jar which are
using some cache.
I need to init cache (cache subscribed to events from different system and
can be reloade) and release cache when the worker will be switched of. Is
it possible to share static cache from multiple topologies and how to shut
down it (in case if I kill one topolgy other still use the cache)? What
should be the event release the cache (worker shutdown)?

Thank you.

Re: How to implement static initialization in storm for several topolgies

[Gmail <dz...@gmail.com>]

Thank you so much.

Sent from my iPhone

> On May 15, 2015, at 8:41 PM, Nathan Leung <nc...@gmail.com> wrote:
> 
> 1) Bolt prepare is good
> 
> 2) If you need to close it, it's best to deactivate the topology first, and then on deactivation have spout broadcast a message to all bolts indicating that shutdown is in progress
> 
> 3) I would still just do lazy initialized singleton; that way whichever bolts need access to the cache can check and then create the connection if necessary.  Once it's created you're good.  I guess you have to check for initialization every time you access the cache but that's cheap compared to talking to another server.
> 
>> On Fri, May 15, 2015 at 7:09 PM, Dzmitry Viarzhbitski <dz...@gmail.com> wrote:
>> 
>> I have cache synchronized with other machines by TCP/IP but I don't know:
>> 
>> 1) where to init it (in bolt prepare? Is it possible to do it for worker?)  
>> 2) and how to dispose it (caches uses a connection to remote http server, it's better to close it after all topologies will be deactivated).
>> 
>> I thinking about creating separate init topology. Is the a normal way to achieve this?
>> How can I guarantee that this bolt will be created on each worker node?  
>> 
>> Thank you.  
>> 
>>> On 15 May 2015 at 15:52, Nathan Leung <nc...@gmail.com> wrote:
>>> Sorry misread original email. If you need to share across topologies then you should consider something like memcache.
>>> 
>>>> On May 15, 2015 6:52 PM, wrote:
>>>> Lazy initialized singleton. Let the os release the memory when you're done :).
>>>> 
>>>>> On May 15, 2015 6:44 PM, "Dzmitry Viarzhbitski" <dz...@gmail.com> wrote:
>>>>> Let's I assume that I have several topologies in the same jar which are using some cache.
>>>>> I need to init cache (cache subscribed to events from different system and can be reloade) and release cache when the worker will be switched of. Is it possible to share static cache from multiple topologies and how to shut down it (in case if I kill one topolgy other still use the cache)? What should be the event release the cache (worker shutdown)?
>>>>> 
>>>>> Thank you.    
> 

Re: How to implement static initialization in storm for several topolgies

[Nathan Leung <nc...@gmail.com>]

1) Bolt prepare is good

2) If you need to close it, it's best to deactivate the topology first, and
then on deactivation have spout broadcast a message to all bolts indicating
that shutdown is in progress

3) I would still just do lazy initialized singleton; that way whichever
bolts need access to the cache can check and then create the connection if
necessary.  Once it's created you're good.  I guess you have to check for
initialization every time you access the cache but that's cheap compared to
talking to another server.

On Fri, May 15, 2015 at 7:09 PM, Dzmitry Viarzhbitski <
dzmitry.viarzhbitski@gmail.com> wrote:

>
> I have cache synchronized with other machines by TCP/IP but I don't know:
>
> 1) where to init it (in bolt prepare? Is it possible to do it for
> worker?)
> 2) and how to dispose it (caches uses a connection to remote http server,
> it's better to close it after all topologies will be deactivated).
>
> I thinking about creating separate init topology. Is the a normal way to
> achieve this?
> How can I guarantee that this bolt will be created on each worker node?
>
> Thank you.
>
> On 15 May 2015 at 15:52, Nathan Leung <nc...@gmail.com> wrote:
>
>> Sorry misread original email. If you need to share across topologies then
>> you should consider something like memcache.
>> On May 15, 2015 6:52 PM, wrote:
>>
>>> Lazy initialized singleton. Let the os release the memory when you're
>>> done :).
>>> On May 15, 2015 6:44 PM, "Dzmitry Viarzhbitski" <
>>> dzmitry.viarzhbitski@gmail.com> wrote:
>>>
>>>> Let's I assume that I have several topologies in the same jar which are
>>>> using some cache.
>>>> I need to init cache (cache subscribed to events from different system
>>>> and can be reloade) and release cache when the worker will be switched of.
>>>> Is it possible to share static cache from multiple topologies and how to
>>>> shut down it (in case if I kill one topolgy other still use the cache)?
>>>> What should be the event release the cache (worker shutdown)?
>>>>
>>>> Thank you.
>>>>
>>>
>

Re: How to implement static initialization in storm for several topolgies

[Dzmitry Viarzhbitski <dz...@gmail.com>]

I have cache synchronized with other machines by TCP/IP but I don't know:

1) where to init it (in bolt prepare? Is it possible to do it for worker?)
2) and how to dispose it (caches uses a connection to remote http server,
it's better to close it after all topologies will be deactivated).

I thinking about creating separate init topology. Is the a normal way to
achieve this?
How can I guarantee that this bolt will be created on each worker node?

Thank you.

On 15 May 2015 at 15:52, Nathan Leung <nc...@gmail.com> wrote:

> Sorry misread original email. If you need to share across topologies then
> you should consider something like memcache.
> On May 15, 2015 6:52 PM, wrote:
>
>> Lazy initialized singleton. Let the os release the memory when you're
>> done :).
>> On May 15, 2015 6:44 PM, "Dzmitry Viarzhbitski" <
>> dzmitry.viarzhbitski@gmail.com> wrote:
>>
>>> Let's I assume that I have several topologies in the same jar which are
>>> using some cache.
>>> I need to init cache (cache subscribed to events from different system
>>> and can be reloade) and release cache when the worker will be switched of.
>>> Is it possible to share static cache from multiple topologies and how to
>>> shut down it (in case if I kill one topolgy other still use the cache)?
>>> What should be the event release the cache (worker shutdown)?
>>>
>>> Thank you.
>>>
>>

Re: How to implement static initialization in storm for several topolgies

[Nathan Leung <nc...@gmail.com>]

Sorry misread original email. If you need to share across topologies then
you should consider something like memcache.
On May 15, 2015 6:52 PM, wrote:

> Lazy initialized singleton. Let the os release the memory when you're done
> :).
> On May 15, 2015 6:44 PM, "Dzmitry Viarzhbitski" <
> dzmitry.viarzhbitski@gmail.com> wrote:
>
>> Let's I assume that I have several topologies in the same jar which are
>> using some cache.
>> I need to init cache (cache subscribed to events from different system
>> and can be reloade) and release cache when the worker will be switched of.
>> Is it possible to share static cache from multiple topologies and how to
>> shut down it (in case if I kill one topolgy other still use the cache)?
>> What should be the event release the cache (worker shutdown)?
>>
>> Thank you.
>>
>

Re: How to implement static initialization in storm for several topolgies

[Nathan Leung <nc...@gmail.com>]

Lazy initialized singleton. Let the os release the memory when you're done
:).
On May 15, 2015 6:44 PM, "Dzmitry Viarzhbitski" <
dzmitry.viarzhbitski@gmail.com> wrote:

> Let's I assume that I have several topologies in the same jar which are
> using some cache.
> I need to init cache (cache subscribed to events from different system and
> can be reloade) and release cache when the worker will be switched of. Is
> it possible to share static cache from multiple topologies and how to shut
> down it (in case if I kill one topolgy other still use the cache)? What
> should be the event release the cache (worker shutdown)?
>
> Thank you.
>