You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Aleksandar Stojadinovic <al...@nissatech.com> on 2014/06/10 15:11:26 UTC
Storm initialization on startup
Hello,
I have a topology with two bolt types (among others) which I am not certain about how will work. One of the bolts has a custom made stateful component The other bolt type, the "configurator" configures that component under certain conditions and on input from a spout, prepares a configuration, and communicates with a database in the process. That works fine, but the point that bothers me is that I have to configure my stateful component on start-up with some values in the database, but I want to keep the logic in that bolt clear, without database access. So my solution is to access the database and read the configuration on the "configurator" startup and emit it. But what if the stateful bolt is not up yet? Will the messages get persisted (no sign of something like that in the Storm documents), or the prepare command is executed after the topology is set up?
In short, how to handle this situation?
Best regards
Re: Storm initialization on startup
Posted by Michael Rose <mi...@fullcontact.com>.
We tend to push that kind of initialization logic into the prepare() method
of our bolts. Most times, you only need one bolt to do it, so you can push
all of your initialization logic into a different class and guard the init
with double-check locks. e.g.:
@Override
public void prepare(Map config, TopologyContext topologyContext,
OutputCollector outputCollector) {
this.statefulService = StatefulService.getInstance(config);
}
...
class StatefulService {
private static volatile StatefulService INSTANCE = null;
public StatefulService getInstance(Map config) {
if (INSTANCE == null) {
synchronized(StatefulService.class) {
if (INSTANCE == null) {
INSTANCE = new StatefulService(config); // do some init
}
}
}
return INSTANCE;
}
}
Other patterns are pretty similar. Guard the init code, call it in all your
bolt's prepare methods. The first executor to start per JVM wins.
Storm has some gotchas (bolts are serialized, so do your init in
prepare()), but in general things that work in a normal Java application
will end up working in Storm.
Michael
Michael Rose (@Xorlev <https://twitter.com/xorlev>)
Senior Platform Engineer, FullContact <http://www.fullcontact.com/>
michael@fullcontact.com
On Wed, Jun 11, 2014 at 1:23 AM, Aleksandar Stojadinovic <
aleksandar.stojadinovic@nissatech.com> wrote:
> Hi,
>
> Thanks for your answer. It sounds like a good idea. I can put the
> parameters (read from database) in the configuration map in my main
> function and then read them in the configurator. Not too ideal, but a whole
> lot cleaner then any other I thought of. I'll try something like that. Yet
> I'm still open for new answers because I guess I'm not the first and only
> with this problem. There can certainly be a pattern derived.
>
> Best regards,
> Aleksandar
>
>
>
> ------------------------------
> From: qincuidhu@gmail.com
> Subject: Re: Storm initialization on startup
> Date: Tue, 10 Jun 2014 20:07:40 +0200
> To: user@storm.incubator.apache.org
>
>
> Hi Aleksandar,
> If I understand you correctly, you want to configure your stateful
> component. You could try to use the Config class in Storm, something like
> "Config.put(key,value)". Then in the prepare function of bolt you can get
> the parameters you set up,like conf.get(key). Hope it will help you.
>
> Best,
> Cui Qin
>
> Sent from my iPhone
>
> On 10 Jun 2014, at 15:11, Aleksandar Stojadinovic <
> aleksandar.stojadinovic@nissatech.com> wrote:
>
> Hello,
>
> I have a topology with two bolt types (among others) which I am not
> certain about how will work. One of the bolts has a custom made stateful
> component The other bolt type, the "configurator" configures that
> component under certain conditions and on input from a spout, prepares a
> configuration, and communicates with a database in the process. That works
> fine, but the point that bothers me is that I have to configure my stateful
> component on start-up with some values in the database, but I want to keep
> the logic in that bolt clear, without database access. So my solution is to
> access the database and read the configuration on the "configurator"
> startup and emit it. But what if the stateful bolt is not up yet? Will the
> messages get persisted (no sign of something like that in the Storm
> documents), or the prepare command is executed after the topology is set up?
>
> In short, how to handle this situation?
>
> *Best regards*
>
>
RE: Storm initialization on startup
Posted by Aleksandar Stojadinovic <al...@nissatech.com>.
Hi,
Thanks for your answer. It sounds like a good idea. I can put the parameters (read from database) in the configuration map in my main function and then read them in the configurator. Not too ideal, but a whole lot cleaner then any other I thought of. I'll try something like that. Yet I'm still open for new answers because I guess I'm not the first and only with this problem. There can certainly be a pattern derived.
Best regards,Aleksandar
From: qincuidhu@gmail.com
Subject: Re: Storm initialization on startup
Date: Tue, 10 Jun 2014 20:07:40 +0200
To: user@storm.incubator.apache.org
Hi Aleksandar, If I understand you correctly, you want to configure your stateful component. You could try to use the Config class in Storm, something like "Config.put(key,value)". Then in the prepare function of bolt you can get the parameters you set up,like conf.get(key). Hope it will help you. Best,Cui Qin
Sent from my iPhone
On 10 Jun 2014, at 15:11, Aleksandar Stojadinovic <al...@nissatech.com> wrote:
Hello,
I have a topology with two bolt types (among others) which I am not certain about how will work. One of the bolts has a custom made stateful component The other bolt type, the "configurator" configures that component under certain conditions and on input from a spout, prepares a configuration, and communicates with a database in the process. That works fine, but the point that bothers me is that I have to configure my stateful component on start-up with some values in the database, but I want to keep the logic in that bolt clear, without database access. So my solution is to access the database and read the configuration on the "configurator" startup and emit it. But what if the stateful bolt is not up yet? Will the messages get persisted (no sign of something like that in the Storm documents), or the prepare command is executed after the topology is set up?
In short, how to handle this situation?
Best regards
Re: Storm initialization on startup
Posted by qi...@gmail.com.
Hi Aleksandar,
If I understand you correctly, you want to configure your stateful component. You could try to use the Config class in Storm, something like "Config.put(key,value)". Then in the prepare function of bolt you can get the parameters you set up,like conf.get(key). Hope it will help you.
Best,
Cui Qin
Sent from my iPhone
> On 10 Jun 2014, at 15:11, Aleksandar Stojadinovic <al...@nissatech.com> wrote:
>
> Hello,
>
> I have a topology with two bolt types (among others) which I am not certain about how will work. One of the bolts has a custom made stateful component The other bolt type, the "configurator" configures that component under certain conditions and on input from a spout, prepares a configuration, and communicates with a database in the process. That works fine, but the point that bothers me is that I have to configure my stateful component on start-up with some values in the database, but I want to keep the logic in that bolt clear, without database access. So my solution is to access the database and read the configuration on the "configurator" startup and emit it. But what if the stateful bolt is not up yet? Will the messages get persisted (no sign of something like that in the Storm documents), or the prepare command is executed after the topology is set up?
>
> In short, how to handle this situation?
>
> Best regards