You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by 王立 <wa...@gmail.com> on 2015/08/14 08:59:35 UTC
Should spouts and bolts be stateless to guarantee fault tolerance?
Hi all,
I have a question regarding to the fault tolerance mechanism in storm.
I believe the key idea behind fault tolerance mechanism is the statelessness of Nimbus and Supervisors, i.e., storing the state on zookeepers, such that the failure on Nimbus and Supervisors does not result in state loss and they can be restarted as nothing happened.
My question is: does this means the user has to ensure their spouts and bolts to be stateless, in order to guarantee fault tolerance?
Any comment will be highly appreciated! Thanks.
Regards,
Li
Re: Should spouts and bolts be stateless to guarantee fault
tolerance?
Posted by Bobby Evans <ev...@yahoo-inc.com.INVALID>.
To truly guarantee exact recovery your code needs to be stateless, but in most cases with a lambda architecture, which is really what storm was initially designed for, you can deal with keeping some or all of the data only in memory. For example if I am doing a rolling window aggregation and I get a failure the worst case is that the currently active windows will be missing some data. If you have a batch job to correct the error later on you may not need to worry about it.
- Bobby
On Friday, August 14, 2015 8:27 AM, 임정택 <ka...@gmail.com> wrote:
Hi.
First of all, you may want to read "Guaranteeing Message Processing"
<http://storm.apache.org/documentation/Guaranteeing-message-processing.html> to
see how it works.
In short, Spout and Bolt should also be stateless since worker could be
killed and restarted. (at any time)
Also data source for Spout is responsible for replaying the messages when
Spout is killed and restarted.
If data source cannot support it, your spout is responsible for. One way to
support it is storing processed offset to external system.
Hope this helps.
Regards,
Jungtaek Lim (HeartSaVioR)
2015-08-14 15:59 GMT+09:00 王立 <wa...@gmail.com>:
> Hi all,
>
> I have a question regarding to the fault tolerance mechanism in storm.
>
> I believe the key idea behind fault tolerance mechanism is the
> statelessness of Nimbus and Supervisors, i.e., storing the state on
> zookeepers, such that the failure on Nimbus and Supervisors does not result
> in state loss and they can be restarted as nothing happened.
>
> My question is: does this means the user has to ensure their spouts and
> bolts to be stateless, in order to guarantee fault tolerance?
>
> Any comment will be highly appreciated! Thanks.
>
> Regards,
> Li
>
>
--
Name : 임 정택
Blog : http://www.heartsavior.net / http://dev.heartsavior.net
Twitter : http://twitter.com/heartsavior
LinkedIn : http://www.linkedin.com/in/heartsavior
Re: Should spouts and bolts be stateless to guarantee fault tolerance?
Posted by 임정택 <ka...@gmail.com>.
Hi.
First of all, you may want to read "Guaranteeing Message Processing"
<http://storm.apache.org/documentation/Guaranteeing-message-processing.html> to
see how it works.
In short, Spout and Bolt should also be stateless since worker could be
killed and restarted. (at any time)
Also data source for Spout is responsible for replaying the messages when
Spout is killed and restarted.
If data source cannot support it, your spout is responsible for. One way to
support it is storing processed offset to external system.
Hope this helps.
Regards,
Jungtaek Lim (HeartSaVioR)
2015-08-14 15:59 GMT+09:00 王立 <wa...@gmail.com>:
> Hi all,
>
> I have a question regarding to the fault tolerance mechanism in storm.
>
> I believe the key idea behind fault tolerance mechanism is the
> statelessness of Nimbus and Supervisors, i.e., storing the state on
> zookeepers, such that the failure on Nimbus and Supervisors does not result
> in state loss and they can be restarted as nothing happened.
>
> My question is: does this means the user has to ensure their spouts and
> bolts to be stateless, in order to guarantee fault tolerance?
>
> Any comment will be highly appreciated! Thanks.
>
> Regards,
> Li
>
>
--
Name : 임 정택
Blog : http://www.heartsavior.net / http://dev.heartsavior.net
Twitter : http://twitter.com/heartsavior
LinkedIn : http://www.linkedin.com/in/heartsavior
Re: Should spouts and bolts be stateless to guarantee fault tolerance?
Posted by 王立 <wa...@gmail.com>.
Sorry for this abrupt question. I should have learnt about Trident before asking.
> On Aug 14, 2015, at 14:59, 王立 <wa...@gmail.com> wrote:
>
> Hi all,
>
> I have a question regarding to the fault tolerance mechanism in storm.
>
> I believe the key idea behind fault tolerance mechanism is the statelessness of Nimbus and Supervisors, i.e., storing the state on zookeepers, such that the failure on Nimbus and Supervisors does not result in state loss and they can be restarted as nothing happened.
>
> My question is: does this means the user has to ensure their spouts and bolts to be stateless, in order to guarantee fault tolerance?
>
> Any comment will be highly appreciated! Thanks.
>
> Regards,
> Li
>