You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by ? ?? <wa...@outlook.com> on 2017/08/04 05:45:01 UTC
Executor Change
Hi,
It is observed that the executor of a task changes to a different worker on a different supervisor. And the computation, which is similar to wordcount, stops and starts over. After several tests, I find that this is caused by the lost of the task state during executor change. So I wonder:
1. Why does the change happen?
2. When the change happen, does the executor use a new component(bolt) instance? How does the instance come into being? By constructor or deserialization?
3. In this case, how to persist the task state?
________________________________
wangchunchao@outlook.com
Re: Re: Executor Change
Posted by Bobby Evans <ev...@yahoo-inc.com>.
The close method is best effort. It is not guaranteed to be called, and in fact in storm we only give the entire worker a second to shut down cleanly before we start to get mean about it (aka kill -9 and Runtime.halt). If you want the state saved you will then need to do it periodically yourself, possibly on a background thread in your bolt.
- Bobby
On Saturday, August 5, 2017, 4:25:46 AM CDT, 王 纯超 <wa...@outlook.com> wrote:
Thanks, Bobby. Stateful bolt requires reliable processing guarantee, but in my case, it is abandoned for my specific requirement. Thus I tried to save the state myself in the bolt's close method, which did not work. The state was not saved. So how to store the state and when is close method called? wangchunchao@outlook.com
From: Bobby EvansDate: 2017-08-04 21:47To: user@storm.apache.orgCC: zhangxl_bdsSubject: Re: Executor Change#yiv2620919326 div.yiv2620919326FoxDIV_20170805165101711 {font-size:10.5pt;}#yiv2620919326 div.yiv2620919326FoxDIV_20170805165101711 body {line-height:1.5;}#yiv2620919326 div.yiv2620919326FoxDIV_20170805165101711 body {font-size:10.5pt;color:rgb(0, 0, 0);line-height:1.5;}Storm is by default stateless. If you have state to store it is up to you to store that state in an external system, in a way that meets your topologies requirements. At least one vs at most once. In newer versions of storm the concept of stateful bolts was added.
http://storm.apache.org/releases/1.0.3/State-checkpointing.html
is the documentation for that feature. It still saves the state in an external system (that you can pick) but also helps to fix some of the complexity associated with checkpointing and restoring state in line with replay of failed messages.
- Bobby
On Friday, August 4, 2017, 12:45:17 AM CDT, ? ?? <wa...@outlook.com> wrote:
Hi,
It is observed that the executor of a task changes to a different worker on a different supervisor. And the computation, which is similar to wordcount, stops and starts over. After several tests, I find that this is caused by the lost of the task state during executor change. So I wonder:
- Why does the change happen?
- When the change happen, does the executor use a new component(bolt) instance? How does the instance come into being? By constructor or deserialization?
- In this case, how to persist the task state?
wangchunchao@outlook.com
Re: Re: Executor Change
Posted by 王 纯超 <wa...@outlook.com>.
Thanks, Bobby. Stateful bolt requires reliable processing guarantee, but in my case, it is abandoned for my specific requirement. Thus I tried to save the state myself in the bolt's close method, which did not work. The state was not saved. So how to store the state and when is close method called?
________________________________
wangchunchao@outlook.com
From: Bobby Evans<ma...@yahoo-inc.com>
Date: 2017-08-04 21:47
To: user@storm.apache.org<ma...@storm.apache.org>
CC: zhangxl_bds<ma...@si-tech.com.cn>
Subject: Re: Executor Change
Storm is by default stateless. If you have state to store it is up to you to store that state in an external system, in a way that meets your topologies requirements. At least one vs at most once. In newer versions of storm the concept of stateful bolts was added.
http://storm.apache.org/releases/1.0.3/State-checkpointing.html
is the documentation for that feature. It still saves the state in an external system (that you can pick) but also helps to fix some of the complexity associated with checkpointing and restoring state in line with replay of failed messages.
- Bobby
On Friday, August 4, 2017, 12:45:17 AM CDT, ? ?? <wa...@outlook.com> wrote:
Hi,
It is observed that the executor of a task changes to a different worker on a different supervisor. And the computation, which is similar to wordcount, stops and starts over. After several tests, I find that this is caused by the lost of the task state during executor change. So I wonder:
1. Why does the change happen?
2. When the change happen, does the executor use a new component(bolt) instance? How does the instance come into being? By constructor or deserialization?
3. In this case, how to persist the task state?
________________________________
wangchunchao@outlook.com
Re: Executor Change
Posted by Bobby Evans <ev...@yahoo-inc.com>.
Storm is by default stateless. If you have state to store it is up to you to store that state in an external system, in a way that meets your topologies requirements. At least one vs at most once. In newer versions of storm the concept of stateful bolts was added.
http://storm.apache.org/releases/1.0.3/State-checkpointing.html
is the documentation for that feature. It still saves the state in an external system (that you can pick) but also helps to fix some of the complexity associated with checkpointing and restoring state in line with replay of failed messages.
- Bobby
On Friday, August 4, 2017, 12:45:17 AM CDT, ? ?? <wa...@outlook.com> wrote:
#yiv5765339309 body {line-height:1.5;}#yiv5765339309 body {font-size:10.5pt;font-family:'Microsoft YaHei UI';color:rgb(0, 0, 0);line-height:1.5;}Hi,
It is observed that the executor of a task changes to a different worker on a different supervisor. And the computation, which is similar to wordcount, stops and starts over. After several tests, I find that this is caused by the lost of the task state during executor change. So I wonder:
- Why does the change happen?
- When the change happen, does the executor use a new component(bolt) instance? How does the instance come into being? By constructor or deserialization?
- In this case, how to persist the task state?
wangchunchao@outlook.com