You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Arun Mahadevan <ar...@apache.org> on 2017/02/07 06:10:43 UTC
Re: implmement state management with Apache Ignite
> #1 when a worker is died or killed by manually, storm framework will restart this worker, is there a ID which doesn't change for the new worker and the died worker? if there is, how to get it?
Task Id does not change. The worker can be restarted in the same or different host and the tasks would be reassigned.
> #2 when a worker is died, is there a case storm framework can't start a new worker? In this case, how about the grouped message handled by this worker, it will auto take over by other workers?
As long as the state is per task it can be handled by other worker.
> #3 suppose we have started a topology with 3 workers, rebalance topology can change the worker numbers?
Rebalance can change the number of workers, but the overall number of tasks remains the same. Only that the task assignment to workers could change.
Not sure if you are building something different, but storm already has basic support for stateful bolts (see https://github.com/apache/storm/blob/master/docs/State-checkpointing.md).
Thanks,
Arun
From: "shawn.du" <sh...@neulion.com.cn>
Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
Date: Tuesday, February 7, 2017 at 11:25 AM
To: user <us...@storm.apache.org>
Subject: implmement state management with Apache Ignite
Hi community,
Recently we are trying implement state management using apache ignite.
Before do the real work, I have several questions, please help.
#1 when a worker is died or killed by manually, storm framework will restart this worker, is there a ID which doesn't change for the new worker and the died worker? if there is, how to get it?
#2 when a worker is died, is there a case storm framework can't start a new worker? In this case, how about the grouped message handled by this worker, it will auto take over by other workers?
#3 suppose we have started a topology with 3 workers, rebalance topology can change the worker numbers?
Thanks
Shawn
Re: implmement state management with Apache Ignite
Posted by Arun Mahadevan <ar...@apache.org>.
> every task have its own state which gets lost on rebalance operation
The state is not lost as long as the number of tasks remain the same. The state is typically backed by persistent store (like redis).
> 1- How this state correlates to <key,val> pair in these stateful tasks mentioned in the links. Is it separate for every task ?
Each state instance belongs to a “namespace”, which is currently mapped to the task-id. So each task gets a separate state (e.g. a separate k-v map). But all of it could be stored in the same store underneath (e.g. different redis maps in the same redis cluster).
2- Whats is the impact fo rebalance operation on these bolts.
As long as the number of tasks does not change there is no effect. The tasks may get re-assigned to different workers, but would still be mapped to the same state since its task id does not change.
- Arun
From: anshu shukla <an...@gmail.com>
Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
Date: Tuesday, February 7, 2017 at 4:28 PM
To: "user@storm.apache.org" <us...@storm.apache.org>
Subject: Re: implmement state management with Apache Ignite
Hey ,
Nice discussion, but as per my knowledge, every task have its own state which gets lost on rebalance operation(this is the reason why tasks can not be increased at run time).
Mentioned @ https://github.com/apache/storm/pull/1874
So I have 2 questions -
1- How this state correlates to <key,val> pair in these stateful tasks mentioned in the links. Is it separate for every task ?
2- Whats is the impact fo rebalance operation on these bolts.
On Tue, Feb 7, 2017 at 11:40 AM, Arun Mahadevan <ar...@apache.org> wrote:
> #1 when a worker is died or killed by manually, storm framework will restart this worker, is there a ID which doesn't change for the new worker and the died worker? if there is, how to get it?
Task Id does not change. The worker can be restarted in the same or different host and the tasks would be reassigned.
> #2 when a worker is died, is there a case storm framework can't start a new worker? In this case, how about the grouped message handled by this worker, it will auto take over by other workers?
As long as the state is per task it can be handled by other worker.
> #3 suppose we have started a topology with 3 workers, rebalance topology can change the worker numbers?
Rebalance can change the number of workers, but the overall number of tasks remains the same. Only that the task assignment to workers could change.
Not sure if you are building something different, but storm already has basic support for stateful bolts (see https://github.com/apache/storm/blob/master/docs/State-checkpointing.md).
Thanks,
Arun
From: "shawn.du" <sh...@neulion.com.cn>
Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
Date: Tuesday, February 7, 2017 at 11:25 AM
To: user <us...@storm.apache.org>
Subject: implmement state management with Apache Ignite
Hi community,
Recently we are trying implement state management using apache ignite.
Before do the real work, I have several questions, please help.
#1 when a worker is died or killed by manually, storm framework will restart this worker, is there a ID which doesn't change for the new worker and the died worker? if there is, how to get it?
#2 when a worker is died, is there a case storm framework can't start a new worker? In this case, how about the grouped message handled by this worker, it will auto take over by other workers?
#3 suppose we have started a topology with 3 workers, rebalance topology can change the worker numbers?
Thanks
Shawn
--
Thanks & Regards,
Anshu Shukla
Re: implmement state management with Apache Ignite
Posted by anshu shukla <an...@gmail.com>.
Hey ,
Nice discussion, but as per my knowledge, every *task have its own state*
which gets lost on rebalance operation(this is the reason why tasks can not
be increased at run time).
Mentioned @ https://github.com/apache/storm/pull/1874
So I have 2 questions -
*1- How this state correlates to <key,val> pair in these stateful tasks
mentioned in the links. *Is it separate for every task ?
*2- Whats is the impact fo rebalance operation on these bolts.*
On Tue, Feb 7, 2017 at 11:40 AM, Arun Mahadevan <ar...@apache.org> wrote:
> > #1 when a worker is died or killed by manually, storm framework will
> restart this worker, is there a ID which doesn't change for the new worker
> and the died worker? if there is, how to get it?
>
> Task Id does not change. The worker can be restarted in the same or
> different host and the tasks would be reassigned.
>
>
>
> > #2 when a worker is died, is there a case storm framework can't start a
> new worker? In this case, how about the grouped message handled by this
> worker, it will auto take over by other workers?
>
> As long as the state is per task it can be handled by other worker.
>
>
>
> > #3 suppose we have started a topology with 3 workers, rebalance topology
> can change the worker numbers?
>
> Rebalance can change the number of workers, but the overall number of
> tasks remains the same. Only that the task assignment to workers could
> change.
>
>
>
> Not sure if you are building something different, but storm already has
> basic support for stateful bolts (see https://github.com/apache/
> storm/blob/master/docs/State-checkpointing.md).
>
> Thanks,
>
> Arun
>
>
>
>
>
> *From: *"shawn.du" <sh...@neulion.com.cn>
> *Reply-To: *"user@storm.apache.org" <us...@storm.apache.org>
> *Date: *Tuesday, February 7, 2017 at 11:25 AM
> *To: *user <us...@storm.apache.org>
> *Subject: *implmement state management with Apache Ignite
>
>
>
> Hi community,
>
>
>
> Recently we are trying implement state management using apache ignite.
>
> Before do the real work, I have several questions, please help.
>
>
>
> #1 when a worker is died or killed by manually, storm framework will
> restart this worker, is there a ID which doesn't change for the new worker
> and the died worker? if there is, how to get it?
>
> #2 when a worker is died, is there a case storm framework can't start a
> new worker? In this case, how about the grouped message handled by this
> worker, it will auto take over by other workers?
>
> #3 suppose we have started a topology with 3 workers, rebalance topology
> can change the worker numbers?
>
>
>
> Thanks
>
> Shawn
>
>
>
--
Thanks & Regards,
Anshu Shukla
Re: implmement state management with Apache Ignite
Posted by Arun Mahadevan <ar...@apache.org>.
Hi Shawn,
> After study, We will implement IgniteKeyValueState and IgniteStateProvider.
Sure, it will be great if you can implement and contribute it to the storm community.
>Could storm add support for ignite?
>porting RedisKeyValueState to IgniteKeyValueState is rather easy.
Once you have the implementation, you can raise a pull request and it will get reviewed and merged to the storm codebase. In case you are not sure about the process, check this out - https://github.com/apache/storm/blob/master/DEVELOPER.md
Thanks,
Arun
From: "shawn.du" <sh...@neulion.com.cn>
Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
Date: Thursday, February 16, 2017 at 7:30 AM
To: user <us...@storm.apache.org>
Cc: user <us...@storm.apache.org>
Subject: Re: implmement state management with Apache Ignite
Hi,
After study, We will implement IgniteKeyValueState and IgniteStateProvider.
I am a ignite user, I think Ignite has more wonderful features than redis.
Could storm add support for ignite?
porting RedisKeyValueState to IgniteKeyValueState is rather easy.
Thanks
Shawn
On 02/7/2017 14:39，shawn.du<sh...@neulion.com.cn> wrote：
Hi Arun,
Thanks your reply. I have read the document your mentioned.
For current storm 1.0.2 only provide a simple KeyValueState interface, We are still considering whether it can fulfill our requirements.
Thanks
Shawn
On 02/7/2017 14:08，Arun Mahadevan<ar...@apache.org> wrote：
> #1 when a worker is died or killed by manually, storm framework will restart this worker, is there a ID which doesn't change for the new worker and the died worker? if there is, how to get it?
Task Id does not change. The worker can be restarted in the same or different host and the tasks would be reassigned.
> #2 when a worker is died, is there a case storm framework can't start a new worker? In this case, how about the grouped message handled by this worker, it will auto take over by other workers?
As long as the state is per task it can be handled by other worker.
> #3 suppose we have started a topology with 3 workers, rebalance topology can change the worker numbers?
Rebalance can change the number of workers, but the overall number of tasks remains the same. Only that the task assignment to workers could change.
Not sure if you are building something different, but storm already has basic support for stateful bolts (see https://github.com/apache/storm/blob/master/docs/State-checkpointing.md).
Thanks,
Arun
From: "shawn.du" <sh...@neulion.com.cn>
Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
Date: Tuesday, February 7, 2017 at 11:25 AM
To: user <us...@storm.apache.org>
Subject: implmement state management with Apache Ignite
Hi community,
Recently we are trying implement state management using apache ignite.
Before do the real work, I have several questions, please help.
#1 when a worker is died or killed by manually, storm framework will restart this worker, is there a ID which doesn't change for the new worker and the died worker? if there is, how to get it?
#2 when a worker is died, is there a case storm framework can't start a new worker? In this case, how about the grouped message handled by this worker, it will auto take over by other workers?
#3 suppose we have started a topology with 3 workers, rebalance topology can change the worker numbers?
Thanks
Shawn