You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stratos.apache.org by Reka Thirunavukkarasu <re...@wso2.com> on 2014/01/09 14:00:43 UTC
Graceful shutdown for the instances when scale down
Hi
This is to discuss on $subject. Why stratos need to support graceful
shutdown is that when an instance wants to perform any cleaning up task
before the actual termination, then we can add a termination hook in the
stratos and let the instances to clean the data before the termination.
Here is what i suggest as the propose approach to implement this:
- when Autoscaler takes the decision to terminate an instance, it can call
SM to notify the instances via instance-notifier.
- SM send the cleanup notification the topic
- Member will receive the event and check the member filter to perform the
cleanup task
- Member will send the ReadyToShutDown event to instance-status topic once
it is done with the cleaning up
- CC will receive the event and perform the actual termination of the
member.
FYI: Graceful shutdown can be performed only when scaling down the
instances or unsubscribe. But autoscaler can't call the SM to notify in
other cases like instance crashed, agent crashed and application crashed,
since the instance won't be in a good state to clean up the data. But
later, if we specifically identify application crashed, then we can do
cleaning up for application crashed case.
Thanks,
Reka
--
Reka Thirunavukkarasu
Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007
Re: Graceful shutdown for the instances when scale down
Posted by Reka Thirunavukkarasu <re...@wso2.com>.
+1. IN_MAINTENANCE makes more sense in this case..
Thanks,
Reka
On Wed, Feb 19, 2014 at 8:52 AM, Nirmal Fernando <ni...@gmail.com>wrote:
> Agree for IN_MAINTENANCE state.
> Thanks.
>
>
> On Wed, Feb 19, 2014 at 8:24 AM, Afkham Azeez <af...@gmail.com> wrote:
>
>> Yes, this is a necessary step. Servers that support maintenance mode
>> generally have this state called IN_MAINTENANCE.
>> On Feb 17, 2014 6:46 PM, "Reka Thirunavukkarasu" <re...@wso2.com> wrote:
>>
>>> Hi all,
>>>
>>> In the graceful shutdown, we keep the member as active until it
>>> receives ReadyToShutdown event. In between these state changes, member will
>>> become unavailable. So the LB needs to aware of that, in order to avoid
>>> sending requests to a member which is going to graceful shutdown. I think
>>> that if we introduce another state say Maintenance in the Topology, before
>>> the actual shut down of the application inside the member takes place, then
>>> the LB will act upon the Maintenance state and remove the member from the
>>> active members list.
>>>
>>> So the state changes of a member in the case of graceful shutdown would
>>> be:
>>>
>>> Created --> Starting --> Activated --> Maintenance --> ReadyToShutDown
>>> -->Terminated
>>>
>>> WDYT?
>>>
>>> Thanks,
>>> Reka
>>>
>>>
>>> On Mon, Jan 20, 2014 at 10:52 PM, Imesh Gunaratne <im...@apache.org>wrote:
>>>
>>>> Thanks Reka! I thought clean.sh is asynchronous. That's why I got that
>>>> question. What you have explained is great.
>>>>
>>>> Thanks
>>>> On Jan 19, 2014 11:52 PM, "Reka Thirunavukkarasu" <re...@wso2.com>
>>>> wrote:
>>>>
>>>>> Hi Imesh,
>>>>>
>>>>> On Sat, Jan 18, 2014 at 3:31 AM, Imesh Gunaratne <im...@apache.org>wrote:
>>>>>
>>>>>> Hi Reka,
>>>>>>
>>>>>> Great work! I have one question regarding the following step:
>>>>>> - Member will send the ReadyToShutDown event to instance-status topic
>>>>>> once it is done with the cleaning up
>>>>>>
>>>>>> Here I could see that we have introduced a new extension called
>>>>>> clean.sh to let the instance to execute a clean operation before the
>>>>>> shutdown happens.
>>>>>> My question is how does the instance communicate back to the
>>>>>> cartridge agent (Bash to Java) once its process is complete to send the
>>>>>> ReadyToShutDown event?
>>>>>>
>>>>>
>>>>> Yah. It is a good finding. I have handled it as the way where we
>>>>> handled to start servers via startServerExtension using start-server.sh and
>>>>> then publish Activated event. I believe that the executeCleanupExtension
>>>>> and the publishInstanceReadyToShutdownEvent are synchronous. So, after
>>>>> successfully executed the executeCleanupExtension, the method call returns
>>>>> and publishInstanceReadyToShutdownEvent will get executed. Please do
>>>>> correct me, if i'm wrong..
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Reka
>>>>>
>>>>>>
>>>>>> Thanks
>>>>>> Imesh
>>>>>>
>>>>>>
>>>>>> On Wed, Jan 15, 2014 at 1:48 PM, Lahiru Sandaruwan <la...@wso2.com>wrote:
>>>>>>
>>>>>>> Hi Reka,
>>>>>>>
>>>>>>> Yes, I'm done. Sorry for the late reply.
>>>>>>> Following table explains it.
>>>>>>>
>>>>>>>
>>>>>>> Spawn invoked from AS
>>>>>>>
>>>>>>> Adding a pending member
>>>>>>>
>>>>>>> Member Activated event(from CC)
>>>>>>>
>>>>>>> Pending -> Active
>>>>>>>
>>>>>>> Graceful shutdown event sent from AS
>>>>>>>
>>>>>>> Active -> Termination pending
>>>>>>>
>>>>>>> Member Terminated event(from CC)
>>>>>>>
>>>>>>> Removing the termination pending member
>>>>>>>
>>>>>>> Member fault event(from CEP)
>>>>>>>
>>>>>>> Active Member -> Obsolete member
>>>>>>>
>>>>>>> I'm not sure if we can use ready shutdown state here. What
>>>>>>> autoscaler need to know is whether the member needs to be considered as a
>>>>>>> Active serving member or it is on the way to shut down(or already shut
>>>>>>> down).
>>>>>>>
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jan 15, 2014 at 1:59 PM, Reka Thirunavukkarasu <
>>>>>>> reka@wso2.com> wrote:
>>>>>>>
>>>>>>>> Hi Lahiru,
>>>>>>>>
>>>>>>>> Have you done with the changes on this improvement? If so, Can you
>>>>>>>> explain that how have you done with it?
>>>>>>>> Else, I believe that we can use the already existing
>>>>>>>> ReadyToShutDown state to improve the instance termination in autoscaler?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Reka
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Jan 14, 2014 at 12:07 AM, Lahiru Sandaruwan <
>>>>>>>> lahirus@wso2.com> wrote:
>>>>>>>>
>>>>>>>>> HI Reka,
>>>>>>>>>
>>>>>>>>> I think we have to do a improvement in autosclaer side since this
>>>>>>>>> will take some time to terminate. May be introducing "pending terminate".
>>>>>>>>>
>>>>>>>>> Otherwise scaling down rule wont stop scaling down properly...
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Jan 10, 2014 at 10:40 AM, Reka Thirunavukkarasu <
>>>>>>>>> reka@wso2.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Udara,
>>>>>>>>>>
>>>>>>>>>> +1. It is a good point. I will consider introducing a timeout in
>>>>>>>>>> Autoscaler to decide on the actual termination after making the call to SM.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Reka
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Jan 10, 2014 at 8:26 AM, Udara Liyanage <ud...@wso2.com>wrote:
>>>>>>>>>>
>>>>>>>>>>> Let's say Instance cleanup hang due to some reason(a file could
>>>>>>>>>>> not be deleted). Then instance won't send the readyToTerminate message. So
>>>>>>>>>>> isn't it better instance is terminated by force after timeout when the
>>>>>>>>>>> graceful shutdown is issued.
>>>>>>>>>>>
>>>>>>>>>>> Touched, not typed. Erroneous words are a feature, not a typo.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Reka Thirunavukkarasu
>>>>>>>>>> Software Engineer,
>>>>>>>>>> WSO2, Inc.:http://wso2.com,
>>>>>>>>>> Mobile: +94776442007
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> --
>>>>>>>>> Lahiru Sandaruwan
>>>>>>>>> Software Engineer,
>>>>>>>>> Platform Technologies,
>>>>>>>>> WSO2 Inc., http://wso2.com
>>>>>>>>> lean.enterprise.middleware
>>>>>>>>>
>>>>>>>>> email: lahirus@wso2.com cell: (+94) 773 325 954
>>>>>>>>> blog: http://lahiruwrites.blogspot.com/
>>>>>>>>> twitter: http://twitter.com/lahirus
>>>>>>>>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Reka Thirunavukkarasu
>>>>>>>> Software Engineer,
>>>>>>>> WSO2, Inc.:http://wso2.com,
>>>>>>>> Mobile: +94776442007
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> --
>>>>>>> Lahiru Sandaruwan
>>>>>>> Software Engineer,
>>>>>>> Platform Technologies,
>>>>>>> WSO2 Inc., http://wso2.com
>>>>>>> lean.enterprise.middleware
>>>>>>>
>>>>>>> email: lahirus@wso2.com cell: (+94) 773 325 954
>>>>>>> blog: http://lahiruwrites.blogspot.com/
>>>>>>> twitter: http://twitter.com/lahirus
>>>>>>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Reka Thirunavukkarasu
>>>>> Software Engineer,
>>>>> WSO2, Inc.:http://wso2.com,
>>>>> Mobile: +94776442007
>>>>>
>>>>>
>>>>> ,
>>>>>
>>>>
>>>
>>>
>>> --
>>> Reka Thirunavukkarasu
>>> Software Engineer,
>>> WSO2, Inc.:http://wso2.com,
>>> Mobile: +94776442007
>>>
>>>
>>>
>
>
> --
> Best Regards,
> Nirmal
>
> Nirmal Fernando.
> PPMC Member & Committer of Apache Stratos,
> Senior Software Engineer, WSO2 Inc.
>
> Blog: http://nirmalfdo.blogspot.com/
>
--
Reka Thirunavukkarasu
Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007
Re: Graceful shutdown for the instances when scale down
Posted by Nirmal Fernando <ni...@gmail.com>.
Agree for IN_MAINTENANCE state.
Thanks.
On Wed, Feb 19, 2014 at 8:24 AM, Afkham Azeez <af...@gmail.com> wrote:
> Yes, this is a necessary step. Servers that support maintenance mode
> generally have this state called IN_MAINTENANCE.
> On Feb 17, 2014 6:46 PM, "Reka Thirunavukkarasu" <re...@wso2.com> wrote:
>
>> Hi all,
>>
>> In the graceful shutdown, we keep the member as active until it receives
>> ReadyToShutdown event. In between these state changes, member will become
>> unavailable. So the LB needs to aware of that, in order to avoid sending
>> requests to a member which is going to graceful shutdown. I think that if
>> we introduce another state say Maintenance in the Topology, before the
>> actual shut down of the application inside the member takes place, then the
>> LB will act upon the Maintenance state and remove the member from the
>> active members list.
>>
>> So the state changes of a member in the case of graceful shutdown would
>> be:
>>
>> Created --> Starting --> Activated --> Maintenance --> ReadyToShutDown
>> -->Terminated
>>
>> WDYT?
>>
>> Thanks,
>> Reka
>>
>>
>> On Mon, Jan 20, 2014 at 10:52 PM, Imesh Gunaratne <im...@apache.org>wrote:
>>
>>> Thanks Reka! I thought clean.sh is asynchronous. That's why I got that
>>> question. What you have explained is great.
>>>
>>> Thanks
>>> On Jan 19, 2014 11:52 PM, "Reka Thirunavukkarasu" <re...@wso2.com> wrote:
>>>
>>>> Hi Imesh,
>>>>
>>>> On Sat, Jan 18, 2014 at 3:31 AM, Imesh Gunaratne <im...@apache.org>wrote:
>>>>
>>>>> Hi Reka,
>>>>>
>>>>> Great work! I have one question regarding the following step:
>>>>> - Member will send the ReadyToShutDown event to instance-status topic
>>>>> once it is done with the cleaning up
>>>>>
>>>>> Here I could see that we have introduced a new extension called
>>>>> clean.sh to let the instance to execute a clean operation before the
>>>>> shutdown happens.
>>>>> My question is how does the instance communicate back to the cartridge
>>>>> agent (Bash to Java) once its process is complete to send the
>>>>> ReadyToShutDown event?
>>>>>
>>>>
>>>> Yah. It is a good finding. I have handled it as the way where we
>>>> handled to start servers via startServerExtension using start-server.sh and
>>>> then publish Activated event. I believe that the executeCleanupExtension
>>>> and the publishInstanceReadyToShutdownEvent are synchronous. So, after
>>>> successfully executed the executeCleanupExtension, the method call returns
>>>> and publishInstanceReadyToShutdownEvent will get executed. Please do
>>>> correct me, if i'm wrong..
>>>>
>>>>
>>>> Thanks,
>>>> Reka
>>>>
>>>>>
>>>>> Thanks
>>>>> Imesh
>>>>>
>>>>>
>>>>> On Wed, Jan 15, 2014 at 1:48 PM, Lahiru Sandaruwan <la...@wso2.com>wrote:
>>>>>
>>>>>> Hi Reka,
>>>>>>
>>>>>> Yes, I'm done. Sorry for the late reply.
>>>>>> Following table explains it.
>>>>>>
>>>>>>
>>>>>> Spawn invoked from AS
>>>>>>
>>>>>> Adding a pending member
>>>>>>
>>>>>> Member Activated event(from CC)
>>>>>>
>>>>>> Pending -> Active
>>>>>>
>>>>>> Graceful shutdown event sent from AS
>>>>>>
>>>>>> Active -> Termination pending
>>>>>>
>>>>>> Member Terminated event(from CC)
>>>>>>
>>>>>> Removing the termination pending member
>>>>>>
>>>>>> Member fault event(from CEP)
>>>>>>
>>>>>> Active Member -> Obsolete member
>>>>>>
>>>>>> I'm not sure if we can use ready shutdown state here. What autoscaler
>>>>>> need to know is whether the member needs to be considered as a Active
>>>>>> serving member or it is on the way to shut down(or already shut down).
>>>>>>
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>>
>>>>>> On Wed, Jan 15, 2014 at 1:59 PM, Reka Thirunavukkarasu <reka@wso2.com
>>>>>> > wrote:
>>>>>>
>>>>>>> Hi Lahiru,
>>>>>>>
>>>>>>> Have you done with the changes on this improvement? If so, Can you
>>>>>>> explain that how have you done with it?
>>>>>>> Else, I believe that we can use the already existing ReadyToShutDown
>>>>>>> state to improve the instance termination in autoscaler?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Reka
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Jan 14, 2014 at 12:07 AM, Lahiru Sandaruwan <
>>>>>>> lahirus@wso2.com> wrote:
>>>>>>>
>>>>>>>> HI Reka,
>>>>>>>>
>>>>>>>> I think we have to do a improvement in autosclaer side since this
>>>>>>>> will take some time to terminate. May be introducing "pending terminate".
>>>>>>>>
>>>>>>>> Otherwise scaling down rule wont stop scaling down properly...
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Jan 10, 2014 at 10:40 AM, Reka Thirunavukkarasu <
>>>>>>>> reka@wso2.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Udara,
>>>>>>>>>
>>>>>>>>> +1. It is a good point. I will consider introducing a timeout in
>>>>>>>>> Autoscaler to decide on the actual termination after making the call to SM.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Reka
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Jan 10, 2014 at 8:26 AM, Udara Liyanage <ud...@wso2.com>wrote:
>>>>>>>>>
>>>>>>>>>> Let's say Instance cleanup hang due to some reason(a file could
>>>>>>>>>> not be deleted). Then instance won't send the readyToTerminate message. So
>>>>>>>>>> isn't it better instance is terminated by force after timeout when the
>>>>>>>>>> graceful shutdown is issued.
>>>>>>>>>>
>>>>>>>>>> Touched, not typed. Erroneous words are a feature, not a typo.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Reka Thirunavukkarasu
>>>>>>>>> Software Engineer,
>>>>>>>>> WSO2, Inc.:http://wso2.com,
>>>>>>>>> Mobile: +94776442007
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> --
>>>>>>>> Lahiru Sandaruwan
>>>>>>>> Software Engineer,
>>>>>>>> Platform Technologies,
>>>>>>>> WSO2 Inc., http://wso2.com
>>>>>>>> lean.enterprise.middleware
>>>>>>>>
>>>>>>>> email: lahirus@wso2.com cell: (+94) 773 325 954
>>>>>>>> blog: http://lahiruwrites.blogspot.com/
>>>>>>>> twitter: http://twitter.com/lahirus
>>>>>>>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Reka Thirunavukkarasu
>>>>>>> Software Engineer,
>>>>>>> WSO2, Inc.:http://wso2.com,
>>>>>>> Mobile: +94776442007
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> --
>>>>>> Lahiru Sandaruwan
>>>>>> Software Engineer,
>>>>>> Platform Technologies,
>>>>>> WSO2 Inc., http://wso2.com
>>>>>> lean.enterprise.middleware
>>>>>>
>>>>>> email: lahirus@wso2.com cell: (+94) 773 325 954
>>>>>> blog: http://lahiruwrites.blogspot.com/
>>>>>> twitter: http://twitter.com/lahirus
>>>>>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Reka Thirunavukkarasu
>>>> Software Engineer,
>>>> WSO2, Inc.:http://wso2.com,
>>>> Mobile: +94776442007
>>>>
>>>>
>>>> ,
>>>>
>>>
>>
>>
>> --
>> Reka Thirunavukkarasu
>> Software Engineer,
>> WSO2, Inc.:http://wso2.com,
>> Mobile: +94776442007
>>
>>
>>
--
Best Regards,
Nirmal
Nirmal Fernando.
PPMC Member & Committer of Apache Stratos,
Senior Software Engineer, WSO2 Inc.
Blog: http://nirmalfdo.blogspot.com/
Re: Graceful shutdown for the instances when scale down
Posted by Afkham Azeez <af...@gmail.com>.
Yes, this is a necessary step. Servers that support maintenance mode
generally have this state called IN_MAINTENANCE.
On Feb 17, 2014 6:46 PM, "Reka Thirunavukkarasu" <re...@wso2.com> wrote:
> Hi all,
>
> In the graceful shutdown, we keep the member as active until it receives
> ReadyToShutdown event. In between these state changes, member will become
> unavailable. So the LB needs to aware of that, in order to avoid sending
> requests to a member which is going to graceful shutdown. I think that if
> we introduce another state say Maintenance in the Topology, before the
> actual shut down of the application inside the member takes place, then the
> LB will act upon the Maintenance state and remove the member from the
> active members list.
>
> So the state changes of a member in the case of graceful shutdown would be:
>
> Created --> Starting --> Activated --> Maintenance --> ReadyToShutDown
> -->Terminated
>
> WDYT?
>
> Thanks,
> Reka
>
>
> On Mon, Jan 20, 2014 at 10:52 PM, Imesh Gunaratne <im...@apache.org>wrote:
>
>> Thanks Reka! I thought clean.sh is asynchronous. That's why I got that
>> question. What you have explained is great.
>>
>> Thanks
>> On Jan 19, 2014 11:52 PM, "Reka Thirunavukkarasu" <re...@wso2.com> wrote:
>>
>>> Hi Imesh,
>>>
>>> On Sat, Jan 18, 2014 at 3:31 AM, Imesh Gunaratne <im...@apache.org>wrote:
>>>
>>>> Hi Reka,
>>>>
>>>> Great work! I have one question regarding the following step:
>>>> - Member will send the ReadyToShutDown event to instance-status topic
>>>> once it is done with the cleaning up
>>>>
>>>> Here I could see that we have introduced a new extension called
>>>> clean.sh to let the instance to execute a clean operation before the
>>>> shutdown happens.
>>>> My question is how does the instance communicate back to the cartridge
>>>> agent (Bash to Java) once its process is complete to send the
>>>> ReadyToShutDown event?
>>>>
>>>
>>> Yah. It is a good finding. I have handled it as the way where we handled
>>> to start servers via startServerExtension using start-server.sh and then
>>> publish Activated event. I believe that the executeCleanupExtension and the
>>> publishInstanceReadyToShutdownEvent are synchronous. So, after successfully
>>> executed the executeCleanupExtension, the method call returns and
>>> publishInstanceReadyToShutdownEvent will get executed. Please do correct
>>> me, if i'm wrong..
>>>
>>>
>>> Thanks,
>>> Reka
>>>
>>>>
>>>> Thanks
>>>> Imesh
>>>>
>>>>
>>>> On Wed, Jan 15, 2014 at 1:48 PM, Lahiru Sandaruwan <la...@wso2.com>wrote:
>>>>
>>>>> Hi Reka,
>>>>>
>>>>> Yes, I'm done. Sorry for the late reply.
>>>>> Following table explains it.
>>>>>
>>>>>
>>>>> Spawn invoked from AS
>>>>>
>>>>> Adding a pending member
>>>>>
>>>>> Member Activated event(from CC)
>>>>>
>>>>> Pending -> Active
>>>>>
>>>>> Graceful shutdown event sent from AS
>>>>>
>>>>> Active -> Termination pending
>>>>>
>>>>> Member Terminated event(from CC)
>>>>>
>>>>> Removing the termination pending member
>>>>>
>>>>> Member fault event(from CEP)
>>>>>
>>>>> Active Member -> Obsolete member
>>>>>
>>>>> I'm not sure if we can use ready shutdown state here. What autoscaler
>>>>> need to know is whether the member needs to be considered as a Active
>>>>> serving member or it is on the way to shut down(or already shut down).
>>>>>
>>>>>
>>>>> Thanks.
>>>>>
>>>>>
>>>>> On Wed, Jan 15, 2014 at 1:59 PM, Reka Thirunavukkarasu <re...@wso2.com>wrote:
>>>>>
>>>>>> Hi Lahiru,
>>>>>>
>>>>>> Have you done with the changes on this improvement? If so, Can you
>>>>>> explain that how have you done with it?
>>>>>> Else, I believe that we can use the already existing ReadyToShutDown
>>>>>> state to improve the instance termination in autoscaler?
>>>>>>
>>>>>> Thanks,
>>>>>> Reka
>>>>>>
>>>>>>
>>>>>> On Tue, Jan 14, 2014 at 12:07 AM, Lahiru Sandaruwan <lahirus@wso2.com
>>>>>> > wrote:
>>>>>>
>>>>>>> HI Reka,
>>>>>>>
>>>>>>> I think we have to do a improvement in autosclaer side since this
>>>>>>> will take some time to terminate. May be introducing "pending terminate".
>>>>>>>
>>>>>>> Otherwise scaling down rule wont stop scaling down properly...
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Jan 10, 2014 at 10:40 AM, Reka Thirunavukkarasu <
>>>>>>> reka@wso2.com> wrote:
>>>>>>>
>>>>>>>> Hi Udara,
>>>>>>>>
>>>>>>>> +1. It is a good point. I will consider introducing a timeout in
>>>>>>>> Autoscaler to decide on the actual termination after making the call to SM.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Reka
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Jan 10, 2014 at 8:26 AM, Udara Liyanage <ud...@wso2.com>wrote:
>>>>>>>>
>>>>>>>>> Let's say Instance cleanup hang due to some reason(a file could
>>>>>>>>> not be deleted). Then instance won't send the readyToTerminate message. So
>>>>>>>>> isn't it better instance is terminated by force after timeout when the
>>>>>>>>> graceful shutdown is issued.
>>>>>>>>>
>>>>>>>>> Touched, not typed. Erroneous words are a feature, not a typo.
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Reka Thirunavukkarasu
>>>>>>>> Software Engineer,
>>>>>>>> WSO2, Inc.:http://wso2.com,
>>>>>>>> Mobile: +94776442007
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> --
>>>>>>> Lahiru Sandaruwan
>>>>>>> Software Engineer,
>>>>>>> Platform Technologies,
>>>>>>> WSO2 Inc., http://wso2.com
>>>>>>> lean.enterprise.middleware
>>>>>>>
>>>>>>> email: lahirus@wso2.com cell: (+94) 773 325 954
>>>>>>> blog: http://lahiruwrites.blogspot.com/
>>>>>>> twitter: http://twitter.com/lahirus
>>>>>>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Reka Thirunavukkarasu
>>>>>> Software Engineer,
>>>>>> WSO2, Inc.:http://wso2.com,
>>>>>> Mobile: +94776442007
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> --
>>>>> Lahiru Sandaruwan
>>>>> Software Engineer,
>>>>> Platform Technologies,
>>>>> WSO2 Inc., http://wso2.com
>>>>> lean.enterprise.middleware
>>>>>
>>>>> email: lahirus@wso2.com cell: (+94) 773 325 954
>>>>> blog: http://lahiruwrites.blogspot.com/
>>>>> twitter: http://twitter.com/lahirus
>>>>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Reka Thirunavukkarasu
>>> Software Engineer,
>>> WSO2, Inc.:http://wso2.com,
>>> Mobile: +94776442007
>>>
>>>
>>> ,
>>>
>>
>
>
> --
> Reka Thirunavukkarasu
> Software Engineer,
> WSO2, Inc.:http://wso2.com,
> Mobile: +94776442007
>
>
>
Re: Graceful shutdown for the instances when scale down
Posted by Reka Thirunavukkarasu <re...@wso2.com>.
Hi all,
In the graceful shutdown, we keep the member as active until it receives
ReadyToShutdown event. In between these state changes, member will become
unavailable. So the LB needs to aware of that, in order to avoid sending
requests to a member which is going to graceful shutdown. I think that if
we introduce another state say Maintenance in the Topology, before the
actual shut down of the application inside the member takes place, then the
LB will act upon the Maintenance state and remove the member from the
active members list.
So the state changes of a member in the case of graceful shutdown would be:
Created --> Starting --> Activated --> Maintenance --> ReadyToShutDown
-->Terminated
WDYT?
Thanks,
Reka
On Mon, Jan 20, 2014 at 10:52 PM, Imesh Gunaratne <im...@apache.org> wrote:
> Thanks Reka! I thought clean.sh is asynchronous. That's why I got that
> question. What you have explained is great.
>
> Thanks
> On Jan 19, 2014 11:52 PM, "Reka Thirunavukkarasu" <re...@wso2.com> wrote:
>
>> Hi Imesh,
>>
>> On Sat, Jan 18, 2014 at 3:31 AM, Imesh Gunaratne <im...@apache.org>wrote:
>>
>>> Hi Reka,
>>>
>>> Great work! I have one question regarding the following step:
>>> - Member will send the ReadyToShutDown event to instance-status topic
>>> once it is done with the cleaning up
>>>
>>> Here I could see that we have introduced a new extension called clean.sh
>>> to let the instance to execute a clean operation before the shutdown
>>> happens.
>>> My question is how does the instance communicate back to the cartridge
>>> agent (Bash to Java) once its process is complete to send the
>>> ReadyToShutDown event?
>>>
>>
>> Yah. It is a good finding. I have handled it as the way where we handled
>> to start servers via startServerExtension using start-server.sh and then
>> publish Activated event. I believe that the executeCleanupExtension and the
>> publishInstanceReadyToShutdownEvent are synchronous. So, after successfully
>> executed the executeCleanupExtension, the method call returns and
>> publishInstanceReadyToShutdownEvent will get executed. Please do correct
>> me, if i'm wrong..
>>
>>
>> Thanks,
>> Reka
>>
>>>
>>> Thanks
>>> Imesh
>>>
>>>
>>> On Wed, Jan 15, 2014 at 1:48 PM, Lahiru Sandaruwan <la...@wso2.com>wrote:
>>>
>>>> Hi Reka,
>>>>
>>>> Yes, I'm done. Sorry for the late reply.
>>>> Following table explains it.
>>>>
>>>>
>>>> Spawn invoked from AS
>>>>
>>>> Adding a pending member
>>>>
>>>> Member Activated event(from CC)
>>>>
>>>> Pending -> Active
>>>>
>>>> Graceful shutdown event sent from AS
>>>>
>>>> Active -> Termination pending
>>>>
>>>> Member Terminated event(from CC)
>>>>
>>>> Removing the termination pending member
>>>>
>>>> Member fault event(from CEP)
>>>>
>>>> Active Member -> Obsolete member
>>>>
>>>> I'm not sure if we can use ready shutdown state here. What autoscaler
>>>> need to know is whether the member needs to be considered as a Active
>>>> serving member or it is on the way to shut down(or already shut down).
>>>>
>>>>
>>>> Thanks.
>>>>
>>>>
>>>> On Wed, Jan 15, 2014 at 1:59 PM, Reka Thirunavukkarasu <re...@wso2.com>wrote:
>>>>
>>>>> Hi Lahiru,
>>>>>
>>>>> Have you done with the changes on this improvement? If so, Can you
>>>>> explain that how have you done with it?
>>>>> Else, I believe that we can use the already existing ReadyToShutDown
>>>>> state to improve the instance termination in autoscaler?
>>>>>
>>>>> Thanks,
>>>>> Reka
>>>>>
>>>>>
>>>>> On Tue, Jan 14, 2014 at 12:07 AM, Lahiru Sandaruwan <la...@wso2.com>wrote:
>>>>>
>>>>>> HI Reka,
>>>>>>
>>>>>> I think we have to do a improvement in autosclaer side since this
>>>>>> will take some time to terminate. May be introducing "pending terminate".
>>>>>>
>>>>>> Otherwise scaling down rule wont stop scaling down properly...
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>>
>>>>>> On Fri, Jan 10, 2014 at 10:40 AM, Reka Thirunavukkarasu <
>>>>>> reka@wso2.com> wrote:
>>>>>>
>>>>>>> Hi Udara,
>>>>>>>
>>>>>>> +1. It is a good point. I will consider introducing a timeout in
>>>>>>> Autoscaler to decide on the actual termination after making the call to SM.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Reka
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Jan 10, 2014 at 8:26 AM, Udara Liyanage <ud...@wso2.com>wrote:
>>>>>>>
>>>>>>>> Let's say Instance cleanup hang due to some reason(a file could not
>>>>>>>> be deleted). Then instance won't send the readyToTerminate message. So
>>>>>>>> isn't it better instance is terminated by force after timeout when the
>>>>>>>> graceful shutdown is issued.
>>>>>>>>
>>>>>>>> Touched, not typed. Erroneous words are a feature, not a typo.
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Reka Thirunavukkarasu
>>>>>>> Software Engineer,
>>>>>>> WSO2, Inc.:http://wso2.com,
>>>>>>> Mobile: +94776442007
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> --
>>>>>> Lahiru Sandaruwan
>>>>>> Software Engineer,
>>>>>> Platform Technologies,
>>>>>> WSO2 Inc., http://wso2.com
>>>>>> lean.enterprise.middleware
>>>>>>
>>>>>> email: lahirus@wso2.com cell: (+94) 773 325 954
>>>>>> blog: http://lahiruwrites.blogspot.com/
>>>>>> twitter: http://twitter.com/lahirus
>>>>>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Reka Thirunavukkarasu
>>>>> Software Engineer,
>>>>> WSO2, Inc.:http://wso2.com,
>>>>> Mobile: +94776442007
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> --
>>>> Lahiru Sandaruwan
>>>> Software Engineer,
>>>> Platform Technologies,
>>>> WSO2 Inc., http://wso2.com
>>>> lean.enterprise.middleware
>>>>
>>>> email: lahirus@wso2.com cell: (+94) 773 325 954
>>>> blog: http://lahiruwrites.blogspot.com/
>>>> twitter: http://twitter.com/lahirus
>>>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>>>
>>>>
>>>
>>
>>
>> --
>> Reka Thirunavukkarasu
>> Software Engineer,
>> WSO2, Inc.:http://wso2.com,
>> Mobile: +94776442007
>>
>>
>> ,
>>
>
--
Reka Thirunavukkarasu
Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007
Re: Graceful shutdown for the instances when scale down
Posted by Imesh Gunaratne <im...@apache.org>.
Thanks Reka! I thought clean.sh is asynchronous. That's why I got that
question. What you have explained is great.
Thanks
On Jan 19, 2014 11:52 PM, "Reka Thirunavukkarasu" <re...@wso2.com> wrote:
> Hi Imesh,
>
> On Sat, Jan 18, 2014 at 3:31 AM, Imesh Gunaratne <im...@apache.org> wrote:
>
>> Hi Reka,
>>
>> Great work! I have one question regarding the following step:
>> - Member will send the ReadyToShutDown event to instance-status topic
>> once it is done with the cleaning up
>>
>> Here I could see that we have introduced a new extension called clean.sh
>> to let the instance to execute a clean operation before the shutdown
>> happens.
>> My question is how does the instance communicate back to the cartridge
>> agent (Bash to Java) once its process is complete to send the
>> ReadyToShutDown event?
>>
>
> Yah. It is a good finding. I have handled it as the way where we handled
> to start servers via startServerExtension using start-server.sh and then
> publish Activated event. I believe that the executeCleanupExtension and the
> publishInstanceReadyToShutdownEvent are synchronous. So, after successfully
> executed the executeCleanupExtension, the method call returns and
> publishInstanceReadyToShutdownEvent will get executed. Please do correct
> me, if i'm wrong..
>
>
> Thanks,
> Reka
>
>>
>> Thanks
>> Imesh
>>
>>
>> On Wed, Jan 15, 2014 at 1:48 PM, Lahiru Sandaruwan <la...@wso2.com>wrote:
>>
>>> Hi Reka,
>>>
>>> Yes, I'm done. Sorry for the late reply.
>>> Following table explains it.
>>>
>>>
>>> Spawn invoked from AS
>>>
>>> Adding a pending member
>>>
>>> Member Activated event(from CC)
>>>
>>> Pending -> Active
>>>
>>> Graceful shutdown event sent from AS
>>>
>>> Active -> Termination pending
>>>
>>> Member Terminated event(from CC)
>>>
>>> Removing the termination pending member
>>>
>>> Member fault event(from CEP)
>>>
>>> Active Member -> Obsolete member
>>>
>>> I'm not sure if we can use ready shutdown state here. What autoscaler
>>> need to know is whether the member needs to be considered as a Active
>>> serving member or it is on the way to shut down(or already shut down).
>>>
>>>
>>> Thanks.
>>>
>>>
>>> On Wed, Jan 15, 2014 at 1:59 PM, Reka Thirunavukkarasu <re...@wso2.com>wrote:
>>>
>>>> Hi Lahiru,
>>>>
>>>> Have you done with the changes on this improvement? If so, Can you
>>>> explain that how have you done with it?
>>>> Else, I believe that we can use the already existing ReadyToShutDown
>>>> state to improve the instance termination in autoscaler?
>>>>
>>>> Thanks,
>>>> Reka
>>>>
>>>>
>>>> On Tue, Jan 14, 2014 at 12:07 AM, Lahiru Sandaruwan <la...@wso2.com>wrote:
>>>>
>>>>> HI Reka,
>>>>>
>>>>> I think we have to do a improvement in autosclaer side since this will
>>>>> take some time to terminate. May be introducing "pending terminate".
>>>>>
>>>>> Otherwise scaling down rule wont stop scaling down properly...
>>>>>
>>>>> Thanks.
>>>>>
>>>>>
>>>>> On Fri, Jan 10, 2014 at 10:40 AM, Reka Thirunavukkarasu <reka@wso2.com
>>>>> > wrote:
>>>>>
>>>>>> Hi Udara,
>>>>>>
>>>>>> +1. It is a good point. I will consider introducing a timeout in
>>>>>> Autoscaler to decide on the actual termination after making the call to SM.
>>>>>>
>>>>>> Thanks,
>>>>>> Reka
>>>>>>
>>>>>>
>>>>>> On Fri, Jan 10, 2014 at 8:26 AM, Udara Liyanage <ud...@wso2.com>wrote:
>>>>>>
>>>>>>> Let's say Instance cleanup hang due to some reason(a file could not
>>>>>>> be deleted). Then instance won't send the readyToTerminate message. So
>>>>>>> isn't it better instance is terminated by force after timeout when the
>>>>>>> graceful shutdown is issued.
>>>>>>>
>>>>>>> Touched, not typed. Erroneous words are a feature, not a typo.
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Reka Thirunavukkarasu
>>>>>> Software Engineer,
>>>>>> WSO2, Inc.:http://wso2.com,
>>>>>> Mobile: +94776442007
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> --
>>>>> Lahiru Sandaruwan
>>>>> Software Engineer,
>>>>> Platform Technologies,
>>>>> WSO2 Inc., http://wso2.com
>>>>> lean.enterprise.middleware
>>>>>
>>>>> email: lahirus@wso2.com cell: (+94) 773 325 954
>>>>> blog: http://lahiruwrites.blogspot.com/
>>>>> twitter: http://twitter.com/lahirus
>>>>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Reka Thirunavukkarasu
>>>> Software Engineer,
>>>> WSO2, Inc.:http://wso2.com,
>>>> Mobile: +94776442007
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> --
>>> Lahiru Sandaruwan
>>> Software Engineer,
>>> Platform Technologies,
>>> WSO2 Inc., http://wso2.com
>>> lean.enterprise.middleware
>>>
>>> email: lahirus@wso2.com cell: (+94) 773 325 954
>>> blog: http://lahiruwrites.blogspot.com/
>>> twitter: http://twitter.com/lahirus
>>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>>
>>>
>>
>
>
> --
> Reka Thirunavukkarasu
> Software Engineer,
> WSO2, Inc.:http://wso2.com,
> Mobile: +94776442007
>
>
> ,
>
Re: Graceful shutdown for the instances when scale down
Posted by Reka Thirunavukkarasu <re...@wso2.com>.
Hi Imesh,
On Sat, Jan 18, 2014 at 3:31 AM, Imesh Gunaratne <im...@apache.org> wrote:
> Hi Reka,
>
> Great work! I have one question regarding the following step:
> - Member will send the ReadyToShutDown event to instance-status topic once
> it is done with the cleaning up
>
> Here I could see that we have introduced a new extension called clean.sh
> to let the instance to execute a clean operation before the shutdown
> happens.
> My question is how does the instance communicate back to the cartridge
> agent (Bash to Java) once its process is complete to send the
> ReadyToShutDown event?
>
Yah. It is a good finding. I have handled it as the way where we handled to
start servers via startServerExtension using start-server.sh and then
publish Activated event. I believe that the executeCleanupExtension and the
publishInstanceReadyToShutdownEvent are synchronous. So, after successfully
executed the executeCleanupExtension, the method call returns and
publishInstanceReadyToShutdownEvent will get executed. Please do correct
me, if i'm wrong..
Thanks,
Reka
>
> Thanks
> Imesh
>
>
> On Wed, Jan 15, 2014 at 1:48 PM, Lahiru Sandaruwan <la...@wso2.com>wrote:
>
>> Hi Reka,
>>
>> Yes, I'm done. Sorry for the late reply.
>> Following table explains it.
>>
>>
>> Spawn invoked from AS
>>
>> Adding a pending member
>>
>> Member Activated event(from CC)
>>
>> Pending -> Active
>>
>> Graceful shutdown event sent from AS
>>
>> Active -> Termination pending
>>
>> Member Terminated event(from CC)
>>
>> Removing the termination pending member
>>
>> Member fault event(from CEP)
>>
>> Active Member -> Obsolete member
>>
>> I'm not sure if we can use ready shutdown state here. What autoscaler
>> need to know is whether the member needs to be considered as a Active
>> serving member or it is on the way to shut down(or already shut down).
>>
>>
>> Thanks.
>>
>>
>> On Wed, Jan 15, 2014 at 1:59 PM, Reka Thirunavukkarasu <re...@wso2.com>wrote:
>>
>>> Hi Lahiru,
>>>
>>> Have you done with the changes on this improvement? If so, Can you
>>> explain that how have you done with it?
>>> Else, I believe that we can use the already existing ReadyToShutDown
>>> state to improve the instance termination in autoscaler?
>>>
>>> Thanks,
>>> Reka
>>>
>>>
>>> On Tue, Jan 14, 2014 at 12:07 AM, Lahiru Sandaruwan <la...@wso2.com>wrote:
>>>
>>>> HI Reka,
>>>>
>>>> I think we have to do a improvement in autosclaer side since this will
>>>> take some time to terminate. May be introducing "pending terminate".
>>>>
>>>> Otherwise scaling down rule wont stop scaling down properly...
>>>>
>>>> Thanks.
>>>>
>>>>
>>>> On Fri, Jan 10, 2014 at 10:40 AM, Reka Thirunavukkarasu <re...@wso2.com>wrote:
>>>>
>>>>> Hi Udara,
>>>>>
>>>>> +1. It is a good point. I will consider introducing a timeout in
>>>>> Autoscaler to decide on the actual termination after making the call to SM.
>>>>>
>>>>> Thanks,
>>>>> Reka
>>>>>
>>>>>
>>>>> On Fri, Jan 10, 2014 at 8:26 AM, Udara Liyanage <ud...@wso2.com>wrote:
>>>>>
>>>>>> Let's say Instance cleanup hang due to some reason(a file could not
>>>>>> be deleted). Then instance won't send the readyToTerminate message. So
>>>>>> isn't it better instance is terminated by force after timeout when the
>>>>>> graceful shutdown is issued.
>>>>>>
>>>>>> Touched, not typed. Erroneous words are a feature, not a typo.
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Reka Thirunavukkarasu
>>>>> Software Engineer,
>>>>> WSO2, Inc.:http://wso2.com,
>>>>> Mobile: +94776442007
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> --
>>>> Lahiru Sandaruwan
>>>> Software Engineer,
>>>> Platform Technologies,
>>>> WSO2 Inc., http://wso2.com
>>>> lean.enterprise.middleware
>>>>
>>>> email: lahirus@wso2.com cell: (+94) 773 325 954
>>>> blog: http://lahiruwrites.blogspot.com/
>>>> twitter: http://twitter.com/lahirus
>>>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>>>
>>>>
>>>
>>>
>>> --
>>> Reka Thirunavukkarasu
>>> Software Engineer,
>>> WSO2, Inc.:http://wso2.com,
>>> Mobile: +94776442007
>>>
>>>
>>>
>>
>>
>> --
>> --
>> Lahiru Sandaruwan
>> Software Engineer,
>> Platform Technologies,
>> WSO2 Inc., http://wso2.com
>> lean.enterprise.middleware
>>
>> email: lahirus@wso2.com cell: (+94) 773 325 954
>> blog: http://lahiruwrites.blogspot.com/
>> twitter: http://twitter.com/lahirus
>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>
>>
>
--
Reka Thirunavukkarasu
Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007
,
Re: Graceful shutdown for the instances when scale down
Posted by Imesh Gunaratne <im...@apache.org>.
Hi Reka,
Great work! I have one question regarding the following step:
- Member will send the ReadyToShutDown event to instance-status topic once
it is done with the cleaning up
Here I could see that we have introduced a new extension called clean.sh to
let the instance to execute a clean operation before the shutdown happens.
My question is how does the instance communicate back to the cartridge
agent (Bash to Java) once its process is complete to send the
ReadyToShutDown event?
Thanks
Imesh
On Wed, Jan 15, 2014 at 1:48 PM, Lahiru Sandaruwan <la...@wso2.com> wrote:
> Hi Reka,
>
> Yes, I'm done. Sorry for the late reply.
> Following table explains it.
>
>
> Spawn invoked from AS
>
> Adding a pending member
>
> Member Activated event(from CC)
>
> Pending -> Active
>
> Graceful shutdown event sent from AS
>
> Active -> Termination pending
>
> Member Terminated event(from CC)
>
> Removing the termination pending member
>
> Member fault event(from CEP)
>
> Active Member -> Obsolete member
>
> I'm not sure if we can use ready shutdown state here. What autoscaler need
> to know is whether the member needs to be considered as a Active serving
> member or it is on the way to shut down(or already shut down).
>
>
> Thanks.
>
>
> On Wed, Jan 15, 2014 at 1:59 PM, Reka Thirunavukkarasu <re...@wso2.com>wrote:
>
>> Hi Lahiru,
>>
>> Have you done with the changes on this improvement? If so, Can you
>> explain that how have you done with it?
>> Else, I believe that we can use the already existing ReadyToShutDown
>> state to improve the instance termination in autoscaler?
>>
>> Thanks,
>> Reka
>>
>>
>> On Tue, Jan 14, 2014 at 12:07 AM, Lahiru Sandaruwan <la...@wso2.com>wrote:
>>
>>> HI Reka,
>>>
>>> I think we have to do a improvement in autosclaer side since this will
>>> take some time to terminate. May be introducing "pending terminate".
>>>
>>> Otherwise scaling down rule wont stop scaling down properly...
>>>
>>> Thanks.
>>>
>>>
>>> On Fri, Jan 10, 2014 at 10:40 AM, Reka Thirunavukkarasu <re...@wso2.com>wrote:
>>>
>>>> Hi Udara,
>>>>
>>>> +1. It is a good point. I will consider introducing a timeout in
>>>> Autoscaler to decide on the actual termination after making the call to SM.
>>>>
>>>> Thanks,
>>>> Reka
>>>>
>>>>
>>>> On Fri, Jan 10, 2014 at 8:26 AM, Udara Liyanage <ud...@wso2.com> wrote:
>>>>
>>>>> Let's say Instance cleanup hang due to some reason(a file could not be
>>>>> deleted). Then instance won't send the readyToTerminate message. So isn't
>>>>> it better instance is terminated by force after timeout when the graceful
>>>>> shutdown is issued.
>>>>>
>>>>> Touched, not typed. Erroneous words are a feature, not a typo.
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Reka Thirunavukkarasu
>>>> Software Engineer,
>>>> WSO2, Inc.:http://wso2.com,
>>>> Mobile: +94776442007
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> --
>>> Lahiru Sandaruwan
>>> Software Engineer,
>>> Platform Technologies,
>>> WSO2 Inc., http://wso2.com
>>> lean.enterprise.middleware
>>>
>>> email: lahirus@wso2.com cell: (+94) 773 325 954
>>> blog: http://lahiruwrites.blogspot.com/
>>> twitter: http://twitter.com/lahirus
>>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>>
>>>
>>
>>
>> --
>> Reka Thirunavukkarasu
>> Software Engineer,
>> WSO2, Inc.:http://wso2.com,
>> Mobile: +94776442007
>>
>>
>>
>
>
> --
> --
> Lahiru Sandaruwan
> Software Engineer,
> Platform Technologies,
> WSO2 Inc., http://wso2.com
> lean.enterprise.middleware
>
> email: lahirus@wso2.com cell: (+94) 773 325 954
> blog: http://lahiruwrites.blogspot.com/
> twitter: http://twitter.com/lahirus
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
>
Re: Graceful shutdown for the instances when scale down
Posted by Lahiru Sandaruwan <la...@wso2.com>.
Hi Reka,
Yes, I'm done. Sorry for the late reply.
Following table explains it.
Spawn invoked from AS
Adding a pending member
Member Activated event(from CC)
Pending -> Active
Graceful shutdown event sent from AS
Active -> Termination pending
Member Terminated event(from CC)
Removing the termination pending member
Member fault event(from CEP)
Active Member -> Obsolete member
I'm not sure if we can use ready shutdown state here. What autoscaler need
to know is whether the member needs to be considered as a Active serving
member or it is on the way to shut down(or already shut down).
Thanks.
On Wed, Jan 15, 2014 at 1:59 PM, Reka Thirunavukkarasu <re...@wso2.com>wrote:
> Hi Lahiru,
>
> Have you done with the changes on this improvement? If so, Can you explain
> that how have you done with it?
> Else, I believe that we can use the already existing ReadyToShutDown state
> to improve the instance termination in autoscaler?
>
> Thanks,
> Reka
>
>
> On Tue, Jan 14, 2014 at 12:07 AM, Lahiru Sandaruwan <la...@wso2.com>wrote:
>
>> HI Reka,
>>
>> I think we have to do a improvement in autosclaer side since this will
>> take some time to terminate. May be introducing "pending terminate".
>>
>> Otherwise scaling down rule wont stop scaling down properly...
>>
>> Thanks.
>>
>>
>> On Fri, Jan 10, 2014 at 10:40 AM, Reka Thirunavukkarasu <re...@wso2.com>wrote:
>>
>>> Hi Udara,
>>>
>>> +1. It is a good point. I will consider introducing a timeout in
>>> Autoscaler to decide on the actual termination after making the call to SM.
>>>
>>> Thanks,
>>> Reka
>>>
>>>
>>> On Fri, Jan 10, 2014 at 8:26 AM, Udara Liyanage <ud...@wso2.com> wrote:
>>>
>>>> Let's say Instance cleanup hang due to some reason(a file could not be
>>>> deleted). Then instance won't send the readyToTerminate message. So isn't
>>>> it better instance is terminated by force after timeout when the graceful
>>>> shutdown is issued.
>>>>
>>>> Touched, not typed. Erroneous words are a feature, not a typo.
>>>>
>>>
>>>
>>>
>>> --
>>> Reka Thirunavukkarasu
>>> Software Engineer,
>>> WSO2, Inc.:http://wso2.com,
>>> Mobile: +94776442007
>>>
>>>
>>>
>>
>>
>> --
>> --
>> Lahiru Sandaruwan
>> Software Engineer,
>> Platform Technologies,
>> WSO2 Inc., http://wso2.com
>> lean.enterprise.middleware
>>
>> email: lahirus@wso2.com cell: (+94) 773 325 954
>> blog: http://lahiruwrites.blogspot.com/
>> twitter: http://twitter.com/lahirus
>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>
>>
>
>
> --
> Reka Thirunavukkarasu
> Software Engineer,
> WSO2, Inc.:http://wso2.com,
> Mobile: +94776442007
>
>
>
--
--
Lahiru Sandaruwan
Software Engineer,
Platform Technologies,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
email: lahirus@wso2.com cell: (+94) 773 325 954
blog: http://lahiruwrites.blogspot.com/
twitter: http://twitter.com/lahirus
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
Re: Graceful shutdown for the instances when scale down
Posted by Reka Thirunavukkarasu <re...@wso2.com>.
Hi Lahiru,
Have you done with the changes on this improvement? If so, Can you explain
that how have you done with it?
Else, I believe that we can use the already existing ReadyToShutDown state
to improve the instance termination in autoscaler?
Thanks,
Reka
On Tue, Jan 14, 2014 at 12:07 AM, Lahiru Sandaruwan <la...@wso2.com>wrote:
> HI Reka,
>
> I think we have to do a improvement in autosclaer side since this will
> take some time to terminate. May be introducing "pending terminate".
>
> Otherwise scaling down rule wont stop scaling down properly...
>
> Thanks.
>
>
> On Fri, Jan 10, 2014 at 10:40 AM, Reka Thirunavukkarasu <re...@wso2.com>wrote:
>
>> Hi Udara,
>>
>> +1. It is a good point. I will consider introducing a timeout in
>> Autoscaler to decide on the actual termination after making the call to SM.
>>
>> Thanks,
>> Reka
>>
>>
>> On Fri, Jan 10, 2014 at 8:26 AM, Udara Liyanage <ud...@wso2.com> wrote:
>>
>>> Let's say Instance cleanup hang due to some reason(a file could not be
>>> deleted). Then instance won't send the readyToTerminate message. So isn't
>>> it better instance is terminated by force after timeout when the graceful
>>> shutdown is issued.
>>>
>>> Touched, not typed. Erroneous words are a feature, not a typo.
>>>
>>
>>
>>
>> --
>> Reka Thirunavukkarasu
>> Software Engineer,
>> WSO2, Inc.:http://wso2.com,
>> Mobile: +94776442007
>>
>>
>>
>
>
> --
> --
> Lahiru Sandaruwan
> Software Engineer,
> Platform Technologies,
> WSO2 Inc., http://wso2.com
> lean.enterprise.middleware
>
> email: lahirus@wso2.com cell: (+94) 773 325 954
> blog: http://lahiruwrites.blogspot.com/
> twitter: http://twitter.com/lahirus
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
>
--
Reka Thirunavukkarasu
Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007
Re: Graceful shutdown for the instances when scale down
Posted by Lahiru Sandaruwan <la...@wso2.com>.
HI Reka,
I think we have to do a improvement in autosclaer side since this will take
some time to terminate. May be introducing "pending terminate".
Otherwise scaling down rule wont stop scaling down properly...
Thanks.
On Fri, Jan 10, 2014 at 10:40 AM, Reka Thirunavukkarasu <re...@wso2.com>wrote:
> Hi Udara,
>
> +1. It is a good point. I will consider introducing a timeout in
> Autoscaler to decide on the actual termination after making the call to SM.
>
> Thanks,
> Reka
>
>
> On Fri, Jan 10, 2014 at 8:26 AM, Udara Liyanage <ud...@wso2.com> wrote:
>
>> Let's say Instance cleanup hang due to some reason(a file could not be
>> deleted). Then instance won't send the readyToTerminate message. So isn't
>> it better instance is terminated by force after timeout when the graceful
>> shutdown is issued.
>>
>> Touched, not typed. Erroneous words are a feature, not a typo.
>>
>
>
>
> --
> Reka Thirunavukkarasu
> Software Engineer,
> WSO2, Inc.:http://wso2.com,
> Mobile: +94776442007
>
>
>
--
--
Lahiru Sandaruwan
Software Engineer,
Platform Technologies,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
email: lahirus@wso2.com cell: (+94) 773 325 954
blog: http://lahiruwrites.blogspot.com/
twitter: http://twitter.com/lahirus
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
Re: Graceful shutdown for the instances when scale down
Posted by Reka Thirunavukkarasu <re...@wso2.com>.
Hi Udara,
+1. It is a good point. I will consider introducing a timeout in Autoscaler
to decide on the actual termination after making the call to SM.
Thanks,
Reka
On Fri, Jan 10, 2014 at 8:26 AM, Udara Liyanage <ud...@wso2.com> wrote:
> Let's say Instance cleanup hang due to some reason(a file could not be
> deleted). Then instance won't send the readyToTerminate message. So isn't
> it better instance is terminated by force after timeout when the graceful
> shutdown is issued.
>
> Touched, not typed. Erroneous words are a feature, not a typo.
>
--
Reka Thirunavukkarasu
Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007
Re: Graceful shutdown for the instances when scale down
Posted by Udara Liyanage <ud...@wso2.com>.
Let's say Instance cleanup hang due to some reason(a file could not be
deleted). Then instance won't send the readyToTerminate message. So isn't
it better instance is terminated by force after timeout when the graceful
shutdown is issued.
Touched, not typed. Erroneous words are a feature, not a typo.