You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stratos.apache.org by "Jeffrey Nguyen (jeffrngu)" <je...@cisco.com> on 2014/11/20 21:35:53 UTC
Subscription removed from subscription list before VMs terminated
by CC
Hi Devs,
I noticed when we unsubscribe a cartridge, after Stratos puts subscription in maintenance mode and removes it from subscription list, it takes 30 seconds for CC to start terminating the instance(s) and about 5 seconds for it to confirm the instance was terminated. Please see log snippet below for sequence of events.
We're using the Rest API call with resource URL "admin/cartridge/list/subscribed" to determine if all instances spawned as a result of subscriptions have been killed. This is apparently not reliable due to the delays in processing I mentioned earlier.
Questions:
* Can we keep the subscription that in maintenance mode in the subscription list and wait until the spawn instances have been terminated before removing it from subscription list?
* What causes the 30 second delay between subscription being in maintenance mode and CC picking up the status and starts terminating the spawned instances?
* Is there another REST API we can use to reliably tell if all the spawn instances have been terminated?
Snippet from wso2carbon log:
TID: [0] [STRATOS] [2014-11-20 11:05:42,755] INFO {org.apache.stratos.autoscaler.message.receiver.topology.AutoscalerTopologyEventReceiver} - Event received: ClusterMaintenanceModeEvent [serviceName=cisco-sample-vm2, clusterStatus=In_Maintenance]
TID: [0] [STRATOS] [2014-11-20 11:05:42,790] INFO {org.apache.stratos.manager.subscription.utils.CartridgeSubscriptionUtils} - Publishing tenant un-subscribed event: [tenant-id] -1234 [service] cisco-sample-vm2
TID: [0] [STRATOS] [2014-11-20 11:06:12,764] INFO {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - Starting to terminate an instance with member id : cisco-sample-vm2.cisco-sample-f817ea17-6248-495f-abfa-4fef92352e68 in partition id: RegionOne-Core of cluster id: cisco-sample-vm2.cisco-sample- and of cartridge type: cisco-sample-vm2
TID: [0] [STRATOS] [2014-11-20 11:06:17,329] INFO {org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor} - Member terminated: [service] cisco-sample-vm2 [cluster] cisco-sample-vm2.cisco-sample- [member] cisco-sample-vm2.cisco-sample-f817ea17-6248-495f-abfa-4fef92352e68
Regards,
-Jeffrey
Re: Subscription removed from subscription list before VMs
terminated by CC
Posted by "Jeffrey Nguyen (jeffrngu)" <je...@cisco.com>.
Hi Udara,
It's for 4.0. Still the question is does Stratos provide any Rest API for us to query when all instances have been terminated?
The same question should apply to 4.1: we need to have an API to determine if instances spawn by composite applications have been terminated.
-Jeffrey
From: Udara Liyanage <ud...@wso2.com>>
Reply-To: "dev@stratos.apache.org<ma...@stratos.apache.org>" <de...@stratos.apache.org>>
Date: Thursday, November 20, 2014 8:00 PM
To: dev <de...@stratos.apache.org>>
Subject: Re: Subscription removed from subscription list before VMs terminated by CC
Hi,
This delay may be due to graceful shutdown if instances. Upon unsubscription instances are set to in-maintaimace mode. Then instances do whatever cleanup things before termination. After cleanup is over, instances notify CC by an event(ReadyToTerminate AFAIR). Then only CC terminates the instances.
Touched, not typed. Erroneous words are a feature, not a typo.
On Nov 21, 2014 7:46 AM, "Udara Liyanage" <ud...@wso2.com>> wrote:
Hi Jeffrey,
Cartridge subscription is no longer in Stratos now, instead deploy composite application.
cartridge/list/subscribed endpoint is deprecated in 410.
Are you talking about Stratos 400 or current master branch?
Touched, not typed. Erroneous words are a feature, not a typo.
On Nov 21, 2014 2:13 AM, "Jeffrey Nguyen (jeffrngu)" <je...@cisco.com>> wrote:
Hi Devs,
I noticed when we unsubscribe a cartridge, after Stratos puts subscription in maintenance mode and removes it from subscription list, it takes 30 seconds for CC to start terminating the instance(s) and about 5 seconds for it to confirm the instance was terminated. Please see log snippet below for sequence of events.
We're using the Rest API call with resource URL "admin/cartridge/list/subscribed" to determine if all instances spawned as a result of subscriptions have been killed. This is apparently not reliable due to the delays in processing I mentioned earlier.
Questions:
* Can we keep the subscription that in maintenance mode in the subscription list and wait until the spawn instances have been terminated before removing it from subscription list?
* What causes the 30 second delay between subscription being in maintenance mode and CC picking up the status and starts terminating the spawned instances?
* Is there another REST API we can use to reliably tell if all the spawn instances have been terminated?
Snippet from wso2carbon log:
TID: [0] [STRATOS] [2014-11-20 11:05:42,755] INFO {org.apache.stratos.autoscaler.message.receiver.topology.AutoscalerTopologyEventReceiver} - Event received: ClusterMaintenanceModeEvent [serviceName=cisco-sample-vm2, clusterStatus=In_Maintenance]
TID: [0] [STRATOS] [2014-11-20 11:05:42,790] INFO {org.apache.stratos.manager.subscription.utils.CartridgeSubscriptionUtils} - Publishing tenant un-subscribed event: [tenant-id] -1234 [service] cisco-sample-vm2
TID: [0] [STRATOS] [2014-11-20 11:06:12,764] INFO {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - Starting to terminate an instance with member id : cisco-sample-vm2.cisco-sample-f817ea17-6248-495f-abfa-4fef92352e68 in partition id: RegionOne-Core of cluster id: cisco-sample-vm2.cisco-sample- and of cartridge type: cisco-sample-vm2
TID: [0] [STRATOS] [2014-11-20 11:06:17,329] INFO {org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor} - Member terminated: [service] cisco-sample-vm2 [cluster] cisco-sample-vm2.cisco-sample- [member] cisco-sample-vm2.cisco-sample-f817ea17-6248-495f-abfa-4fef92352e68
Regards,
-Jeffrey
Re: Subscription removed from subscription list before VMs terminated
by CC
Posted by Udara Liyanage <ud...@wso2.com>.
Hi,
This delay may be due to graceful shutdown if instances. Upon
unsubscription instances are set to in-maintaimace mode. Then instances do
whatever cleanup things before termination. After cleanup is over,
instances notify CC by an event(ReadyToTerminate AFAIR). Then only CC
terminates the instances.
Touched, not typed. Erroneous words are a feature, not a typo.
On Nov 21, 2014 7:46 AM, "Udara Liyanage" <ud...@wso2.com> wrote:
> Hi Jeffrey,
>
> Cartridge subscription is no longer in Stratos now, instead deploy
> composite application.
> cartridge/list/subscribed endpoint is deprecated in 410.
> Are you talking about Stratos 400 or current master branch?
>
>
>
>
> Touched, not typed. Erroneous words are a feature, not a typo.
> On Nov 21, 2014 2:13 AM, "Jeffrey Nguyen (jeffrngu)" <je...@cisco.com>
> wrote:
>
>> Hi Devs,
>>
>> I noticed when we unsubscribe a cartridge, after Stratos puts
>> subscription in maintenance mode and removes it from subscription list, it
>> takes 30 seconds for CC to start terminating the instance(s) and about 5
>> seconds for it to confirm the instance was terminated. Please see log
>> snippet below for sequence of events.
>>
>> We're using the Rest API call with resource URL "
>> *admin/cartridge/list/subscribed*" to determine if all instances spawned
>> as a result of subscriptions have been killed. This is apparently not
>> reliable due to the delays in processing I mentioned earlier.
>>
>> Questions:
>>
>> - Can we keep the subscription that in maintenance mode in the
>> subscription list and wait until the spawn instances have been terminated
>> before removing it from subscription list?
>> - What causes the 30 second delay between subscription being in
>> maintenance mode and CC picking up the status and starts terminating the
>> spawned instances?
>> - Is there another REST API we can use to reliably tell if all the
>> spawn instances have been terminated?
>>
>>
>> Snippet from wso2carbon log:
>>
>> TID: [0] [STRATOS] [2014-11-20 11:05:42,755] INFO
>> {org.apache.stratos.autoscaler.message.receiver.topology.AutoscalerTopologyEventReceiver}
>> - Event received: ClusterMaintenanceModeEvent
>> [serviceName=cisco-sample-vm2, clusterStatus=In_Maintenance]
>>
>>
>> TID: [0] [STRATOS] [2014-11-20 11:05:42,790] INFO
>> {org.apache.stratos.manager.subscription.utils.CartridgeSubscriptionUtils}
>> - Publishing tenant un-subscribed event: [tenant-id] -1234 [service]
>> cisco-sample-vm2
>>
>>
>>
>> TID: [0] [STRATOS] [2014-11-20 11:06:12,764] INFO
>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>> Starting to terminate an instance with member id :
>> cisco-sample-vm2.cisco-sample-f817ea17-6248-495f-abfa-4fef92352e68 in
>> partition id: RegionOne-Core of cluster id: cisco-sample-vm2.cisco-sample-
>> and of cartridge type: cisco-sample-vm2
>>
>>
>> TID: [0] [STRATOS] [2014-11-20 11:06:17,329] INFO
>> {org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor}
>> - Member terminated: [service] cisco-sample-vm2 [cluster]
>> cisco-sample-vm2.cisco-sample- [member]
>> cisco-sample-vm2.cisco-sample-f817ea17-6248-495f-abfa-4fef92352e68
>>
>> Regards,
>> -Jeffrey
>>
>
Re: Subscription removed from subscription list before VMs terminated
by CC
Posted by Udara Liyanage <ud...@wso2.com>.
Hi Jeffrey,
Cartridge subscription is no longer in Stratos now, instead deploy
composite application.
cartridge/list/subscribed endpoint is deprecated in 410.
Are you talking about Stratos 400 or current master branch?
Touched, not typed. Erroneous words are a feature, not a typo.
On Nov 21, 2014 2:13 AM, "Jeffrey Nguyen (jeffrngu)" <je...@cisco.com>
wrote:
> Hi Devs,
>
> I noticed when we unsubscribe a cartridge, after Stratos puts
> subscription in maintenance mode and removes it from subscription list, it
> takes 30 seconds for CC to start terminating the instance(s) and about 5
> seconds for it to confirm the instance was terminated. Please see log
> snippet below for sequence of events.
>
> We're using the Rest API call with resource URL "
> *admin/cartridge/list/subscribed*" to determine if all instances spawned
> as a result of subscriptions have been killed. This is apparently not
> reliable due to the delays in processing I mentioned earlier.
>
> Questions:
>
> - Can we keep the subscription that in maintenance mode in the
> subscription list and wait until the spawn instances have been terminated
> before removing it from subscription list?
> - What causes the 30 second delay between subscription being in
> maintenance mode and CC picking up the status and starts terminating the
> spawned instances?
> - Is there another REST API we can use to reliably tell if all the
> spawn instances have been terminated?
>
>
> Snippet from wso2carbon log:
>
> TID: [0] [STRATOS] [2014-11-20 11:05:42,755] INFO
> {org.apache.stratos.autoscaler.message.receiver.topology.AutoscalerTopologyEventReceiver}
> - Event received: ClusterMaintenanceModeEvent
> [serviceName=cisco-sample-vm2, clusterStatus=In_Maintenance]
>
>
> TID: [0] [STRATOS] [2014-11-20 11:05:42,790] INFO
> {org.apache.stratos.manager.subscription.utils.CartridgeSubscriptionUtils}
> - Publishing tenant un-subscribed event: [tenant-id] -1234 [service]
> cisco-sample-vm2
>
>
>
> TID: [0] [STRATOS] [2014-11-20 11:06:12,764] INFO
> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
> Starting to terminate an instance with member id :
> cisco-sample-vm2.cisco-sample-f817ea17-6248-495f-abfa-4fef92352e68 in
> partition id: RegionOne-Core of cluster id: cisco-sample-vm2.cisco-sample-
> and of cartridge type: cisco-sample-vm2
>
>
> TID: [0] [STRATOS] [2014-11-20 11:06:17,329] INFO
> {org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor}
> - Member terminated: [service] cisco-sample-vm2 [cluster]
> cisco-sample-vm2.cisco-sample- [member]
> cisco-sample-vm2.cisco-sample-f817ea17-6248-495f-abfa-4fef92352e68
>
> Regards,
> -Jeffrey
>