You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stratos.apache.org by "Martin Eppel (meppel)" <me...@cisco.com> on 2015/06/30 22:57:38 UTC
application removal issue
Hi Reka,
Here is the issue with application removal in one of our use cases:
In the scenario there are 5 applications deployed, cartridge-proxy, di-000-001, di-000-002, di-000-003, di-000-004
The scenario tries to remove application , di-000-001, di-000-002, di-000-003, di-000-004.
Removing , di-000-001 succeeds but, at least at first sight, removing di-000-002 ends up with the application di-000-002 stuck in "undeploying" mode and the cartridge in maintenance mode. However, as it turned out, di-000-002 is being removed successfully but it took 32 minutes.
I think the question is why does it take that long, is it expected and what are the conditions for it to take that long (in comparison, removing di-000-002 only took a few seconds).
Also, the application were removed using the force flag. Shouldn't force remove go quick in any case ?
Removing di-000-001:
"
TID: [0] [STRATOS] [2015-06-30 15:52:09,073] INFO {org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} - Application undeployment process started: [application-id] di-000-001
...
TID: [0] [STRATOS] [2015-06-30 15:52:12,982] INFO {org.apache.stratos.autoscaler.applications.topic.ApplicationBuilder} - Application un-deployed successfully: [application-id] di-000-001
"
Removing di-000-002
"
...
TID: [0] [STRATOS] [2015-06-30 15:52:50,383] INFO {org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} - Application undeployment process started: [application-id] di-000-002
...
TID: [0] [STRATOS] [2015-06-30 16:24:04,083] INFO {org.apache.stratos.autoscaler.applications.topic.ApplicationBuilder} - Application un-deployed successfully: [application-id] di-000-002
...
"
Application jsons are listed below, wso2carbon.log is attached.
Thanks
Martin
stratos> list-applications
Applications found:
+-----------------+-----------------+-------------+
| Application ID | Alias | Status |
+-----------------+-----------------+-------------+
| cartridge-proxy | cartridge-proxy | Deployed |
+-----------------+-----------------+-------------+
| di-000-002 | di-000-002 | Undeploying |
+-----------------+-----------------+-------------+
| di-000-003 | di-000-003 | Deployed |
+-----------------+-----------------+-------------+
| di-000-004 | di-000-004 | Deployed |
Waiting for a long time:
+-----------------+-----------------+----------+
| Application ID | Alias | Status |
+-----------------+-----------------+----------+
| cartridge-proxy | cartridge-proxy | Deployed |
+-----------------+-----------------+----------+
| di-000-002 | di-000-002 | Created |
+-----------------+-----------------+----------+
| di-000-003 | di-000-003 | Deployed |
+-----------------+-----------------+----------+
| di-000-004 | di-000-004 | Deployed |
+-----------------+-----------------+----------+
See below json files for the applications:
stratos> describe-application di-000-001
Application not found: di-000-001
stratos> describe-application di-000-002
Application: di-000-002
{
"applicationId": "di-000-002",
"multiTenant": false,
"alias": "di-000-002",
"status": "Undeploying",
"components": {
"cartridges": [
{
"type": "cisco-qvpc-cf-02-0",
"cartridgeMin": 1,
"cartridgeMax": 1,
"subscribableInfo": {
"alias": "di-000-002",
"deploymentPolicy": "static-1",
"autoscalingPolicy": "economyPolicy",
"maxMembers": 0,
"minMembers": 0,
"artifactRepository": {
"alias": "di-000-002",
"privateRepo": true,
"repoUrl": "http://octl.qmog.cisco.com:10080/git/default.git",
"repoUsername": "user",
"repoPassword": "password"
},
"property": [
{
"name": "payload_parameter.VOLUME_INFO",
"value": "di-000-002:ca6bbb1b-47a5-44c5-af41-6264eb2bdcfd"
}
],
"persistence": {
"isRequired": false
}
}
}
]
}
}
////
Application: di-000-003
{
"applicationId": "di-000-003",
"multiTenant": false,
"alias": "di-000-003",
"status": "Deployed",
"components": {
"cartridges": [
{
"type": "cisco-qvpc-sf-0",
"cartridgeMin": 1,
"cartridgeMax": 1,
"subscribableInfo": {
"alias": "di-000-003",
"deploymentPolicy": "static-1",
"autoscalingPolicy": "economyPolicy",
"maxMembers": 0,
"minMembers": 0,
"artifactRepository": {
"alias": "di-000-003",
"privateRepo": true,
"repoUrl": "http://octl.qmog.cisco.com:10080/git/default.git",
"repoUsername": "user",
"repoPassword": "password"
}
}
}
]
}
}
/////
stratos> describe-application di-000-004
Application: di-000-004
{
"applicationId": "di-000-004",
"multiTenant": false,
"alias": "di-000-004",
"status": "Deployed",
"components": {
"cartridges": [
{
"type": "cisco-qvpc-sf-0",
"cartridgeMin": 1,
"cartridgeMax": 1,
"subscribableInfo": {
"alias": "di-000-004",
"deploymentPolicy": "static-1",
"autoscalingPolicy": "economyPolicy",
"maxMembers": 0,
"minMembers": 0,
"artifactRepository": {
"alias": "di-000-004",
"privateRepo": true,
"repoUrl": "http://octl.qmog.cisco.com:10080/git/default.git",
"repoUsername": "user",
"repoPassword": "password"
}
}
}
]
}
}
////
Application: cartridge-proxy
{
"applicationId": "cartridge-proxy",
"multiTenant": false,
"alias": "cartridge-proxy",
"status": "Deployed",
"components": {
"cartridges": [
{
"type": "cartridge-proxy",
"cartridgeMin": 1,
"cartridgeMax": 1,
"subscribableInfo": {
"alias": "cartridge-proxy",
"deploymentPolicy": "static-1",
"autoscalingPolicy": "economyPolicy",
"maxMembers": 0,
"minMembers": 0,
"artifactRepository": {
"alias": "cartridge-proxy",
"privateRepo": true,
"repoUrl": "http://octl.qmog.cisco.com:10080/git/default.git",
"repoUsername": "user",
"repoPassword": "password"
}
}
}
]
}
}
Re: application removal issue
Posted by Reka Thirunavukkarasu <re...@wso2.com>.
Thanks for the update Martin..It's good that the issue is identified.!
Thanks,
Reka
On Thu, Jul 2, 2015 at 8:40 PM, Martin Eppel (meppel) <me...@cisco.com>
wrote:
> Hi Reka,
>
>
>
> Just a heads up, the issue turned out to be an issue with the clean up
> script. Thanks for the pointers,
>
>
>
> Regards
>
>
>
> Martin
>
>
>
> *From:* Reka Thirunavukkarasu [mailto:reka@wso2.com]
> *Sent:* Wednesday, July 01, 2015 2:08 AM
> *To:* Martin Eppel (meppel)
> *Cc:* dev@stratos.apache.org; Ryan Du Plessis (rdupless)
> *Subject:* Re: application removal issue
>
>
>
> Hi Martin,
>
> As i have gone through your log, please see below with my findings. There
> seems to be not anything to do with undeployment process for this
> particular scenario as it is not a bug in undeployment. We can fix it by
> changing a configuration. Also, we will need to investigate at cartridge
> agent side as well.
>
> 1. After you invoke the undeploy, member sent the in-maintenance event.
>
> 2. Autoscaler received it and mark the member as terminatingPending
>
> 3. Stratos is waiting for ReadToShutdown event from the member. But member
> didn't send this event due to network issue or some cleanup process issue
>
> 4. Autoscaler has a pendingTerminationMemberExpiryTimeout in
> autoscaler.xml(SRTATOS-HOME/repository/conf). I believe that this timeout
> is 30min(according to the log below) in your setup. After this timeout,
> autoscaler will move this member to obsolete list. So that the member will
> get terminated immediately. Please see the log as i found:
>
> TID: [0] [STRATOS] [2015-06-30 16:22:56,505] INFO
> {org.apache.stratos.autoscaler.context.partition.ClusterLevelPartitionContext$TerminationPendingMemberWatcher}
> - * Termination pending state of member is expired, member will be moved
> to obsolete list* [termination pending member]
> di-000-002.di-000-002.cisco-qvpc-cf-02-0.domain2b83bd51-a85a-473b-b143-adc2c8feac94
> [expiry time] 1800000 [cluster]
> di-000-002.di-000-002.cisco-qvpc-cf-02-0.domain [cluster instance]
> di-000-002-1
>
> 5. Member got terminated successfully. Hence cluster and then application
> also got terminated successfully.
>
> As the solution, can you try to reduce
> pendingTerminationMemberExpiryTimeout according to your suitable value?
> Also, if you detect this issue again, it is better to get the cartridge
> agent logs as well. So that we can try to find out why cartridge agent
> failed to send the ReadyToShutdown event.
>
> Thanks,
>
> Reka
>
>
>
> On Wed, Jul 1, 2015 at 2:27 AM, Martin Eppel (meppel) <me...@cisco.com>
> wrote:
>
> Hi Reka,
>
>
>
> Here is the issue with application removal in one of our use cases:
>
>
>
> In the scenario there are 5 applications deployed, cartridge-proxy,
> di-000-001, di-000-002, di-000-003, di-000-004
>
>
>
> The scenario tries to remove application , di-000-001, di-000-002,
> di-000-003, di-000-004.
>
>
>
> Removing , di-000-001 succeeds but, at least at first sight, removing
> di-000-002 ends up with the application di-000-002 stuck in “undeploying”
> mode and the cartridge in maintenance mode. However, as it turned out,
> di-000-002 is being removed successfully but it took *32 minutes*.
>
> I think the question is why does it take that long, is it expected and
> what are the conditions for it to take that long (in comparison, removing
> di-000-002 only took a few seconds).
>
> Also, the application were removed using the force flag. Shouldn’t force
> remove go quick in any case ?
>
>
>
> Removing di-000-001:
>
> “
>
> *TID: [0] [STRATOS] [2015-06-30 15:52:09,073] INFO
> {org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} -
> Application undeployment process started: [application-id] di-000-001*
>
> *…*
>
> *TID: [0] [STRATOS] [2015-06-30 15:52:12,982] INFO
> {org.apache.stratos.autoscaler.applications.topic.ApplicationBuilder} -
> Application un-deployed successfully: [application-id] di-000-001*
>
> “
>
>
>
> Removing di-000-002
>
> “
>
> …
>
> *TID: [0] [STRATOS] [2015-06-30 15:52:50,383] INFO
> {org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} -
> Application undeployment process started: [application-id] di-000-002*
>
> *…*
>
> *TID: [0] [STRATOS] [2015-06-30 16:24:04,083] INFO
> {org.apache.stratos.autoscaler.applications.topic.ApplicationBuilder} -
> Application un-deployed successfully: [application-id] di-000-002*
>
> *…*
>
> “
>
>
>
> Application jsons are listed below, wso2carbon.log is attached.
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
>
>
> stratos> list-applications
>
> Applications found:
>
> +-----------------+-----------------+-------------+
>
> | Application ID | Alias | Status |
>
> +-----------------+-----------------+-------------+
>
> | cartridge-proxy | cartridge-proxy | Deployed |
>
> +-----------------+-----------------+-------------+
>
> | di-000-002 | di-000-002 | Undeploying |
>
> +-----------------+-----------------+-------------+
>
> | di-000-003 | di-000-003 | Deployed |
>
> +-----------------+-----------------+-------------+
>
> | di-000-004 | di-000-004 | Deployed |
>
>
>
>
>
>
>
> Waiting for a long time:
>
>
>
> +-----------------+-----------------+----------+
>
> | Application ID | Alias | Status |
>
> +-----------------+-----------------+----------+
>
> | cartridge-proxy | cartridge-proxy | Deployed |
>
> +-----------------+-----------------+----------+
>
> | di-000-002 | di-000-002 | Created |
>
> +-----------------+-----------------+----------+
>
> | di-000-003 | di-000-003 | Deployed |
>
> +-----------------+-----------------+----------+
>
> | di-000-004 | di-000-004 | Deployed |
>
> +-----------------+-----------------+----------+
>
>
>
>
>
> See below json files for the applications:
>
>
>
> stratos> describe-application di-000-001
>
> Application not found: di-000-001
>
>
>
>
>
> stratos> describe-application di-000-002
>
> Application: di-000-002
>
> {
>
> "applicationId": "di-000-002",
>
> "multiTenant": false,
>
> "alias": "di-000-002",
>
> "status": "Undeploying",
>
> "components": {
>
> "cartridges": [
>
> {
>
> "type": "cisco-qvpc-cf-02-0",
>
> "cartridgeMin": 1,
>
> "cartridgeMax": 1,
>
> "subscribableInfo": {
>
> "alias": "di-000-002",
>
> "deploymentPolicy": "static-1",
>
> "autoscalingPolicy": "economyPolicy",
>
> "maxMembers": 0,
>
> "minMembers": 0,
>
> "artifactRepository": {
>
> "alias": "di-000-002",
>
> "privateRepo": true,
>
> "repoUrl": "http://octl.qmog.cisco.com:10080/git/default.git",
>
> "repoUsername": "user",
>
> "repoPassword": "password"
>
> },
>
> "property": [
>
> {
>
> "name": "payload_parameter.VOLUME_INFO",
>
> "value": "di-000-002:ca6bbb1b-47a5-44c5-af41-6264eb2bdcfd"
>
> }
>
> ],
>
> "persistence": {
>
> "isRequired": false
>
> }
>
> }
>
> }
>
> ]
>
> }
>
> }
>
>
>
> ////
>
> Application: di-000-003
>
> {
>
> "applicationId": "di-000-003",
>
> "multiTenant": false,
>
> "alias": "di-000-003",
>
> "status": "Deployed",
>
> "components": {
>
> "cartridges": [
>
> {
>
> "type": "cisco-qvpc-sf-0",
>
> "cartridgeMin": 1,
>
> "cartridgeMax": 1,
>
> "subscribableInfo": {
>
> "alias": "di-000-003",
>
> "deploymentPolicy": "static-1",
>
> "autoscalingPolicy": "economyPolicy",
>
> "maxMembers": 0,
>
> "minMembers": 0,
>
> "artifactRepository": {
>
> "alias": "di-000-003",
>
> "privateRepo": true,
>
> "repoUrl": "http://octl.qmog.cisco.com:10080/git/default.git",
>
> "repoUsername": "user",
>
> "repoPassword": "password"
>
> }
>
> }
>
> }
>
> ]
>
> }
>
> }
>
>
>
> /////
>
> stratos> describe-application di-000-004
>
> Application: di-000-004
>
> {
>
> "applicationId": "di-000-004",
>
> "multiTenant": false,
>
> "alias": "di-000-004",
>
> "status": "Deployed",
>
> "components": {
>
> "cartridges": [
>
> {
>
> "type": "cisco-qvpc-sf-0",
>
> "cartridgeMin": 1,
>
> "cartridgeMax": 1,
>
> "subscribableInfo": {
>
> "alias": "di-000-004",
>
> "deploymentPolicy": "static-1",
>
> "autoscalingPolicy": "economyPolicy",
>
> "maxMembers": 0,
>
> "minMembers": 0,
>
> "artifactRepository": {
>
> "alias": "di-000-004",
>
> "privateRepo": true,
>
> "repoUrl": "http://octl.qmog.cisco.com:10080/git/default.git",
>
> "repoUsername": "user",
>
> "repoPassword": "password"
>
> }
>
> }
>
> }
>
> ]
>
> }
>
> }
>
>
>
> ////
>
>
>
> Application: cartridge-proxy
>
> {
>
> "applicationId": "cartridge-proxy",
>
> "multiTenant": false,
>
> "alias": "cartridge-proxy",
>
> "status": "Deployed",
>
> "components": {
>
> "cartridges": [
>
> {
>
> "type": "cartridge-proxy",
>
> "cartridgeMin": 1,
>
> "cartridgeMax": 1,
>
> "subscribableInfo": {
>
> "alias": "cartridge-proxy",
>
> "deploymentPolicy": "static-1",
>
> "autoscalingPolicy": "economyPolicy",
>
> "maxMembers": 0,
>
> "minMembers": 0,
>
> "artifactRepository": {
>
> "alias": "cartridge-proxy",
>
> "privateRepo": true,
>
> "repoUrl": "http://octl.qmog.cisco.com:10080/git/default.git",
>
> "repoUsername": "user",
>
> "repoPassword": "password"
>
> }
>
> }
>
> }
>
> ]
>
> }
>
> }
>
>
>
>
>
>
>
>
>
>
>
>
> --
>
> Reka Thirunavukkarasu
> Senior Software Engineer,
> WSO2, Inc.:http://wso2.com,
>
> Mobile: +94776442007
>
>
>
--
Reka Thirunavukkarasu
Senior Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007
RE: application removal issue
Posted by "Martin Eppel (meppel)" <me...@cisco.com>.
Hi Reka,
Just a heads up, the issue turned out to be an issue with the clean up script. Thanks for the pointers,
Regards
Martin
From: Reka Thirunavukkarasu [mailto:reka@wso2.com]
Sent: Wednesday, July 01, 2015 2:08 AM
To: Martin Eppel (meppel)
Cc: dev@stratos.apache.org; Ryan Du Plessis (rdupless)
Subject: Re: application removal issue
Hi Martin,
As i have gone through your log, please see below with my findings. There seems to be not anything to do with undeployment process for this particular scenario as it is not a bug in undeployment. We can fix it by changing a configuration. Also, we will need to investigate at cartridge agent side as well.
1. After you invoke the undeploy, member sent the in-maintenance event.
2. Autoscaler received it and mark the member as terminatingPending
3. Stratos is waiting for ReadToShutdown event from the member. But member didn't send this event due to network issue or some cleanup process issue
4. Autoscaler has a pendingTerminationMemberExpiryTimeout in autoscaler.xml(SRTATOS-HOME/repository/conf). I believe that this timeout is 30min(according to the log below) in your setup. After this timeout, autoscaler will move this member to obsolete list. So that the member will get terminated immediately. Please see the log as i found:
TID: [0] [STRATOS] [2015-06-30 16:22:56,505] INFO {org.apache.stratos.autoscaler.context.partition.ClusterLevelPartitionContext$TerminationPendingMemberWatcher} - Termination pending state of member is expired, member will be moved to obsolete list [termination pending member] di-000-002.di-000-002.cisco-qvpc-cf-02-0.domain2b83bd51-a85a-473b-b143-adc2c8feac94 [expiry time] 1800000 [cluster] di-000-002.di-000-002.cisco-qvpc-cf-02-0.domain [cluster instance] di-000-002-1
5. Member got terminated successfully. Hence cluster and then application also got terminated successfully.
As the solution, can you try to reduce pendingTerminationMemberExpiryTimeout according to your suitable value? Also, if you detect this issue again, it is better to get the cartridge agent logs as well. So that we can try to find out why cartridge agent failed to send the ReadyToShutdown event.
Thanks,
Reka
On Wed, Jul 1, 2015 at 2:27 AM, Martin Eppel (meppel) <me...@cisco.com>> wrote:
Hi Reka,
Here is the issue with application removal in one of our use cases:
In the scenario there are 5 applications deployed, cartridge-proxy, di-000-001, di-000-002, di-000-003, di-000-004
The scenario tries to remove application , di-000-001, di-000-002, di-000-003, di-000-004.
Removing , di-000-001 succeeds but, at least at first sight, removing di-000-002 ends up with the application di-000-002 stuck in “undeploying” mode and the cartridge in maintenance mode. However, as it turned out, di-000-002 is being removed successfully but it took 32 minutes.
I think the question is why does it take that long, is it expected and what are the conditions for it to take that long (in comparison, removing di-000-002 only took a few seconds).
Also, the application were removed using the force flag. Shouldn’t force remove go quick in any case ?
Removing di-000-001:
“
TID: [0] [STRATOS] [2015-06-30 15:52:09,073] INFO {org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} - Application undeployment process started: [application-id] di-000-001
…
TID: [0] [STRATOS] [2015-06-30 15:52:12,982] INFO {org.apache.stratos.autoscaler.applications.topic.ApplicationBuilder} - Application un-deployed successfully: [application-id] di-000-001
“
Removing di-000-002
“
…
TID: [0] [STRATOS] [2015-06-30 15:52:50,383] INFO {org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} - Application undeployment process started: [application-id] di-000-002
…
TID: [0] [STRATOS] [2015-06-30 16:24:04,083] INFO {org.apache.stratos.autoscaler.applications.topic.ApplicationBuilder} - Application un-deployed successfully: [application-id] di-000-002
…
“
Application jsons are listed below, wso2carbon.log is attached.
Thanks
Martin
stratos> list-applications
Applications found:
+-----------------+-----------------+-------------+
| Application ID | Alias | Status |
+-----------------+-----------------+-------------+
| cartridge-proxy | cartridge-proxy | Deployed |
+-----------------+-----------------+-------------+
| di-000-002 | di-000-002 | Undeploying |
+-----------------+-----------------+-------------+
| di-000-003 | di-000-003 | Deployed |
+-----------------+-----------------+-------------+
| di-000-004 | di-000-004 | Deployed |
Waiting for a long time:
+-----------------+-----------------+----------+
| Application ID | Alias | Status |
+-----------------+-----------------+----------+
| cartridge-proxy | cartridge-proxy | Deployed |
+-----------------+-----------------+----------+
| di-000-002 | di-000-002 | Created |
+-----------------+-----------------+----------+
| di-000-003 | di-000-003 | Deployed |
+-----------------+-----------------+----------+
| di-000-004 | di-000-004 | Deployed |
+-----------------+-----------------+----------+
See below json files for the applications:
stratos> describe-application di-000-001
Application not found: di-000-001
stratos> describe-application di-000-002
Application: di-000-002
{
"applicationId": "di-000-002",
"multiTenant": false,
"alias": "di-000-002",
"status": "Undeploying",
"components": {
"cartridges": [
{
"type": "cisco-qvpc-cf-02-0",
"cartridgeMin": 1,
"cartridgeMax": 1,
"subscribableInfo": {
"alias": "di-000-002",
"deploymentPolicy": "static-1",
"autoscalingPolicy": "economyPolicy",
"maxMembers": 0,
"minMembers": 0,
"artifactRepository": {
"alias": "di-000-002",
"privateRepo": true,
"repoUrl": "http://octl.qmog.cisco.com:10080/git/default.git",
"repoUsername": "user",
"repoPassword": "password"
},
"property": [
{
"name": "payload_parameter.VOLUME_INFO",
"value": "di-000-002:ca6bbb1b-47a5-44c5-af41-6264eb2bdcfd"
}
],
"persistence": {
"isRequired": false
}
}
}
]
}
}
////
Application: di-000-003
{
"applicationId": "di-000-003",
"multiTenant": false,
"alias": "di-000-003",
"status": "Deployed",
"components": {
"cartridges": [
{
"type": "cisco-qvpc-sf-0",
"cartridgeMin": 1,
"cartridgeMax": 1,
"subscribableInfo": {
"alias": "di-000-003",
"deploymentPolicy": "static-1",
"autoscalingPolicy": "economyPolicy",
"maxMembers": 0,
"minMembers": 0,
"artifactRepository": {
"alias": "di-000-003",
"privateRepo": true,
"repoUrl": "http://octl.qmog.cisco.com:10080/git/default.git",
"repoUsername": "user",
"repoPassword": "password"
}
}
}
]
}
}
/////
stratos> describe-application di-000-004
Application: di-000-004
{
"applicationId": "di-000-004",
"multiTenant": false,
"alias": "di-000-004",
"status": "Deployed",
"components": {
"cartridges": [
{
"type": "cisco-qvpc-sf-0",
"cartridgeMin": 1,
"cartridgeMax": 1,
"subscribableInfo": {
"alias": "di-000-004",
"deploymentPolicy": "static-1",
"autoscalingPolicy": "economyPolicy",
"maxMembers": 0,
"minMembers": 0,
"artifactRepository": {
"alias": "di-000-004",
"privateRepo": true,
"repoUrl": "http://octl.qmog.cisco.com:10080/git/default.git",
"repoUsername": "user",
"repoPassword": "password"
}
}
}
]
}
}
////
Application: cartridge-proxy
{
"applicationId": "cartridge-proxy",
"multiTenant": false,
"alias": "cartridge-proxy",
"status": "Deployed",
"components": {
"cartridges": [
{
"type": "cartridge-proxy",
"cartridgeMin": 1,
"cartridgeMax": 1,
"subscribableInfo": {
"alias": "cartridge-proxy",
"deploymentPolicy": "static-1",
"autoscalingPolicy": "economyPolicy",
"maxMembers": 0,
"minMembers": 0,
"artifactRepository": {
"alias": "cartridge-proxy",
"privateRepo": true,
"repoUrl": "http://octl.qmog.cisco.com:10080/git/default.git",
"repoUsername": "user",
"repoPassword": "password"
}
}
}
]
}
}
--
Reka Thirunavukkarasu
Senior Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007
Re: application removal issue
Posted by Reka Thirunavukkarasu <re...@wso2.com>.
Hi Martin,
As i have gone through your log, please see below with my findings. There
seems to be not anything to do with undeployment process for this
particular scenario as it is not a bug in undeployment. We can fix it by
changing a configuration. Also, we will need to investigate at cartridge
agent side as well.
1. After you invoke the undeploy, member sent the in-maintenance event.
2. Autoscaler received it and mark the member as terminatingPending
3. Stratos is waiting for ReadToShutdown event from the member. But member
didn't send this event due to network issue or some cleanup process issue
4. Autoscaler has a pendingTerminationMemberExpiryTimeout in
autoscaler.xml(SRTATOS-HOME/repository/conf). I believe that this timeout
is 30min(according to the log below) in your setup. After this timeout,
autoscaler will move this member to obsolete list. So that the member will
get terminated immediately. Please see the log as i found:
TID: [0] [STRATOS] [2015-06-30 16:22:56,505] INFO
{org.apache.stratos.autoscaler.context.partition.ClusterLevelPartitionContext$TerminationPendingMemberWatcher}
- *Termination pending state of member is expired, member will be moved to
obsolete list* [termination pending member]
di-000-002.di-000-002.cisco-qvpc-cf-02-0.domain2b83bd51-a85a-473b-b143-adc2c8feac94
[expiry time] 1800000 [cluster]
di-000-002.di-000-002.cisco-qvpc-cf-02-0.domain [cluster instance]
di-000-002-1
5. Member got terminated successfully. Hence cluster and then application
also got terminated successfully.
As the solution, can you try to reduce
pendingTerminationMemberExpiryTimeout according to your suitable value?
Also, if you detect this issue again, it is better to get the cartridge
agent logs as well. So that we can try to find out why cartridge agent
failed to send the ReadyToShutdown event.
Thanks,
Reka
On Wed, Jul 1, 2015 at 2:27 AM, Martin Eppel (meppel) <me...@cisco.com>
wrote:
> Hi Reka,
>
>
>
> Here is the issue with application removal in one of our use cases:
>
>
>
> In the scenario there are 5 applications deployed, cartridge-proxy,
> di-000-001, di-000-002, di-000-003, di-000-004
>
>
>
> The scenario tries to remove application , di-000-001, di-000-002,
> di-000-003, di-000-004.
>
>
>
> Removing , di-000-001 succeeds but, at least at first sight, removing
> di-000-002 ends up with the application di-000-002 stuck in “undeploying”
> mode and the cartridge in maintenance mode. However, as it turned out,
> di-000-002 is being removed successfully but it took *32 minutes*.
>
> I think the question is why does it take that long, is it expected and
> what are the conditions for it to take that long (in comparison, removing
> di-000-002 only took a few seconds).
>
> Also, the application were removed using the force flag. Shouldn’t force
> remove go quick in any case ?
>
>
>
> Removing di-000-001:
>
> “
>
> *TID: [0] [STRATOS] [2015-06-30 15:52:09,073] INFO
> {org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} -
> Application undeployment process started: [application-id] di-000-001*
>
> *…*
>
> *TID: [0] [STRATOS] [2015-06-30 15:52:12,982] INFO
> {org.apache.stratos.autoscaler.applications.topic.ApplicationBuilder} -
> Application un-deployed successfully: [application-id] di-000-001*
>
> “
>
>
>
> Removing di-000-002
>
> “
>
> …
>
> *TID: [0] [STRATOS] [2015-06-30 15:52:50,383] INFO
> {org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} -
> Application undeployment process started: [application-id] di-000-002*
>
> *…*
>
> *TID: [0] [STRATOS] [2015-06-30 16:24:04,083] INFO
> {org.apache.stratos.autoscaler.applications.topic.ApplicationBuilder} -
> Application un-deployed successfully: [application-id] di-000-002*
>
> *…*
>
> “
>
>
>
> Application jsons are listed below, wso2carbon.log is attached.
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
>
>
> stratos> list-applications
>
> Applications found:
>
> +-----------------+-----------------+-------------+
>
> | Application ID | Alias | Status |
>
> +-----------------+-----------------+-------------+
>
> | cartridge-proxy | cartridge-proxy | Deployed |
>
> +-----------------+-----------------+-------------+
>
> | di-000-002 | di-000-002 | Undeploying |
>
> +-----------------+-----------------+-------------+
>
> | di-000-003 | di-000-003 | Deployed |
>
> +-----------------+-----------------+-------------+
>
> | di-000-004 | di-000-004 | Deployed |
>
>
>
>
>
>
>
> Waiting for a long time:
>
>
>
> +-----------------+-----------------+----------+
>
> | Application ID | Alias | Status |
>
> +-----------------+-----------------+----------+
>
> | cartridge-proxy | cartridge-proxy | Deployed |
>
> +-----------------+-----------------+----------+
>
> | di-000-002 | di-000-002 | Created |
>
> +-----------------+-----------------+----------+
>
> | di-000-003 | di-000-003 | Deployed |
>
> +-----------------+-----------------+----------+
>
> | di-000-004 | di-000-004 | Deployed |
>
> +-----------------+-----------------+----------+
>
>
>
>
>
> See below json files for the applications:
>
>
>
> stratos> describe-application di-000-001
>
> Application not found: di-000-001
>
>
>
>
>
> stratos> describe-application di-000-002
>
> Application: di-000-002
>
> {
>
> "applicationId": "di-000-002",
>
> "multiTenant": false,
>
> "alias": "di-000-002",
>
> "status": "Undeploying",
>
> "components": {
>
> "cartridges": [
>
> {
>
> "type": "cisco-qvpc-cf-02-0",
>
> "cartridgeMin": 1,
>
> "cartridgeMax": 1,
>
> "subscribableInfo": {
>
> "alias": "di-000-002",
>
> "deploymentPolicy": "static-1",
>
> "autoscalingPolicy": "economyPolicy",
>
> "maxMembers": 0,
>
> "minMembers": 0,
>
> "artifactRepository": {
>
> "alias": "di-000-002",
>
> "privateRepo": true,
>
> "repoUrl": "http://octl.qmog.cisco.com:10080/git/default.git",
>
> "repoUsername": "user",
>
> "repoPassword": "password"
>
> },
>
> "property": [
>
> {
>
> "name": "payload_parameter.VOLUME_INFO",
>
> "value": "di-000-002:ca6bbb1b-47a5-44c5-af41-6264eb2bdcfd"
>
> }
>
> ],
>
> "persistence": {
>
> "isRequired": false
>
> }
>
> }
>
> }
>
> ]
>
> }
>
> }
>
>
>
> ////
>
> Application: di-000-003
>
> {
>
> "applicationId": "di-000-003",
>
> "multiTenant": false,
>
> "alias": "di-000-003",
>
> "status": "Deployed",
>
> "components": {
>
> "cartridges": [
>
> {
>
> "type": "cisco-qvpc-sf-0",
>
> "cartridgeMin": 1,
>
> "cartridgeMax": 1,
>
> "subscribableInfo": {
>
> "alias": "di-000-003",
>
> "deploymentPolicy": "static-1",
>
> "autoscalingPolicy": "economyPolicy",
>
> "maxMembers": 0,
>
> "minMembers": 0,
>
> "artifactRepository": {
>
> "alias": "di-000-003",
>
> "privateRepo": true,
>
> "repoUrl": "http://octl.qmog.cisco.com:10080/git/default.git",
>
> "repoUsername": "user",
>
> "repoPassword": "password"
>
> }
>
> }
>
> }
>
> ]
>
> }
>
> }
>
>
>
> /////
>
> stratos> describe-application di-000-004
>
> Application: di-000-004
>
> {
>
> "applicationId": "di-000-004",
>
> "multiTenant": false,
>
> "alias": "di-000-004",
>
> "status": "Deployed",
>
> "components": {
>
> "cartridges": [
>
> {
>
> "type": "cisco-qvpc-sf-0",
>
> "cartridgeMin": 1,
>
> "cartridgeMax": 1,
>
> "subscribableInfo": {
>
> "alias": "di-000-004",
>
> "deploymentPolicy": "static-1",
>
> "autoscalingPolicy": "economyPolicy",
>
> "maxMembers": 0,
>
> "minMembers": 0,
>
> "artifactRepository": {
>
> "alias": "di-000-004",
>
> "privateRepo": true,
>
> "repoUrl": "http://octl.qmog.cisco.com:10080/git/default.git",
>
> "repoUsername": "user",
>
> "repoPassword": "password"
>
> }
>
> }
>
> }
>
> ]
>
> }
>
> }
>
>
>
> ////
>
>
>
> Application: cartridge-proxy
>
> {
>
> "applicationId": "cartridge-proxy",
>
> "multiTenant": false,
>
> "alias": "cartridge-proxy",
>
> "status": "Deployed",
>
> "components": {
>
> "cartridges": [
>
> {
>
> "type": "cartridge-proxy",
>
> "cartridgeMin": 1,
>
> "cartridgeMax": 1,
>
> "subscribableInfo": {
>
> "alias": "cartridge-proxy",
>
> "deploymentPolicy": "static-1",
>
> "autoscalingPolicy": "economyPolicy",
>
> "maxMembers": 0,
>
> "minMembers": 0,
>
> "artifactRepository": {
>
> "alias": "cartridge-proxy",
>
> "privateRepo": true,
>
> "repoUrl": "http://octl.qmog.cisco.com:10080/git/default.git",
>
> "repoUsername": "user",
>
> "repoPassword": "password"
>
> }
>
> }
>
> }
>
> ]
>
> }
>
> }
>
>
>
>
>
>
>
>
>
--
Reka Thirunavukkarasu
Senior Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007