You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stratos.apache.org by "Martin Eppel (meppel)" <me...@cisco.com> on 2015/04/01 03:24:47 UTC
RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going
active ... was RE: Stratos 4.1.0 : Issue when deploying application with
single group and 3 cartridges, no dependencies ...
Hi Lahiru,
I retested the scenario to make sure proper unique aliases are provided and they are, please double check the provided artifacts. Issue remains the same, one VM gets stuck in INITIALIZED until terminated and restarted. As pointed out before, the cartridge alias in the cartridge agent is the same in 2 cartridges (although provided aliases in the application json is unique). After restarting the faulty member cartridge agent alias has the proper alias and goes into active.
I also provided the cartridge-agent logs for the member which is faulty, before restart and after restart (cartridge-agent-70~INITIALIZED, ACTIVE)
Please see attached logs, artifacts,
Thanks
Martin
From: Martin Eppel (meppel)
Sent: Tuesday, March 31, 2015 1:22 PM
To: dev@stratos.apache.org
Subject: RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
Lahiru,
The aliases are all unique, please check the the application json, and as I said in my analysis it I think this is what I saw but I can’t be sure. So this is not an issue of starting the application with duplicate aliases but rather something else ,
Thanks
Martin
From: Lahiru Sandaruwan [mailto:lahirus@wso2.com]
Sent: Tuesday, March 31, 2015 12:58 PM
To: dev
Subject: Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
Hi Martin,
On Wed, Apr 1, 2015 at 12:34 AM, Martin Eppel (meppel) <me...@cisco.com>> wrote:
Hi Lahiru,
I tested the fix and noticed the following issue:
From the 4 cartridges 3 instances go Active, however one instance got stuck in INITIALIZED state.
One of the things I noticed while the member was stuck in INITIALIZED state is that the system received 2 active events for the same membered:
TID: [0] [STRATOS] [2015-03-31 18:10:27,645] INFO {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher} - Publishing member activated event: [service-name] c2 [cluster-id] subscription-G1-G2-G3.c2-1x1.c2.domain [cluster-instance-id] subscription-G1-G2-G3-1 [member-id] subscription-G1-G2-G3.c2-1x1.c2.domainf939e8dc-e0cc-408a-8343-ac86e479549f [network-partition-id] RegionOne [partition-id] whole-region
TID: [0] [STRATOS] [2015-03-31 18:10:31,059] INFO {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher} - Publishing member activated event: [service-name] c2 [cluster-id] subscription-G1-G2-G3.c2-1x1.c2.domain [cluster-instance-id] subscription-G1-G2-G3-1 [member-id] subscription-G1-G2-G3.c2-1x1.c2.domainf939e8dc-e0cc-408a-8343-ac86e479549f [network-partition-id] RegionOne [partition-id] whole-region
As another note, but I can’t be sure since the log got overwritten is that it appeared to me that at one point (while the cartridge was stuck in INITIALIZED state is that the cartridge alias of 2 cartridges was the same (~71, ~70: 71: CARTRIDGE_ALIAS=c2-1x1), but I can’t say for sure since when I double checked a new instance of the cartridge was created (which went ACTIVE).
Can i assume this occurred when two cartridges use same alias? May be we have to put a validation on Application as we cannot use same alias in two cartridges.
If so, could you please try to reproduce this with different aliases?
Thanks.
Eventually the system terminated the “faulty” member and the cartridge came up and went into active state:
“Pending state of member expired, member will be moved to obsolete list. [pending member] subscription-G1-G2-G3.c2-1x0.c2.domainfe9a30cf-fffc-4a6c-9b3b-f419498895b9”
…
“Member activated: [service] c2 [cluster] subscription-G1-G2-G3.c2-1x0.c2.domain [member] subscription-G1-G2-G3.c2-1x0.c2.domain44a2ae3c-9968-41de-9e10-7212ef3788c2”
Cartridge alias after all 4 cartridges went ACTIVE:
70: CARTRIDGE_ALIAS=c2-1x0
71: CARTRIDGE_ALIAS=c2-1x1
69: CARTRIDGE_ALIAS=c1-1x0
68: CARTRIDGE_ALIAS=c1-1x1
I attached the logs and artefacts
Thanks
Martin
From: Lahiru Sandaruwan [mailto:lahirus@wso2.com<ma...@wso2.com>]
Sent: Tuesday, March 31, 2015 2:24 AM
To: dev
Subject: Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
Hi Martin,
I have reproduced the issue and pushed a fix. Please verify.
Thanks.
On Tue, Mar 31, 2015 at 8:32 AM, Lahiru Sandaruwan <la...@wso2.com>> wrote:
Hi Martin,
On Tue, Mar 31, 2015 at 5:41 AM, Martin Eppel (meppel) <me...@cisco.com>> wrote:
Hi Lahiru,
I tested a new scenario, where I have 2 groups with cartridges and no start up dependencies and I see again the issue with cartridge instances being stuck in “started” state. I also noticed the following log message:
“Artifact repositories not found for application signup, artifact updated event not sent: [application-id] subscription-G1-G2-G3”.
Will have a look.
Thanks.
Please find attached the artefacts and log messages,
Thanks
Martin
From: Martin Eppel (meppel)
Sent: Monday, March 30, 2015 2:36 PM
To: dev@stratos.apache.org<ma...@stratos.apache.org>
Subject: RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
Hi Lahiru,
I tested both scenarios,
1. Single cartridge in single group and it worked, cartridge instance starts up and goes active,
2. Multiple cartridges with startup dependencies :
All instances are started up (in right sequence) and go active
JIRA https://issues.apache.org/jira/browse/STRATOS-1292[http://wikicentral.cisco.com/images/icons/linkext7.gif] is being tested and verified
Removing / Un-deploying the application caused an issue as one of the VMs did not get terminated (throwing exceptions in the log). I am not sure if this is already a known issue related to to the ongoing discussion of application removal / un-deployment.
Just in case, I zipped up the logs and attached it to the email thread (single-group-multiple-cartridges-startup.zip). Let me know if the un-deployment issue is the same already discussed or if this is a new issue ?
Same error occurred in the scenario where multiple cartridges were started up (all go active) and then application was removed, I attached the log for this scenario as well (wscarbon-application-single-group-multiple-cartridges-no-startup-remval-error.log.zip)
Thanks
Martin
From: Martin Eppel (meppel)
Sent: Monday, March 30, 2015 7:47 AM
To: dev@stratos.apache.org<ma...@stratos.apache.org>
Subject: RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
Thanks Lahiru,
Will do and let you know
Martin
From: Lahiru Sandaruwan [mailto:lahirus@wso2.com]
Sent: Monday, March 30, 2015 3:32 AM
To: dev
Subject: Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
Hi Martin,
I was able to reproduce the issue and committed a fix.
Could you please verify?
Thanks.
On Sun, Mar 29, 2015 at 5:43 PM, Lahiru Sandaruwan <la...@wso2.com>> wrote:
Hi Martin,
According to the logs attached, it seems the cartridge agent waits for the artifact updated event, and somehow it is not sent.
Looking into it.
Thanks.
On Sat, Mar 28, 2015 at 6:48 AM, Imesh Gunaratne <im...@apache.org>> wrote:
Hi Martin,
Thanks for the quick response on this!
I will use these artifacts to re-produce the problem locally and will let you know the results soon.
Thanks
On Sat, Mar 28, 2015 at 4:19 AM, Martin Eppel (meppel) <me...@cisco.com>> wrote:
Hi Imesh,
As discussed please find attached the “real” artefacts (application, cartridge group, cartridge):
Attached it as well to the jira
Thanks
Martin
From: Martin Eppel (meppel)
Sent: Friday, March 27, 2015 11:37 AM
To: dev@stratos.apache.org<ma...@stratos.apache.org>
Subject: RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
Hi Imesh,
Originally I tested with 3 cartridges but then realized that it also occurs with a single cartridge in a group – see also the JIRA for the respective notes and artefacts (single group, single cartridge)
Thanks
Martin
From: Imesh Gunaratne [mailto:imesh@apache.org]
Sent: Friday, March 27, 2015 11:34 AM
To: dev
Subject: Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
Hi Martin,
Thanks for reporting this problem. You have mentioned that there are three cartridges in the application but in the given application.json I only can see one cartridge. Are we missing anything here? Will it be possible for you to attach the complete set of Stratos artifacts that you are using in a zip file?
Thanks
Imesh
On Fri, Mar 27, 2015 at 11:51 PM, Martin Eppel (meppel) <me...@cisco.com>> wrote:
Forgot to mention, this is the same issue as reported in JIRA https://issues.apache.org/jira/browse/STRATOS-1292
Thanks
Martin
From: Martin Eppel (meppel)
Sent: Friday, March 27, 2015 10:52 AM
To: dev@stratos.apache.org<ma...@stratos.apache.org>
Subject: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
I retested with a similar scenario (single cartridges in group) with debug logs turned on, please see logs attached.
Cartridge is stuck in “Starting” state
Thanks
Martin
From: Martin Eppel (meppel)
Sent: Monday, March 23, 2015 7:15 PM
To: dev@stratos.apache.org<ma...@stratos.apache.org>
Subject: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
Hi,
I see an issue when I subscribe a single group with 3 cartridges, no dependencies – they never go active instead they all are stuck in “Starting”. I attached the server log and the cartridge-agent logs
Member status:
subscription-c1-c2-c3: applicationInstances 1, groupInstances 1, clusterInstances 3, members 3 (Starting 3)
c2: 172.16.2.18/172.28.40.69<http://172.16.2.18/172.28.40.69>: status Starting
c3: 172.16.2.19/172.28.40.70<http://172.16.2.19/172.28.40.70>: status Starting
c1: 172.16.2.17/172.28.40.68<http://172.16.2.17/172.28.40.68>: status Starting
Let me know if I should go ahead and open a JIRA
Thanks
Martin
--
Imesh Gunaratne
Technical Lead, WSO2
Committer & PMC Member, Apache Stratos
--
Imesh Gunaratne
Technical Lead, WSO2
Committer & PMC Member, Apache Stratos
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
phone: +94773325954<tel:%2B94773325954>
email: lahirus@wso2.com<ma...@wso2.com> blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
phone: +94773325954<tel:%2B94773325954>
email: lahirus@wso2.com<ma...@wso2.com> blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
phone: +94773325954<tel:%2B94773325954>
email: lahirus@wso2.com<ma...@wso2.com> blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
phone: +94773325954<tel:%2B94773325954>
email: lahirus@wso2.com<ma...@wso2.com> blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
phone: +94773325954
email: lahirus@wso2.com<ma...@wso2.com> blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going
active ... was RE: Stratos 4.1.0 : Issue when deploying application with
single group and 3 cartridges, no dependencies ...
Posted by Lahiru Sandaruwan <la...@wso2.com>.
Hi Isuru,
I did a fix to send the payload separately to place we call jclouds,
without setting it to iaasProvider early.
Thanks.
On Thu, Apr 2, 2015 at 9:37 AM, Isuru Haththotuwa <is...@apache.org> wrote:
> Hi Lahiru,
>
> On Thu, Apr 2, 2015 at 9:11 AM, Lahiru Sandaruwan <la...@wso2.com>
> wrote:
>
>> Thanks Martin for the logs. So the reason is clear why happens this. We
>> keep the payload par cartridge type in iaas provider object, in cloud
>> controller context.
>>
>> When second member approaches the instance started method, it rewrites
>> the payload in iaasProvider object. But the first instance is still not
>> really created in iaas, and it runs in a different thread. Therefore the
>> payload belong to second instance, is passed to second instance as well as
>> the first instance.
>>
> Nice work finding this! So the resolution would be to change the cartridge
> type -> payload mapping to alias -> payload?
>
>>
>> I will fix it.
>>
>> Thanks.
>>
>> On Wed, Apr 1, 2015 at 11:38 PM, Martin Eppel (meppel) <me...@cisco.com>
>> wrote:
>>
>>> Hi Lahiru,
>>>
>>>
>>>
>>> Attached are the logs,
>>>
>>>
>>>
>>> Thanks
>>>
>>>
>>>
>>> Martin
>>>
>>>
>>>
>>> *From:* Lahiru Sandaruwan [mailto:lahirus@wso2.com]
>>> *Sent:* Wednesday, April 01, 2015 4:11 AM
>>>
>>> *To:* dev
>>> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
>>> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
>>> with single group and 3 cartridges, no dependencies ...
>>>
>>>
>>>
>>> HI Martin,
>>>
>>>
>>>
>>> Could you please send the debug logs enabling debug on following
>>> classes? I have improved few debug logs to track payload, therefore please
>>> take a pull before reproducing with debug logs.
>>>
>>>
>>> log4j.logger.org.apache.stratos.autoscaler.applications.parser.DefaultApplicationParser=DEBUG
>>>
>>>
>>> log4j.logger.org.apache.stratos.cloud.controller.services.impl.InstanceCreator=DEBUG
>>>
>>>
>>> log4j.logger.org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl=DEBUG
>>>
>>> Thanks.
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Apr 1, 2015 at 10:50 AM, Lahiru Sandaruwan <la...@wso2.com>
>>> wrote:
>>>
>>> I will have a look Martin.
>>>
>>>
>>>
>>> Thanks.
>>>
>>>
>>>
>>> On Wed, Apr 1, 2015 at 6:54 AM, Martin Eppel (meppel) <me...@cisco.com>
>>> wrote:
>>>
>>> Hi Lahiru,
>>>
>>>
>>>
>>> I retested the scenario to make sure proper unique aliases are provided
>>> and they are, please double check the provided artifacts. Issue remains the
>>> same, one VM gets stuck in INITIALIZED until terminated and restarted. As
>>> pointed out before, the cartridge alias in the cartridge agent is the same
>>> in 2 cartridges (although provided aliases in the application json is
>>> unique). After restarting the faulty member cartridge agent alias has the
>>> proper alias and goes into active.
>>>
>>> I also provided the cartridge-agent logs for the member which is faulty,
>>> before restart and after restart (cartridge-agent-70~INITIALIZED, ACTIVE)
>>>
>>>
>>>
>>> Please see attached logs, artifacts,
>>>
>>>
>>>
>>> Thanks
>>>
>>>
>>>
>>> Martin
>>>
>>>
>>>
>>> *From:* Martin Eppel (meppel)
>>> *Sent:* Tuesday, March 31, 2015 1:22 PM
>>>
>>>
>>> *To:* dev@stratos.apache.org
>>> *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
>>> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
>>> with single group and 3 cartridges, no dependencies ...
>>>
>>>
>>>
>>> Lahiru,
>>>
>>>
>>>
>>> The aliases are all unique, please check the the application json, and
>>> as I said in my analysis it I think this is what I saw but I can’t be sure.
>>> So this is not an issue of starting the application with duplicate aliases
>>> but rather something else ,
>>>
>>>
>>>
>>> Thanks
>>>
>>>
>>>
>>> Martin
>>>
>>>
>>>
>>> *From:* Lahiru Sandaruwan [mailto:lahirus@wso2.com <la...@wso2.com>]
>>> *Sent:* Tuesday, March 31, 2015 12:58 PM
>>> *To:* dev
>>> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
>>> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
>>> with single group and 3 cartridges, no dependencies ...
>>>
>>>
>>>
>>> Hi Martin,
>>>
>>>
>>>
>>> On Wed, Apr 1, 2015 at 12:34 AM, Martin Eppel (meppel) <me...@cisco.com>
>>> wrote:
>>>
>>> Hi Lahiru,
>>>
>>>
>>>
>>> I tested the fix and noticed the following issue:
>>>
>>>
>>>
>>> From the 4 cartridges 3 instances go Active, however one instance got
>>> stuck in INITIALIZED state.
>>>
>>>
>>>
>>> One of the things I noticed while the member was stuck in INITIALIZED
>>> state is that the system received 2 active events for the same membered:
>>>
>>>
>>>
>>> *TID: [0] [STRATOS] [2015-03-31 18:10:27,645] INFO
>>> {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
>>> - Publishing member activated event: [service-name] c2 [cluster-id]
>>> subscription-G1-G2-G3.c2-1x1.c2.domain [cluster-instance-id]
>>> subscription-G1-G2-G3-1 [member-id]
>>> subscription-G1-G2-G3.c2-1x1.c2.domainf939e8dc-e0cc-408a-8343-ac86e479549f
>>> [network-partition-id] RegionOne [partition-id] whole-region*
>>>
>>> *TID: [0] [STRATOS] [2015-03-31 18:10:31,059] INFO
>>> {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
>>> - Publishing member activated event: [service-name] c2 [cluster-id]
>>> subscription-G1-G2-G3.c2-1x1.c2.domain [cluster-instance-id]
>>> subscription-G1-G2-G3-1 [member-id]
>>> subscription-G1-G2-G3.c2-1x1.c2.domainf939e8dc-e0cc-408a-8343-ac86e479549f
>>> [network-partition-id] RegionOne [partition-id] whole-region*
>>>
>>>
>>>
>>> As another note, but I can’t be sure since the log got overwritten is
>>> that it appeared to me that at one point (while the cartridge was stuck in
>>> INITIALIZED state is that the cartridge alias of 2 cartridges was the same
>>> (~71, ~70: 71: CARTRIDGE_ALIAS=c2-1x1), but I can’t say for sure since when
>>> I double checked a new instance of the cartridge was created (which went
>>> ACTIVE).
>>>
>>>
>>>
>>>
>>>
>>> Can i assume this occurred when two cartridges use same alias? May be we
>>> have to put a validation on Application as we cannot use same alias in two
>>> cartridges.
>>>
>>>
>>>
>>> If so, could you please try to reproduce this with different aliases?
>>>
>>>
>>>
>>> Thanks.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Eventually the system terminated the “faulty” member and the cartridge
>>> came up and went into active state:
>>>
>>>
>>>
>>> *“Pending state of member expired, member will be moved to obsolete
>>> list. [pending member]
>>> subscription-G1-G2-G3.c2-1x0.c2.domainfe9a30cf-fffc-4a6c-9b3b-f419498895b9*
>>> ”
>>>
>>> …
>>>
>>> “*Member activated: [service] c2 [cluster]
>>> subscription-G1-G2-G3.c2-1x0.c2.domain [member]
>>> subscription-G1-G2-G3.c2-1x0.c2.domain44a2ae3c-9968-41de-9e10-7212ef3788c2*
>>> ”
>>>
>>>
>>>
>>> Cartridge alias after all 4 cartridges went ACTIVE:
>>>
>>>
>>>
>>> 70: CARTRIDGE_ALIAS=c2-1x0
>>>
>>> 71: CARTRIDGE_ALIAS=c2-1x1
>>>
>>>
>>>
>>> 69: CARTRIDGE_ALIAS=c1-1x0
>>>
>>> 68: CARTRIDGE_ALIAS=c1-1x1
>>>
>>>
>>>
>>> I attached the logs and artefacts
>>>
>>>
>>>
>>> Thanks
>>>
>>>
>>>
>>> Martin
>>>
>>>
>>>
>>> *From:* Lahiru Sandaruwan [mailto:lahirus@wso2.com]
>>> *Sent:* Tuesday, March 31, 2015 2:24 AM
>>>
>>>
>>> *To:* dev
>>> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
>>> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
>>> with single group and 3 cartridges, no dependencies ...
>>>
>>>
>>>
>>> Hi Martin,
>>>
>>>
>>>
>>> I have reproduced the issue and pushed a fix. Please verify.
>>>
>>>
>>>
>>> Thanks.
>>>
>>>
>>>
>>> On Tue, Mar 31, 2015 at 8:32 AM, Lahiru Sandaruwan <la...@wso2.com>
>>> wrote:
>>>
>>> Hi Martin,
>>>
>>>
>>>
>>> On Tue, Mar 31, 2015 at 5:41 AM, Martin Eppel (meppel) <me...@cisco.com>
>>> wrote:
>>>
>>> Hi Lahiru,
>>>
>>>
>>>
>>> I tested a new scenario, where I have 2 groups with cartridges and no
>>> start up dependencies and I see again the issue with cartridge instances
>>> being stuck in “started” state. I also noticed the following log message:
>>>
>>> *“Artifact repositories not found for application signup, artifact
>>> updated event not sent: [application-id] subscription-G1-G2-G3*”.
>>>
>>>
>>>
>>>
>>>
>>> Will have a look.
>>>
>>>
>>>
>>> Thanks.
>>>
>>> Please find attached the artefacts and log messages,
>>>
>>>
>>>
>>> Thanks
>>>
>>>
>>>
>>> Martin
>>>
>>>
>>>
>>> *From:* Martin Eppel (meppel)
>>> *Sent:* Monday, March 30, 2015 2:36 PM
>>>
>>>
>>> *To:* dev@stratos.apache.org
>>> *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
>>> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
>>> with single group and 3 cartridges, no dependencies ...
>>>
>>>
>>>
>>> Hi Lahiru,
>>>
>>>
>>>
>>> I tested both scenarios,
>>>
>>>
>>>
>>> 1. Single cartridge in single group and it worked, cartridge
>>> instance starts up and goes active,
>>>
>>> 2. Multiple cartridges with startup dependencies :
>>>
>>> All instances are started up (in right sequence) and go active
>>>
>>> JIRA https://issues.apache.org/jira/browse/STRATOS-1292[image:
>>> http://wikicentral.cisco.com/images/icons/linkext7.gif]
>>> <https://issues.apache.org/jira/browse/STRATOS-1292> is being tested
>>> and verified
>>>
>>>
>>>
>>> Removing / Un-deploying the application caused an issue as one of the
>>> VMs did not get terminated (throwing exceptions in the log). I am not sure
>>> if this is already a known issue related to to the ongoing discussion of
>>> application removal / un-deployment.
>>>
>>> Just in case, I zipped up the logs and attached it to the email thread
>>> (single-group-multiple-cartridges-startup.zip). Let me know if the
>>> un-deployment issue is the same already discussed or if this is a new issue
>>> ?
>>>
>>>
>>>
>>> Same error occurred in the scenario where multiple cartridges were
>>> started up (all go active) and then application was removed, I attached the
>>> log for this scenario as well
>>> (wscarbon-application-single-group-multiple-cartridges-no-startup-remval-error.log.zip)
>>>
>>>
>>>
>>>
>>>
>>> Thanks
>>>
>>>
>>>
>>> Martin
>>>
>>>
>>>
>>>
>>>
>>> *From:* Martin Eppel (meppel)
>>> *Sent:* Monday, March 30, 2015 7:47 AM
>>> *To:* dev@stratos.apache.org
>>> *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
>>> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
>>> with single group and 3 cartridges, no dependencies ...
>>>
>>>
>>>
>>> Thanks Lahiru,
>>>
>>>
>>>
>>> Will do and let you know
>>>
>>>
>>>
>>> Martin
>>>
>>>
>>>
>>> *From:* Lahiru Sandaruwan [mailto:lahirus@wso2.com <la...@wso2.com>]
>>> *Sent:* Monday, March 30, 2015 3:32 AM
>>> *To:* dev
>>> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
>>> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
>>> with single group and 3 cartridges, no dependencies ...
>>>
>>>
>>>
>>> Hi Martin,
>>>
>>>
>>>
>>> I was able to reproduce the issue and committed a fix.
>>>
>>> Could you please verify?
>>>
>>>
>>>
>>> Thanks.
>>>
>>>
>>>
>>> On Sun, Mar 29, 2015 at 5:43 PM, Lahiru Sandaruwan <la...@wso2.com>
>>> wrote:
>>>
>>> Hi Martin,
>>>
>>>
>>>
>>> According to the logs attached, it seems the cartridge agent waits for
>>> the artifact updated event, and somehow it is not sent.
>>>
>>>
>>>
>>> Looking into it.
>>>
>>>
>>>
>>> Thanks.
>>>
>>>
>>>
>>> On Sat, Mar 28, 2015 at 6:48 AM, Imesh Gunaratne <im...@apache.org>
>>> wrote:
>>>
>>> Hi Martin,
>>>
>>>
>>>
>>> Thanks for the quick response on this!
>>>
>>> I will use these artifacts to re-produce the problem locally and will
>>> let you know the results soon.
>>>
>>>
>>>
>>> Thanks
>>>
>>>
>>>
>>> On Sat, Mar 28, 2015 at 4:19 AM, Martin Eppel (meppel) <me...@cisco.com>
>>> wrote:
>>>
>>> Hi Imesh,
>>>
>>>
>>>
>>> As discussed please find attached the “real” artefacts (application,
>>> cartridge group, cartridge):
>>>
>>>
>>>
>>> Attached it as well to the jira
>>>
>>>
>>>
>>> Thanks
>>>
>>>
>>>
>>> Martin
>>>
>>>
>>>
>>>
>>>
>>> *From:* Martin Eppel (meppel)
>>> *Sent:* Friday, March 27, 2015 11:37 AM
>>> *To:* dev@stratos.apache.org
>>> *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
>>> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
>>> with single group and 3 cartridges, no dependencies ...
>>>
>>>
>>>
>>> Hi Imesh,
>>>
>>>
>>>
>>> Originally I tested with 3 cartridges but then realized that it also
>>> occurs with a single cartridge in a group – see also the JIRA for the
>>> respective notes and artefacts (single group, single cartridge)
>>>
>>>
>>>
>>> Thanks
>>>
>>>
>>>
>>> Martin
>>>
>>>
>>>
>>> *From:* Imesh Gunaratne [mailto:imesh@apache.org <im...@apache.org>]
>>> *Sent:* Friday, March 27, 2015 11:34 AM
>>> *To:* dev
>>> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
>>> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
>>> with single group and 3 cartridges, no dependencies ...
>>>
>>>
>>>
>>> Hi Martin,
>>>
>>>
>>>
>>> Thanks for reporting this problem. You have mentioned that there are
>>> three cartridges in the application but in the given application.json I
>>> only can see one cartridge. Are we missing anything here? Will it be
>>> possible for you to attach the complete set of Stratos artifacts that you
>>> are using in a zip file?
>>>
>>>
>>>
>>> Thanks
>>>
>>> Imesh
>>>
>>>
>>>
>>> On Fri, Mar 27, 2015 at 11:51 PM, Martin Eppel (meppel) <
>>> meppel@cisco.com> wrote:
>>>
>>> Forgot to mention, this is the same issue as reported in JIRA
>>> https://issues.apache.org/jira/browse/STRATOS-1292
>>>
>>>
>>>
>>> Thanks
>>>
>>>
>>>
>>> Martin
>>>
>>>
>>>
>>> *From:* Martin Eppel (meppel)
>>> *Sent:* Friday, March 27, 2015 10:52 AM
>>> *To:* dev@stratos.apache.org
>>> *Subject:* Issue (?): Stratos 4.1.0 - Cartridges in a group are not
>>> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
>>> with single group and 3 cartridges, no dependencies ...
>>>
>>>
>>>
>>> I retested with a similar scenario (single cartridges in group) with
>>> debug logs turned on, please see logs attached.
>>>
>>>
>>>
>>> Cartridge is stuck in “Starting” state
>>>
>>>
>>>
>>> Thanks
>>>
>>>
>>>
>>> Martin
>>>
>>>
>>>
>>> *From:* Martin Eppel (meppel)
>>> *Sent:* Monday, March 23, 2015 7:15 PM
>>> *To:* dev@stratos.apache.org
>>> *Subject:* Stratos 4.1.0 : Issue when deploying application with single
>>> group and 3 cartridges, no dependencies ...
>>>
>>>
>>>
>>> Hi,
>>>
>>>
>>>
>>> I see an issue when I subscribe a single group with 3 cartridges, no
>>> dependencies – they never go active instead they all are stuck in
>>> “Starting”. I attached the server log and the cartridge-agent logs
>>>
>>>
>>>
>>>
>>>
>>> Member status:
>>>
>>> subscription-c1-c2-c3: applicationInstances 1, groupInstances 1,
>>> clusterInstances 3, members 3 (Starting 3)
>>>
>>> c2: 172.16.2.18/172.28.40.69: status Starting
>>>
>>> c3: 172.16.2.19/172.28.40.70: status Starting
>>>
>>> c1: 172.16.2.17/172.28.40.68: status Starting
>>>
>>>
>>>
>>> Let me know if I should go ahead and open a JIRA
>>>
>>>
>>>
>>>
>>>
>>> Thanks
>>>
>>>
>>>
>>> Martin
>>>
>>>
>>>
>>>
>>>
>>> --
>>>
>>> Imesh Gunaratne
>>>
>>>
>>>
>>> Technical Lead, WSO2
>>>
>>> Committer & PMC Member, Apache Stratos
>>>
>>>
>>>
>>>
>>>
>>> --
>>>
>>> Imesh Gunaratne
>>>
>>>
>>>
>>> Technical Lead, WSO2
>>>
>>> Committer & PMC Member, Apache Stratos
>>>
>>>
>>>
>>>
>>>
>>> --
>>>
>>> --
>>> Lahiru Sandaruwan
>>>
>>> Committer and PMC member, Apache Stratos,
>>> Senior Software Engineer,
>>> WSO2 Inc., http://wso2.com
>>>
>>> lean.enterprise.middleware
>>>
>>> phone: +94773325954
>>> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
>>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>>
>>> --
>>> Lahiru Sandaruwan
>>>
>>> Committer and PMC member, Apache Stratos,
>>> Senior Software Engineer,
>>> WSO2 Inc., http://wso2.com
>>>
>>> lean.enterprise.middleware
>>>
>>> phone: +94773325954
>>> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
>>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>>
>>> --
>>> Lahiru Sandaruwan
>>>
>>> Committer and PMC member, Apache Stratos,
>>> Senior Software Engineer,
>>> WSO2 Inc., http://wso2.com
>>>
>>> lean.enterprise.middleware
>>>
>>> phone: +94773325954
>>> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
>>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>>
>>> --
>>> Lahiru Sandaruwan
>>>
>>> Committer and PMC member, Apache Stratos,
>>> Senior Software Engineer,
>>> WSO2 Inc., http://wso2.com
>>>
>>> lean.enterprise.middleware
>>>
>>> phone: +94773325954
>>> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
>>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>>
>>> --
>>> Lahiru Sandaruwan
>>>
>>> Committer and PMC member, Apache Stratos,
>>> Senior Software Engineer,
>>> WSO2 Inc., http://wso2.com
>>>
>>> lean.enterprise.middleware
>>>
>>> phone: +94773325954
>>> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
>>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>>
>>> --
>>> Lahiru Sandaruwan
>>>
>>> Committer and PMC member, Apache Stratos,
>>> Senior Software Engineer,
>>> WSO2 Inc., http://wso2.com
>>>
>>> lean.enterprise.middleware
>>>
>>> phone: +94773325954
>>> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
>>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>>
>>> --
>>> Lahiru Sandaruwan
>>>
>>> Committer and PMC member, Apache Stratos,
>>> Senior Software Engineer,
>>> WSO2 Inc., http://wso2.com
>>>
>>> lean.enterprise.middleware
>>>
>>> phone: +94773325954
>>> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
>>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>>
>>>
>>>
>>
>>
>>
>> --
>> --
>> Lahiru Sandaruwan
>> Committer and PMC member, Apache Stratos,
>> Senior Software Engineer,
>> WSO2 Inc., http://wso2.com
>> lean.enterprise.middleware
>>
>> phone: +94773325954
>> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
>> linked-in:
>> http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>
>> --
>> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>
>> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>
>> Thanks and Regards,
>>
>> Isuru H.
>> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>
>> +94 716 358 048 <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>*
>> <http://wso2.com/>*
>>
>>
>> * <http://wso2.com/>*
>>
>>
>>
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
phone: +94773325954
email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going
active ... was RE: Stratos 4.1.0 : Issue when deploying application with
single group and 3 cartridges, no dependencies ...
Posted by "Martin Eppel (meppel)" <me...@cisco.com>.
Hi Lahiru,
Tested the fix and it works, nice work !
I attached he log files in case you want to double check,
Regards
Martin
From: Lahiru Sandaruwan [mailto:lahirus@wso2.com]
Sent: Thursday, April 02, 2015 1:45 AM
To: dev
Subject: Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
Hi Martin,
Could you please try the fix i have committed?
Thanks,
On Thu, Apr 2, 2015 at 10:35 AM, Martin Eppel (meppel) <me...@cisco.com>> wrote:
Great,
Thanks
Martin
From: isuruh@wso2.com<ma...@wso2.com> [mailto:isuruh@wso2.com<ma...@wso2.com>] On Behalf Of Isuru Haththotuwa
Sent: Wednesday, April 01, 2015 9:08 PM
To: dev
Subject: Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
Hi Lahiru,
On Thu, Apr 2, 2015 at 9:11 AM, Lahiru Sandaruwan <la...@wso2.com>> wrote:
Thanks Martin for the logs. So the reason is clear why happens this. We keep the payload par cartridge type in iaas provider object, in cloud controller context.
When second member approaches the instance started method, it rewrites the payload in iaasProvider object. But the first instance is still not really created in iaas, and it runs in a different thread. Therefore the payload belong to second instance, is passed to second instance as well as the first instance.
Nice work finding this! So the resolution would be to change the cartridge type -> payload mapping to alias -> payload?
I will fix it.
Thanks.
On Wed, Apr 1, 2015 at 11:38 PM, Martin Eppel (meppel) <me...@cisco.com>> wrote:
Hi Lahiru,
Attached are the logs,
Thanks
Martin
From: Lahiru Sandaruwan [mailto:lahirus@wso2.com<ma...@wso2.com>]
Sent: Wednesday, April 01, 2015 4:11 AM
To: dev
Subject: Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
HI Martin,
Could you please send the debug logs enabling debug on following classes? I have improved few debug logs to track payload, therefore please take a pull before reproducing with debug logs.
log4j.logger.org.apache.stratos.autoscaler.applications.parser.DefaultApplicationParser=DEBUG
log4j.logger.org.apache.stratos.cloud.controller.services.impl.InstanceCreator=DEBUG
log4j.logger.org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl=DEBUG
Thanks.
On Wed, Apr 1, 2015 at 10:50 AM, Lahiru Sandaruwan <la...@wso2.com>> wrote:
I will have a look Martin.
Thanks.
On Wed, Apr 1, 2015 at 6:54 AM, Martin Eppel (meppel) <me...@cisco.com>> wrote:
Hi Lahiru,
I retested the scenario to make sure proper unique aliases are provided and they are, please double check the provided artifacts. Issue remains the same, one VM gets stuck in INITIALIZED until terminated and restarted. As pointed out before, the cartridge alias in the cartridge agent is the same in 2 cartridges (although provided aliases in the application json is unique). After restarting the faulty member cartridge agent alias has the proper alias and goes into active.
I also provided the cartridge-agent logs for the member which is faulty, before restart and after restart (cartridge-agent-70~INITIALIZED, ACTIVE)
Please see attached logs, artifacts,
Thanks
Martin
From: Martin Eppel (meppel)
Sent: Tuesday, March 31, 2015 1:22 PM
To: dev@stratos.apache.org<ma...@stratos.apache.org>
Subject: RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
Lahiru,
The aliases are all unique, please check the the application json, and as I said in my analysis it I think this is what I saw but I can’t be sure. So this is not an issue of starting the application with duplicate aliases but rather something else ,
Thanks
Martin
From: Lahiru Sandaruwan [mailto:lahirus@wso2.com]
Sent: Tuesday, March 31, 2015 12:58 PM
To: dev
Subject: Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
Hi Martin,
On Wed, Apr 1, 2015 at 12:34 AM, Martin Eppel (meppel) <me...@cisco.com>> wrote:
Hi Lahiru,
I tested the fix and noticed the following issue:
From the 4 cartridges 3 instances go Active, however one instance got stuck in INITIALIZED state.
One of the things I noticed while the member was stuck in INITIALIZED state is that the system received 2 active events for the same membered:
TID: [0] [STRATOS] [2015-03-31 18:10:27,645] INFO {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher} - Publishing member activated event: [service-name] c2 [cluster-id] subscription-G1-G2-G3.c2-1x1.c2.domain [cluster-instance-id] subscription-G1-G2-G3-1 [member-id] subscription-G1-G2-G3.c2-1x1.c2.domainf939e8dc-e0cc-408a-8343-ac86e479549f [network-partition-id] RegionOne [partition-id] whole-region
TID: [0] [STRATOS] [2015-03-31 18:10:31,059] INFO {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher} - Publishing member activated event: [service-name] c2 [cluster-id] subscription-G1-G2-G3.c2-1x1.c2.domain [cluster-instance-id] subscription-G1-G2-G3-1 [member-id] subscription-G1-G2-G3.c2-1x1.c2.domainf939e8dc-e0cc-408a-8343-ac86e479549f [network-partition-id] RegionOne [partition-id] whole-region
As another note, but I can’t be sure since the log got overwritten is that it appeared to me that at one point (while the cartridge was stuck in INITIALIZED state is that the cartridge alias of 2 cartridges was the same (~71, ~70: 71: CARTRIDGE_ALIAS=c2-1x1), but I can’t say for sure since when I double checked a new instance of the cartridge was created (which went ACTIVE).
Can i assume this occurred when two cartridges use same alias? May be we have to put a validation on Application as we cannot use same alias in two cartridges.
If so, could you please try to reproduce this with different aliases?
Thanks.
Eventually the system terminated the “faulty” member and the cartridge came up and went into active state:
“Pending state of member expired, member will be moved to obsolete list. [pending member] subscription-G1-G2-G3.c2-1x0.c2.domainfe9a30cf-fffc-4a6c-9b3b-f419498895b9”
…
“Member activated: [service] c2 [cluster] subscription-G1-G2-G3.c2-1x0.c2.domain [member] subscription-G1-G2-G3.c2-1x0.c2.domain44a2ae3c-9968-41de-9e10-7212ef3788c2”
Cartridge alias after all 4 cartridges went ACTIVE:
70: CARTRIDGE_ALIAS=c2-1x0
71: CARTRIDGE_ALIAS=c2-1x1
69: CARTRIDGE_ALIAS=c1-1x0
68: CARTRIDGE_ALIAS=c1-1x1
I attached the logs and artefacts
Thanks
Martin
From: Lahiru Sandaruwan [mailto:lahirus@wso2.com<ma...@wso2.com>]
Sent: Tuesday, March 31, 2015 2:24 AM
To: dev
Subject: Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
Hi Martin,
I have reproduced the issue and pushed a fix. Please verify.
Thanks.
On Tue, Mar 31, 2015 at 8:32 AM, Lahiru Sandaruwan <la...@wso2.com>> wrote:
Hi Martin,
On Tue, Mar 31, 2015 at 5:41 AM, Martin Eppel (meppel) <me...@cisco.com>> wrote:
Hi Lahiru,
I tested a new scenario, where I have 2 groups with cartridges and no start up dependencies and I see again the issue with cartridge instances being stuck in “started” state. I also noticed the following log message:
“Artifact repositories not found for application signup, artifact updated event not sent: [application-id] subscription-G1-G2-G3”.
Will have a look.
Thanks.
Please find attached the artefacts and log messages,
Thanks
Martin
From: Martin Eppel (meppel)
Sent: Monday, March 30, 2015 2:36 PM
To: dev@stratos.apache.org<ma...@stratos.apache.org>
Subject: RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
Hi Lahiru,
I tested both scenarios,
1. Single cartridge in single group and it worked, cartridge instance starts up and goes active,
2. Multiple cartridges with startup dependencies :
All instances are started up (in right sequence) and go active
JIRA https://issues.apache.org/jira/browse/STRATOS-1292[http://wikicentral.cisco.com/images/icons/linkext7.gif] is being tested and verified
Removing / Un-deploying the application caused an issue as one of the VMs did not get terminated (throwing exceptions in the log). I am not sure if this is already a known issue related to to the ongoing discussion of application removal / un-deployment.
Just in case, I zipped up the logs and attached it to the email thread (single-group-multiple-cartridges-startup.zip). Let me know if the un-deployment issue is the same already discussed or if this is a new issue ?
Same error occurred in the scenario where multiple cartridges were started up (all go active) and then application was removed, I attached the log for this scenario as well (wscarbon-application-single-group-multiple-cartridges-no-startup-remval-error.log.zip)
Thanks
Martin
From: Martin Eppel (meppel)
Sent: Monday, March 30, 2015 7:47 AM
To: dev@stratos.apache.org<ma...@stratos.apache.org>
Subject: RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
Thanks Lahiru,
Will do and let you know
Martin
From: Lahiru Sandaruwan [mailto:lahirus@wso2.com]
Sent: Monday, March 30, 2015 3:32 AM
To: dev
Subject: Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
Hi Martin,
I was able to reproduce the issue and committed a fix.
Could you please verify?
Thanks.
On Sun, Mar 29, 2015 at 5:43 PM, Lahiru Sandaruwan <la...@wso2.com>> wrote:
Hi Martin,
According to the logs attached, it seems the cartridge agent waits for the artifact updated event, and somehow it is not sent.
Looking into it.
Thanks.
On Sat, Mar 28, 2015 at 6:48 AM, Imesh Gunaratne <im...@apache.org>> wrote:
Hi Martin,
Thanks for the quick response on this!
I will use these artifacts to re-produce the problem locally and will let you know the results soon.
Thanks
On Sat, Mar 28, 2015 at 4:19 AM, Martin Eppel (meppel) <me...@cisco.com>> wrote:
Hi Imesh,
As discussed please find attached the “real” artefacts (application, cartridge group, cartridge):
Attached it as well to the jira
Thanks
Martin
From: Martin Eppel (meppel)
Sent: Friday, March 27, 2015 11:37 AM
To: dev@stratos.apache.org<ma...@stratos.apache.org>
Subject: RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
Hi Imesh,
Originally I tested with 3 cartridges but then realized that it also occurs with a single cartridge in a group – see also the JIRA for the respective notes and artefacts (single group, single cartridge)
Thanks
Martin
From: Imesh Gunaratne [mailto:imesh@apache.org]
Sent: Friday, March 27, 2015 11:34 AM
To: dev
Subject: Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
Hi Martin,
Thanks for reporting this problem. You have mentioned that there are three cartridges in the application but in the given application.json I only can see one cartridge. Are we missing anything here? Will it be possible for you to attach the complete set of Stratos artifacts that you are using in a zip file?
Thanks
Imesh
On Fri, Mar 27, 2015 at 11:51 PM, Martin Eppel (meppel) <me...@cisco.com>> wrote:
Forgot to mention, this is the same issue as reported in JIRA https://issues.apache.org/jira/browse/STRATOS-1292
Thanks
Martin
From: Martin Eppel (meppel)
Sent: Friday, March 27, 2015 10:52 AM
To: dev@stratos.apache.org<ma...@stratos.apache.org>
Subject: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
I retested with a similar scenario (single cartridges in group) with debug logs turned on, please see logs attached.
Cartridge is stuck in “Starting” state
Thanks
Martin
From: Martin Eppel (meppel)
Sent: Monday, March 23, 2015 7:15 PM
To: dev@stratos.apache.org<ma...@stratos.apache.org>
Subject: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
Hi,
I see an issue when I subscribe a single group with 3 cartridges, no dependencies – they never go active instead they all are stuck in “Starting”. I attached the server log and the cartridge-agent logs
Member status:
subscription-c1-c2-c3: applicationInstances 1, groupInstances 1, clusterInstances 3, members 3 (Starting 3)
c2: 172.16.2.18/172.28.40.69<http://172.16.2.18/172.28.40.69>: status Starting
c3: 172.16.2.19/172.28.40.70<http://172.16.2.19/172.28.40.70>: status Starting
c1: 172.16.2.17/172.28.40.68<http://172.16.2.17/172.28.40.68>: status Starting
Let me know if I should go ahead and open a JIRA
Thanks
Martin
--
Imesh Gunaratne
Technical Lead, WSO2
Committer & PMC Member, Apache Stratos
--
Imesh Gunaratne
Technical Lead, WSO2
Committer & PMC Member, Apache Stratos
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
phone: +94773325954<tel:%2B94773325954>
email: lahirus@wso2.com<ma...@wso2.com> blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
phone: +94773325954<tel:%2B94773325954>
email: lahirus@wso2.com<ma...@wso2.com> blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
phone: +94773325954<tel:%2B94773325954>
email: lahirus@wso2.com<ma...@wso2.com> blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
phone: +94773325954<tel:%2B94773325954>
email: lahirus@wso2.com<ma...@wso2.com> blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
phone: +94773325954<tel:%2B94773325954>
email: lahirus@wso2.com<ma...@wso2.com> blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
phone: +94773325954<tel:%2B94773325954>
email: lahirus@wso2.com<ma...@wso2.com> blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
phone: +94773325954<tel:%2B94773325954>
email: lahirus@wso2.com<ma...@wso2.com> blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
phone: +94773325954<tel:%2B94773325954>
email: lahirus@wso2.com<ma...@wso2.com> blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
--
<http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>
Thanks and Regards,
Isuru H.
<http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>
+94 716 358 048<http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
phone: +94773325954
email: lahirus@wso2.com<ma...@wso2.com> blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going
active ... was RE: Stratos 4.1.0 : Issue when deploying application with
single group and 3 cartridges, no dependencies ...
Posted by Lahiru Sandaruwan <la...@wso2.com>.
Hi Martin,
Could you please try the fix i have committed?
Thanks,
On Thu, Apr 2, 2015 at 10:35 AM, Martin Eppel (meppel) <me...@cisco.com>
wrote:
> Great,
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* isuruh@wso2.com [mailto:isuruh@wso2.com] *On Behalf Of *Isuru
> Haththotuwa
> *Sent:* Wednesday, April 01, 2015 9:08 PM
>
> *To:* dev
> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> Hi Lahiru,
>
>
> On Thu, Apr 2, 2015 at 9:11 AM, Lahiru Sandaruwan <la...@wso2.com>
> wrote:
>
> Thanks Martin for the logs. So the reason is clear why happens this. We
> keep the payload par cartridge type in iaas provider object, in cloud
> controller context.
>
>
>
> When second member approaches the instance started method, it rewrites the
> payload in iaasProvider object. But the first instance is still not really
> created in iaas, and it runs in a different thread. Therefore the payload
> belong to second instance, is passed to second instance as well as the
> first instance.
>
> Nice work finding this! So the resolution would be to change the cartridge
> type -> payload mapping to alias -> payload?
>
>
>
> I will fix it.
>
>
>
> Thanks.
>
>
>
> On Wed, Apr 1, 2015 at 11:38 PM, Martin Eppel (meppel) <me...@cisco.com>
> wrote:
>
> Hi Lahiru,
>
>
>
> Attached are the logs,
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Lahiru Sandaruwan [mailto:lahirus@wso2.com]
> *Sent:* Wednesday, April 01, 2015 4:11 AM
>
>
> *To:* dev
> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> HI Martin,
>
>
>
> Could you please send the debug logs enabling debug on following classes?
> I have improved few debug logs to track payload, therefore please take a
> pull before reproducing with debug logs.
>
>
> log4j.logger.org.apache.stratos.autoscaler.applications.parser.DefaultApplicationParser=DEBUG
>
>
> log4j.logger.org.apache.stratos.cloud.controller.services.impl.InstanceCreator=DEBUG
>
>
> log4j.logger.org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl=DEBUG
>
> Thanks.
>
>
>
>
>
> On Wed, Apr 1, 2015 at 10:50 AM, Lahiru Sandaruwan <la...@wso2.com>
> wrote:
>
> I will have a look Martin.
>
>
>
> Thanks.
>
>
>
> On Wed, Apr 1, 2015 at 6:54 AM, Martin Eppel (meppel) <me...@cisco.com>
> wrote:
>
> Hi Lahiru,
>
>
>
> I retested the scenario to make sure proper unique aliases are provided
> and they are, please double check the provided artifacts. Issue remains the
> same, one VM gets stuck in INITIALIZED until terminated and restarted. As
> pointed out before, the cartridge alias in the cartridge agent is the same
> in 2 cartridges (although provided aliases in the application json is
> unique). After restarting the faulty member cartridge agent alias has the
> proper alias and goes into active.
>
> I also provided the cartridge-agent logs for the member which is faulty,
> before restart and after restart (cartridge-agent-70~INITIALIZED, ACTIVE)
>
>
>
> Please see attached logs, artifacts,
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Martin Eppel (meppel)
> *Sent:* Tuesday, March 31, 2015 1:22 PM
>
>
> *To:* dev@stratos.apache.org
> *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> Lahiru,
>
>
>
> The aliases are all unique, please check the the application json, and as
> I said in my analysis it I think this is what I saw but I can’t be sure. So
> this is not an issue of starting the application with duplicate aliases but
> rather something else ,
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Lahiru Sandaruwan [mailto:lahirus@wso2.com <la...@wso2.com>]
> *Sent:* Tuesday, March 31, 2015 12:58 PM
> *To:* dev
> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> Hi Martin,
>
>
>
> On Wed, Apr 1, 2015 at 12:34 AM, Martin Eppel (meppel) <me...@cisco.com>
> wrote:
>
> Hi Lahiru,
>
>
>
> I tested the fix and noticed the following issue:
>
>
>
> From the 4 cartridges 3 instances go Active, however one instance got
> stuck in INITIALIZED state.
>
>
>
> One of the things I noticed while the member was stuck in INITIALIZED
> state is that the system received 2 active events for the same membered:
>
>
>
> *TID: [0] [STRATOS] [2015-03-31 18:10:27,645] INFO
> {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
> - Publishing member activated event: [service-name] c2 [cluster-id]
> subscription-G1-G2-G3.c2-1x1.c2.domain [cluster-instance-id]
> subscription-G1-G2-G3-1 [member-id]
> subscription-G1-G2-G3.c2-1x1.c2.domainf939e8dc-e0cc-408a-8343-ac86e479549f
> [network-partition-id] RegionOne [partition-id] whole-region*
>
> *TID: [0] [STRATOS] [2015-03-31 18:10:31,059] INFO
> {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
> - Publishing member activated event: [service-name] c2 [cluster-id]
> subscription-G1-G2-G3.c2-1x1.c2.domain [cluster-instance-id]
> subscription-G1-G2-G3-1 [member-id]
> subscription-G1-G2-G3.c2-1x1.c2.domainf939e8dc-e0cc-408a-8343-ac86e479549f
> [network-partition-id] RegionOne [partition-id] whole-region*
>
>
>
> As another note, but I can’t be sure since the log got overwritten is that
> it appeared to me that at one point (while the cartridge was stuck in
> INITIALIZED state is that the cartridge alias of 2 cartridges was the same
> (~71, ~70: 71: CARTRIDGE_ALIAS=c2-1x1), but I can’t say for sure since when
> I double checked a new instance of the cartridge was created (which went
> ACTIVE).
>
>
>
>
>
> Can i assume this occurred when two cartridges use same alias? May be we
> have to put a validation on Application as we cannot use same alias in two
> cartridges.
>
>
>
> If so, could you please try to reproduce this with different aliases?
>
>
>
> Thanks.
>
>
>
>
>
>
>
> Eventually the system terminated the “faulty” member and the cartridge
> came up and went into active state:
>
>
>
> *“Pending state of member expired, member will be moved to obsolete list.
> [pending member]
> subscription-G1-G2-G3.c2-1x0.c2.domainfe9a30cf-fffc-4a6c-9b3b-f419498895b9*
> ”
>
> …
>
> “*Member activated: [service] c2 [cluster]
> subscription-G1-G2-G3.c2-1x0.c2.domain [member]
> subscription-G1-G2-G3.c2-1x0.c2.domain44a2ae3c-9968-41de-9e10-7212ef3788c2*
> ”
>
>
>
> Cartridge alias after all 4 cartridges went ACTIVE:
>
>
>
> 70: CARTRIDGE_ALIAS=c2-1x0
>
> 71: CARTRIDGE_ALIAS=c2-1x1
>
>
>
> 69: CARTRIDGE_ALIAS=c1-1x0
>
> 68: CARTRIDGE_ALIAS=c1-1x1
>
>
>
> I attached the logs and artefacts
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Lahiru Sandaruwan [mailto:lahirus@wso2.com]
> *Sent:* Tuesday, March 31, 2015 2:24 AM
>
>
> *To:* dev
> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> Hi Martin,
>
>
>
> I have reproduced the issue and pushed a fix. Please verify.
>
>
>
> Thanks.
>
>
>
> On Tue, Mar 31, 2015 at 8:32 AM, Lahiru Sandaruwan <la...@wso2.com>
> wrote:
>
> Hi Martin,
>
>
>
> On Tue, Mar 31, 2015 at 5:41 AM, Martin Eppel (meppel) <me...@cisco.com>
> wrote:
>
> Hi Lahiru,
>
>
>
> I tested a new scenario, where I have 2 groups with cartridges and no
> start up dependencies and I see again the issue with cartridge instances
> being stuck in “started” state. I also noticed the following log message:
>
> *“Artifact repositories not found for application signup, artifact updated
> event not sent: [application-id] subscription-G1-G2-G3*”.
>
>
>
>
>
> Will have a look.
>
>
>
> Thanks.
>
> Please find attached the artefacts and log messages,
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Martin Eppel (meppel)
> *Sent:* Monday, March 30, 2015 2:36 PM
>
>
> *To:* dev@stratos.apache.org
> *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> Hi Lahiru,
>
>
>
> I tested both scenarios,
>
>
>
> 1. Single cartridge in single group and it worked, cartridge
> instance starts up and goes active,
>
> 2. Multiple cartridges with startup dependencies :
>
> All instances are started up (in right sequence) and go active
>
> JIRA https://issues.apache.org/jira/browse/STRATOS-1292[image:
> http://wikicentral.cisco.com/images/icons/linkext7.gif]
> <https://issues.apache.org/jira/browse/STRATOS-1292> is being tested and
> verified
>
>
>
> Removing / Un-deploying the application caused an issue as one of the VMs
> did not get terminated (throwing exceptions in the log). I am not sure if
> this is already a known issue related to to the ongoing discussion of
> application removal / un-deployment.
>
> Just in case, I zipped up the logs and attached it to the email thread
> (single-group-multiple-cartridges-startup.zip). Let me know if the
> un-deployment issue is the same already discussed or if this is a new issue
> ?
>
>
>
> Same error occurred in the scenario where multiple cartridges were started
> up (all go active) and then application was removed, I attached the log for
> this scenario as well
> (wscarbon-application-single-group-multiple-cartridges-no-startup-remval-error.log.zip)
>
>
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
>
>
> *From:* Martin Eppel (meppel)
> *Sent:* Monday, March 30, 2015 7:47 AM
> *To:* dev@stratos.apache.org
> *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> Thanks Lahiru,
>
>
>
> Will do and let you know
>
>
>
> Martin
>
>
>
> *From:* Lahiru Sandaruwan [mailto:lahirus@wso2.com <la...@wso2.com>]
> *Sent:* Monday, March 30, 2015 3:32 AM
> *To:* dev
> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> Hi Martin,
>
>
>
> I was able to reproduce the issue and committed a fix.
>
> Could you please verify?
>
>
>
> Thanks.
>
>
>
> On Sun, Mar 29, 2015 at 5:43 PM, Lahiru Sandaruwan <la...@wso2.com>
> wrote:
>
> Hi Martin,
>
>
>
> According to the logs attached, it seems the cartridge agent waits for the
> artifact updated event, and somehow it is not sent.
>
>
>
> Looking into it.
>
>
>
> Thanks.
>
>
>
> On Sat, Mar 28, 2015 at 6:48 AM, Imesh Gunaratne <im...@apache.org> wrote:
>
> Hi Martin,
>
>
>
> Thanks for the quick response on this!
>
> I will use these artifacts to re-produce the problem locally and will let
> you know the results soon.
>
>
>
> Thanks
>
>
>
> On Sat, Mar 28, 2015 at 4:19 AM, Martin Eppel (meppel) <me...@cisco.com>
> wrote:
>
> Hi Imesh,
>
>
>
> As discussed please find attached the “real” artefacts (application,
> cartridge group, cartridge):
>
>
>
> Attached it as well to the jira
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
>
>
> *From:* Martin Eppel (meppel)
> *Sent:* Friday, March 27, 2015 11:37 AM
> *To:* dev@stratos.apache.org
> *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> Hi Imesh,
>
>
>
> Originally I tested with 3 cartridges but then realized that it also
> occurs with a single cartridge in a group – see also the JIRA for the
> respective notes and artefacts (single group, single cartridge)
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Imesh Gunaratne [mailto:imesh@apache.org <im...@apache.org>]
> *Sent:* Friday, March 27, 2015 11:34 AM
> *To:* dev
> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> Hi Martin,
>
>
>
> Thanks for reporting this problem. You have mentioned that there are three
> cartridges in the application but in the given application.json I only can
> see one cartridge. Are we missing anything here? Will it be possible for
> you to attach the complete set of Stratos artifacts that you are using in a
> zip file?
>
>
>
> Thanks
>
> Imesh
>
>
>
> On Fri, Mar 27, 2015 at 11:51 PM, Martin Eppel (meppel) <me...@cisco.com>
> wrote:
>
> Forgot to mention, this is the same issue as reported in JIRA
> https://issues.apache.org/jira/browse/STRATOS-1292
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Martin Eppel (meppel)
> *Sent:* Friday, March 27, 2015 10:52 AM
> *To:* dev@stratos.apache.org
> *Subject:* Issue (?): Stratos 4.1.0 - Cartridges in a group are not going
> active ... was RE: Stratos 4.1.0 : Issue when deploying application with
> single group and 3 cartridges, no dependencies ...
>
>
>
> I retested with a similar scenario (single cartridges in group) with
> debug logs turned on, please see logs attached.
>
>
>
> Cartridge is stuck in “Starting” state
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Martin Eppel (meppel)
> *Sent:* Monday, March 23, 2015 7:15 PM
> *To:* dev@stratos.apache.org
> *Subject:* Stratos 4.1.0 : Issue when deploying application with single
> group and 3 cartridges, no dependencies ...
>
>
>
> Hi,
>
>
>
> I see an issue when I subscribe a single group with 3 cartridges, no
> dependencies – they never go active instead they all are stuck in
> “Starting”. I attached the server log and the cartridge-agent logs
>
>
>
>
>
> Member status:
>
> subscription-c1-c2-c3: applicationInstances 1, groupInstances 1,
> clusterInstances 3, members 3 (Starting 3)
>
> c2: 172.16.2.18/172.28.40.69: status Starting
>
> c3: 172.16.2.19/172.28.40.70: status Starting
>
> c1: 172.16.2.17/172.28.40.68: status Starting
>
>
>
> Let me know if I should go ahead and open a JIRA
>
>
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
>
>
> --
>
> Imesh Gunaratne
>
>
>
> Technical Lead, WSO2
>
> Committer & PMC Member, Apache Stratos
>
>
>
>
>
> --
>
> Imesh Gunaratne
>
>
>
> Technical Lead, WSO2
>
> Committer & PMC Member, Apache Stratos
>
>
>
>
>
> --
>
> --
> Lahiru Sandaruwan
>
> Committer and PMC member, Apache Stratos,
> Senior Software Engineer,
> WSO2 Inc., http://wso2.com
>
> lean.enterprise.middleware
>
> phone: +94773325954
> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
>
>
>
>
>
>
> --
>
> --
> Lahiru Sandaruwan
>
> Committer and PMC member, Apache Stratos,
> Senior Software Engineer,
> WSO2 Inc., http://wso2.com
>
> lean.enterprise.middleware
>
> phone: +94773325954
> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
>
>
>
>
>
>
> --
>
> --
> Lahiru Sandaruwan
>
> Committer and PMC member, Apache Stratos,
> Senior Software Engineer,
> WSO2 Inc., http://wso2.com
>
> lean.enterprise.middleware
>
> phone: +94773325954
> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
>
>
>
>
>
>
> --
>
> --
> Lahiru Sandaruwan
>
> Committer and PMC member, Apache Stratos,
> Senior Software Engineer,
> WSO2 Inc., http://wso2.com
>
> lean.enterprise.middleware
>
> phone: +94773325954
> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
>
>
>
>
>
>
> --
>
> --
> Lahiru Sandaruwan
>
> Committer and PMC member, Apache Stratos,
> Senior Software Engineer,
> WSO2 Inc., http://wso2.com
>
> lean.enterprise.middleware
>
> phone: +94773325954
> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
>
>
>
>
>
>
> --
>
> --
> Lahiru Sandaruwan
>
> Committer and PMC member, Apache Stratos,
> Senior Software Engineer,
> WSO2 Inc., http://wso2.com
>
> lean.enterprise.middleware
>
> phone: +94773325954
> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
>
>
>
>
>
>
> --
>
> --
> Lahiru Sandaruwan
>
> Committer and PMC member, Apache Stratos,
> Senior Software Engineer,
> WSO2 Inc., http://wso2.com
>
> lean.enterprise.middleware
>
> phone: +94773325954
> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
>
>
>
>
>
>
> --
>
> --
> Lahiru Sandaruwan
>
> Committer and PMC member, Apache Stratos,
> Senior Software Engineer,
> WSO2 Inc., http://wso2.com
>
> lean.enterprise.middleware
>
> phone: +94773325954
> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
> --
> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>
>
> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>
>
>
>
>
>
> *Thanks and Regards, Isuru H.
> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>*
>
>
>
> +94 716 358 048 <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>
>
>
>
>
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
phone: +94773325954
email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going
active ... was RE: Stratos 4.1.0 : Issue when deploying application with
single group and 3 cartridges, no dependencies ...
Posted by "Martin Eppel (meppel)" <me...@cisco.com>.
Great,
Thanks
Martin
From: isuruh@wso2.com [mailto:isuruh@wso2.com] On Behalf Of Isuru Haththotuwa
Sent: Wednesday, April 01, 2015 9:08 PM
To: dev
Subject: Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
Hi Lahiru,
On Thu, Apr 2, 2015 at 9:11 AM, Lahiru Sandaruwan <la...@wso2.com>> wrote:
Thanks Martin for the logs. So the reason is clear why happens this. We keep the payload par cartridge type in iaas provider object, in cloud controller context.
When second member approaches the instance started method, it rewrites the payload in iaasProvider object. But the first instance is still not really created in iaas, and it runs in a different thread. Therefore the payload belong to second instance, is passed to second instance as well as the first instance.
Nice work finding this! So the resolution would be to change the cartridge type -> payload mapping to alias -> payload?
I will fix it.
Thanks.
On Wed, Apr 1, 2015 at 11:38 PM, Martin Eppel (meppel) <me...@cisco.com>> wrote:
Hi Lahiru,
Attached are the logs,
Thanks
Martin
From: Lahiru Sandaruwan [mailto:lahirus@wso2.com<ma...@wso2.com>]
Sent: Wednesday, April 01, 2015 4:11 AM
To: dev
Subject: Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
HI Martin,
Could you please send the debug logs enabling debug on following classes? I have improved few debug logs to track payload, therefore please take a pull before reproducing with debug logs.
log4j.logger.org.apache.stratos.autoscaler.applications.parser.DefaultApplicationParser=DEBUG
log4j.logger.org.apache.stratos.cloud.controller.services.impl.InstanceCreator=DEBUG
log4j.logger.org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl=DEBUG
Thanks.
On Wed, Apr 1, 2015 at 10:50 AM, Lahiru Sandaruwan <la...@wso2.com>> wrote:
I will have a look Martin.
Thanks.
On Wed, Apr 1, 2015 at 6:54 AM, Martin Eppel (meppel) <me...@cisco.com>> wrote:
Hi Lahiru,
I retested the scenario to make sure proper unique aliases are provided and they are, please double check the provided artifacts. Issue remains the same, one VM gets stuck in INITIALIZED until terminated and restarted. As pointed out before, the cartridge alias in the cartridge agent is the same in 2 cartridges (although provided aliases in the application json is unique). After restarting the faulty member cartridge agent alias has the proper alias and goes into active.
I also provided the cartridge-agent logs for the member which is faulty, before restart and after restart (cartridge-agent-70~INITIALIZED, ACTIVE)
Please see attached logs, artifacts,
Thanks
Martin
From: Martin Eppel (meppel)
Sent: Tuesday, March 31, 2015 1:22 PM
To: dev@stratos.apache.org<ma...@stratos.apache.org>
Subject: RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
Lahiru,
The aliases are all unique, please check the the application json, and as I said in my analysis it I think this is what I saw but I can’t be sure. So this is not an issue of starting the application with duplicate aliases but rather something else ,
Thanks
Martin
From: Lahiru Sandaruwan [mailto:lahirus@wso2.com]
Sent: Tuesday, March 31, 2015 12:58 PM
To: dev
Subject: Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
Hi Martin,
On Wed, Apr 1, 2015 at 12:34 AM, Martin Eppel (meppel) <me...@cisco.com>> wrote:
Hi Lahiru,
I tested the fix and noticed the following issue:
From the 4 cartridges 3 instances go Active, however one instance got stuck in INITIALIZED state.
One of the things I noticed while the member was stuck in INITIALIZED state is that the system received 2 active events for the same membered:
TID: [0] [STRATOS] [2015-03-31 18:10:27,645] INFO {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher} - Publishing member activated event: [service-name] c2 [cluster-id] subscription-G1-G2-G3.c2-1x1.c2.domain [cluster-instance-id] subscription-G1-G2-G3-1 [member-id] subscription-G1-G2-G3.c2-1x1.c2.domainf939e8dc-e0cc-408a-8343-ac86e479549f [network-partition-id] RegionOne [partition-id] whole-region
TID: [0] [STRATOS] [2015-03-31 18:10:31,059] INFO {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher} - Publishing member activated event: [service-name] c2 [cluster-id] subscription-G1-G2-G3.c2-1x1.c2.domain [cluster-instance-id] subscription-G1-G2-G3-1 [member-id] subscription-G1-G2-G3.c2-1x1.c2.domainf939e8dc-e0cc-408a-8343-ac86e479549f [network-partition-id] RegionOne [partition-id] whole-region
As another note, but I can’t be sure since the log got overwritten is that it appeared to me that at one point (while the cartridge was stuck in INITIALIZED state is that the cartridge alias of 2 cartridges was the same (~71, ~70: 71: CARTRIDGE_ALIAS=c2-1x1), but I can’t say for sure since when I double checked a new instance of the cartridge was created (which went ACTIVE).
Can i assume this occurred when two cartridges use same alias? May be we have to put a validation on Application as we cannot use same alias in two cartridges.
If so, could you please try to reproduce this with different aliases?
Thanks.
Eventually the system terminated the “faulty” member and the cartridge came up and went into active state:
“Pending state of member expired, member will be moved to obsolete list. [pending member] subscription-G1-G2-G3.c2-1x0.c2.domainfe9a30cf-fffc-4a6c-9b3b-f419498895b9”
…
“Member activated: [service] c2 [cluster] subscription-G1-G2-G3.c2-1x0.c2.domain [member] subscription-G1-G2-G3.c2-1x0.c2.domain44a2ae3c-9968-41de-9e10-7212ef3788c2”
Cartridge alias after all 4 cartridges went ACTIVE:
70: CARTRIDGE_ALIAS=c2-1x0
71: CARTRIDGE_ALIAS=c2-1x1
69: CARTRIDGE_ALIAS=c1-1x0
68: CARTRIDGE_ALIAS=c1-1x1
I attached the logs and artefacts
Thanks
Martin
From: Lahiru Sandaruwan [mailto:lahirus@wso2.com<ma...@wso2.com>]
Sent: Tuesday, March 31, 2015 2:24 AM
To: dev
Subject: Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
Hi Martin,
I have reproduced the issue and pushed a fix. Please verify.
Thanks.
On Tue, Mar 31, 2015 at 8:32 AM, Lahiru Sandaruwan <la...@wso2.com>> wrote:
Hi Martin,
On Tue, Mar 31, 2015 at 5:41 AM, Martin Eppel (meppel) <me...@cisco.com>> wrote:
Hi Lahiru,
I tested a new scenario, where I have 2 groups with cartridges and no start up dependencies and I see again the issue with cartridge instances being stuck in “started” state. I also noticed the following log message:
“Artifact repositories not found for application signup, artifact updated event not sent: [application-id] subscription-G1-G2-G3”.
Will have a look.
Thanks.
Please find attached the artefacts and log messages,
Thanks
Martin
From: Martin Eppel (meppel)
Sent: Monday, March 30, 2015 2:36 PM
To: dev@stratos.apache.org<ma...@stratos.apache.org>
Subject: RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
Hi Lahiru,
I tested both scenarios,
1. Single cartridge in single group and it worked, cartridge instance starts up and goes active,
2. Multiple cartridges with startup dependencies :
All instances are started up (in right sequence) and go active
JIRA https://issues.apache.org/jira/browse/STRATOS-1292[http://wikicentral.cisco.com/images/icons/linkext7.gif] is being tested and verified
Removing / Un-deploying the application caused an issue as one of the VMs did not get terminated (throwing exceptions in the log). I am not sure if this is already a known issue related to to the ongoing discussion of application removal / un-deployment.
Just in case, I zipped up the logs and attached it to the email thread (single-group-multiple-cartridges-startup.zip). Let me know if the un-deployment issue is the same already discussed or if this is a new issue ?
Same error occurred in the scenario where multiple cartridges were started up (all go active) and then application was removed, I attached the log for this scenario as well (wscarbon-application-single-group-multiple-cartridges-no-startup-remval-error.log.zip)
Thanks
Martin
From: Martin Eppel (meppel)
Sent: Monday, March 30, 2015 7:47 AM
To: dev@stratos.apache.org<ma...@stratos.apache.org>
Subject: RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
Thanks Lahiru,
Will do and let you know
Martin
From: Lahiru Sandaruwan [mailto:lahirus@wso2.com]
Sent: Monday, March 30, 2015 3:32 AM
To: dev
Subject: Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
Hi Martin,
I was able to reproduce the issue and committed a fix.
Could you please verify?
Thanks.
On Sun, Mar 29, 2015 at 5:43 PM, Lahiru Sandaruwan <la...@wso2.com>> wrote:
Hi Martin,
According to the logs attached, it seems the cartridge agent waits for the artifact updated event, and somehow it is not sent.
Looking into it.
Thanks.
On Sat, Mar 28, 2015 at 6:48 AM, Imesh Gunaratne <im...@apache.org>> wrote:
Hi Martin,
Thanks for the quick response on this!
I will use these artifacts to re-produce the problem locally and will let you know the results soon.
Thanks
On Sat, Mar 28, 2015 at 4:19 AM, Martin Eppel (meppel) <me...@cisco.com>> wrote:
Hi Imesh,
As discussed please find attached the “real” artefacts (application, cartridge group, cartridge):
Attached it as well to the jira
Thanks
Martin
From: Martin Eppel (meppel)
Sent: Friday, March 27, 2015 11:37 AM
To: dev@stratos.apache.org<ma...@stratos.apache.org>
Subject: RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
Hi Imesh,
Originally I tested with 3 cartridges but then realized that it also occurs with a single cartridge in a group – see also the JIRA for the respective notes and artefacts (single group, single cartridge)
Thanks
Martin
From: Imesh Gunaratne [mailto:imesh@apache.org]
Sent: Friday, March 27, 2015 11:34 AM
To: dev
Subject: Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
Hi Martin,
Thanks for reporting this problem. You have mentioned that there are three cartridges in the application but in the given application.json I only can see one cartridge. Are we missing anything here? Will it be possible for you to attach the complete set of Stratos artifacts that you are using in a zip file?
Thanks
Imesh
On Fri, Mar 27, 2015 at 11:51 PM, Martin Eppel (meppel) <me...@cisco.com>> wrote:
Forgot to mention, this is the same issue as reported in JIRA https://issues.apache.org/jira/browse/STRATOS-1292
Thanks
Martin
From: Martin Eppel (meppel)
Sent: Friday, March 27, 2015 10:52 AM
To: dev@stratos.apache.org<ma...@stratos.apache.org>
Subject: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
I retested with a similar scenario (single cartridges in group) with debug logs turned on, please see logs attached.
Cartridge is stuck in “Starting” state
Thanks
Martin
From: Martin Eppel (meppel)
Sent: Monday, March 23, 2015 7:15 PM
To: dev@stratos.apache.org<ma...@stratos.apache.org>
Subject: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
Hi,
I see an issue when I subscribe a single group with 3 cartridges, no dependencies – they never go active instead they all are stuck in “Starting”. I attached the server log and the cartridge-agent logs
Member status:
subscription-c1-c2-c3: applicationInstances 1, groupInstances 1, clusterInstances 3, members 3 (Starting 3)
c2: 172.16.2.18/172.28.40.69<http://172.16.2.18/172.28.40.69>: status Starting
c3: 172.16.2.19/172.28.40.70<http://172.16.2.19/172.28.40.70>: status Starting
c1: 172.16.2.17/172.28.40.68<http://172.16.2.17/172.28.40.68>: status Starting
Let me know if I should go ahead and open a JIRA
Thanks
Martin
--
Imesh Gunaratne
Technical Lead, WSO2
Committer & PMC Member, Apache Stratos
--
Imesh Gunaratne
Technical Lead, WSO2
Committer & PMC Member, Apache Stratos
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
phone: +94773325954<tel:%2B94773325954>
email: lahirus@wso2.com<ma...@wso2.com> blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
phone: +94773325954<tel:%2B94773325954>
email: lahirus@wso2.com<ma...@wso2.com> blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
phone: +94773325954<tel:%2B94773325954>
email: lahirus@wso2.com<ma...@wso2.com> blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
phone: +94773325954<tel:%2B94773325954>
email: lahirus@wso2.com<ma...@wso2.com> blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
phone: +94773325954<tel:%2B94773325954>
email: lahirus@wso2.com<ma...@wso2.com> blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
phone: +94773325954<tel:%2B94773325954>
email: lahirus@wso2.com<ma...@wso2.com> blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
phone: +94773325954<tel:%2B94773325954>
email: lahirus@wso2.com<ma...@wso2.com> blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
phone: +94773325954<tel:%2B94773325954>
email: lahirus@wso2.com<ma...@wso2.com> blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
--
<http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>
<http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>
Thanks and Regards,
Isuru H.
<http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>
+94 716 358 048<http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>
Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going
active ... was RE: Stratos 4.1.0 : Issue when deploying application with
single group and 3 cartridges, no dependencies ...
Posted by Isuru Haththotuwa <is...@apache.org>.
Hi Lahiru,
On Thu, Apr 2, 2015 at 9:11 AM, Lahiru Sandaruwan <la...@wso2.com> wrote:
> Thanks Martin for the logs. So the reason is clear why happens this. We
> keep the payload par cartridge type in iaas provider object, in cloud
> controller context.
>
> When second member approaches the instance started method, it rewrites the
> payload in iaasProvider object. But the first instance is still not really
> created in iaas, and it runs in a different thread. Therefore the payload
> belong to second instance, is passed to second instance as well as the
> first instance.
>
Nice work finding this! So the resolution would be to change the cartridge
type -> payload mapping to alias -> payload?
>
> I will fix it.
>
> Thanks.
>
> On Wed, Apr 1, 2015 at 11:38 PM, Martin Eppel (meppel) <me...@cisco.com>
> wrote:
>
>> Hi Lahiru,
>>
>>
>>
>> Attached are the logs,
>>
>>
>>
>> Thanks
>>
>>
>>
>> Martin
>>
>>
>>
>> *From:* Lahiru Sandaruwan [mailto:lahirus@wso2.com]
>> *Sent:* Wednesday, April 01, 2015 4:11 AM
>>
>> *To:* dev
>> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
>> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
>> with single group and 3 cartridges, no dependencies ...
>>
>>
>>
>> HI Martin,
>>
>>
>>
>> Could you please send the debug logs enabling debug on following classes?
>> I have improved few debug logs to track payload, therefore please take a
>> pull before reproducing with debug logs.
>>
>>
>> log4j.logger.org.apache.stratos.autoscaler.applications.parser.DefaultApplicationParser=DEBUG
>>
>>
>> log4j.logger.org.apache.stratos.cloud.controller.services.impl.InstanceCreator=DEBUG
>>
>>
>> log4j.logger.org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl=DEBUG
>>
>> Thanks.
>>
>>
>>
>>
>>
>> On Wed, Apr 1, 2015 at 10:50 AM, Lahiru Sandaruwan <la...@wso2.com>
>> wrote:
>>
>> I will have a look Martin.
>>
>>
>>
>> Thanks.
>>
>>
>>
>> On Wed, Apr 1, 2015 at 6:54 AM, Martin Eppel (meppel) <me...@cisco.com>
>> wrote:
>>
>> Hi Lahiru,
>>
>>
>>
>> I retested the scenario to make sure proper unique aliases are provided
>> and they are, please double check the provided artifacts. Issue remains the
>> same, one VM gets stuck in INITIALIZED until terminated and restarted. As
>> pointed out before, the cartridge alias in the cartridge agent is the same
>> in 2 cartridges (although provided aliases in the application json is
>> unique). After restarting the faulty member cartridge agent alias has the
>> proper alias and goes into active.
>>
>> I also provided the cartridge-agent logs for the member which is faulty,
>> before restart and after restart (cartridge-agent-70~INITIALIZED, ACTIVE)
>>
>>
>>
>> Please see attached logs, artifacts,
>>
>>
>>
>> Thanks
>>
>>
>>
>> Martin
>>
>>
>>
>> *From:* Martin Eppel (meppel)
>> *Sent:* Tuesday, March 31, 2015 1:22 PM
>>
>>
>> *To:* dev@stratos.apache.org
>> *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
>> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
>> with single group and 3 cartridges, no dependencies ...
>>
>>
>>
>> Lahiru,
>>
>>
>>
>> The aliases are all unique, please check the the application json, and as
>> I said in my analysis it I think this is what I saw but I can’t be sure. So
>> this is not an issue of starting the application with duplicate aliases but
>> rather something else ,
>>
>>
>>
>> Thanks
>>
>>
>>
>> Martin
>>
>>
>>
>> *From:* Lahiru Sandaruwan [mailto:lahirus@wso2.com <la...@wso2.com>]
>> *Sent:* Tuesday, March 31, 2015 12:58 PM
>> *To:* dev
>> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
>> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
>> with single group and 3 cartridges, no dependencies ...
>>
>>
>>
>> Hi Martin,
>>
>>
>>
>> On Wed, Apr 1, 2015 at 12:34 AM, Martin Eppel (meppel) <me...@cisco.com>
>> wrote:
>>
>> Hi Lahiru,
>>
>>
>>
>> I tested the fix and noticed the following issue:
>>
>>
>>
>> From the 4 cartridges 3 instances go Active, however one instance got
>> stuck in INITIALIZED state.
>>
>>
>>
>> One of the things I noticed while the member was stuck in INITIALIZED
>> state is that the system received 2 active events for the same membered:
>>
>>
>>
>> *TID: [0] [STRATOS] [2015-03-31 18:10:27,645] INFO
>> {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
>> - Publishing member activated event: [service-name] c2 [cluster-id]
>> subscription-G1-G2-G3.c2-1x1.c2.domain [cluster-instance-id]
>> subscription-G1-G2-G3-1 [member-id]
>> subscription-G1-G2-G3.c2-1x1.c2.domainf939e8dc-e0cc-408a-8343-ac86e479549f
>> [network-partition-id] RegionOne [partition-id] whole-region*
>>
>> *TID: [0] [STRATOS] [2015-03-31 18:10:31,059] INFO
>> {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
>> - Publishing member activated event: [service-name] c2 [cluster-id]
>> subscription-G1-G2-G3.c2-1x1.c2.domain [cluster-instance-id]
>> subscription-G1-G2-G3-1 [member-id]
>> subscription-G1-G2-G3.c2-1x1.c2.domainf939e8dc-e0cc-408a-8343-ac86e479549f
>> [network-partition-id] RegionOne [partition-id] whole-region*
>>
>>
>>
>> As another note, but I can’t be sure since the log got overwritten is
>> that it appeared to me that at one point (while the cartridge was stuck in
>> INITIALIZED state is that the cartridge alias of 2 cartridges was the same
>> (~71, ~70: 71: CARTRIDGE_ALIAS=c2-1x1), but I can’t say for sure since when
>> I double checked a new instance of the cartridge was created (which went
>> ACTIVE).
>>
>>
>>
>>
>>
>> Can i assume this occurred when two cartridges use same alias? May be we
>> have to put a validation on Application as we cannot use same alias in two
>> cartridges.
>>
>>
>>
>> If so, could you please try to reproduce this with different aliases?
>>
>>
>>
>> Thanks.
>>
>>
>>
>>
>>
>>
>>
>> Eventually the system terminated the “faulty” member and the cartridge
>> came up and went into active state:
>>
>>
>>
>> *“Pending state of member expired, member will be moved to obsolete list.
>> [pending member]
>> subscription-G1-G2-G3.c2-1x0.c2.domainfe9a30cf-fffc-4a6c-9b3b-f419498895b9*
>> ”
>>
>> …
>>
>> “*Member activated: [service] c2 [cluster]
>> subscription-G1-G2-G3.c2-1x0.c2.domain [member]
>> subscription-G1-G2-G3.c2-1x0.c2.domain44a2ae3c-9968-41de-9e10-7212ef3788c2*
>> ”
>>
>>
>>
>> Cartridge alias after all 4 cartridges went ACTIVE:
>>
>>
>>
>> 70: CARTRIDGE_ALIAS=c2-1x0
>>
>> 71: CARTRIDGE_ALIAS=c2-1x1
>>
>>
>>
>> 69: CARTRIDGE_ALIAS=c1-1x0
>>
>> 68: CARTRIDGE_ALIAS=c1-1x1
>>
>>
>>
>> I attached the logs and artefacts
>>
>>
>>
>> Thanks
>>
>>
>>
>> Martin
>>
>>
>>
>> *From:* Lahiru Sandaruwan [mailto:lahirus@wso2.com]
>> *Sent:* Tuesday, March 31, 2015 2:24 AM
>>
>>
>> *To:* dev
>> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
>> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
>> with single group and 3 cartridges, no dependencies ...
>>
>>
>>
>> Hi Martin,
>>
>>
>>
>> I have reproduced the issue and pushed a fix. Please verify.
>>
>>
>>
>> Thanks.
>>
>>
>>
>> On Tue, Mar 31, 2015 at 8:32 AM, Lahiru Sandaruwan <la...@wso2.com>
>> wrote:
>>
>> Hi Martin,
>>
>>
>>
>> On Tue, Mar 31, 2015 at 5:41 AM, Martin Eppel (meppel) <me...@cisco.com>
>> wrote:
>>
>> Hi Lahiru,
>>
>>
>>
>> I tested a new scenario, where I have 2 groups with cartridges and no
>> start up dependencies and I see again the issue with cartridge instances
>> being stuck in “started” state. I also noticed the following log message:
>>
>> *“Artifact repositories not found for application signup, artifact
>> updated event not sent: [application-id] subscription-G1-G2-G3*”.
>>
>>
>>
>>
>>
>> Will have a look.
>>
>>
>>
>> Thanks.
>>
>> Please find attached the artefacts and log messages,
>>
>>
>>
>> Thanks
>>
>>
>>
>> Martin
>>
>>
>>
>> *From:* Martin Eppel (meppel)
>> *Sent:* Monday, March 30, 2015 2:36 PM
>>
>>
>> *To:* dev@stratos.apache.org
>> *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
>> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
>> with single group and 3 cartridges, no dependencies ...
>>
>>
>>
>> Hi Lahiru,
>>
>>
>>
>> I tested both scenarios,
>>
>>
>>
>> 1. Single cartridge in single group and it worked, cartridge
>> instance starts up and goes active,
>>
>> 2. Multiple cartridges with startup dependencies :
>>
>> All instances are started up (in right sequence) and go active
>>
>> JIRA https://issues.apache.org/jira/browse/STRATOS-1292[image:
>> http://wikicentral.cisco.com/images/icons/linkext7.gif]
>> <https://issues.apache.org/jira/browse/STRATOS-1292> is being tested and
>> verified
>>
>>
>>
>> Removing / Un-deploying the application caused an issue as one of the VMs
>> did not get terminated (throwing exceptions in the log). I am not sure if
>> this is already a known issue related to to the ongoing discussion of
>> application removal / un-deployment.
>>
>> Just in case, I zipped up the logs and attached it to the email thread
>> (single-group-multiple-cartridges-startup.zip). Let me know if the
>> un-deployment issue is the same already discussed or if this is a new issue
>> ?
>>
>>
>>
>> Same error occurred in the scenario where multiple cartridges were
>> started up (all go active) and then application was removed, I attached the
>> log for this scenario as well
>> (wscarbon-application-single-group-multiple-cartridges-no-startup-remval-error.log.zip)
>>
>>
>>
>>
>>
>> Thanks
>>
>>
>>
>> Martin
>>
>>
>>
>>
>>
>> *From:* Martin Eppel (meppel)
>> *Sent:* Monday, March 30, 2015 7:47 AM
>> *To:* dev@stratos.apache.org
>> *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
>> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
>> with single group and 3 cartridges, no dependencies ...
>>
>>
>>
>> Thanks Lahiru,
>>
>>
>>
>> Will do and let you know
>>
>>
>>
>> Martin
>>
>>
>>
>> *From:* Lahiru Sandaruwan [mailto:lahirus@wso2.com <la...@wso2.com>]
>> *Sent:* Monday, March 30, 2015 3:32 AM
>> *To:* dev
>> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
>> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
>> with single group and 3 cartridges, no dependencies ...
>>
>>
>>
>> Hi Martin,
>>
>>
>>
>> I was able to reproduce the issue and committed a fix.
>>
>> Could you please verify?
>>
>>
>>
>> Thanks.
>>
>>
>>
>> On Sun, Mar 29, 2015 at 5:43 PM, Lahiru Sandaruwan <la...@wso2.com>
>> wrote:
>>
>> Hi Martin,
>>
>>
>>
>> According to the logs attached, it seems the cartridge agent waits for
>> the artifact updated event, and somehow it is not sent.
>>
>>
>>
>> Looking into it.
>>
>>
>>
>> Thanks.
>>
>>
>>
>> On Sat, Mar 28, 2015 at 6:48 AM, Imesh Gunaratne <im...@apache.org>
>> wrote:
>>
>> Hi Martin,
>>
>>
>>
>> Thanks for the quick response on this!
>>
>> I will use these artifacts to re-produce the problem locally and will let
>> you know the results soon.
>>
>>
>>
>> Thanks
>>
>>
>>
>> On Sat, Mar 28, 2015 at 4:19 AM, Martin Eppel (meppel) <me...@cisco.com>
>> wrote:
>>
>> Hi Imesh,
>>
>>
>>
>> As discussed please find attached the “real” artefacts (application,
>> cartridge group, cartridge):
>>
>>
>>
>> Attached it as well to the jira
>>
>>
>>
>> Thanks
>>
>>
>>
>> Martin
>>
>>
>>
>>
>>
>> *From:* Martin Eppel (meppel)
>> *Sent:* Friday, March 27, 2015 11:37 AM
>> *To:* dev@stratos.apache.org
>> *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
>> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
>> with single group and 3 cartridges, no dependencies ...
>>
>>
>>
>> Hi Imesh,
>>
>>
>>
>> Originally I tested with 3 cartridges but then realized that it also
>> occurs with a single cartridge in a group – see also the JIRA for the
>> respective notes and artefacts (single group, single cartridge)
>>
>>
>>
>> Thanks
>>
>>
>>
>> Martin
>>
>>
>>
>> *From:* Imesh Gunaratne [mailto:imesh@apache.org <im...@apache.org>]
>> *Sent:* Friday, March 27, 2015 11:34 AM
>> *To:* dev
>> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
>> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
>> with single group and 3 cartridges, no dependencies ...
>>
>>
>>
>> Hi Martin,
>>
>>
>>
>> Thanks for reporting this problem. You have mentioned that there are
>> three cartridges in the application but in the given application.json I
>> only can see one cartridge. Are we missing anything here? Will it be
>> possible for you to attach the complete set of Stratos artifacts that you
>> are using in a zip file?
>>
>>
>>
>> Thanks
>>
>> Imesh
>>
>>
>>
>> On Fri, Mar 27, 2015 at 11:51 PM, Martin Eppel (meppel) <me...@cisco.com>
>> wrote:
>>
>> Forgot to mention, this is the same issue as reported in JIRA
>> https://issues.apache.org/jira/browse/STRATOS-1292
>>
>>
>>
>> Thanks
>>
>>
>>
>> Martin
>>
>>
>>
>> *From:* Martin Eppel (meppel)
>> *Sent:* Friday, March 27, 2015 10:52 AM
>> *To:* dev@stratos.apache.org
>> *Subject:* Issue (?): Stratos 4.1.0 - Cartridges in a group are not
>> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
>> with single group and 3 cartridges, no dependencies ...
>>
>>
>>
>> I retested with a similar scenario (single cartridges in group) with
>> debug logs turned on, please see logs attached.
>>
>>
>>
>> Cartridge is stuck in “Starting” state
>>
>>
>>
>> Thanks
>>
>>
>>
>> Martin
>>
>>
>>
>> *From:* Martin Eppel (meppel)
>> *Sent:* Monday, March 23, 2015 7:15 PM
>> *To:* dev@stratos.apache.org
>> *Subject:* Stratos 4.1.0 : Issue when deploying application with single
>> group and 3 cartridges, no dependencies ...
>>
>>
>>
>> Hi,
>>
>>
>>
>> I see an issue when I subscribe a single group with 3 cartridges, no
>> dependencies – they never go active instead they all are stuck in
>> “Starting”. I attached the server log and the cartridge-agent logs
>>
>>
>>
>>
>>
>> Member status:
>>
>> subscription-c1-c2-c3: applicationInstances 1, groupInstances 1,
>> clusterInstances 3, members 3 (Starting 3)
>>
>> c2: 172.16.2.18/172.28.40.69: status Starting
>>
>> c3: 172.16.2.19/172.28.40.70: status Starting
>>
>> c1: 172.16.2.17/172.28.40.68: status Starting
>>
>>
>>
>> Let me know if I should go ahead and open a JIRA
>>
>>
>>
>>
>>
>> Thanks
>>
>>
>>
>> Martin
>>
>>
>>
>>
>>
>> --
>>
>> Imesh Gunaratne
>>
>>
>>
>> Technical Lead, WSO2
>>
>> Committer & PMC Member, Apache Stratos
>>
>>
>>
>>
>>
>> --
>>
>> Imesh Gunaratne
>>
>>
>>
>> Technical Lead, WSO2
>>
>> Committer & PMC Member, Apache Stratos
>>
>>
>>
>>
>>
>> --
>>
>> --
>> Lahiru Sandaruwan
>>
>> Committer and PMC member, Apache Stratos,
>> Senior Software Engineer,
>> WSO2 Inc., http://wso2.com
>>
>> lean.enterprise.middleware
>>
>> phone: +94773325954
>> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>> --
>> Lahiru Sandaruwan
>>
>> Committer and PMC member, Apache Stratos,
>> Senior Software Engineer,
>> WSO2 Inc., http://wso2.com
>>
>> lean.enterprise.middleware
>>
>> phone: +94773325954
>> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>> --
>> Lahiru Sandaruwan
>>
>> Committer and PMC member, Apache Stratos,
>> Senior Software Engineer,
>> WSO2 Inc., http://wso2.com
>>
>> lean.enterprise.middleware
>>
>> phone: +94773325954
>> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>> --
>> Lahiru Sandaruwan
>>
>> Committer and PMC member, Apache Stratos,
>> Senior Software Engineer,
>> WSO2 Inc., http://wso2.com
>>
>> lean.enterprise.middleware
>>
>> phone: +94773325954
>> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>> --
>> Lahiru Sandaruwan
>>
>> Committer and PMC member, Apache Stratos,
>> Senior Software Engineer,
>> WSO2 Inc., http://wso2.com
>>
>> lean.enterprise.middleware
>>
>> phone: +94773325954
>> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>> --
>> Lahiru Sandaruwan
>>
>> Committer and PMC member, Apache Stratos,
>> Senior Software Engineer,
>> WSO2 Inc., http://wso2.com
>>
>> lean.enterprise.middleware
>>
>> phone: +94773325954
>> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>> --
>> Lahiru Sandaruwan
>>
>> Committer and PMC member, Apache Stratos,
>> Senior Software Engineer,
>> WSO2 Inc., http://wso2.com
>>
>> lean.enterprise.middleware
>>
>> phone: +94773325954
>> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>
>>
>>
>
>
>
> --
> --
> Lahiru Sandaruwan
> Committer and PMC member, Apache Stratos,
> Senior Software Engineer,
> WSO2 Inc., http://wso2.com
> lean.enterprise.middleware
>
> phone: +94773325954
> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
> --
> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>
> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>
> Thanks and Regards,
>
> Isuru H.
> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>
> +94 716 358 048 <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>*
> <http://wso2.com/>*
>
>
> * <http://wso2.com/>*
>
>
>
Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going
active ... was RE: Stratos 4.1.0 : Issue when deploying application with
single group and 3 cartridges, no dependencies ...
Posted by Lahiru Sandaruwan <la...@wso2.com>.
Thanks Martin for the logs. So the reason is clear why happens this. We
keep the payload par cartridge type in iaas provider object, in cloud
controller context.
When second member approaches the instance started method, it rewrites the
payload in iaasProvider object. But the first instance is still not really
created in iaas, and it runs in a different thread. Therefore the payload
belong to second instance, is passed to second instance as well as the
first instance.
I will fix it.
Thanks.
On Wed, Apr 1, 2015 at 11:38 PM, Martin Eppel (meppel) <me...@cisco.com>
wrote:
> Hi Lahiru,
>
>
>
> Attached are the logs,
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Lahiru Sandaruwan [mailto:lahirus@wso2.com]
> *Sent:* Wednesday, April 01, 2015 4:11 AM
>
> *To:* dev
> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> HI Martin,
>
>
>
> Could you please send the debug logs enabling debug on following classes?
> I have improved few debug logs to track payload, therefore please take a
> pull before reproducing with debug logs.
>
>
> log4j.logger.org.apache.stratos.autoscaler.applications.parser.DefaultApplicationParser=DEBUG
>
>
> log4j.logger.org.apache.stratos.cloud.controller.services.impl.InstanceCreator=DEBUG
>
>
> log4j.logger.org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl=DEBUG
>
> Thanks.
>
>
>
>
>
> On Wed, Apr 1, 2015 at 10:50 AM, Lahiru Sandaruwan <la...@wso2.com>
> wrote:
>
> I will have a look Martin.
>
>
>
> Thanks.
>
>
>
> On Wed, Apr 1, 2015 at 6:54 AM, Martin Eppel (meppel) <me...@cisco.com>
> wrote:
>
> Hi Lahiru,
>
>
>
> I retested the scenario to make sure proper unique aliases are provided
> and they are, please double check the provided artifacts. Issue remains the
> same, one VM gets stuck in INITIALIZED until terminated and restarted. As
> pointed out before, the cartridge alias in the cartridge agent is the same
> in 2 cartridges (although provided aliases in the application json is
> unique). After restarting the faulty member cartridge agent alias has the
> proper alias and goes into active.
>
> I also provided the cartridge-agent logs for the member which is faulty,
> before restart and after restart (cartridge-agent-70~INITIALIZED, ACTIVE)
>
>
>
> Please see attached logs, artifacts,
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Martin Eppel (meppel)
> *Sent:* Tuesday, March 31, 2015 1:22 PM
>
>
> *To:* dev@stratos.apache.org
> *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> Lahiru,
>
>
>
> The aliases are all unique, please check the the application json, and as
> I said in my analysis it I think this is what I saw but I can’t be sure. So
> this is not an issue of starting the application with duplicate aliases but
> rather something else ,
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Lahiru Sandaruwan [mailto:lahirus@wso2.com <la...@wso2.com>]
> *Sent:* Tuesday, March 31, 2015 12:58 PM
> *To:* dev
> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> Hi Martin,
>
>
>
> On Wed, Apr 1, 2015 at 12:34 AM, Martin Eppel (meppel) <me...@cisco.com>
> wrote:
>
> Hi Lahiru,
>
>
>
> I tested the fix and noticed the following issue:
>
>
>
> From the 4 cartridges 3 instances go Active, however one instance got
> stuck in INITIALIZED state.
>
>
>
> One of the things I noticed while the member was stuck in INITIALIZED
> state is that the system received 2 active events for the same membered:
>
>
>
> *TID: [0] [STRATOS] [2015-03-31 18:10:27,645] INFO
> {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
> - Publishing member activated event: [service-name] c2 [cluster-id]
> subscription-G1-G2-G3.c2-1x1.c2.domain [cluster-instance-id]
> subscription-G1-G2-G3-1 [member-id]
> subscription-G1-G2-G3.c2-1x1.c2.domainf939e8dc-e0cc-408a-8343-ac86e479549f
> [network-partition-id] RegionOne [partition-id] whole-region*
>
> *TID: [0] [STRATOS] [2015-03-31 18:10:31,059] INFO
> {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
> - Publishing member activated event: [service-name] c2 [cluster-id]
> subscription-G1-G2-G3.c2-1x1.c2.domain [cluster-instance-id]
> subscription-G1-G2-G3-1 [member-id]
> subscription-G1-G2-G3.c2-1x1.c2.domainf939e8dc-e0cc-408a-8343-ac86e479549f
> [network-partition-id] RegionOne [partition-id] whole-region*
>
>
>
> As another note, but I can’t be sure since the log got overwritten is that
> it appeared to me that at one point (while the cartridge was stuck in
> INITIALIZED state is that the cartridge alias of 2 cartridges was the same
> (~71, ~70: 71: CARTRIDGE_ALIAS=c2-1x1), but I can’t say for sure since when
> I double checked a new instance of the cartridge was created (which went
> ACTIVE).
>
>
>
>
>
> Can i assume this occurred when two cartridges use same alias? May be we
> have to put a validation on Application as we cannot use same alias in two
> cartridges.
>
>
>
> If so, could you please try to reproduce this with different aliases?
>
>
>
> Thanks.
>
>
>
>
>
>
>
> Eventually the system terminated the “faulty” member and the cartridge
> came up and went into active state:
>
>
>
> *“Pending state of member expired, member will be moved to obsolete list.
> [pending member]
> subscription-G1-G2-G3.c2-1x0.c2.domainfe9a30cf-fffc-4a6c-9b3b-f419498895b9*
> ”
>
> …
>
> “*Member activated: [service] c2 [cluster]
> subscription-G1-G2-G3.c2-1x0.c2.domain [member]
> subscription-G1-G2-G3.c2-1x0.c2.domain44a2ae3c-9968-41de-9e10-7212ef3788c2*
> ”
>
>
>
> Cartridge alias after all 4 cartridges went ACTIVE:
>
>
>
> 70: CARTRIDGE_ALIAS=c2-1x0
>
> 71: CARTRIDGE_ALIAS=c2-1x1
>
>
>
> 69: CARTRIDGE_ALIAS=c1-1x0
>
> 68: CARTRIDGE_ALIAS=c1-1x1
>
>
>
> I attached the logs and artefacts
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Lahiru Sandaruwan [mailto:lahirus@wso2.com]
> *Sent:* Tuesday, March 31, 2015 2:24 AM
>
>
> *To:* dev
> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> Hi Martin,
>
>
>
> I have reproduced the issue and pushed a fix. Please verify.
>
>
>
> Thanks.
>
>
>
> On Tue, Mar 31, 2015 at 8:32 AM, Lahiru Sandaruwan <la...@wso2.com>
> wrote:
>
> Hi Martin,
>
>
>
> On Tue, Mar 31, 2015 at 5:41 AM, Martin Eppel (meppel) <me...@cisco.com>
> wrote:
>
> Hi Lahiru,
>
>
>
> I tested a new scenario, where I have 2 groups with cartridges and no
> start up dependencies and I see again the issue with cartridge instances
> being stuck in “started” state. I also noticed the following log message:
>
> *“Artifact repositories not found for application signup, artifact updated
> event not sent: [application-id] subscription-G1-G2-G3*”.
>
>
>
>
>
> Will have a look.
>
>
>
> Thanks.
>
> Please find attached the artefacts and log messages,
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Martin Eppel (meppel)
> *Sent:* Monday, March 30, 2015 2:36 PM
>
>
> *To:* dev@stratos.apache.org
> *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> Hi Lahiru,
>
>
>
> I tested both scenarios,
>
>
>
> 1. Single cartridge in single group and it worked, cartridge
> instance starts up and goes active,
>
> 2. Multiple cartridges with startup dependencies :
>
> All instances are started up (in right sequence) and go active
>
> JIRA https://issues.apache.org/jira/browse/STRATOS-1292[image:
> http://wikicentral.cisco.com/images/icons/linkext7.gif]
> <https://issues.apache.org/jira/browse/STRATOS-1292> is being tested and
> verified
>
>
>
> Removing / Un-deploying the application caused an issue as one of the VMs
> did not get terminated (throwing exceptions in the log). I am not sure if
> this is already a known issue related to to the ongoing discussion of
> application removal / un-deployment.
>
> Just in case, I zipped up the logs and attached it to the email thread
> (single-group-multiple-cartridges-startup.zip). Let me know if the
> un-deployment issue is the same already discussed or if this is a new issue
> ?
>
>
>
> Same error occurred in the scenario where multiple cartridges were started
> up (all go active) and then application was removed, I attached the log for
> this scenario as well
> (wscarbon-application-single-group-multiple-cartridges-no-startup-remval-error.log.zip)
>
>
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
>
>
> *From:* Martin Eppel (meppel)
> *Sent:* Monday, March 30, 2015 7:47 AM
> *To:* dev@stratos.apache.org
> *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> Thanks Lahiru,
>
>
>
> Will do and let you know
>
>
>
> Martin
>
>
>
> *From:* Lahiru Sandaruwan [mailto:lahirus@wso2.com <la...@wso2.com>]
> *Sent:* Monday, March 30, 2015 3:32 AM
> *To:* dev
> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> Hi Martin,
>
>
>
> I was able to reproduce the issue and committed a fix.
>
> Could you please verify?
>
>
>
> Thanks.
>
>
>
> On Sun, Mar 29, 2015 at 5:43 PM, Lahiru Sandaruwan <la...@wso2.com>
> wrote:
>
> Hi Martin,
>
>
>
> According to the logs attached, it seems the cartridge agent waits for the
> artifact updated event, and somehow it is not sent.
>
>
>
> Looking into it.
>
>
>
> Thanks.
>
>
>
> On Sat, Mar 28, 2015 at 6:48 AM, Imesh Gunaratne <im...@apache.org> wrote:
>
> Hi Martin,
>
>
>
> Thanks for the quick response on this!
>
> I will use these artifacts to re-produce the problem locally and will let
> you know the results soon.
>
>
>
> Thanks
>
>
>
> On Sat, Mar 28, 2015 at 4:19 AM, Martin Eppel (meppel) <me...@cisco.com>
> wrote:
>
> Hi Imesh,
>
>
>
> As discussed please find attached the “real” artefacts (application,
> cartridge group, cartridge):
>
>
>
> Attached it as well to the jira
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
>
>
> *From:* Martin Eppel (meppel)
> *Sent:* Friday, March 27, 2015 11:37 AM
> *To:* dev@stratos.apache.org
> *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> Hi Imesh,
>
>
>
> Originally I tested with 3 cartridges but then realized that it also
> occurs with a single cartridge in a group – see also the JIRA for the
> respective notes and artefacts (single group, single cartridge)
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Imesh Gunaratne [mailto:imesh@apache.org <im...@apache.org>]
> *Sent:* Friday, March 27, 2015 11:34 AM
> *To:* dev
> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> Hi Martin,
>
>
>
> Thanks for reporting this problem. You have mentioned that there are three
> cartridges in the application but in the given application.json I only can
> see one cartridge. Are we missing anything here? Will it be possible for
> you to attach the complete set of Stratos artifacts that you are using in a
> zip file?
>
>
>
> Thanks
>
> Imesh
>
>
>
> On Fri, Mar 27, 2015 at 11:51 PM, Martin Eppel (meppel) <me...@cisco.com>
> wrote:
>
> Forgot to mention, this is the same issue as reported in JIRA
> https://issues.apache.org/jira/browse/STRATOS-1292
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Martin Eppel (meppel)
> *Sent:* Friday, March 27, 2015 10:52 AM
> *To:* dev@stratos.apache.org
> *Subject:* Issue (?): Stratos 4.1.0 - Cartridges in a group are not going
> active ... was RE: Stratos 4.1.0 : Issue when deploying application with
> single group and 3 cartridges, no dependencies ...
>
>
>
> I retested with a similar scenario (single cartridges in group) with
> debug logs turned on, please see logs attached.
>
>
>
> Cartridge is stuck in “Starting” state
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Martin Eppel (meppel)
> *Sent:* Monday, March 23, 2015 7:15 PM
> *To:* dev@stratos.apache.org
> *Subject:* Stratos 4.1.0 : Issue when deploying application with single
> group and 3 cartridges, no dependencies ...
>
>
>
> Hi,
>
>
>
> I see an issue when I subscribe a single group with 3 cartridges, no
> dependencies – they never go active instead they all are stuck in
> “Starting”. I attached the server log and the cartridge-agent logs
>
>
>
>
>
> Member status:
>
> subscription-c1-c2-c3: applicationInstances 1, groupInstances 1,
> clusterInstances 3, members 3 (Starting 3)
>
> c2: 172.16.2.18/172.28.40.69: status Starting
>
> c3: 172.16.2.19/172.28.40.70: status Starting
>
> c1: 172.16.2.17/172.28.40.68: status Starting
>
>
>
> Let me know if I should go ahead and open a JIRA
>
>
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
>
>
> --
>
> Imesh Gunaratne
>
>
>
> Technical Lead, WSO2
>
> Committer & PMC Member, Apache Stratos
>
>
>
>
>
> --
>
> Imesh Gunaratne
>
>
>
> Technical Lead, WSO2
>
> Committer & PMC Member, Apache Stratos
>
>
>
>
>
> --
>
> --
> Lahiru Sandaruwan
>
> Committer and PMC member, Apache Stratos,
> Senior Software Engineer,
> WSO2 Inc., http://wso2.com
>
> lean.enterprise.middleware
>
> phone: +94773325954
> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
>
>
>
>
>
>
> --
>
> --
> Lahiru Sandaruwan
>
> Committer and PMC member, Apache Stratos,
> Senior Software Engineer,
> WSO2 Inc., http://wso2.com
>
> lean.enterprise.middleware
>
> phone: +94773325954
> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
>
>
>
>
>
>
> --
>
> --
> Lahiru Sandaruwan
>
> Committer and PMC member, Apache Stratos,
> Senior Software Engineer,
> WSO2 Inc., http://wso2.com
>
> lean.enterprise.middleware
>
> phone: +94773325954
> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
>
>
>
>
>
>
> --
>
> --
> Lahiru Sandaruwan
>
> Committer and PMC member, Apache Stratos,
> Senior Software Engineer,
> WSO2 Inc., http://wso2.com
>
> lean.enterprise.middleware
>
> phone: +94773325954
> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
>
>
>
>
>
>
> --
>
> --
> Lahiru Sandaruwan
>
> Committer and PMC member, Apache Stratos,
> Senior Software Engineer,
> WSO2 Inc., http://wso2.com
>
> lean.enterprise.middleware
>
> phone: +94773325954
> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
>
>
>
>
>
>
> --
>
> --
> Lahiru Sandaruwan
>
> Committer and PMC member, Apache Stratos,
> Senior Software Engineer,
> WSO2 Inc., http://wso2.com
>
> lean.enterprise.middleware
>
> phone: +94773325954
> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
>
>
>
>
>
>
> --
>
> --
> Lahiru Sandaruwan
>
> Committer and PMC member, Apache Stratos,
> Senior Software Engineer,
> WSO2 Inc., http://wso2.com
>
> lean.enterprise.middleware
>
> phone: +94773325954
> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
>
>
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
phone: +94773325954
email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going
active ... was RE: Stratos 4.1.0 : Issue when deploying application with
single group and 3 cartridges, no dependencies ...
Posted by "Martin Eppel (meppel)" <me...@cisco.com>.
Hi Lahiru,
Attached are the logs,
Thanks
Martin
From: Lahiru Sandaruwan [mailto:lahirus@wso2.com]
Sent: Wednesday, April 01, 2015 4:11 AM
To: dev
Subject: Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
HI Martin,
Could you please send the debug logs enabling debug on following classes? I have improved few debug logs to track payload, therefore please take a pull before reproducing with debug logs.
log4j.logger.org.apache.stratos.autoscaler.applications.parser.DefaultApplicationParser=DEBUG
log4j.logger.org.apache.stratos.cloud.controller.services.impl.InstanceCreator=DEBUG
log4j.logger.org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl=DEBUG
Thanks.
On Wed, Apr 1, 2015 at 10:50 AM, Lahiru Sandaruwan <la...@wso2.com>> wrote:
I will have a look Martin.
Thanks.
On Wed, Apr 1, 2015 at 6:54 AM, Martin Eppel (meppel) <me...@cisco.com>> wrote:
Hi Lahiru,
I retested the scenario to make sure proper unique aliases are provided and they are, please double check the provided artifacts. Issue remains the same, one VM gets stuck in INITIALIZED until terminated and restarted. As pointed out before, the cartridge alias in the cartridge agent is the same in 2 cartridges (although provided aliases in the application json is unique). After restarting the faulty member cartridge agent alias has the proper alias and goes into active.
I also provided the cartridge-agent logs for the member which is faulty, before restart and after restart (cartridge-agent-70~INITIALIZED, ACTIVE)
Please see attached logs, artifacts,
Thanks
Martin
From: Martin Eppel (meppel)
Sent: Tuesday, March 31, 2015 1:22 PM
To: dev@stratos.apache.org<ma...@stratos.apache.org>
Subject: RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
Lahiru,
The aliases are all unique, please check the the application json, and as I said in my analysis it I think this is what I saw but I can’t be sure. So this is not an issue of starting the application with duplicate aliases but rather something else ,
Thanks
Martin
From: Lahiru Sandaruwan [mailto:lahirus@wso2.com]
Sent: Tuesday, March 31, 2015 12:58 PM
To: dev
Subject: Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
Hi Martin,
On Wed, Apr 1, 2015 at 12:34 AM, Martin Eppel (meppel) <me...@cisco.com>> wrote:
Hi Lahiru,
I tested the fix and noticed the following issue:
From the 4 cartridges 3 instances go Active, however one instance got stuck in INITIALIZED state.
One of the things I noticed while the member was stuck in INITIALIZED state is that the system received 2 active events for the same membered:
TID: [0] [STRATOS] [2015-03-31 18:10:27,645] INFO {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher} - Publishing member activated event: [service-name] c2 [cluster-id] subscription-G1-G2-G3.c2-1x1.c2.domain [cluster-instance-id] subscription-G1-G2-G3-1 [member-id] subscription-G1-G2-G3.c2-1x1.c2.domainf939e8dc-e0cc-408a-8343-ac86e479549f [network-partition-id] RegionOne [partition-id] whole-region
TID: [0] [STRATOS] [2015-03-31 18:10:31,059] INFO {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher} - Publishing member activated event: [service-name] c2 [cluster-id] subscription-G1-G2-G3.c2-1x1.c2.domain [cluster-instance-id] subscription-G1-G2-G3-1 [member-id] subscription-G1-G2-G3.c2-1x1.c2.domainf939e8dc-e0cc-408a-8343-ac86e479549f [network-partition-id] RegionOne [partition-id] whole-region
As another note, but I can’t be sure since the log got overwritten is that it appeared to me that at one point (while the cartridge was stuck in INITIALIZED state is that the cartridge alias of 2 cartridges was the same (~71, ~70: 71: CARTRIDGE_ALIAS=c2-1x1), but I can’t say for sure since when I double checked a new instance of the cartridge was created (which went ACTIVE).
Can i assume this occurred when two cartridges use same alias? May be we have to put a validation on Application as we cannot use same alias in two cartridges.
If so, could you please try to reproduce this with different aliases?
Thanks.
Eventually the system terminated the “faulty” member and the cartridge came up and went into active state:
“Pending state of member expired, member will be moved to obsolete list. [pending member] subscription-G1-G2-G3.c2-1x0.c2.domainfe9a30cf-fffc-4a6c-9b3b-f419498895b9”
…
“Member activated: [service] c2 [cluster] subscription-G1-G2-G3.c2-1x0.c2.domain [member] subscription-G1-G2-G3.c2-1x0.c2.domain44a2ae3c-9968-41de-9e10-7212ef3788c2”
Cartridge alias after all 4 cartridges went ACTIVE:
70: CARTRIDGE_ALIAS=c2-1x0
71: CARTRIDGE_ALIAS=c2-1x1
69: CARTRIDGE_ALIAS=c1-1x0
68: CARTRIDGE_ALIAS=c1-1x1
I attached the logs and artefacts
Thanks
Martin
From: Lahiru Sandaruwan [mailto:lahirus@wso2.com<ma...@wso2.com>]
Sent: Tuesday, March 31, 2015 2:24 AM
To: dev
Subject: Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
Hi Martin,
I have reproduced the issue and pushed a fix. Please verify.
Thanks.
On Tue, Mar 31, 2015 at 8:32 AM, Lahiru Sandaruwan <la...@wso2.com>> wrote:
Hi Martin,
On Tue, Mar 31, 2015 at 5:41 AM, Martin Eppel (meppel) <me...@cisco.com>> wrote:
Hi Lahiru,
I tested a new scenario, where I have 2 groups with cartridges and no start up dependencies and I see again the issue with cartridge instances being stuck in “started” state. I also noticed the following log message:
“Artifact repositories not found for application signup, artifact updated event not sent: [application-id] subscription-G1-G2-G3”.
Will have a look.
Thanks.
Please find attached the artefacts and log messages,
Thanks
Martin
From: Martin Eppel (meppel)
Sent: Monday, March 30, 2015 2:36 PM
To: dev@stratos.apache.org<ma...@stratos.apache.org>
Subject: RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
Hi Lahiru,
I tested both scenarios,
1. Single cartridge in single group and it worked, cartridge instance starts up and goes active,
2. Multiple cartridges with startup dependencies :
All instances are started up (in right sequence) and go active
JIRA https://issues.apache.org/jira/browse/STRATOS-1292[http://wikicentral.cisco.com/images/icons/linkext7.gif] is being tested and verified
Removing / Un-deploying the application caused an issue as one of the VMs did not get terminated (throwing exceptions in the log). I am not sure if this is already a known issue related to to the ongoing discussion of application removal / un-deployment.
Just in case, I zipped up the logs and attached it to the email thread (single-group-multiple-cartridges-startup.zip). Let me know if the un-deployment issue is the same already discussed or if this is a new issue ?
Same error occurred in the scenario where multiple cartridges were started up (all go active) and then application was removed, I attached the log for this scenario as well (wscarbon-application-single-group-multiple-cartridges-no-startup-remval-error.log.zip)
Thanks
Martin
From: Martin Eppel (meppel)
Sent: Monday, March 30, 2015 7:47 AM
To: dev@stratos.apache.org<ma...@stratos.apache.org>
Subject: RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
Thanks Lahiru,
Will do and let you know
Martin
From: Lahiru Sandaruwan [mailto:lahirus@wso2.com]
Sent: Monday, March 30, 2015 3:32 AM
To: dev
Subject: Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
Hi Martin,
I was able to reproduce the issue and committed a fix.
Could you please verify?
Thanks.
On Sun, Mar 29, 2015 at 5:43 PM, Lahiru Sandaruwan <la...@wso2.com>> wrote:
Hi Martin,
According to the logs attached, it seems the cartridge agent waits for the artifact updated event, and somehow it is not sent.
Looking into it.
Thanks.
On Sat, Mar 28, 2015 at 6:48 AM, Imesh Gunaratne <im...@apache.org>> wrote:
Hi Martin,
Thanks for the quick response on this!
I will use these artifacts to re-produce the problem locally and will let you know the results soon.
Thanks
On Sat, Mar 28, 2015 at 4:19 AM, Martin Eppel (meppel) <me...@cisco.com>> wrote:
Hi Imesh,
As discussed please find attached the “real” artefacts (application, cartridge group, cartridge):
Attached it as well to the jira
Thanks
Martin
From: Martin Eppel (meppel)
Sent: Friday, March 27, 2015 11:37 AM
To: dev@stratos.apache.org<ma...@stratos.apache.org>
Subject: RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
Hi Imesh,
Originally I tested with 3 cartridges but then realized that it also occurs with a single cartridge in a group – see also the JIRA for the respective notes and artefacts (single group, single cartridge)
Thanks
Martin
From: Imesh Gunaratne [mailto:imesh@apache.org]
Sent: Friday, March 27, 2015 11:34 AM
To: dev
Subject: Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
Hi Martin,
Thanks for reporting this problem. You have mentioned that there are three cartridges in the application but in the given application.json I only can see one cartridge. Are we missing anything here? Will it be possible for you to attach the complete set of Stratos artifacts that you are using in a zip file?
Thanks
Imesh
On Fri, Mar 27, 2015 at 11:51 PM, Martin Eppel (meppel) <me...@cisco.com>> wrote:
Forgot to mention, this is the same issue as reported in JIRA https://issues.apache.org/jira/browse/STRATOS-1292
Thanks
Martin
From: Martin Eppel (meppel)
Sent: Friday, March 27, 2015 10:52 AM
To: dev@stratos.apache.org<ma...@stratos.apache.org>
Subject: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
I retested with a similar scenario (single cartridges in group) with debug logs turned on, please see logs attached.
Cartridge is stuck in “Starting” state
Thanks
Martin
From: Martin Eppel (meppel)
Sent: Monday, March 23, 2015 7:15 PM
To: dev@stratos.apache.org<ma...@stratos.apache.org>
Subject: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ...
Hi,
I see an issue when I subscribe a single group with 3 cartridges, no dependencies – they never go active instead they all are stuck in “Starting”. I attached the server log and the cartridge-agent logs
Member status:
subscription-c1-c2-c3: applicationInstances 1, groupInstances 1, clusterInstances 3, members 3 (Starting 3)
c2: 172.16.2.18/172.28.40.69<http://172.16.2.18/172.28.40.69>: status Starting
c3: 172.16.2.19/172.28.40.70<http://172.16.2.19/172.28.40.70>: status Starting
c1: 172.16.2.17/172.28.40.68<http://172.16.2.17/172.28.40.68>: status Starting
Let me know if I should go ahead and open a JIRA
Thanks
Martin
--
Imesh Gunaratne
Technical Lead, WSO2
Committer & PMC Member, Apache Stratos
--
Imesh Gunaratne
Technical Lead, WSO2
Committer & PMC Member, Apache Stratos
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
phone: +94773325954<tel:%2B94773325954>
email: lahirus@wso2.com<ma...@wso2.com> blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
phone: +94773325954<tel:%2B94773325954>
email: lahirus@wso2.com<ma...@wso2.com> blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
phone: +94773325954<tel:%2B94773325954>
email: lahirus@wso2.com<ma...@wso2.com> blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
phone: +94773325954<tel:%2B94773325954>
email: lahirus@wso2.com<ma...@wso2.com> blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
phone: +94773325954<tel:%2B94773325954>
email: lahirus@wso2.com<ma...@wso2.com> blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
phone: +94773325954<tel:%2B94773325954>
email: lahirus@wso2.com<ma...@wso2.com> blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
phone: +94773325954
email: lahirus@wso2.com<ma...@wso2.com> blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going
active ... was RE: Stratos 4.1.0 : Issue when deploying application with
single group and 3 cartridges, no dependencies ...
Posted by Lahiru Sandaruwan <la...@wso2.com>.
HI Martin,
Could you please send the debug logs enabling debug on following classes? I
have improved few debug logs to track payload, therefore please take a pull
before reproducing with debug logs.
log4j.logger.org.apache.stratos.autoscaler.applications.parser.DefaultApplicationParser=DEBUG
log4j.logger.org.apache.stratos.cloud.controller.services.impl.InstanceCreator=DEBUG
log4j.logger.org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl=DEBUG
Thanks.
On Wed, Apr 1, 2015 at 10:50 AM, Lahiru Sandaruwan <la...@wso2.com> wrote:
> I will have a look Martin.
>
> Thanks.
>
> On Wed, Apr 1, 2015 at 6:54 AM, Martin Eppel (meppel) <me...@cisco.com>
> wrote:
>
>> Hi Lahiru,
>>
>>
>>
>> I retested the scenario to make sure proper unique aliases are provided
>> and they are, please double check the provided artifacts. Issue remains the
>> same, one VM gets stuck in INITIALIZED until terminated and restarted. As
>> pointed out before, the cartridge alias in the cartridge agent is the same
>> in 2 cartridges (although provided aliases in the application json is
>> unique). After restarting the faulty member cartridge agent alias has the
>> proper alias and goes into active.
>>
>> I also provided the cartridge-agent logs for the member which is faulty,
>> before restart and after restart (cartridge-agent-70~INITIALIZED, ACTIVE)
>>
>>
>>
>> Please see attached logs, artifacts,
>>
>>
>>
>> Thanks
>>
>>
>>
>> Martin
>>
>>
>>
>> *From:* Martin Eppel (meppel)
>> *Sent:* Tuesday, March 31, 2015 1:22 PM
>>
>> *To:* dev@stratos.apache.org
>> *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
>> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
>> with single group and 3 cartridges, no dependencies ...
>>
>>
>>
>> Lahiru,
>>
>>
>>
>> The aliases are all unique, please check the the application json, and as
>> I said in my analysis it I think this is what I saw but I can’t be sure. So
>> this is not an issue of starting the application with duplicate aliases but
>> rather something else ,
>>
>>
>>
>> Thanks
>>
>>
>>
>> Martin
>>
>>
>>
>> *From:* Lahiru Sandaruwan [mailto:lahirus@wso2.com <la...@wso2.com>]
>> *Sent:* Tuesday, March 31, 2015 12:58 PM
>> *To:* dev
>> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
>> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
>> with single group and 3 cartridges, no dependencies ...
>>
>>
>>
>> Hi Martin,
>>
>>
>>
>> On Wed, Apr 1, 2015 at 12:34 AM, Martin Eppel (meppel) <me...@cisco.com>
>> wrote:
>>
>> Hi Lahiru,
>>
>>
>>
>> I tested the fix and noticed the following issue:
>>
>>
>>
>> From the 4 cartridges 3 instances go Active, however one instance got
>> stuck in INITIALIZED state.
>>
>>
>>
>> One of the things I noticed while the member was stuck in INITIALIZED
>> state is that the system received 2 active events for the same membered:
>>
>>
>>
>> *TID: [0] [STRATOS] [2015-03-31 18:10:27,645] INFO
>> {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
>> - Publishing member activated event: [service-name] c2 [cluster-id]
>> subscription-G1-G2-G3.c2-1x1.c2.domain [cluster-instance-id]
>> subscription-G1-G2-G3-1 [member-id]
>> subscription-G1-G2-G3.c2-1x1.c2.domainf939e8dc-e0cc-408a-8343-ac86e479549f
>> [network-partition-id] RegionOne [partition-id] whole-region*
>>
>> *TID: [0] [STRATOS] [2015-03-31 18:10:31,059] INFO
>> {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
>> - Publishing member activated event: [service-name] c2 [cluster-id]
>> subscription-G1-G2-G3.c2-1x1.c2.domain [cluster-instance-id]
>> subscription-G1-G2-G3-1 [member-id]
>> subscription-G1-G2-G3.c2-1x1.c2.domainf939e8dc-e0cc-408a-8343-ac86e479549f
>> [network-partition-id] RegionOne [partition-id] whole-region*
>>
>>
>>
>> As another note, but I can’t be sure since the log got overwritten is
>> that it appeared to me that at one point (while the cartridge was stuck in
>> INITIALIZED state is that the cartridge alias of 2 cartridges was the same
>> (~71, ~70: 71: CARTRIDGE_ALIAS=c2-1x1), but I can’t say for sure since when
>> I double checked a new instance of the cartridge was created (which went
>> ACTIVE).
>>
>>
>>
>>
>>
>> Can i assume this occurred when two cartridges use same alias? May be we
>> have to put a validation on Application as we cannot use same alias in two
>> cartridges.
>>
>>
>>
>> If so, could you please try to reproduce this with different aliases?
>>
>>
>>
>> Thanks.
>>
>>
>>
>>
>>
>>
>>
>> Eventually the system terminated the “faulty” member and the cartridge
>> came up and went into active state:
>>
>>
>>
>> *“Pending state of member expired, member will be moved to obsolete list.
>> [pending member]
>> subscription-G1-G2-G3.c2-1x0.c2.domainfe9a30cf-fffc-4a6c-9b3b-f419498895b9*
>> ”
>>
>> …
>>
>> “*Member activated: [service] c2 [cluster]
>> subscription-G1-G2-G3.c2-1x0.c2.domain [member]
>> subscription-G1-G2-G3.c2-1x0.c2.domain44a2ae3c-9968-41de-9e10-7212ef3788c2*
>> ”
>>
>>
>>
>> Cartridge alias after all 4 cartridges went ACTIVE:
>>
>>
>>
>> 70: CARTRIDGE_ALIAS=c2-1x0
>>
>> 71: CARTRIDGE_ALIAS=c2-1x1
>>
>>
>>
>> 69: CARTRIDGE_ALIAS=c1-1x0
>>
>> 68: CARTRIDGE_ALIAS=c1-1x1
>>
>>
>>
>> I attached the logs and artefacts
>>
>>
>>
>> Thanks
>>
>>
>>
>> Martin
>>
>>
>>
>> *From:* Lahiru Sandaruwan [mailto:lahirus@wso2.com]
>> *Sent:* Tuesday, March 31, 2015 2:24 AM
>>
>>
>> *To:* dev
>> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
>> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
>> with single group and 3 cartridges, no dependencies ...
>>
>>
>>
>> Hi Martin,
>>
>>
>>
>> I have reproduced the issue and pushed a fix. Please verify.
>>
>>
>>
>> Thanks.
>>
>>
>>
>> On Tue, Mar 31, 2015 at 8:32 AM, Lahiru Sandaruwan <la...@wso2.com>
>> wrote:
>>
>> Hi Martin,
>>
>>
>>
>> On Tue, Mar 31, 2015 at 5:41 AM, Martin Eppel (meppel) <me...@cisco.com>
>> wrote:
>>
>> Hi Lahiru,
>>
>>
>>
>> I tested a new scenario, where I have 2 groups with cartridges and no
>> start up dependencies and I see again the issue with cartridge instances
>> being stuck in “started” state. I also noticed the following log message:
>>
>> *“Artifact repositories not found for application signup, artifact
>> updated event not sent: [application-id] subscription-G1-G2-G3*”.
>>
>>
>>
>>
>>
>> Will have a look.
>>
>>
>>
>> Thanks.
>>
>> Please find attached the artefacts and log messages,
>>
>>
>>
>> Thanks
>>
>>
>>
>> Martin
>>
>>
>>
>> *From:* Martin Eppel (meppel)
>> *Sent:* Monday, March 30, 2015 2:36 PM
>>
>>
>> *To:* dev@stratos.apache.org
>> *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
>> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
>> with single group and 3 cartridges, no dependencies ...
>>
>>
>>
>> Hi Lahiru,
>>
>>
>>
>> I tested both scenarios,
>>
>>
>>
>> 1. Single cartridge in single group and it worked, cartridge
>> instance starts up and goes active,
>>
>> 2. Multiple cartridges with startup dependencies :
>>
>> All instances are started up (in right sequence) and go active
>>
>> JIRA https://issues.apache.org/jira/browse/STRATOS-1292[image:
>> http://wikicentral.cisco.com/images/icons/linkext7.gif]
>> <https://issues.apache.org/jira/browse/STRATOS-1292> is being tested and
>> verified
>>
>>
>>
>> Removing / Un-deploying the application caused an issue as one of the VMs
>> did not get terminated (throwing exceptions in the log). I am not sure if
>> this is already a known issue related to to the ongoing discussion of
>> application removal / un-deployment.
>>
>> Just in case, I zipped up the logs and attached it to the email thread
>> (single-group-multiple-cartridges-startup.zip). Let me know if the
>> un-deployment issue is the same already discussed or if this is a new issue
>> ?
>>
>>
>>
>> Same error occurred in the scenario where multiple cartridges were
>> started up (all go active) and then application was removed, I attached the
>> log for this scenario as well
>> (wscarbon-application-single-group-multiple-cartridges-no-startup-remval-error.log.zip)
>>
>>
>>
>>
>>
>> Thanks
>>
>>
>>
>> Martin
>>
>>
>>
>>
>>
>> *From:* Martin Eppel (meppel)
>> *Sent:* Monday, March 30, 2015 7:47 AM
>> *To:* dev@stratos.apache.org
>> *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
>> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
>> with single group and 3 cartridges, no dependencies ...
>>
>>
>>
>> Thanks Lahiru,
>>
>>
>>
>> Will do and let you know
>>
>>
>>
>> Martin
>>
>>
>>
>> *From:* Lahiru Sandaruwan [mailto:lahirus@wso2.com <la...@wso2.com>]
>> *Sent:* Monday, March 30, 2015 3:32 AM
>> *To:* dev
>> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
>> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
>> with single group and 3 cartridges, no dependencies ...
>>
>>
>>
>> Hi Martin,
>>
>>
>>
>> I was able to reproduce the issue and committed a fix.
>>
>> Could you please verify?
>>
>>
>>
>> Thanks.
>>
>>
>>
>> On Sun, Mar 29, 2015 at 5:43 PM, Lahiru Sandaruwan <la...@wso2.com>
>> wrote:
>>
>> Hi Martin,
>>
>>
>>
>> According to the logs attached, it seems the cartridge agent waits for
>> the artifact updated event, and somehow it is not sent.
>>
>>
>>
>> Looking into it.
>>
>>
>>
>> Thanks.
>>
>>
>>
>> On Sat, Mar 28, 2015 at 6:48 AM, Imesh Gunaratne <im...@apache.org>
>> wrote:
>>
>> Hi Martin,
>>
>>
>>
>> Thanks for the quick response on this!
>>
>> I will use these artifacts to re-produce the problem locally and will let
>> you know the results soon.
>>
>>
>>
>> Thanks
>>
>>
>>
>> On Sat, Mar 28, 2015 at 4:19 AM, Martin Eppel (meppel) <me...@cisco.com>
>> wrote:
>>
>> Hi Imesh,
>>
>>
>>
>> As discussed please find attached the “real” artefacts (application,
>> cartridge group, cartridge):
>>
>>
>>
>> Attached it as well to the jira
>>
>>
>>
>> Thanks
>>
>>
>>
>> Martin
>>
>>
>>
>>
>>
>> *From:* Martin Eppel (meppel)
>> *Sent:* Friday, March 27, 2015 11:37 AM
>> *To:* dev@stratos.apache.org
>> *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
>> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
>> with single group and 3 cartridges, no dependencies ...
>>
>>
>>
>> Hi Imesh,
>>
>>
>>
>> Originally I tested with 3 cartridges but then realized that it also
>> occurs with a single cartridge in a group – see also the JIRA for the
>> respective notes and artefacts (single group, single cartridge)
>>
>>
>>
>> Thanks
>>
>>
>>
>> Martin
>>
>>
>>
>> *From:* Imesh Gunaratne [mailto:imesh@apache.org <im...@apache.org>]
>> *Sent:* Friday, March 27, 2015 11:34 AM
>> *To:* dev
>> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
>> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
>> with single group and 3 cartridges, no dependencies ...
>>
>>
>>
>> Hi Martin,
>>
>>
>>
>> Thanks for reporting this problem. You have mentioned that there are
>> three cartridges in the application but in the given application.json I
>> only can see one cartridge. Are we missing anything here? Will it be
>> possible for you to attach the complete set of Stratos artifacts that you
>> are using in a zip file?
>>
>>
>>
>> Thanks
>>
>> Imesh
>>
>>
>>
>> On Fri, Mar 27, 2015 at 11:51 PM, Martin Eppel (meppel) <me...@cisco.com>
>> wrote:
>>
>> Forgot to mention, this is the same issue as reported in JIRA
>> https://issues.apache.org/jira/browse/STRATOS-1292
>>
>>
>>
>> Thanks
>>
>>
>>
>> Martin
>>
>>
>>
>> *From:* Martin Eppel (meppel)
>> *Sent:* Friday, March 27, 2015 10:52 AM
>> *To:* dev@stratos.apache.org
>> *Subject:* Issue (?): Stratos 4.1.0 - Cartridges in a group are not
>> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
>> with single group and 3 cartridges, no dependencies ...
>>
>>
>>
>> I retested with a similar scenario (single cartridges in group) with
>> debug logs turned on, please see logs attached.
>>
>>
>>
>> Cartridge is stuck in “Starting” state
>>
>>
>>
>> Thanks
>>
>>
>>
>> Martin
>>
>>
>>
>> *From:* Martin Eppel (meppel)
>> *Sent:* Monday, March 23, 2015 7:15 PM
>> *To:* dev@stratos.apache.org
>> *Subject:* Stratos 4.1.0 : Issue when deploying application with single
>> group and 3 cartridges, no dependencies ...
>>
>>
>>
>> Hi,
>>
>>
>>
>> I see an issue when I subscribe a single group with 3 cartridges, no
>> dependencies – they never go active instead they all are stuck in
>> “Starting”. I attached the server log and the cartridge-agent logs
>>
>>
>>
>>
>>
>> Member status:
>>
>> subscription-c1-c2-c3: applicationInstances 1, groupInstances 1,
>> clusterInstances 3, members 3 (Starting 3)
>>
>> c2: 172.16.2.18/172.28.40.69: status Starting
>>
>> c3: 172.16.2.19/172.28.40.70: status Starting
>>
>> c1: 172.16.2.17/172.28.40.68: status Starting
>>
>>
>>
>> Let me know if I should go ahead and open a JIRA
>>
>>
>>
>>
>>
>> Thanks
>>
>>
>>
>> Martin
>>
>>
>>
>>
>>
>> --
>>
>> Imesh Gunaratne
>>
>>
>>
>> Technical Lead, WSO2
>>
>> Committer & PMC Member, Apache Stratos
>>
>>
>>
>>
>>
>> --
>>
>> Imesh Gunaratne
>>
>>
>>
>> Technical Lead, WSO2
>>
>> Committer & PMC Member, Apache Stratos
>>
>>
>>
>>
>>
>> --
>>
>> --
>> Lahiru Sandaruwan
>>
>> Committer and PMC member, Apache Stratos,
>> Senior Software Engineer,
>> WSO2 Inc., http://wso2.com
>>
>> lean.enterprise.middleware
>>
>> phone: +94773325954
>> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>> --
>> Lahiru Sandaruwan
>>
>> Committer and PMC member, Apache Stratos,
>> Senior Software Engineer,
>> WSO2 Inc., http://wso2.com
>>
>> lean.enterprise.middleware
>>
>> phone: +94773325954
>> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>> --
>> Lahiru Sandaruwan
>>
>> Committer and PMC member, Apache Stratos,
>> Senior Software Engineer,
>> WSO2 Inc., http://wso2.com
>>
>> lean.enterprise.middleware
>>
>> phone: +94773325954
>> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>> --
>> Lahiru Sandaruwan
>>
>> Committer and PMC member, Apache Stratos,
>> Senior Software Engineer,
>> WSO2 Inc., http://wso2.com
>>
>> lean.enterprise.middleware
>>
>> phone: +94773325954
>> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>> --
>> Lahiru Sandaruwan
>>
>> Committer and PMC member, Apache Stratos,
>> Senior Software Engineer,
>> WSO2 Inc., http://wso2.com
>>
>> lean.enterprise.middleware
>>
>> phone: +94773325954
>> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>
>>
>>
>
>
>
> --
> --
> Lahiru Sandaruwan
> Committer and PMC member, Apache Stratos,
> Senior Software Engineer,
> WSO2 Inc., http://wso2.com
> lean.enterprise.middleware
>
> phone: +94773325954
> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
>
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
phone: +94773325954
email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going
active ... was RE: Stratos 4.1.0 : Issue when deploying application with
single group and 3 cartridges, no dependencies ...
Posted by Lahiru Sandaruwan <la...@wso2.com>.
I will have a look Martin.
Thanks.
On Wed, Apr 1, 2015 at 6:54 AM, Martin Eppel (meppel) <me...@cisco.com>
wrote:
> Hi Lahiru,
>
>
>
> I retested the scenario to make sure proper unique aliases are provided
> and they are, please double check the provided artifacts. Issue remains the
> same, one VM gets stuck in INITIALIZED until terminated and restarted. As
> pointed out before, the cartridge alias in the cartridge agent is the same
> in 2 cartridges (although provided aliases in the application json is
> unique). After restarting the faulty member cartridge agent alias has the
> proper alias and goes into active.
>
> I also provided the cartridge-agent logs for the member which is faulty,
> before restart and after restart (cartridge-agent-70~INITIALIZED, ACTIVE)
>
>
>
> Please see attached logs, artifacts,
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Martin Eppel (meppel)
> *Sent:* Tuesday, March 31, 2015 1:22 PM
>
> *To:* dev@stratos.apache.org
> *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> Lahiru,
>
>
>
> The aliases are all unique, please check the the application json, and as
> I said in my analysis it I think this is what I saw but I can’t be sure. So
> this is not an issue of starting the application with duplicate aliases but
> rather something else ,
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Lahiru Sandaruwan [mailto:lahirus@wso2.com <la...@wso2.com>]
> *Sent:* Tuesday, March 31, 2015 12:58 PM
> *To:* dev
> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> Hi Martin,
>
>
>
> On Wed, Apr 1, 2015 at 12:34 AM, Martin Eppel (meppel) <me...@cisco.com>
> wrote:
>
> Hi Lahiru,
>
>
>
> I tested the fix and noticed the following issue:
>
>
>
> From the 4 cartridges 3 instances go Active, however one instance got
> stuck in INITIALIZED state.
>
>
>
> One of the things I noticed while the member was stuck in INITIALIZED
> state is that the system received 2 active events for the same membered:
>
>
>
> *TID: [0] [STRATOS] [2015-03-31 18:10:27,645] INFO
> {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
> - Publishing member activated event: [service-name] c2 [cluster-id]
> subscription-G1-G2-G3.c2-1x1.c2.domain [cluster-instance-id]
> subscription-G1-G2-G3-1 [member-id]
> subscription-G1-G2-G3.c2-1x1.c2.domainf939e8dc-e0cc-408a-8343-ac86e479549f
> [network-partition-id] RegionOne [partition-id] whole-region*
>
> *TID: [0] [STRATOS] [2015-03-31 18:10:31,059] INFO
> {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
> - Publishing member activated event: [service-name] c2 [cluster-id]
> subscription-G1-G2-G3.c2-1x1.c2.domain [cluster-instance-id]
> subscription-G1-G2-G3-1 [member-id]
> subscription-G1-G2-G3.c2-1x1.c2.domainf939e8dc-e0cc-408a-8343-ac86e479549f
> [network-partition-id] RegionOne [partition-id] whole-region*
>
>
>
> As another note, but I can’t be sure since the log got overwritten is that
> it appeared to me that at one point (while the cartridge was stuck in
> INITIALIZED state is that the cartridge alias of 2 cartridges was the same
> (~71, ~70: 71: CARTRIDGE_ALIAS=c2-1x1), but I can’t say for sure since when
> I double checked a new instance of the cartridge was created (which went
> ACTIVE).
>
>
>
>
>
> Can i assume this occurred when two cartridges use same alias? May be we
> have to put a validation on Application as we cannot use same alias in two
> cartridges.
>
>
>
> If so, could you please try to reproduce this with different aliases?
>
>
>
> Thanks.
>
>
>
>
>
>
>
> Eventually the system terminated the “faulty” member and the cartridge
> came up and went into active state:
>
>
>
> *“Pending state of member expired, member will be moved to obsolete list.
> [pending member]
> subscription-G1-G2-G3.c2-1x0.c2.domainfe9a30cf-fffc-4a6c-9b3b-f419498895b9*
> ”
>
> …
>
> “*Member activated: [service] c2 [cluster]
> subscription-G1-G2-G3.c2-1x0.c2.domain [member]
> subscription-G1-G2-G3.c2-1x0.c2.domain44a2ae3c-9968-41de-9e10-7212ef3788c2*
> ”
>
>
>
> Cartridge alias after all 4 cartridges went ACTIVE:
>
>
>
> 70: CARTRIDGE_ALIAS=c2-1x0
>
> 71: CARTRIDGE_ALIAS=c2-1x1
>
>
>
> 69: CARTRIDGE_ALIAS=c1-1x0
>
> 68: CARTRIDGE_ALIAS=c1-1x1
>
>
>
> I attached the logs and artefacts
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Lahiru Sandaruwan [mailto:lahirus@wso2.com]
> *Sent:* Tuesday, March 31, 2015 2:24 AM
>
>
> *To:* dev
> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> Hi Martin,
>
>
>
> I have reproduced the issue and pushed a fix. Please verify.
>
>
>
> Thanks.
>
>
>
> On Tue, Mar 31, 2015 at 8:32 AM, Lahiru Sandaruwan <la...@wso2.com>
> wrote:
>
> Hi Martin,
>
>
>
> On Tue, Mar 31, 2015 at 5:41 AM, Martin Eppel (meppel) <me...@cisco.com>
> wrote:
>
> Hi Lahiru,
>
>
>
> I tested a new scenario, where I have 2 groups with cartridges and no
> start up dependencies and I see again the issue with cartridge instances
> being stuck in “started” state. I also noticed the following log message:
>
> *“Artifact repositories not found for application signup, artifact updated
> event not sent: [application-id] subscription-G1-G2-G3*”.
>
>
>
>
>
> Will have a look.
>
>
>
> Thanks.
>
> Please find attached the artefacts and log messages,
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Martin Eppel (meppel)
> *Sent:* Monday, March 30, 2015 2:36 PM
>
>
> *To:* dev@stratos.apache.org
> *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> Hi Lahiru,
>
>
>
> I tested both scenarios,
>
>
>
> 1. Single cartridge in single group and it worked, cartridge
> instance starts up and goes active,
>
> 2. Multiple cartridges with startup dependencies :
>
> All instances are started up (in right sequence) and go active
>
> JIRA https://issues.apache.org/jira/browse/STRATOS-1292[image:
> http://wikicentral.cisco.com/images/icons/linkext7.gif]
> <https://issues.apache.org/jira/browse/STRATOS-1292> is being tested and
> verified
>
>
>
> Removing / Un-deploying the application caused an issue as one of the VMs
> did not get terminated (throwing exceptions in the log). I am not sure if
> this is already a known issue related to to the ongoing discussion of
> application removal / un-deployment.
>
> Just in case, I zipped up the logs and attached it to the email thread
> (single-group-multiple-cartridges-startup.zip). Let me know if the
> un-deployment issue is the same already discussed or if this is a new issue
> ?
>
>
>
> Same error occurred in the scenario where multiple cartridges were started
> up (all go active) and then application was removed, I attached the log for
> this scenario as well
> (wscarbon-application-single-group-multiple-cartridges-no-startup-remval-error.log.zip)
>
>
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
>
>
> *From:* Martin Eppel (meppel)
> *Sent:* Monday, March 30, 2015 7:47 AM
> *To:* dev@stratos.apache.org
> *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> Thanks Lahiru,
>
>
>
> Will do and let you know
>
>
>
> Martin
>
>
>
> *From:* Lahiru Sandaruwan [mailto:lahirus@wso2.com <la...@wso2.com>]
> *Sent:* Monday, March 30, 2015 3:32 AM
> *To:* dev
> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> Hi Martin,
>
>
>
> I was able to reproduce the issue and committed a fix.
>
> Could you please verify?
>
>
>
> Thanks.
>
>
>
> On Sun, Mar 29, 2015 at 5:43 PM, Lahiru Sandaruwan <la...@wso2.com>
> wrote:
>
> Hi Martin,
>
>
>
> According to the logs attached, it seems the cartridge agent waits for the
> artifact updated event, and somehow it is not sent.
>
>
>
> Looking into it.
>
>
>
> Thanks.
>
>
>
> On Sat, Mar 28, 2015 at 6:48 AM, Imesh Gunaratne <im...@apache.org> wrote:
>
> Hi Martin,
>
>
>
> Thanks for the quick response on this!
>
> I will use these artifacts to re-produce the problem locally and will let
> you know the results soon.
>
>
>
> Thanks
>
>
>
> On Sat, Mar 28, 2015 at 4:19 AM, Martin Eppel (meppel) <me...@cisco.com>
> wrote:
>
> Hi Imesh,
>
>
>
> As discussed please find attached the “real” artefacts (application,
> cartridge group, cartridge):
>
>
>
> Attached it as well to the jira
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
>
>
> *From:* Martin Eppel (meppel)
> *Sent:* Friday, March 27, 2015 11:37 AM
> *To:* dev@stratos.apache.org
> *Subject:* RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> Hi Imesh,
>
>
>
> Originally I tested with 3 cartridges but then realized that it also
> occurs with a single cartridge in a group – see also the JIRA for the
> respective notes and artefacts (single group, single cartridge)
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Imesh Gunaratne [mailto:imesh@apache.org <im...@apache.org>]
> *Sent:* Friday, March 27, 2015 11:34 AM
> *To:* dev
> *Subject:* Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not
> going active ... was RE: Stratos 4.1.0 : Issue when deploying application
> with single group and 3 cartridges, no dependencies ...
>
>
>
> Hi Martin,
>
>
>
> Thanks for reporting this problem. You have mentioned that there are three
> cartridges in the application but in the given application.json I only can
> see one cartridge. Are we missing anything here? Will it be possible for
> you to attach the complete set of Stratos artifacts that you are using in a
> zip file?
>
>
>
> Thanks
>
> Imesh
>
>
>
> On Fri, Mar 27, 2015 at 11:51 PM, Martin Eppel (meppel) <me...@cisco.com>
> wrote:
>
> Forgot to mention, this is the same issue as reported in JIRA
> https://issues.apache.org/jira/browse/STRATOS-1292
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Martin Eppel (meppel)
> *Sent:* Friday, March 27, 2015 10:52 AM
> *To:* dev@stratos.apache.org
> *Subject:* Issue (?): Stratos 4.1.0 - Cartridges in a group are not going
> active ... was RE: Stratos 4.1.0 : Issue when deploying application with
> single group and 3 cartridges, no dependencies ...
>
>
>
> I retested with a similar scenario (single cartridges in group) with
> debug logs turned on, please see logs attached.
>
>
>
> Cartridge is stuck in “Starting” state
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> *From:* Martin Eppel (meppel)
> *Sent:* Monday, March 23, 2015 7:15 PM
> *To:* dev@stratos.apache.org
> *Subject:* Stratos 4.1.0 : Issue when deploying application with single
> group and 3 cartridges, no dependencies ...
>
>
>
> Hi,
>
>
>
> I see an issue when I subscribe a single group with 3 cartridges, no
> dependencies – they never go active instead they all are stuck in
> “Starting”. I attached the server log and the cartridge-agent logs
>
>
>
>
>
> Member status:
>
> subscription-c1-c2-c3: applicationInstances 1, groupInstances 1,
> clusterInstances 3, members 3 (Starting 3)
>
> c2: 172.16.2.18/172.28.40.69: status Starting
>
> c3: 172.16.2.19/172.28.40.70: status Starting
>
> c1: 172.16.2.17/172.28.40.68: status Starting
>
>
>
> Let me know if I should go ahead and open a JIRA
>
>
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
>
>
> --
>
> Imesh Gunaratne
>
>
>
> Technical Lead, WSO2
>
> Committer & PMC Member, Apache Stratos
>
>
>
>
>
> --
>
> Imesh Gunaratne
>
>
>
> Technical Lead, WSO2
>
> Committer & PMC Member, Apache Stratos
>
>
>
>
>
> --
>
> --
> Lahiru Sandaruwan
>
> Committer and PMC member, Apache Stratos,
> Senior Software Engineer,
> WSO2 Inc., http://wso2.com
>
> lean.enterprise.middleware
>
> phone: +94773325954
> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
>
>
>
>
>
>
> --
>
> --
> Lahiru Sandaruwan
>
> Committer and PMC member, Apache Stratos,
> Senior Software Engineer,
> WSO2 Inc., http://wso2.com
>
> lean.enterprise.middleware
>
> phone: +94773325954
> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
>
>
>
>
>
>
> --
>
> --
> Lahiru Sandaruwan
>
> Committer and PMC member, Apache Stratos,
> Senior Software Engineer,
> WSO2 Inc., http://wso2.com
>
> lean.enterprise.middleware
>
> phone: +94773325954
> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
>
>
>
>
>
>
> --
>
> --
> Lahiru Sandaruwan
>
> Committer and PMC member, Apache Stratos,
> Senior Software Engineer,
> WSO2 Inc., http://wso2.com
>
> lean.enterprise.middleware
>
> phone: +94773325954
> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
>
>
>
>
>
>
> --
>
> --
> Lahiru Sandaruwan
>
> Committer and PMC member, Apache Stratos,
> Senior Software Engineer,
> WSO2 Inc., http://wso2.com
>
> lean.enterprise.middleware
>
> phone: +94773325954
> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
>
>
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
phone: +94773325954
email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146